LoRA_are_lie_detectors/notebooks/101_dev_llama.ipynb

In [1]:

from transformers import AutoTokenizer, AutoModelForCausalLM
# model_name = "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T"
model_name = "TinyLlama/TinyLlama-1.1B-Chat-v0.6"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
model

Out [1]:

Downloading tokenizer_config.json:   0%|          | 0.00/1.29k [00:00<?, ?B/s]

Downloading tokenizer.model:   0%|          | 0.00/500k [00:00<?, ?B/s]

Downloading tokenizer.json:   0%|          | 0.00/1.84M [00:00<?, ?B/s]

Downloading (…)cial_tokens_map.json:   0%|          | 0.00/551 [00:00<?, ?B/s]

Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.

LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(32000, 2048)
    (layers): ModuleList(
      (0-21): 22 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear(in_features=2048, out_features=2048, bias=False)
          (k_proj): Linear(in_features=2048, out_features=256, bias=False)
          (v_proj): Linear(in_features=2048, out_features=256, bias=False)
          (o_proj): Linear(in_features=2048, out_features=2048, bias=False)
          (rotary_emb): LlamaRotaryEmbedding()
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=2048, out_features=5632, bias=False)
          (up_proj): Linear(in_features=2048, out_features=5632, bias=False)
          (down_proj): Linear(in_features=5632, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): LlamaRMSNorm()
        (post_attention_layernorm): LlamaRMSNorm()
      )
    )
    (norm): LlamaRMSNorm()
  )
  (lm_head): Linear(in_features=2048, out_features=32000, bias=False)
)

The Kernel crashed while executing code in the the current cell or a previous cell. Please review the code in the cell(s) to identify a possible cause of the failure. Click <a href='https://aka.ms/vscodeJupyterKernelCrash'>here</a> for more info. View Jupyter <a href='command:jupyter.viewOutput'>log</a> for further details.

• https://github.com/jzhang38/TinyLlama/blob/main/sft/finetune.py
• https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py

In [29]:

import torch
x=  torch.rand((9))
x1 = torch.diff(x, dim=0, prepend=torch.zeros_like(x)[:1])
x2 = torch.diff(x, dim=0)
x2 = torch.cat([x[:1], x2], dim=0)
x.shape, x1.shape, x2.shape

Out [29]:

(torch.Size([9]), torch.Size([9]), torch.Size([9]))

In [31]:

x1[0], x2[0]

Out [31]:

(tensor(0.1303), tensor(0.1303))

In [30]:

from matplotlib import pyplot as plt
# plt.plot(x)
plt.plot(x1, linestyle='-')
plt.plot(x2, linestyle='-.')

Out [30]:

[<matplotlib.lines.Line2D at 0x7f0288e9a450>]

In [ ]: