wassname/discovering_latent_knowledge
1
0
Fork 0
mirror of https://github.com/wassname/discovering_latent_knowledge.git synced 2026-09-11 12:10:11 +08:00
Code Issues Packages Projects Releases Wiki Activity

wip

Browse Source
This commit is contained in:
deep1
2023-09-03 16:44:31 +08:00
parent 2e73d16932
commit 8522e268a0
5 changed files with 668 additions and 1575 deletions
Download Patch File Download Diff File Expand all files Collapse all files
mjc_notes.md
+48 -1
View File
@@ -1041,7 +1041,7 @@ But wait, how much does the preamble contribute to the lies, and how much does t
# 2023-08-25 08:59:59
On discord someone point out that my approach of taking deception as: wrong answer where it could otherwise answer them has a flaw. What if you then increase the answer with CoT/MultiShot/better prompting. Then it turns out it could answer all along, it's just that your prompting was confusing. The examples where this is true seem to be a case of the model being confused, rather than deceptive.
On discord someone pointed that my approach of taking deception as: wrong answer where it could otherwise answer them has a flaw. What if you then increase the answer with CoT/MultiShot/better prompting. Then it turns out it could answer all along, it's just that your prompting was confusing. The examples where this is true seem to be a case of the model being confused, rather than deceptive.
We have these categories of examples:
- that it can always solve "sentiment of terrible"
@@ -1140,3 +1140,50 @@ Let me think how to set this up. So let's say we know:
But if we give it the gradient from the loss, if that has the rigth answer in then it's data leakage and wont work during deployment
Hmm it's not so easy as there are many layers. And each is huge. I may need to use captum.
maybe one of these techniques
- https://captum.ai/api/neuron.html#neuron-guided-backprop omputes the gradient of the target neuron with respect to the input
- https://captum.ai/api/neuron.html#neuron-gradient output of a particular neuron with respect to the inputs of the network.
I may need to modify a method! https://github.com/pytorch/captum/blob/master/captum/_utils/gradient.py
# 2023-09-02 12:41:43
Problem: how to actually get gradioents?
- [ ] Counterfactual?
- what is it?
- Counterfactuals, hypothetical examples that show people how to obtain a different prediction.
- [ ] Neuron attribution? I would need to change from input to output
- [ ] Can I find a simple repo?
- somehow they schoe input or output Computes the gradient of the output of a particular neuron with respect to the inputs of the network.
torch.autograd.grad
OK it's too hard how about this
- just do each layer
- just do the last MLP
- and it's output neurons (need to work out how to do this... maybe reshape them sum over input?)
For having gradient I need bf16, which is 4x as large. That means I cannot fit a 15B model like starcoder. Which 7b model to try?
- https://huggingface.co/digitalpipelines/llama2_7b_chat_uncensored
- https://huggingface.co/WizardLM/WizardCoder-Python-7B-V1.0
there is also the 3b and 1b coding models
- https://huggingface.co/WizardLM/WizardCoder-3B-V1.0
- https://huggingface.co/WizardLM/WizardCoder-1B-V1.0
# which layers... this is an interesting choice
https://www.lesswrong.com/posts/kuQfnotjkQA4Kkfou/inference-time-intervention-eliciting-truthful-answers-from?commentId=bzJpeGjbEDAKDdJiX
They use train a linear probe on the for the activations of every attention head (post attention, pre W^O multiplication) to classify T vs F example answers. They see which attention heads they can successfully learn a probe at. They select the top 48 attention heads (by classifier accuracy).
For each of these heads they choose a “truthful direction” based on the difference of means between T and F example answers. (Or by using the direction orthogonal to the probe, but diff of means performs better.)
This is interesting as they do not use hidden states. They use attention head outputs hmm
then they only take the top 48 attentions heads, and only direction
notebooks/03_make_dataset.ipynb
+559 -1520
View File
File diff suppressed because it is too large Load Diff
src/datasets/batch.py
+5 -16
View File
@@ -22,7 +22,7 @@ def batch_hidden_states(model, tokenizer, data: Dataset, batch_size=2, mcdropout
"""
ehs = ExtractHiddenStates(model, tokenizer)
torch_cols = ['input_ids', 'attention_mask']
torch_cols = ['input_ids', 'attention_mask', 'choice_ids']
ds_t_subset = ds_keep_cols(data, torch_cols)
ds_t_subset.set_format(type='torch')
@@ -31,23 +31,12 @@ def batch_hidden_states(model, tokenizer, data: Dataset, batch_size=2, mcdropout
dl = DataLoader(ds_t_subset, batch_size=batch_size, shuffle=False)
for i, batch in enumerate(tqdm(dl, desc='get hidden states')):
input_ids, attention_mask = batch["input_ids"], batch["attention_mask"]
input_ids, attention_mask, choice_ids = batch["input_ids"], batch["attention_mask"], batch["choice_ids"]
nn = len(input_ids)
index = i*batch_size+np.arange(nn)
# different due to dropout
hs0 = ehs.get_batch_of_hidden_states(input_ids=input_ids, attention_mask=attention_mask, use_mcdropout=mcdropout)
if mcdropout:
hs1 = ehs.get_batch_of_hidden_states(input_ids=input_ids, attention_mask=attention_mask, use_mcdropout=mcdropout)
# QC
if i==0:
eps=1e-5
mpe = lambda x,y: np.mean(np.abs(x-y)/(np.abs(x)+np.abs(y)+eps))
a,b=hs1['hidden_states'],hs0['hidden_states']
assert mpe(a,b)>eps, "the hidden state pairs should be different but are not. Check model.config.use_cache==False, check this model has dropout in it's arch"
else:
hs1 = hs0
hs0 = ehs.get_batch_of_hidden_states(input_ids=input_ids, attention_mask=attention_mask, use_mcdropout=mcdropout, choice_ids=choice_ids)
for j in range(nn):
@@ -60,8 +49,8 @@ def batch_hidden_states(model, tokenizer, data: Dataset, batch_size=2, mcdropout
hs0=float_to_int16(torch.from_numpy(hs0['hidden_states'][j])),
scores0=hs0["scores"][j],
hs1=float_to_int16(torch.from_numpy(hs1['hidden_states'][j])),
scores1=hs1["scores"][j],
# hs1=float_to_int16(torch.from_numpy(hs1['hidden_states'][j])),
# scores1=hs1["scores"][j],
ds_index=index[j],
src/datasets/hs.py
+52 -34
View File
@@ -13,10 +13,10 @@ from transformers import (
)
from typing import Optional, List, Tuple, Dict
from transformers import LogitsProcessorList
import functools
from src.helpers.torch import to_numpy
from src.datasets.dropout import enable_dropout
import re
from tqdm.auto import tqdm
# from src.datasets.hs import ExtractHiddenStates
@@ -25,6 +25,23 @@ from datasets import Dataset
import numpy as np
import torch
import torch.nn.functional as F
from src.datasets.scores import choice2id, choice2ids
def get_gradients(model: PreTrainedModel, outputs, token_y, token_n):
model.zero_grad()
assert token_y.shape[1]<2, 'FIXME just use the first token for now'
score_y = torch.index_select(outputs["scores"], 1, token_y[:, 0])
score_n = torch.index_select(outputs["scores"], 1, token_n[:, 0])
# score_n = outputs["scores"][:, token_n]
pred = score_y - score_n
loss = F.mse_loss(pred, -pred)
loss.backward()
ps = model.named_parameters()
grads = {n:g.grad.cpu() for n,g in ps if g.grad is not None}
model.zero_grad()
# model.eval()
return grads
@dataclass
@@ -41,6 +58,7 @@ class ExtractHiddenStates:
input_text: Optional[List[str]] = None,
input_ids: torch.Tensor = None,
attention_mask: Optional[torch.Tensor] = None,
choice_ids: List[torch.Tensor] = None,
truncation_length=999,
use_mcdropout=True,
debug=False,
@@ -62,49 +80,49 @@ class ExtractHiddenStates:
)
input_ids = t.input_ids.to(self.model.device)
attention_mask = t.attention_mask.to(self.model.device)
else:
input_ids = input_ids.to(self.model.device)
attention_mask = attention_mask.to(self.model.device)
choice_ids = choice_ids.to(self.model.device)
# forward pass
last_token = -1
with torch.no_grad():
input_ids = input_ids.to(self.model.device)
self.model.eval()
if use_mcdropout:
enable_dropout(self.model, use_mcdropout)
self.model.train()
# Forward for one step is the same as greedy generation for one step
# https://github.com/huggingface/transformers/blob/234cfefbb083d2614a55f6093b0badfb2efc3b45/src/transformers/generation_utils.py#L1528
model_inputs = self.model.prepare_inputs_for_generation(input_ids=input_ids, attention_mask=attention_mask, use_cache=False)
outputs = self.model.forward(
**model_inputs,
return_dict=True,
output_hidden_states=True,
)
# next_token_logits = outputs.logits[:, -1, :]
# Forward for one step is the same as greedy generation for one step
# https://github.com/huggingface/transformers/blob/234cfefbb083d2614a55f6093b0badfb2efc3b45/src/transformers/generation_utils.py#L1528
model_inputs = self.model.prepare_inputs_for_generation(input_ids=input_ids, attention_mask=attention_mask, use_cache=False)
outputs = self.model.forward(
**model_inputs,
return_dict=True,
output_hidden_states=True,
)
# # pre-process distribution
# next_token_scores = logits_processor(input_ids, next_token_logits)
# next_token_scores = logits_warper(input_ids, next_token_scores)
# probs = nn.functional.softmax(next_token_scores, dim=-1)
outputs["scores"] = outputs.logits[:, last_token, :]
layers = self.get_layer_selection(outputs)
hidden_states = torch.stack(
[outputs["hidden_states"][i] for i in layers], 1
)
# (batch, layers, past_seq, logits) take just the last token so they are same size
hidden_states = hidden_states[
:, :, last_token
]
outputs["scores"] = outputs.logits[:, last_token, :]
layers = self.get_layer_selection(outputs)
token_n = choice_ids[:, 0] # [batch, tokens]
token_y = choice_ids[:, 1]
grads_all = get_gradients(self.model, outputs, token_y, token_n)
p = ".+mlp.c_proj.weight" # get the last weight of each layer (ignore bias)
# p = ".+mlp.c_proj.bias" # get the last weight of each layer
grads = torch.stack([g.mean(1).float() for k,g in grads_all.items() if re.match(p, k)])
hidden_states = torch.stack(
[outputs["hidden_states"][i] for i in layers], 1
)
# (batch, layers, past_seq, logits) take just the last token so they are same size
hidden_states = hidden_states[
:, :, last_token
]
out = dict(
hidden_states=hidden_states,
scores=outputs["scores"],
input_ids=input_ids,
layers=layers,
grads = grads,
)
out = {k: to_numpy(v) for k, v in out.items()}
if debug:
src/models/load.py
+4 -4
View File
@@ -24,19 +24,19 @@ def load_model(model_repo = "HuggingFaceH4/starchat-beta", lora_repo=None, verbo
# elif "llama" in model_repo:
# model, tokenizer = load_llama(model_repo=model_repo, lora_repo=lora_repo)
else:
raise NotImplementedError(f"model_repo {model_repo} not found")
raise NotImplementedError(f"code for model_repo {model_repo} not found")
if verbose: print(model.config)
assert check_for_dropout(model), 'model should have dropout'
return model, tokenizer
def load_starchat(model_repo = "HuggingFaceH4/starchat-beta"):
def load_starchat(model_repo = "HuggingFaceH4/starchat-beta", load_in_4bit=True, torch_dtype=torch.float16):
# see https://github.com/deep-diver/LLM-As-Chatbot/blob/main/models/starchat.py
model_options = dict(
device_map="auto",
load_in_4bit=True,
torch_dtype=torch.float16, # note because datasets pickles the model into numpy to get the unique datasets name, and because numpy doesn't support bfloat16, we need to use float16
load_in_4bit=load_in_4bit,
torch_dtype=torch_dtype, # note because datasets pickles the model into numpy to get the unique datasets name, and because numpy doesn't support bfloat16, we need to use float16
use_safetensors=False,
)