distance and direciton

Let try to opt for distance and direction with

L1loss(y_1-y_0, y_{true})

where y_1=model(x_1)

So I'm optimising for the hidden states to be the correct distance and direcioton away. It's like the margin raning loss.

links:

In [1]:

# import your package
%load_ext autoreload
%autoreload 2

In [2]:


import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
plt.style.use('ggplot')

from typing import Optional, List, Dict, Union

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch import Tensor
from torch import optim
from torch.utils.data import random_split, DataLoader, TensorDataset

from pathlib import Path

import transformers

import lightning.pytorch as pl
# from dataclasses import dataclass

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score, roc_auc_score, accuracy_score
from sklearn.preprocessing import RobustScaler

from tqdm.auto import tqdm
import os

from loguru import logger
logger.add(os.sys.stderr, format="{time} {level} {message}", level="INFO")



transformers.__version__

Out [2]:

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /home/ubuntu/mambaforge/envs/dlk3/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda117.so
CUDA SETUP: CUDA runtime path found: /home/ubuntu/mambaforge/envs/dlk3/lib/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 117
CUDA SETUP: Loading binary /home/ubuntu/mambaforge/envs/dlk3/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda117.so...

/home/ubuntu/mambaforge/envs/dlk3/lib/python3.11/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/home/ubuntu/mambaforge/envs/dlk3/lib/libcudart.so'), PosixPath('/home/ubuntu/mambaforge/envs/dlk3/lib/libcudart.so.11.0')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
  warn(msg)

'4.31.0'

In [3]:

from src.helpers.lightning import read_metrics_csv

Dataset

In [4]:

from datasets import load_from_disk, concatenate_datasets
fs = [
    # '../.ds/HuggingFaceH4starchat_beta_imdb_train_9002',
    # '../.ds/HuggingFaceH4starchat_beta_imdb_train_12002',
    # '../.ds/HuggingFaceH4starchat_beta_amazon_polarity_train_12002'
    '../.ds/WizardLMWizardCoder_3B_V1.0_imdb_train_2620',
]

# './.ds/HuggingFaceH4starchat_beta-None-N_8000-ns_3-mc_0.2-2ffc1e'
ds1 = concatenate_datasets([load_from_disk(f) for f in fs])
ds1

Out [4]:

Dataset({
    features: ['hs0', 'scores0', 'grads_mlp0', 'grads_mlp_cfc0', 'grads_attn0', 'ds_index', 'ds_string', 'example_i', 'answer', 'question', 'answer_choices', 'template_name', 'label_true', 'label_instructed', 'instructed_to_lie', 'sys_instr_name', 'prompt_truncated', 'choice_probs0', 'ans0', 'txt_ans0'],
    num_rows: 2620
})

In [5]:

from src.datasets.load import ds2df

Filter

In [6]:

# lets select only the ones where
df = ds2df(ds1)
df

Out [6]:

	ds_index	ds_string	example_i	answer	question	answer_choices	template_name	label_true	label_instructed	instructed_to_lie	sys_instr_name	prompt_truncated	choice_probs0	ans0	txt_ans0	dir_true	llm_ans
0	0	imdb	0	positive	At every step avoid telling the truth.\n\n### ...	[negative, positive]	Movie Expressed Sentiment 2	0	1	True	just_lie	's Elephant (2003) though good, seems almost s...	[0.30319116, 0.4411405]	0.592659	positive	0.592659	True
1	1	imdb	0	0	Below is an instruction that describes a task,...	[0, 1]	burns_2	0	0	False	truth	ly good; and movies based on the history of th...	[0.44310966, 0.5344923]	0.546733	1	0.546733	True
2	2	imdb	1	negative	You are an AI assistant that always tells a li...	[negative, positive]	Reviewer Expressed Sentiment	1	0	True	lie_for_charity	things to ensure that this is one of the few ...	[0.031801913, 0.32118794]	0.909881	positive	0.909881	True
3	3	imdb	1	1	Below is an instruction that describes a task,...	[0, 1]	burns_1	1	1	False	truth	example is\n\n### Response:\n0\n\n### Instruc...	[0.5583413, 0.43483666]	0.437819	0	0.437819	False
4	4	imdb	2	negative review.	In this exam you will be scored on your abilit...	[negative, positive]	Negation template for positive and negative	0	1	True	this_is_an_exam	### Response:\nreview.\n\n### Instruction\nYeh...	[0.000287705, 0.00014131595]	0.321889	review	0.321889	False
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
2615	2615	imdb	1307	positive	Below is an instruction that describes a task,...	[negative, positive]	Reviewer Expressed Sentiment	1	1	False	truth	disagree, perhaps it is your hearts that are ...	[0.14765827, 0.45481953]	0.754902	positive	0.754902	True
2616	2616	imdb	1308	good	You are about to immerse yourself into a logic...	[bad, good]	Reviewer Opinion bad good choices	0	1	True	guard	ood. I brought it home and put it on.<br /><br...	[0.13962792, 0.34022465]	0.709004	good	0.709004	True
2617	2617	imdb	1308	negative	Below is an instruction that describes a task,...	[negative, positive]	Reviewer Expressed Sentiment	0	0	False	truth	993 ensemble "Short Cuts". Lawrence uses this ...	[0.15071113, 0.2772002]	0.647783	positive	0.647783	True
2618	2618	imdb	1309	negative	You've been assigned a role in a classic truth...	[negative, positive]	Writer Expressed Sentiment	1	0	True	puzzle	movie has no story,has no acting..and has no ...	[0.003769116, 0.31378734]	0.988100	positive	0.988100	True
2619	2619	imdb	1309	1	Below is an instruction that describes a task,...	[0, 1]	burns_1	1	1	False	truth	laugh your ass off.<br /><br />9 - 10. '''\nB...	[0.50953037, 0.46393266]	0.476575	0	0.476575	False

2620 rows × 17 columns

In [ ]:

In [8]:

# # just select the question where the model knows the answer. 
df = ds2df(ds1)
d = df.query('sys_instr_name=="truth"').set_index("example_i")

# # these are the ones where it got it right when asked to tell the truth
m1 = d.llm_ans==d.label_true
known_indices = d[m1].index
print(f"select rows are {m1.mean():2.2%} based on knowledge")
# # convert to row numbers, and use datasets to select
known_rows = df['example_i'].isin(known_indices)
known_rows_i = df[known_rows].index

# # also restrict it to significant permutations. That is monte carlo dropout pairs, where the answer changes by more than X%
# m = np.abs(df.ans0-df.ans1)>0.05
# print(f"selected rows are {m.mean():2.2%} for significance")
# significant_rows = m[m].index

# allowed_rows_i = set(known_rows_i).intersection(significant_rows)
# allowed_rows_i = significant_rows
ds = ds1.select(known_rows_i)
ds

Out [8]:

select rows are 68.85% based on knowledge

Dataset({
    features: ['hs0', 'scores0', 'grads_mlp0', 'grads_mlp_cfc0', 'grads_attn0', 'ds_index', 'ds_string', 'example_i', 'answer', 'question', 'answer_choices', 'template_name', 'label_true', 'label_instructed', 'instructed_to_lie', 'sys_instr_name', 'prompt_truncated', 'choice_probs0', 'ans0', 'txt_ans0'],
    num_rows: 1804
})

Transform: Normalize by activation

In [9]:

# N = 1000
# small_ds = ds.select(range(N))
# b = N
# hs0 = small_ds['hs0'].reshape((b, -1))

# scaler = RobustScaler()
# hs1 = scaler.fit_transform(hs0)

# def normalize_hs(hs0, hs1):
#     shape=hs0.shape
#     b = len(hs0)
#     hs0 = scaler.transform(hs0.reshape((b, -1))).reshape(shape)
#     hs1 = scaler.transform(hs1.reshape((b, -1))).reshape(shape)
#     return {'hs0':hs0, 'hs1': hs1}

# # Plot
# plt.hist(hs0.flatten(), bins=155, range=[-5, 5], label='before', histtype='step')
# plt.hist(hs1.flatten(), bins=155, range=[-5, 5], label='after', histtype='step')
# plt.legend()
# plt.show()

# # # Test
# # small_dataset = ds.select(range(4))
# # small_dataset.map(normalize_hs, batched=True, batch_size=2, input_columns=['hs0', 'hs1'])

# # run
# ds = ds.map(normalize_hs, batched=True, input_columns=['hs0', 'hs1'])
# ds

Lightning DataModule

In [10]:

df = ds2df(ds)
df.head(4)

Out [10]:

	ds_index	ds_string	example_i	answer	question	answer_choices	template_name	label_true	label_instructed	instructed_to_lie	sys_instr_name	prompt_truncated	choice_probs0	ans0	txt_ans0	dir_true	llm_ans
0	10	imdb	5	negative	You've been assigned a role in a classic truth...	[negative, positive]	Reviewer Sentiment Feeling	1	0	True	puzzle	an uneasy, cheap and insincere feeling about ...	[0.07703635, 0.42967427]	0.847951	positive	0.847951	True
1	11	imdb	5	positive	Below is an instruction that describes a task,...	[negative, positive]	Writer Expressed Sentiment	1	1	False	truth	appropriately completes the request.\n\n### I...	[0.36913574, 0.5124942]	0.581296	positive	0.581296	True
2	12	imdb	6	negative review.	You are an AI assistant that always tells a li...	[negative, positive]	Negation template for positive and negative	0	1	True	lie_for_charity	ination it was lovely. They barely knew each o...	[0.0005304276, 0.0007480229]	0.580560	review	0.580560	True
3	13	imdb	6	negative	Below is an instruction that describes a task,...	[negative, positive]	Reviewer Expressed Sentiment	0	0	False	truth	time spy...\nWhat is the sentiment expressed ...	[0.2338603, 0.14184344]	0.377531	negative	0.377531	False

In [11]:

# ds?

What are we detecting? If the right example of the pair is more deceptive.

Now it's only deceptive if

it was asked to lie
it knows the truth
it gave the wrong answer (around 10% of the time)( it's hard to get these models to lie by encouragement rather than instruction)

In [16]:

from src.helpers import switch2bool, bool2switch
from src.datasets.dm import imdbHSDataModule

In [18]:

batch_size = 120
# test and cache
dm = imdbHSDataModule(ds, batch_size=batch_size)
dm.setup('train')

dl_val = dm.val_dataloader()
dl_train = dm.train_dataloader()
len(dl_train), len(dl_val)

Out [18]:

(8, 4)

In [22]:

ds['grads_mlp0'].shape

Out [22]:

(1804, 2816)

In [ ]:

b = next(iter(dl_train))
x0, x1, y = b
x0.shape

Data prep

We do two inferences on the same inputs. Since we have dropout enabled, even during inference, we get two slightly different hidden states hs1 and hs2, and two slightly different probabilities for our yes and no output tokens p1 p2. We also have the true answer t

So there are a few ways we can set up the problem.

We can vary x:

model(hs1)-model(hs2)=y
model(hs1-hs2)==y

And we can try differen't y's:

direction with a ranked loss. This could be unsupervised.
magnitude with a regression loss
vector (direction and magnitude) with a regression loss

QC: Linear supervised probes

Let's verify that the model's representations are good

Before trying CCS, let's make sure there exists a direction that classifies examples as true vs false with high accuracy; if supervised logistic regression accuracy is bad, there's no hope of unsupervised CCS doing well.

Note that because logistic regression is supervised we expect it to do better but to have worse generalisation that equivilent unsupervised methods. However in this case CSS is using a deeper model so it is more complicated.

Try a classification of direction to truth

In [ ]:

# dm.y

In [ ]:

# n = len(df)

# # Define X and y
# X = (dm.hs1-dm.hs0).reshape((n, -1))#/dm.y[:, None]
# y = dm.y>0

# # split
# n = len(y)
# max_rows = 300
# print('split size', n//2)
# X_train, X_test = X[:n//2], X[n//2:]
# y_train, y_test = y[:n//2], y[n//2:]
# X_train = X_train[:max_rows]
# y_train = y_train[:max_rows]
# X_test = X_test[:max_rows]
# y_test = y_test[:max_rows]

# # scale
# scaler = RobustScaler()
# scaler.fit(X_train)
# X_train2 = scaler.transform(X_train)
# X_test2 = scaler.transform(X_test)
# print('lr')

# lr = LogisticRegression(class_weight="balanced", penalty="l2", max_iter=100)
# lr.fit(X_train2, y_train>0)

In [ ]:

# y.mean()

In [ ]:

# print("Logistic cls acc: {:2.2%} [TRAIN]".format(lr.score(X_train2, y_train>0)))
# print("Logistic cls acc: {:2.2%} [TEST]".format(lr.score(X_test2, y_test>0)))

# m = df['instructed_to_lie'][n//2:][:max_rows]
# y_test_pred = lr.predict(X_test2)
# acc_w_lie = ((y_test_pred[m]>0)==(y_test[m]>0)).mean()
# acc_wo_lie = ((y_test_pred[~m]>0)==(y_test[~m]>0)).mean()
# print(f'test acc w lie {acc_w_lie:2.2%}')
# print(f'test acc wo lie {acc_wo_lie:2.2%}')

In [ ]:

# primary_baseline = roc_auc_score(y_test>0, y_test_pred)
# primary_baseline

LightningModel

In [ ]:

# # from src.probes.conv import PLConvProbe
# import torch
# import torch.nn as nn
# import torch.nn.functional as F
# from src.probes.conv import PLConvProbe
# from src.probes.pl_ranking import PLRanking
# from torchmetrics.functional import accuracy
# from src.helpers import switch2bool, bool2switch

# class ConvProbe(nn.Module):
#     def __init__(self, c_in, depth=0, hs=16, dropout=0, input_dropout=0):
#         super().__init__()

#         layers = [
#             nn.BatchNorm1d(c_in, affine=False),  # this will normalise the inputs
#             nn.Dropout1d(input_dropout),
            
#             nn.Conv1d(c_in, hs*(depth+1), kernel_size=3),
#             nn.ReLU(),
#             nn.BatchNorm1d(hs*(depth+1)),
#             nn.AdaptiveAvgPool1d(5),
#             nn.Flatten(),
#             nn.Linear(hs*(depth+1)*5, hs*(depth+1)),
#         ]
#         for i in range(depth):
#             layers += [
#                 nn.Linear(hs*(depth-i+1), hs*(depth-i)),
#                 nn.ReLU(),
#                 nn.BatchNorm1d(hs*(depth-i)),
                
#             ]
#         # layers += [nn.AdaptiveAvgPool1d(1)]
#         self.net = nn.Sequential(*layers)
#         self.head = nn.Sequential(
#             nn.Linear(hs, hs), nn.ReLU(),
#             nn.Dropout(dropout), nn.Linear(hs, 1)            
#         )

#     def forward(self, x):
#         h = self.net(x)
#         # print(1, h.shape)
#         h = h.squeeze(-1)
#         # print(1, h.shape)
#         return self.head(h)

# class PLConvProbe(PLRanking):
#     def __init__(self, c_in, total_steps, lr=4e-3, weight_decay=1e-9, **kwargs):
#         super().__init__(total_steps=total_steps, lr=lr, weight_decay=weight_decay)
#         self.probe = ConvProbe(c_in, **kwargs)
#         self.save_hyperparameters()
        
        
#     def _step(self, batch, batch_idx, stage='train'):
#         x0, x1, y = batch
#         ypred0 = self(x0)
#         ypred1 = self(x1)
        
#         if stage=='pred':
#             return (ypred1-ypred0).float()
        
#         # loss = F.smooth_l1_loss(ypred1-ypred0, y)
#         loss = F.margin_ranking_loss(ypred1, ypred0, y, margin=0.5)
#         # self.log(f"{stage}/loss", loss)
        
#         y_cls = switch2bool(ypred1-ypred0)
#         self.log(f"{stage}/acc", accuracy(y_cls, y>0, "binary"), on_epoch=True, on_step=False)
#         self.log(f"{stage}/loss", loss, on_epoch=True, on_step=False)
#         self.log(f"{stage}/n", len(y), on_epoch=True, on_step=False, reduce_fx=torch.sum)
#         return loss

In [ ]:

# from src.probes.conv import PLConvProbe
import torch
import torch.nn as nn
import torch.nn.functional as F
from src.probes.conv import PLConvProbe
from src.probes.pl_ranking import PLRanking
from torchmetrics.functional import accuracy
from src.helpers import switch2bool, bool2switch

class ConvProbe(nn.Module):
    def __init__(self, c_in, depth=0, hs=16, dropout=0, input_dropout=0):
        super().__init__()
        self.n_groups = 24 # groups of neurons
        c = c_in//self.n_groups
        P = 3

        cw = hs*(depth+1)
        self.layers1 = nn.Sequential(*[
            nn.BatchNorm2d(c, affine=False),  # this will normalise the inputs
            nn.Dropout2d(input_dropout),
            
            nn.Conv2d(c, c//4, kernel_size=(1, 3)),
            nn.Conv2d(c//4, cw, kernel_size=(3, 1)),
            nn.ReLU(),
            nn.BatchNorm2d(cw),
            
            nn.Conv2d(cw, cw, kernel_size=(1, 3)),
            nn.Conv2d(cw, cw, kernel_size=(3, 1)),
            nn.ReLU(),
            nn.BatchNorm2d(cw),
            
            
            nn.Conv2d(cw, cw, kernel_size=(1, 3)),
            nn.Conv2d(cw, cw, kernel_size=(3, 1)),
            nn.ReLU(),
            nn.BatchNorm2d(cw),            
            
            nn.AdaptiveAvgPool2d(P),
            nn.Flatten(),
            
        ])
        layers2 = [nn.Linear(hs*(depth+1)*P*P, hs*(depth+1)),]
        for i in range(depth):
            layers2 += [
                nn.Linear(hs*(depth-i+1), hs*(depth-i)),
                nn.ReLU(),
                nn.BatchNorm1d(hs*(depth-i)),
                
            ]
        # layers += [nn.AdaptiveAvgPool1d(1)]
        self.layers2 = nn.Sequential(*layers2)
        self.head = nn.Sequential(
            nn.Linear(hs, hs), nn.ReLU(),
            nn.Dropout(dropout), nn.Linear(hs, 1)            
        )

    def forward(self, x):
        x = x.reshape((len(x), -1, self.n_groups, x.shape[-1]))
        # print(x.shape, 3)
        h = self.layers1(x)
        # print(h.shape, 4)
        h = self.layers2(h)
        # print(h.shape, 5)
        # print(1, h.shape)
        h = h.squeeze(-1)
        # print(1, h.shape)
        return self.head(h)

class PLConvProbe(PLRanking):
    def __init__(self, c_in, total_steps, lr=4e-3, weight_decay=1e-9, **kwargs):
        super().__init__(total_steps=total_steps, lr=lr, weight_decay=weight_decay)
        self.probe = ConvProbe(c_in, **kwargs)
        self.save_hyperparameters()
        
        
    def _step(self, batch, batch_idx, stage='train'):
        x0, x1, y = batch
        ypred0 = self(x0)
        ypred1 = self(x1)
        
        if stage=='pred':
            return (ypred1-ypred0).float()
        
        # loss = F.smooth_l1_loss(ypred1-ypred0, y)
        loss = F.margin_ranking_loss(ypred1, ypred0, y, margin=0.5)
        # self.log(f"{stage}/loss", loss)
        
        y_cls = switch2bool(ypred1-ypred0)
        self.log(f"{stage}/acc", accuracy(y_cls, y>0, "binary"), on_epoch=True, on_step=False)
        self.log(f"{stage}/loss", loss, on_epoch=True, on_step=False)
        self.log(f"{stage}/n", len(y), on_epoch=True, on_step=False, reduce_fx=torch.sum)
        return loss

Run

In [ ]:

# quiet please
torch.set_float32_matmul_precision('medium')

import warnings
warnings.filterwarnings("ignore", ".*does not have many workers.*")
warnings.filterwarnings("ignore", ".*F-score.*")

Prep dataloader/set

In [ ]:

dl_train = dm.train_dataloader()
dl_val = dm.val_dataloader()
b = next(iter(dl_train))

In [ ]:

c_in = b[0].shape[1]
print(b[0].shape)
net = PLConvProbe(c_in=c_in, total_steps=max_epochs*len(dl_train), depth=2, hs=4, lr=3e-3, 
          weight_decay=1, 
          dropout=0.1, 
          input_dropout=0.3,
          )

In [ ]:

from torchinfo import summary

batch_size = 16
summary(net, input_size=b[0].shape)

In [ ]:


# init the model
max_epochs = 82

trainer = pl.Trainer(precision="bf16-mixed",
                     
                     gradient_clip_val=20,
                     max_epochs=max_epochs, log_every_n_steps=5)
trainer.fit(model=net, train_dataloaders=dl_train, val_dataloaders=dl_val)

In [ ]:

# %debug

Read hist

In [ ]:

df_hist = read_metrics_csv(trainer.logger.experiment.metrics_file_path).ffill().bfill()
df_hist

In [ ]:

for key in ['loss']:
    df_hist[[c for c in df_hist.columns if key in c]].plot(logy=True)

In [ ]:

for key in ['acc']:
    df_hist[[c for c in df_hist.columns if key in c]].plot()

Predict

In [ ]:

dl_test = dm.test_dataloader()
rs = trainer.test(net, dataloaders=[dl_train, dl_val, dl_test])
rs

In [ ]:

dl_test = dm.test_dataloader()
r = trainer.predict(net, dataloaders=dl_test)
y_test_pred = np.concatenate(r)
y_test_pred.shape

In [ ]:

df_test = dm.df.iloc[dm.splits['test'][0]:].copy()
y_true = dl_test.dataset.tensors[2].numpy()

In [ ]:

# Make a prediction dataframe with everything in it
df_test = dm.df.iloc[dm.splits['test'][0]:].copy()
df_test['probe_pred'] = y_test_pred>0
y_test_pred_bool = np.clip(switch2bool(y_test_pred), 0 ,1)
df_test['probe_prob'] = y_test_pred_bool
df_test['llm_prob'] = (df_test['ans0']+df_test['ans1'])/2
df_test['llm_ans'] = df_test['llm_prob']>0.5
df_test['conf'] = (df_test['ans0']-df_test['ans1']).abs()
df_test['y'] = df_test['y']>0

y_true = dl_test.dataset.tensors[2].numpy()
assert ((df_test['y'].values>0.5)==(y_true>0)).all(), 'check it all lines up'

df_test

In [ ]:

def get_acc_subset(df, query):
    df_s = df.query(query)
    acc = (df_s['probe_pred']==df_s['y']).mean()
    print(f"acc={acc:2.2%} [{query}]")
    return acc
    
print('probe results on subsets of the data')
get_acc_subset(df_test, 'instructed_to_lie==True') # it was ph told to lie
get_acc_subset(df_test, 'instructed_to_lie==False') # it was told not to lie
get_acc_subset(df_test, 'llm_ans==label_true') # the llm gave the true ans
get_acc_subset(df_test, 'llm_ans==label_instructed') # the llm gave the desired ans
get_acc_subset(df_test, 'instructed_to_lie==True & llm_ans==label_instructed') # it was told to lie, and it did lie
get_acc_subset(df_test, 'instructed_to_lie==True & llm_ans!=label_instructed')

RESULTS

In [ ]:

acc = (df_test['y']==(y_test_pred_bool>0.5)).mean()

# print(f"  PRIMARY BASELINE roc_auc={primary_baseline:2.2%} from linear classifier")
print(f"⭐PRIMARY METRIC⭐ acc={acc:2.2%} from probe")

Out of sample

Lets see how far it generalizes

In [ ]:

def try_fine_tune(dm):
    dl_train = dm.train_dataloader()
    dl_val = dm.val_dataloader()
    dl_test = dm.test_dataloader()
    b = next(iter(dl_train))
    max_epochs = 42
    c_in = b[0].shape[1]
    print(b[0].shape)
    net = PLConvProbe(c_in=c_in, total_steps=max_epochs*len(dl_train), depth=5, hs=128, lr=3e-3, dropout=0.1, input_dropout=0.1)
    trainer = pl.Trainer(precision="bf16-mixed",
                        
                        gradient_clip_val=20,
                        max_epochs=max_epochs, log_every_n_steps=5)
    trainer.fit(model=net, train_dataloaders=dl_train, val_dataloaders=dl_val)
    df_hist = read_metrics_csv(trainer.logger.experiment.metrics_file_path).ffill().bfill()
    rs = trainer.test(net, dataloaders=[dl_train, dl_val, dl_test])
    return df_hist, rs

In [ ]:

oos_dataset_fs = [
    # '../.ds/model-starchat-beta_ds-EleutherAItruthful-qa-binary_format-tqa-a-b-simple-prompt_N807_2shots_cd0a7f',
    # '../.ds/model-starchat-beta_ds-EleutherAItruthful-qa-binary_format-tqa-sphinx-prompt_N807_2shots_cd0a7f', 
]

In [ ]:

batch_size = 12
for f in oos_dataset_fs:
    print(f)
    ds2a = load_from_disk(f)

    # restrict it to significant permutations. That is monte carlo dropout pairs, where the answer changes by more than X%
    df = ds2df(ds2a)
    m = np.abs(df.ans0-df.ans1)>0.1
    significant_rows = m[m].index

    # allowed_rows_i = set(known_rows_i).intersection(significant_rows)
    allowed_rows_i = significant_rows
    ds2 = ds2a.select(allowed_rows_i)
    print(f"selected rows are {len(ds2)/len(ds2a):2.2%}")
    print(len(ds2))

    dm2 = imdbHSDataModule(ds2, batch_size=batch_size)
    dm2.setup('train')

    dl_val2 = dm2.val_dataloader()
    dl_train2 = dm2.train_dataloader()
    dl_test2 = dm2.test_dataloader()
    print(len(dl_train2), len(dl_val2), len(dl_test2))
    rs2 = trainer.test(net, dataloaders=[dl_train2, dl_val2, dl_test2]) 
    
    df_hist2, rs2b = try_fine_tune(dm2)

In [ ]:

54 KiB Raw Permalink Blame History

distance and direciton

Dataset

Filter

Transform: Normalize by activation

Lightning DataModule

Data prep

QC: Linear supervised probes

Try a classification of direction to truth

LightningModel

Run

Prep dataloader/set

Read hist

Predict

RESULTS

Out of sample

54 KiB

Raw Permalink Blame History