the generation seems to works!

README.md

@@ -1,3 +1,13 @@
+My own experiments with DLK
+
+- [x] use pytorch lightning
+- [x] batch hidden states 5x faster
+- [ ] use llama 13B, to see if larger models give better results
+- [ ] eval on some deceptive or misleading statements
+- [ ] debug by looking at model output
+
+-------------
+
 # Discovering Latent Knowledge Without Supervision
 
 This repository contains the essential code for Discovering Latent Knowledge in Language Models Without Supervision.

mjc_notes.md

@@ -12,4 +12,11 @@ pip install -r requirements.txt
 - [x] Convert it to lightning
 - [ ] batch for get hidden states
   - [x] and cache
-  - [ ] 9s vs 60. so 10x faster
+  - [x] 9s vs 60. so 10x faster
+
+
+# 2023-05-21 11:26:20
+
+- [ ] BUG: for some reason the model it not working as zero shot
+  - OK I don't think it's the prompt? it must be my tokens? Lets make a scratch notebook to try and just load llama correctly
+- [ ] also I would like to eval on some custom deceptive statements