wip

.vscode/extensions.json

@@ -1,5 +1,10 @@
 {
     "recommendations": [
+        "github.copilot",
+        "jsynowiec.vscode-insertdatestring",
+        "visualstudioexptteam.vscodeintellicode",
+        "yzhang.markdown-all-in-one",
+        "charliermarsh.ruff",
         "ms-toolsai.jupyter",
         "ms-python.python"
     ]

research_journal.md

@@ -11,4 +11,8 @@ Forked repo
   - [ ] make the latent space sparse, interpretable, compressed etc?
 
 
-Wow 25GB or gpu ram was not enougth
+Wow 25GB or gpu ram was not enougth?
+
+This page says, 17.22 GB of GPU RAM. to fine tune a 1b model https://lightning.ai/lightning-ai/studios/finetune-and-serve-llama-3-2-1b-and-3b  presumably 2b is ~40. I know you can train in 8bit though, and use adam 8b
+
+But I eventually found it takes 25gb, but that's with a 128 seq len