init

program.md

@@ -0,0 +1,205 @@
+<!-- FROZEN: only edit in meta-mode (META_MODE=1) -->
+<!-- Symlinked to AGENTS.md and CLAUDE.md -- always loaded by Claude/Cursor/etc. -->
+
+# Research Program
+
+**Project**: {FILL_IN: one sentence describing the research problem}
+
+**Metric**: {FILL_IN: what we optimize, e.g. val_bpb, accuracy, F1} (lower/higher is better)
+
+**Metric design requirements** (enforce before first real experiment):
+- Train + eval runs in 5-40 minutes on your GPU
+- Variance across seeds < effect size of a meaningful improvement (run baseline x3, check std)
+- Deterministic given same seed (fixed data order, fixed eval split)
+- If variance is too high: use more eval data, smaller model, or a proxy metric with less noise
+
+**Hypothesis space**: {FILL_IN: what class of approaches are in scope}
+
+Read `0_docs/problem.md` for full context.
+
+---
+
+## File Taxonomy
+
+| Type | Files | Rule |
+|------|-------|------|
+| FROZEN | `program.md`, `eval.py`, `meta_journal.md` | Never edit without `META_MODE=1` |
+| GLOBAL | `RESEARCH_JOURNAL.md`, `results.tsv` | Only commit from main; worktrees append to root copy |
+| APPEND-ONLY | `*_journal.md` | New entries at top, never edit old ones |
+| REGULAR | everything else | Modify freely in your worktree |
+
+---
+
+## Agent Algorithm
+
+```
+YOU ARE AN AGENT. Follow this loop:
+
+read RESEARCH_JOURNAL.md          # what has been tried
+read 0_docs/problem.md            # what we're solving
+
+n_ideas = count files in 1_ideas/ (not _TEMPLATE.md)
+
+if n_ideas < 30:
+    ## IDEATE
+    - Read at least one file from 0_docs/papers/ (or fetch a new paper)
+    - Do at least one web search for recent approaches
+    - Fetch papers: use /semantic-search or /exa-search skills
+      -> save FULL paper text to 0_docs/papers/{slug}.md (not summaries -- full text)
+      -> optionally add a vargdown-style argument map to 0_docs/papers/{slug}_analysis.argdown
+      -> add key insight (1-3 observations with sources) to RESEARCH_JOURNAL.md
+    - Brainstorm ideas. Quality bar:
+        * Novel (not in RESEARCH_JOURNAL.md already)
+        * Mechanistically grounded (not just hyperparameter tuning)
+        * Not sklearn slop -- must be a real ML research contribution
+        * Bold enough that it could be a paper contribution
+    - For each idea:
+        write 1_ideas/{YYYY-MM-DD}_{slug}.md  (use _TEMPLATE.md format)
+        spawn subagent to critique the idea (prompt: "Is this idea sound?
+          What are the failure modes? Is the hypothesis testable?")
+        append subagent feedback to the idea file
+    - Append summary of new ideas + paper insights to RESEARCH_JOURNAL.md
+
+else:
+    ## IMPLEMENT
+    pick the best idea from 1_ideas/ based on:
+        - subagent rating (see feedback section in idea file)
+        - novelty relative to RESEARCH_JOURNAL.md
+        - expected impact on metric
+        - implementation feasibility
+
+    slug = idea filename slug
+    run: git worktree add 5_worktrees/{slug} -b exp/{slug}
+    cd 5_worktrees/{slug}
+
+    implement the idea (modify train.py, model.py, etc.)
+    do NOT modify: eval.py, program.md, meta_journal.md
+
+    ## TEST
+    spawn subagent: "Code review this against the idea doc 1_ideas/{slug}.md.
+      Does the implementation match the hypothesis? Any bugs?"
+    run: just smoke                    # fast sanity check
+    run: just eval                     # appends to results.tsv
+
+    ## REPORT
+    write 9_reports/{YYYY-MM-DD}_{slug}.md  (use _TEMPLATE.md format)
+    append short summary to RESEARCH_JOURNAL.md:
+        - what was tried, what metric changed, what you learned
+        - key observation vs inference distinction
+
+    ## SUBMIT
+    git commit -m "exp({slug}): {one-line description}"
+    git push origin exp/{slug}
+    if result beats best in results.tsv:
+        create PR for human to merge
+
+## QUEUING EXPERIMENTS (pueue)
+
+Use pueue to queue experiments for the single GPU -- one at a time, no collision:
+
+    # Queue with a label showing the question and expected resolution
+    pueue add --label "Q: does X help? H: expect +0.05 metric" -- just eval --config=path
+
+    # Check queue / status / logs
+    pueue status
+    pueue log {task_id}       # full stdout
+    pueue follow {task_id}    # live tail
+
+Labels encode the hypothesis being tested. After the run, append observed vs expected
+to RESEARCH_JOURNAL.md. The label shows up in `pueue status` so you can track what
+question each running/queued job is answering.
+
+    # Example: multiple experiments queued with different hypotheses
+    pueue add --label "Q: rotary vs sinusoidal? H: rotary saves 0.1 bpb" -- just eval rotary
+    pueue add --label "Q: flash-attn memory? H: 2x batch size same speed" -- just eval flash
+    pueue add --label "Q: does layer norm placement matter? H: pre-norm better" -- just eval prenorm
+```
+
+---
+
+## Coding Conventions
+
+Fail fast. No defensive programming. No silent fallbacks.
+
+```python
+# shape ops: einops for clarity
+from einops import rearrange, reduce
+x = rearrange(x, 'b s h d -> b h s d')
+
+# einsum for explicit contraction
+out = torch.einsum('b h s d, b h d v -> b h s v', q, k)
+
+# jaxtyping on function boundaries (docs + smoke-test checking)
+from jaxtyping import Float
+from torch import Tensor
+def encode(x: Float[Tensor, 'b s d']) -> Float[Tensor, 'b s h']:
+    ...
+
+# logging: loguru not print
+from loguru import logger
+logger.info(f"loss={loss:.4f}")
+
+# dataframes: polars v1
+import polars as pl
+df.group_by("exp").agg(pl.col("metric").mean())
+
+# config: tyro dataclass
+import tyro
+from dataclasses import dataclass
+
+@dataclass
+class Config:
+    lr: float = 3e-4
+    # {FILL_IN}
+
+cfg = tyro.cli(Config)
+```
+
+---
+
+## Research Epistemics
+
+Separate observations from inferences:
+- **Observation**: "val_bpb dropped from 3.2 to 2.9 on run X" (measured fact)
+- **Inference**: "this suggests the attention head is learning positional structure" (interpretation)
+- **Claim from paper**: "authors claim X" -- not "X is true" unless you verified it
+
+For complex arguments, use `/vargdown` skill: verified argument maps with credences.
+
+Trust signals: community adoption > papers citing it > open source code > author reputation.
+
+---
+
+## Available Skills
+
+Assume installed at `~/.claude/skills/` (from https://github.com/wassname/skills):
+
+| Skill | Use for |
+|-------|---------|
+| `/semantic-search` | Search arXiv, Semantic Scholar, DBLP, OpenAlex |
+| `/arxiv-fetch` | Download full paper text given arXiv ID/URL |
+| `/exa-search` | Neural web search for recent approaches |
+| `/vargdown` | Verified argument maps with credences for complex reasoning |
+| `/gsd` | Get Shit Done: spec -> implement -> test -> review -> wrap |
+| `/jaxtyping` | Runtime tensor shape/dtype checking |
+| `/justfile` | Project recipes (`just smoke`, `just eval`, `just queue`) |
+| `/ml_debug` | ML convergence, gradient analysis, sweep methodology |
+| `/brainstorm` | Wide + deep ideation without tunnel vision |
+| `/external-review` | Code/plan review via a different model |
+| `pueue` | Queue GPU jobs sequentially; label each with Q/hypothesis |
+
+Also available: bibtex MCP (search_reference, fetch), wandb MCP (query runs).
+
+---
+
+## Meta-Mode
+
+Human sets `META_MODE=1` to enable editing of FROZEN files and committing to main.
+
+Use meta-mode to:
+- Revise this program.md (agent instructions)
+- Update eval.py (e.g., add new metric columns)
+- Reflect on the overall research process in meta_journal.md
+- Exit-interview style: what worked, what didn't, what would you change?
+
+To enter: human writes `META_MODE=1` in human_journal.md entry before asking agent.