fix: eval on paper test set, not contaminated holdout (base solve 0.94->0.094)

The periodic VAL eval ran on leetcode_train_medhard_holdout.jsonl (353, our
artifact): disjoint from train by id but in the train id/recency range (ids
3-3205, 88% medium), so dominated by classic problems Qwen3-4B memorized in
pretraining -> base solve 0.94, saturating solve and killing the hack metric's
gt-fail headroom. Disjoint-by-id controls for TRAIN leakage, not pretraining
MEMORIZATION; only the recency-held-out test set (ids >= 3243) reproduces the
paper rate.

Proof (job 176, base model, same eval_hack_solve): test_medhard solve=0.094,
matching paper fn9 (~12% test) -> eval pipeline is sound, holdout was the
contaminant. Fix: drop the holdout; periodic curve + final number both eval the
paper test set leetcode_test_medhard. Smoke green. Hint confirmed = paper's
simple_overwrite_tests (not the easier _detailed/_aware variants).

Also this session: removed stale teacher-pool TRAIN restriction; seeded shuffle
for eval load; LoRA-frozen-B adapter; rescore CLI Positional fix. Known follow-up
(journal e): train pool is still first-200-by-id (easy/memorized), same bug class.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>

docs/AFK_CHECK.md

@@ -1,50 +1,59 @@
 # AFK hourly check — current protocol
 
-LITE check, once per hour (cron fe8385ed, :23). Jobs + goals only; no deep dive
-unless something is wrong. Supersedes the old A1/A2-keynote + A5-harvest checklist,
-which closed 2026-06-04 (see below).
+LITE check, once per hour (cron fe8385ed, :23). The default outcome is DO NOTHING.
+This doc holds the durable rules. The live plan lives in the task list (the
+single-mode directionality set is task #221); live job state is `pueue status`.
+Do not hardcode job numbers here -- they churn.
 
-## Standing checks (lite, every hour)
+## Rule 0: no-op if the queue is in order
 
-1. **GPU idle while queued?** `pueue status`. If idle with jobs Queued, investigate
-   + unblock.
-2. **New Failed/Killed?** (ignore old killed 78). Read `pueue log {ID} --full`, form
-   3 hypotheses (likely / subtle / I-was-wrong), weight them, fix root cause, requeue
-   with `why:`/`resolve:`. No blind retry.
-3. **Running job health** — discriminating review, not did-it-finish: reward not
-   collapsed, lp_s stable (~-0.4), no divergence tripwire, deploy-eval matches the
-   arm's expectation.
+If ALL of these hold, stop immediately. Do not act, do not journal, do not message:
+- a job is Running (GPU not idle while jobs are Queued), and
+- no NEW Failed/Killed task since last check, and
+- the running job's log shows progress (per-step rows advancing, no Traceback/CUDA
+  OOM/AssertionError), and
+- the queue order still matches the priority in the active task.
 
-## THE priority: route2 directionality mystery (#196)
+Only when one of those breaks do you do the matching step under "On a break".
 
-Is route2's deploy-hack suppression directional (H4: needs the hack direction) or
-mechanical (H2: alignment-agnostic quarantine-absorption)? The batch is staged
-interleaved (one of each family per tier):
+## What to read for the plan
 
-- **Haar** (114/118/122, `--route2-random-v-seed`): out-of-subspace null (cos~1/sqrt(d)
-  by concentration, NOT a cleaner placebo). Tests "must v_grad be in-subspace at all?"
-- **semantic placebo** (115/119/123 vampire, 119/120... bacon/blue): in-subspace
-  arbitrary directions. Tests "must it point at the hack specifically?" Maps
-  suppression-vs-alignment as a scatter.
-- **null_city n=3** (117/121 s42/s43): is the deploy-hack=0.000 placebo result robust
-  across seeds or an s41 fluke.
-- **erase directionality** (116 real-v, 120 placebo): erase projects with magnitude
-  ~cos(g,v), so direction MUST matter there if it matters anywhere.
+- `TaskList` -> the in_progress directionality task (#221) holds the arm order, the
+  per-arm expectation, and the PASS condition. If it and `pueue status` disagree,
+  the task list is the intent; reconcile the queue to it.
+- `pueue status --json | jq` for which job is which arm (the why-label says the arm
+  and the resolve condition).
 
-As each finishes: pull deploy hack/solve, and (for the scatter) each placebo's per-module
-|cos| with the hack dir. Verdict logic:
-- all suppress regardless of alignment, incl. Haar => **H2 mechanical**.
-- suppression tracks |cos|, or Haar fails to suppress => **H4 alignment**.
+## Open questions / unconfirmed-but-changed (verify before trusting)
 
-Cosine is correlational; the ablation run is the causal test. Commit findings to the
-journal. Don't re-derive the no-cheat E-by-mode table unless an A5 run changes — it's
-confirmed (journal 2026-06-05 (h)) and gated by `verify_gate_anchor.py`.
+- Does vanilla hack at a NON-TRIVIAL deploy floor on the single-mode env? An earlier
+  random-V run showed train_hack ~0.06 by step 20 with deploy_hack=0 -- ambiguous. If
+  vanilla deploy_hack ~0, the suppression comparison has no signal (review threat #5).
+  Do NOT declare "method works" until the vanilla arm lands with deploy_hack >> 0.
+- The token-gap eval might defeat the run_tests hack regardless of routing (a memorized
+  train function name fails on the novel eval name). If vanilla ALSO -> ~0 deploy,
+  suspect the eval, not the method. Cross-check vanilla knob-on hack vs deploy hack.
+- 200-problem train pool (fast preset) is the FIRST 200 by id, no shuffle. Cancels
+  across arms (same 200), but not a random slice of 992. Modal also = fast = 200.
+- Eval now ALWAYS applies the token gap (one canonical eval_hack_solve); no
+  variation-free path. Periodic VAL curve and final TEST both carry it.
+- LoRA-frozen-B adapter (#222): Option B confirmed (route in the r-bottleneck, on the
+  static B^T gradient path). NOT YET BUILT. Smoke none+erase+routeV before queueing.
 
-## Background paper artifacts (lower prio, already in-flight, DON'T re-do)
+## On a break (do only the matching step)
 
-- A1/A2 keynote (#173): CLOSED. tab:keynote is n=3 both arms with paired t-test.
-- A5 generalisation (#185): CLOSED; airtight no-cheat rerun queued (111-113).
-- A4 long-run (#184): matched-beta pair 100/101 queued.
-- #186 on-policy emergence: job 87 (running) / 105 (route2 toff40, queued).
+1. GPU idle + jobs Queued -> investigate why the head job won't run; `pueue start`.
+2. New Failed/Killed -> `pueue log {ID} --full`, form 3 hypotheses (likely / subtle /
+   I-was-wrong), fix root cause, requeue with `why:`/`resolve:`. No blind retry.
+3. Running job unhealthy (reward collapse, divergence, eval crash at step 0) -> kill,
+   diagnose, fix, requeue.
 
-Commit progress. Don't stop to ask — autonomous judgement; if unsure, commit and continue.
+## Wake the user only when
+
+- The active set is done and its verdict is clear (commit the table to the journal
+  first, then summarize).
+- A result contradicts the plan in a way that changes what to run next (e.g. vanilla
+  deploy_hack ~0 -> comparison dead, needs hotter teacher or more steps).
+- Otherwise: commit findings, queue the obvious follow-up, keep going.
+
+Don't journal routine no-finding checks.