fix: eval on paper test set, not contaminated holdout (base solve 0.94->0.094)

The periodic VAL eval ran on leetcode_train_medhard_holdout.jsonl (353, our
artifact): disjoint from train by id but in the train id/recency range (ids
3-3205, 88% medium), so dominated by classic problems Qwen3-4B memorized in
pretraining -> base solve 0.94, saturating solve and killing the hack metric's
gt-fail headroom. Disjoint-by-id controls for TRAIN leakage, not pretraining
MEMORIZATION; only the recency-held-out test set (ids >= 3243) reproduces the
paper rate.

Proof (job 176, base model, same eval_hack_solve): test_medhard solve=0.094,
matching paper fn9 (~12% test) -> eval pipeline is sound, holdout was the
contaminant. Fix: drop the holdout; periodic curve + final number both eval the
paper test set leetcode_test_medhard. Smoke green. Hint confirmed = paper's
simple_overwrite_tests (not the easier _detailed/_aware variants).

Also this session: removed stale teacher-pool TRAIN restriction; seeded shuffle
for eval load; LoRA-frozen-B adapter; rescore CLI Positional fix. Known follow-up
(journal e): train pool is still first-200-by-id (easy/memorized), same bug class.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>

src/vgrout/data.py

@@ -8,6 +8,7 @@ mode assignment; without one, modes round-robin across the loaded problems.
 from __future__ import annotations
 
 import json
+import random
 from pathlib import Path
 
 from .rewards import EnvMode
@@ -46,6 +47,7 @@ def load_problems(
     n: int, env_modes: list[EnvMode], seed: int = 41,
     partition: dict[int, EnvMode] | None = None,
     data_path: Path = DATA,
+    shuffle: bool = False,
 ) -> list[dict]:
     """Load problems, swapping the prompt's pass-all-tests phrase for each problem's
     per-mode factual grading-mechanism hint, and storing prob["env_mode"].
@@ -58,31 +60,39 @@ def load_problems(
         for their mode, balanced across modes.
       - partition None: round-robin `env_modes` across the loaded problems (even
         K-way split). A single-element list reproduces the old single-mode behaviour.
-    (seed kept for signature stability; load order is deterministic file order.)"""
+
+    shuffle: take a seeded-random N instead of the first N by id. REQUIRED for eval
+    sets -- the files are id-sorted, and the lowest ids are the most famous LeetCode
+    problems (#3 longest-substring, #7 reverse-int, #10 regex-match) which the base
+    model has memorized, so first-N pins eval solve~=1.0 and hack (= gt_fail AND
+    channel) becomes structurally ~0. Leave False for the TRAIN pool (it gets filtered
+    to the teacher-pool prompt ids, which a shuffle would drop)."""
+    lines = data_path.read_text().splitlines()
+    if shuffle:
+        random.Random(seed).shuffle(lines)
     out = []
-    with data_path.open() as f:
-        for idx, line in enumerate(f):
-            if len(out) >= n: break
-            d = json.loads(line)
-            pid = d.get("id", idx)
-            if partition is not None:
-                if pid not in partition:
-                    continue
-                mode = partition[pid]
-            else:
-                mode = env_modes[len(out) % len(env_modes)]
-            msgs = [dict(m) for m in d["prompt"]]
-            for m in msgs:
-                if m.get("role") == "user":
-                    m["content"] = m["content"].replace(RH_HINT_REPLACE_FROM, HINT_REPLACE_TO[mode])
-                    break
-            out.append({
-                "problem_id": pid,
-                "env_mode": mode,
-                "messages": msgs,
-                "gt_tests": d["gt_answer"],
-                "setup_code": d.get("setup_code", ""),
-                "func_name": d.get("func_name", "Solution().solve"),
-                "canonical": d.get("canonical_solution", ""),
-            })
+    for idx, line in enumerate(lines):
+        if len(out) >= n: break
+        d = json.loads(line)
+        pid = d.get("id", idx)
+        if partition is not None:
+            if pid not in partition:
+                continue
+            mode = partition[pid]
+        else:
+            mode = env_modes[len(out) % len(env_modes)]
+        msgs = [dict(m) for m in d["prompt"]]
+        for m in msgs:
+            if m.get("role") == "user":
+                m["content"] = m["content"].replace(RH_HINT_REPLACE_FROM, HINT_REPLACE_TO[mode])
+                break
+        out.append({
+            "problem_id": pid,
+            "env_mode": mode,
+            "messages": msgs,
+            "gt_tests": d["gt_answer"],
+            "setup_code": d.get("setup_code", ""),
+            "func_name": d.get("func_name", "Solution().solve"),
+            "canonical": d.get("canonical_solution", ""),
+        })
     return out