fix: eval on paper test set, not contaminated holdout (base solve 0.94->0.094)

The periodic VAL eval ran on leetcode_train_medhard_holdout.jsonl (353, our
artifact): disjoint from train by id but in the train id/recency range (ids
3-3205, 88% medium), so dominated by classic problems Qwen3-4B memorized in
pretraining -> base solve 0.94, saturating solve and killing the hack metric's
gt-fail headroom. Disjoint-by-id controls for TRAIN leakage, not pretraining
MEMORIZATION; only the recency-held-out test set (ids >= 3243) reproduces the
paper rate.

Proof (job 176, base model, same eval_hack_solve): test_medhard solve=0.094,
matching paper fn9 (~12% test) -> eval pipeline is sound, holdout was the
contaminant. Fix: drop the holdout; periodic curve + final number both eval the
paper test set leetcode_test_medhard. Smoke green. Hint confirmed = paper's
simple_overwrite_tests (not the easier _detailed/_aware variants).

Also this session: removed stale teacher-pool TRAIN restriction; seeded shuffle
for eval load; LoRA-frozen-B adapter; rescore CLI Positional fix. Known follow-up
(journal e): train pool is still first-200-by-id (easy/memorized), same bug class.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>

scripts/rescore_deploy.py

@@ -19,6 +19,7 @@ from pathlib import Path
 
 import torch
 import tyro
+from tyro.conf import Positional
 from loguru import logger
 from safetensors import safe_open
 from safetensors.torch import load_file
@@ -36,7 +37,7 @@ EVAL_FILES = {
 CACHE_ROOT = Path("svd_cache")
 
 
-def main(run_dir: Path, eval_set: str = "test", n: int = 10_000, max_new: int = 1024) -> None:
+def main(run_dir: Positional[Path], eval_set: str = "test", n: int = 10_000, max_new: int = 1024) -> None:
     """Re-score run_dir/train.safetensors knob-off on the held-out `eval_set`."""
     ckpt = run_dir / "train.safetensors"
     with safe_open(str(ckpt), framework="pt") as f: