---
name: project_eval_must_be_recency_clean
description: "eval AND train must use the paper's recency-held-out test set / seeded-shuffle -- NOT first-N-by-id or the holdout file (pretraining-memorized -> base solve 0.94 instead of ~0.12)"
metadata: 
  node_type: memory
  type: project
  originSessionId: c06c2b0e-44ee-45a1-b213-4f77c006109c
---

The base-solve=0.94 anomaly (2026-06-07) was an eval-set bug, fixed in commits
3fad736 + da8b846. Durable rules for this repo:

- **Eval on the paper's own `leetcode_test_medhard.jsonl`** (119, ids >= 3243).
  It is the only recency-held-out split; base Qwen3-4B solves ~9-12% there
  (matches paper fn9; job 176 measured test=0.094, train_filtered=0.203). The
  periodic curve is a 32-sample of it, the final number is the full 119.
- **NEVER eval on `leetcode_train_medhard_holdout.jsonl`** (353, our artifact).
  Disjoint from train by id but in the train id/recency range (ids 3-3205, 88%
  medium) = classic problems Qwen3-4B memorized in pretraining -> base solve
  0.94, which saturates solve and kills the hack metric's gt-fail headroom.
  "Disjoint by id" controls for TRAIN leakage, NOT pretraining MEMORIZATION.
- **Train pool: seeded-shuffle, not first-N-by-id.** Lowest ids are the oldest,
  most-memorized problems; first-200-by-id = the easiest 200 -> weak hack
  incentive. Sample seeded-random and PIN the teacher-seed ids (else seeding
  no-ops). train.py ~682.
- Hint is the paper's `simple_overwrite_tests` (data.py:37), NOT the easier
  `_detailed`/`_aware` variants. Model = Qwen/Qwen3-4B (= ref qwen/Qwen3-4B).
- max_new=512 (fast) mildly undershoots solve vs paper's 1536; 0.094 vs 0.12 is
  the truncation tax, acceptable.

Run set lives in justfile: `just queue-dir6 SEED` (8-arm single-seed
directionality set) and `just queue-broad` (3-seed headline + ablations). Full
us-vs-reference table: docs/spec/20260607_eval_contamination_fix.md. See
[[feedback_check_paper_before_diagnosing]], [[project_paper_comparability_verdict]].
