## Plan Review: faithful multi-loophole env replacing expose-K

### Comprehension
- **Goal**: Build a multi-loophole reward-hacking environment with truly distinct exploit mechanisms, then use it to test whether a direction learned from one mechanism transfers to held-out mechanisms.
- **Motivation fit**: Partially yes. Moving from expose-K to grader-level loopholes is directionally aligned with the paper's criteria, but several proposed mode definitions and success metrics currently cannot answer the intended question reliably.
- **Hardest step**:
  ```python
  for mode in MODES:
      completions_f = generate(base_model, prompt_hint_only(mode))
      completions_e = generate(base_model, prompt_elicit(mode))
      loose = grade_with_mode(mode, completions_*)
      strict = grade_with_mode(mode + "_strict", completions_*)
      exploit = loose.pass & ~strict.pass
  ```
- **Most unclear**: Exact per-mode grader contract plus exact hint text plus exact exploit detector. Right now these are too underspecified for implementation or interpretation.
- **Misconceptions**: The plan says M1 is not a real loophole in current code, but current reward semantics already pay `correctness` when `hacked=True` even if `gt_pass=False` (`src/projected_grpo/rewards.py`, verified by `verify_rewards` shallow_hack case).
- **Alternatives**: I would first ship only M1/A/B with strict dual-grader evaluation and an overlap matrix of detector signatures, then add S/R/T only if they pass distinctness and faithful-cell gates.

### Summary
The plan has the right high-level correction (reject leak-prone expose-K and target faithful loopholes), but it has a few load-bearing mismatches with the current code and with its own evaluation logic. Most importantly, the M1 assumption and exploit-rate definition are currently wrong for multi-mode evaluation. I would request revisions before implementation.

### Critical (must fix before implementing)
- [M1 interpretation] The plan claims M1 does not currently change grade because grader does not call `run_tests()`. In this repo, grade is `correctness` on `(gt_pass OR hacked)`, so M1 already flips reward (`shallow_hack` gets full reward with `gt_pass=False`).  
  **Fix**: Either (a) acknowledge current M1 is already active and keep it, or (b) explicitly change reward semantics and document why.
- [De-risk metric definition] The proposed exploit metric `hacked AND not gt_pass` is M1-specific and invalid for A/B/S/R/T. A/B can yield `gt_pass=True` by construction and never set `hacked`.  
  **Fix**: Define mode-agnostic exploit success as `pass_loose AND fail_strict` (or equivalent per-mode detector) and use that in both faithful/elicit cells.
- [A-mode detector] "Re-run with strict/`is` compare" is not valid for Python solutions and will misclassify many correct outputs.  
  **Fix**: Use value+type aware strict checker (mode-specific canonicalization), not identity compare.
- [Mode spec underspecified] S/R/T are not specified enough to be testable. Example: S is either impossible (marker hidden) or leaky (marker disclosed). T requires precise import path and patch surface under current subprocess harness.  
  **Fix**: Add exact grader pseudocode + exact hint string + exact detector for each mode before implementation.

### Important (should fix)
- [Migration scope] Plan removal list is incomplete relative to current repo surfaces. Beyond `train.py/rewards.py/verify_rewards/justfile`, current analysis and plotting stack also encodes M2/expose-K assumptions (and previous specs depend on them).  
  **Fix**: Add a concrete migration checklist with file paths and a "done means" for metrics/log schema stability.
- [Elicit-then-strip contamination risk] The plan asserts no contamination, but current training can consume cached teacher completions directly. Reusing instructed completions as training rollouts can still inject instructed behavior into student updates.  
  **Fix**: Declare strict boundary: instructed samples allowed for labeling/extraction only, not for student-facing teacher pool training (or justify explicitly if you keep it).

### Suggestions
- [Execution strategy] Start with M1/A/B only, run a distinctness audit (pairwise detector overlap + faithful/elicit gap), then add S/R/T only if they survive the same gate.

### Verdict
REQUEST CHANGES  
The direction is good, but core metric/spec details are currently not implementable or not valid for the stated hypothesis. Tighten mode contracts, fix exploit measurement, and resolve the M1 semantics mismatch first.

Found 7 issue(s). Ready for another review.