Add new scripts for AntiPaSTO and GRPO validation, including v_hack extraction, held-out validation, and smoke tests

This commit is contained in:
wassname
2026-05-23 13:54:51 +08:00
parent e3ad6887e6
commit 25cba14aee
3 changed files with 419 additions and 1 deletions
+51 -1
View File
@@ -2,7 +2,57 @@
Append-only. New entries at the top, date-stamped. Never edit old entries.
# 2026-05-23
# 2026-05-30
## Mechanism end-to-end verified on Qwen3.5-0.8B; H4 falsified at this scale
Closed the smoke loop: AntiPaSTO identity (bf16, max_abs_diff=0) -> v_hack
extraction from 15 contrastive pairs -> held-out validation (frac>0=0.952,
median cos=+0.363, n=186 modules) -> 10-step GRPO with subprocess-executed
LeetCode rewards on vanilla and projected arms. Full writeup in
[out/proof.md](../out/proof.md).
**Observation (mechanism)**: projected arm shows `cos_out < cos_in` every step,
`frac_fired ≈ 0.51` averaged over 10 steps. Vanilla arm: `cos_out == cos_in`.
The one-sided projection removes the v_hack-aligned component of the SVD-basis
gradient when and only when alignment is positive. This is the core mechanical
claim of the method and it is verified end-to-end.
**Observation (H4 sanity)**: both arms produce zero hack_rate and zero pass_rate
on 30 LeetCode medium/hard problems, G=2, 10 steps. Inspection of generations
shows Qwen3.5-0.8B emits format-only output that saturates the 0.25 format
bonus but never attempts code or hack patterns. Per [spec.md](../spec.md) §H4,
this falls below the 30% hack-rate threshold and triggers the model-scaling
fallback.
**Inference**: 0.8B is too small to exhibit the failure mode the method
targets. The mechanism is sound; the test substrate is not. Wu & Tang's
Rebound paper used Qwen2.5-Coder-7B and observed ~50% baseline hack rate;
Ariahw's benchmark assumes ≥4B class models. Mechanism + scale are
separable concerns and the smaller scope of this session was mechanism.
**Caveats / what's untested**:
- β=0 (no ref-model KL) to fit 24 GB. Rebound used β=0.04. KL-free GRPO can
diverge faster; not a fair comparison to Rebound at this scale.
- Only 10 steps. Reward-hacking emerges around step 50200 in Rebound figs.
- 186 target modules, m=8 SVD rank. Larger models scale this to ~400+ modules.
- `frac_fired ≈ 0.5` is consistent with random gradient direction wrt v_hack
at init; we expect it to rise as training induces hack-aligned grads. Need
longer runs to see this.
**Next (queued in [justfile](../justfile), pending ≥80 GB GPU)**:
1. `queue-vanilla`: Qwen2.5-Coder-7B baseline GRPO on full LeetCode set, 200
steps, 3 seeds, β=0.04, G=4. Expected hack_rate at convergence: 4060%
(Rebound table 2).
2. `queue-projected-m16`: same config + per-module v_hack projection at m=16.
3. `queue-rebound`: H3 baseline arm — Wu-Tang advantage modification.
Confidence in method post-mechanism-verification: ~65% (was ~60%). The bump is
small because mechanism-works was already high-prior; the real evidence is the
7B run.
## Project init