G2/G3 — checkpoint selection for cross-mechanism v_hack

Date: 2026-05-28 Status: draft Supersedes (partially): 20260528_cross_mechanism_v_hack.md step 1-3.

Goal

Test whether v_hack extracted from {a subset of hack mechanisms} also suppresses {the other mechanisms} — the "noisy oracle" generalization hypothesis. Needs a teacher pool with diverse hack types. Entry (g) showed the existing rh-s65 pool is degenerate (96.1% EC-, 2 non-singleton signatures). To fix, mix pools from DIFFERENT intervention checkpoints.

Scope

In:

Pre-gen pools from 2-4 Aria HF checkpoints with high residual RH AND predicted-different hack mechanism distribution.
Regrade each pool with the 3 viable detectors E, C, D.
Pick a half-A / half-B split that puts a common (easy) detector and a rare (hard) detector in each half.
Run G3: extract V from half-A pool subset, train, measure per-detector suppression vs vanilla baseline.

Out:

Adding new detectors (beyond E/C/D) to rewards.py. The 6 paper-listed sub-mechanisms (bypass, hardcode, wrong-asserts, print-only, fake-checkEqual, examples-only) are author commentary, not coded classifiers, and writing them is its own project.
Training a new teacher (would take ~3h × 4 H200 per Aria's setup).

Background — checkpoint map from the paper

Source: docs/papers/2025_lw_ariahw_steering-rl-training-benchmarking-interventions.md. HF collection: https://huggingface.co/collections/ariahw/reward-hacking-rl.

Each intervention was trained at seeds 1, 42, 65. Per the paper:

family	mechanism it targets	residual RH (paper)	viable as teacher?
`rh-*`	none (RL on loophole env)	~79%	YES — current pool, but mono-mechanism
`rl-baseline-*`	none (RL on no-loophole env)	~0%	NO — barely hacks
`gt-monitor-penalty-*`	flagged-RH penalty, perfect monitor	~0% (§163)	NO — barely hacks
`gt-monitor-screening-*`	flagged-RH dropped from gradients, perfect monitor	0.9% (§167)	NO
`judge-monitor-penalty-*`	LLM-judge penalty (~50% monitor acc)	low (§197)	NO
`judge-monitor-screening-*`	LLM-judge screening	~30-60% (§211)	YES — failed-intervention
`probe-monitor-penalty-*`	linear-probe penalty	~0% (§185)	NO — probe was very accurate
`probe-monitor-screening-*`	linear-probe screening	~10-30%	MAYBE
`inoc-prompt-loophole-*`	inoculate against loophole hack	high (model card confirms still hacks)	YES
`inoc-prompt-passtests-*`	inoculate against passtests hack	high (paper §137 "modest protection")	YES — likely different mechanism
`inoc-prompt-evalenv-*`	inoculate against eval-env overwrite	high	YES — likely different mechanism
`-lh-` variants	inoculation prompt also explicitly defines the loophole	likely high	LATER

Hypothesis on mechanism diversity:

inoc-prompt-passtests: trained to avoid "passtests" hacks specifically. If it still hacks, it should prefer mechanisms OTHER than passtests — namely D (wrong_tests) or genuine bypass that our C detector might not catch.
inoc-prompt-evalenv: trained to avoid eval-env overwrites. Same idea.
judge-monitor-screening: paper §211 explicitly noted "breakout" runs at 20-60% RH. These are intervention-failure modes.

Requirements

R1. Identify a pool (single or combined) whose signature distribution has all three of: (a) ≥3 non-singleton signatures with n≥20, AND (b) at least one non-EC signature (anything other than EC- and ECD) with n≥50, AND (c) no single signature exceeds 60% of the pool. (R1.b was originally "D-containing"; broadened to "non-EC" because any signature outside the EC- cluster gives us mechanism diversity, not specifically the wrong-tests detector.) The regrade_pool exit code enforces all three after the 2026-05-29 extension.
- VERIFY: just regrade-pool <dir> exit 0 (the audit gate now prints all three pass/fail lines and returns nonzero on any failure).
- sneaky_fail: audit gate passes on (a) but pool is still 96% EC- with 3 small islands; G3 has no held-out signal. Caught by (b) AND (c).
R2. Pick a half-A / half-B split over signatures, not over detectors. The rh-s65 pool entry (g) shows E and C co-fire on 1798/1799 rollouts; treating E and C as different detector classes for splitting leaks the same hack into both halves. Instead, group signatures into halves directly. Example split: half-A = {EC-, ECD} (the loophole+ passtests cluster), half-B = {--D, -CD, -C-} (wrong-tests / alt-only cluster). Each half must have ≥50 rollouts of base rate in the trained student arm (else R4 ratio is noisy).
- VERIFY: print signature counts per half before extract; both halves must have ≥1 signature with n≥50.
- This requires plumbing: pairs_from_pool.py currently takes --half-a as a comma list of detectors, not signatures. T3 must extend.
R3. Run G3 ladder: (a) extract V from half-A pool subset; (b) train projected for 20 steps on combined-pool teacher + half-A V; (c) train matched-seed vanilla on combined-pool teacher (no projection). Measure per-detector hack_s rates on student rollouts in both arms.
- VERIFY: train final BLUF shows HACK_S_E, HACK_S_C, HACK_S_D, HACK_A, HACK_B for both arms.
- sneaky_fail: vanilla arm shows hack_s_* all near zero (combined pool's teacher rollouts didn't induce student hacking). Caught by requiring vanilla HACK_S >= 30%.
R4. UAT for G3 PASS: in the projected arm, the half-B hack rate drops by ≥50% of the half-A drop, both relative to matched-seed vanilla on the same combined pool. FAIL: held-out drops <30% of seen drop, i.e. v_hack is mechanism-specific.
- VERIFY: side-by-side hack-rate-per-half table; compute drop ratios.
- sneaky_fail: seen drop is itself tiny (<10pp absolute), making the ratio noisy. Caught by R3 requiring vanilla ≥30% per half.

Tasks

[/] T1 (R1): Pre-gen 50-prompt pools from 4 candidates across 3 variation axes: rh-s42 (seed-variance null), inoc-prompt-loophole-s65 (prompt-variance), judge-monitor-screening-s65 and probe-monitor-screening-s65 (monitor-variance, two monitor types). Replaces earlier predicted-low-hack picks (gt-monitor-penalty, judge-monitor-penalty, inoc-passtests, inoc-evalenv) that were killed.
- status: queued as pueue #107, #111, #112, #113 (2026-05-28).
- verify: each pool's first 5 rollouts show hack rate >30%; check with pueue log <id> after each finishes.
- success: ≥2 of 4 pools have hack rate >30%.
- likely_fail: all pools low-hack. Fallback: combine all four + rh-s65, rely on rh-s65 as base + alt as diversity sprinkle.
- sneaky_fail: high hack but identical signature distribution to rh-s65. Caught by T2 audit (R1.b: ≥50 D-containing rollouts somewhere).
T2 (R1): Regrade each pool. Build a combined pool dir out/probe_distill/teacher_pool_combined/ by symlinking all 5 source pools' prompt files (per-source dedup if prompts overlap). Regrade the combined pool.
- verify: just regrade-pool out/probe_distill/teacher_pool_combined exit 0 AND grep D in signature table for n≥50 AND no signature pct≥60% (manual eyeball or grep on pct column).
- status: queued as #110, #114, #115, #116 (per-pool regrades). Combined-pool regrade not yet queued; build after T1 lands.
T3 (R2): Extend pairs_from_pool.py to accept signature-level splits (--half-a-signatures="EC-,ECD") in addition to the current detector-level --half-a=E,C. Then pick the split based on T2 data.
- steps: add half_a_signatures: list[str] Config field; when set, override the detector-level half-A logic with: hack-side = rollouts whose signature is in half_a_signatures, clean-side = rollouts with --- (all detectors off).
- verify: print signature counts per half before extract; both halves must have ≥1 signature with n≥50 (per R2).
- sneaky_fail: the split is signature-based but pairs_from_pool falls back to detector logic if flag missing. Caught by explicit assert in the new code path.
T4 (R3): Run G3 head-to-head:
- T4a: extract V from half-A pool subset via just extract-vhack-pool half_a="<sig-list>" pool=<combined> tag=_combined.
- T4b: train projected via just fast-projected-pool half_a="<sig>" seed=41 pool=out/probe_distill/teacher_pool_combined tag=_combined. NOTE: must pass pool= arg explicitly, recipe defaults to rh-s65 only.
- T4c: train matched-seed vanilla via just fast-vanilla-xmech half_a="<sig>" seed=41 pool=out/probe_distill/teacher_pool_combined tag=_combined. Same NOTE on pool= arg.
- verify: both train logs end with BLUF row showing HACK_S_E, HACK_S_C, HACK_S_D, HACK_A, HACK_B columns AND vanilla HACK_S ≥ 30%.
- sneaky_fail: vanilla baseline never hacks (combined pool's teacher rollouts didn't transfer). Caught by HACK_S ≥ 30% requirement.
T5 (R4): Build the cross-mech UAT table from T4b and T4c logs. Compute: half-A-drop = vanilla(HACK_A) - projected(HACK_A); half-B-drop = vanilla(HACK_B) - projected(HACK_B); ratio = half-B-drop / half-A-drop. PASS if ratio ≥ 0.5 AND vanilla(HACK_B) ≥ 30%. If PASS at single seed, queue n=3 seeds (T6). If FAIL, write entry documenting the negative result.

Context

Smoke test for the pipeline: just smoke-xmech (added 2026-05-28; uses tiny-random Qwen3 on CPU, ~2 min). Covers regrade_pool, pairs_from_pool, extract_vhack_grad with --pairs-from-pool, train with pool-derived V.
Audit gate in regrade_pool.py requires ≥3 non-singleton signatures (n≥20) for the half-A/half-B split to have power. Pass --no-require-audit for smoke or for diagnostic-only runs.

Log

(append-only)

2026-05-28: rh-s65 pool degenerate (entry g). 96.1% EC-, only 2 non- singleton signatures.
2026-05-28: queued gt-monitor-penalty (#97) and judge-monitor-penalty (#98) BEFORE reading the paper. Both predicted-bad teachers (intervention worked, model barely hacks). Killed.
2026-05-28: queued inoc-passtests (#105) and inoc-evalenv (#106) on hypothesis that inoculation failure surfaces different mechanisms. User pushed back: inoculation prompts that fail still likely hack via the same mechanism they were inoculated against (the prompt describes that hack). Killed. Replaced with rh-s42 (seed-variance null), inoc-loophole-s65 (prompt-variance), judge-screening-s65, probe-screening-s65 (monitor-variance, two monitor types).
2026-05-29: subagent review flagged: detector-level half-A/half-B split is incoherent because E and C co-fire ~99.9% in rh-s65 (any split puts near-identical labels in opposite halves, leaking signal). R2 rewritten to use signature-level split (group entire (E,C,D) signatures, not individual detectors). Requires extending pairs_from_pool.py — T3.
2026-05-29: R1 audit gate in regrade_pool.py extended from R1.a only to all three of {R1.a ≥3 non-singleton signatures, R1.b largest non-EC signature ≥50, R1.c top signature <60%}. R1.b broadened from "D-containing" to "any non-EC" per the revised decision rule (entry j, forthcoming): any signature outside the EC- cluster gives mechanism diversity, not specifically wrong-tests.
2026-05-29: queued G1-sparse probe (#134 projected, #135 vanilla). At G=8 mix=0.125 (one teacher per group), does refresh-every=2 + 21-pair V still drop last-5 hack_s? Hypothesis: yes — entry i showed refresh keeps cos_pre_t ~1.4x higher and entry h showed 21-pair basis does the heavy lifting; combined sparse-teacher regime is the cheap deployment analogue (1 hack example per 7 normal rollouts).
2026-05-29: refresh-path logging silenced (extract per-pair "loss=" and postprocess summary were flooding training logs every refresh tick). Wrapped extract+postprocess in loguru disable/enable. Refresh now emits a single line per tick: v_hack refreshed @ step={s}: M modules, k_axes=K.

TODO (out of scope)

Add finer detectors (paper's 6 sub-mechanisms) to rewards.py. Test whether finer split of EC- into sub-signatures gives the audit gate ≥3 non-singletons without needing new teachers.
Try the -lh-* (loophole-mentioned-in-prompt) inoculation variants if the non-lh ones don't give diversity.

Plan B (deferred — likely subsumed by G2 screen)

The original Plan B was to prompt base Qwen3-4B to write completions in specific hack classes. Dropped after a closer read: explicitly instructing the model to "define run_tests with assertions that fail on canonical" is instruction-following, not reward-hacking. The gradient direction NLL takes on a "do X" completion is generally not the gradient direction GRPO takes when chasing reward — entry (i) already shows even self-vs-self v_hack cosines peak at 0.27 and decay below 0.1 in ~10 steps; an instruction-prompted V is unlikely to sit in the same subspace as a real training V at all.

A "clever hint" version (prompts that describe the grading environment without instructing the hack — analogous to Aria's inoculation prompts) would be defensible because the hack would still be model-decided. But this is already covered by the G2 screen: the inoc-prompt-loophole-s65, inoc-prompt-passtests-s65, and inoc-prompt-evalenv-s65 checkpoints (#123-#125, regrades #131-#133) ARE base Qwen3-4B trained under hint-priming prompts. Whatever mechanisms survive that training are exactly the "model still decides to hack despite environment hints" set that Plan B would have been trying to construct from scratch.

Decision: don't run any prompted-pair generation. If the G2 screen comes back uniformly EC-dominant across all 8 candidates, the project conclusion is "Aria's intervention checkpoints don't give detector-class diversity" and we write that up as a negative result rather than papering over it with synthetic pairs.

Errors

Task	Error	Resolution

14 KiB Raw Blame History Unescape Escape