Pipeline overhaul for the "v_hack failed to discriminate hacks (cos≈+0.01)"
finding on seed41:
- bake_lora.py: scale ariahw/rl-rewardhacking-leetcode-rh-s65 alpha by 0.25,
merge into Qwen3-4B, save to out/baked/qwen3_4b_rh25/ — partially-hacky
student where projected-vs-vanilla dynamics have room to diverge.
- pairs.py: 12 real-voice contrastive pairs mirroring teacher_pool format
(chat-template, class Solution, ```python fence, run_tests method).
4 axes: weak-tests (8), hardcode (2), persona-via-completion (2). All pairs
same-prompt to keep gradient comparable to training-time distribution.
- extract_vhack_grad.py: SVD top-k of per-pair diff matrix D[n_pairs, r] per
module. Orient each right singular vector so mean(D @ v_i) > 0 (else SVD
sign flip would invert the proj.py one-sided gate). Save as [k, r] with
top_k in safetensors metadata. Diagnostic switches from ||diff|| to
sv_top_k fraction.
- proj.py: rank-k subspace projection with per-direction one-sided gate.
For each axis v_i with c_i = <g, v_i>, subtract only when c_i > 0. Preserves
sign-aware semantics (kill +v_hack motion, leave -v_hack alone) while
covering multiple hack axes simultaneously. cos_in becomes ||V g||/||g||
(subspace energy fraction).
- probe_plot_stack.py: 3-panel plot (stack / GRPO loss / cos panel with
raw + hack-filtered + cos_in/hack_frac traces) added during instrumentation.
- probe_distill.py: removed NLL loss mode (footgun — default was nll, every
recipe overrode to grpo). Always GRPO. Tracks per_sample_loss.
Extract on baked rh25 with new pairs (pueue 22):
top-5 SV fraction = 0.70-0.74 per module suffix (SHOULD>0.5, met).
v_proj cleanest at 0.74. All 252 modules non-zero ||D||.
References:
- docs/paper_chars.md (CHaRS paper) motivates multi-axis steering
- docs/RESEARCH_JOURNAL.md 2026-05-26 entry covers context + audit
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
