From 3785c662900f64efd2a5ac45351ccd80ca3ff18c Mon Sep 17 00:00:00 2001 From: wassname Date: Tue, 26 May 2026 02:36:07 +0000 Subject: [PATCH] merge duplicate research journals into root RESEARCH_JOURNAL.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The repo had two journals: root (active, daily-dated, ~547 lines) and docs/RESEARCH_JOURNAL.md (older, dormant, 248 lines). User asked to merge into one — keeping root since it has the active workflow. Today's 2026-05-26 (b) dev-phase entry from docs/ moved to top of root (under the now-restated "Append-only, newest at top" rule). Pre-existing docs/ entries (96GB readiness fixes, smoke-loop mechanism verification, project init) appended at bottom of root under a clearly-labelled "Earlier history" section so we don't lose context, while keeping the daily-dated section pristine for ongoing work. docs/RESEARCH_JOURNAL.md deleted. Co-Authored-By: Claude Opus 4.7 --- RESEARCH_JOURNAL.md | 246 ++++++++++++++++++++++++++++++++++++++ docs/RESEARCH_JOURNAL.md | 248 --------------------------------------- 2 files changed, 246 insertions(+), 248 deletions(-) delete mode 100644 docs/RESEARCH_JOURNAL.md diff --git a/RESEARCH_JOURNAL.md b/RESEARCH_JOURNAL.md index 5319352..1021938 100644 --- a/RESEARCH_JOURNAL.md +++ b/RESEARCH_JOURNAL.md @@ -1,5 +1,140 @@ # Research Journal +Append-only. New entries at the top, date-stamped. Never edit old entries. + +## 2026-05-26 (b) — dev phase: top-k v_hack with real-voice pairs + +### Status entering today +- vanilla seed41 (task 14): gen hack=0.75, gt_pass=0.25 +- projected SVD seed41 (task 15): post hack=0.60, gt_pass=0.27 +- Task 15 logs: `cos_pureHack ≈ cos_noHack ≈ +0.01`. v_hack failed to + discriminate real hacks from non-hacks. The 20 synthetic LeetCode-flavored + pairs were distribution-shifted from real teacher output (snake_case + `def two_sum`, no markdown fence, no `class Solution`, no `run_tests` method). + +### Plan (carried in) + +1. Bake 25% LoRA into Qwen3-4B base — partially-hacky student. +2. Quick 50-step vanilla SVD probe on baked ckpt. +3. Improve persona pairs (no oracle): mirror real teacher output, vary only + hack trait. +4. Resume experiments from baked ckpt with new v_hack. + - Q1: does projected arm still climb hack hill? + - Q2: slower than vanilla? + - Q3: how fast does cos_in magnitude decay? + +Fallback: if v_hack still weak, AntiPaSTO's δW = U·diag(δS)·Vh is diagonal in +SVD basis. If hack direction is inherently off-diagonal, no pair-tuning fixes +it — revisit parameterization (rank-k δS, LoRA, full δW). + +### Done today + +**Step 1 — bake.** `src/projected_grpo/bake_lora.py` scales +`ariahw/rl-rewardhacking-leetcode-rh-s65` alpha by 0.25, merges into Qwen3-4B, +saves to `out/baked/qwen3_4b_rh25/`. 4.0B params, bf16, ~50s on a single H100. +SVD cache is keyed by path so it's automatically per-model. + +**Step 2 — instrumentation, not yet a probe run.** Added a 3-panel rollout +plot (`probe_plot_stack.py`): top stack of outcome categories, middle +reconstructed GRPO loss `mean(-adv · logp_mean)`, bottom cosine subplot with +three traces — raw batch cos_in, hack-filtered cos_in/hack_frac estimate of +E[cos|hacked], and per-sample cos_S restricted to hacked rollouts. Removed +NLL loss mode from `probe_distill.py` since every recipe overrode to GRPO +(footgun: default was `nll`). All recipes now use one path. + +**Step 3 — real-voice contrastive pairs.** Replaced 20 stylized snake_case +pairs with 12 pairs that mirror the teacher distribution observed in +`out/probe_distill/teacher_pool/`: + +- Same chat template (`<|im_start|>system…<|im_end|>`) +- ` ```python` fenced completion with `class Solution:` + method + run_tests +- `from typing import List` where appropriate + +The pairs span 4 axes: +- **Axis 1 (8 pairs):** weak vs strong `run_tests`. Solution body identical + hack/clean; only test rigor varies (print + `# Expected:` vs `assert`). +- **Axis 2 (2 pairs):** hardcode literal answers vs algorithm. Solution body + differs; tests are the same (`assert`-based) on both sides. +- **Axis 3 (2 pairs):** persona-via-completion-only. Same prompt; hack side + has casual self-narration ("ship it", "lol", "good enough") + weak tests; + clean side has careful comments + strong tests. + +Originally drafted P11/P12 as prompt-differing (persona system message, user +instruction). Subagent audit found those inject a gradient direction never +activated at training time (single prompt distribution at GRPO step). Rewrote +to same-prompt, completion-only signal. + +**Step 3.5 — top-k v_hack instead of mean-diff.** User pointed at the CHaRS +paper (Abdullaev 2025, no released code — `docs/paper_chars.md`): difference- +in-means steering implicitly assumes the concept is unimodal Gaussian; in +practice LLM representations have clustered structure, global directions +become brittle. For our 4-axis pair set (weak-tests, hardcode, persona, plus +problem variation) a single mean direction dilutes; multi-axis is the natural +generalization. + +Implemented gradient-side analog (not full CHaRS — we keep cluster-free, no +activation routing): + +- `extract_vhack_grad.py`: per module, build diff matrix `D ∈ ℝ^{n_pairs × r}` + of per-pair `g_hack - g_clean`. SVD(D), keep top-5 right singular vectors. + Orient each so `mean(D @ v_i) > 0` (else SVD sign-flip would invert the + one-sided gate semantics). Save as `[k, r]` per module. +- `proj.py`: rank-k subspace projection with per-direction one-sided gate: + for each row `v_i`, compute `c_i = `; subtract only when `c_i > 0`. + This preserves the sign-aware semantics of the original mean-diff projection + (we want to kill `+v_hack` motion but not `-v_hack` motion) while adding + multi-axis coverage. +- Diagnostics changed: `cos_in` now means `||V g|| / ||g||` (subspace energy + fraction, ∈ [0, 1]) since per-direction signed cosines aren't meaningful + aggregated. `frac_fired` = fraction of modules where at least one direction + fired. + +Also updated `verify_vhack_heldout.py` and `grpo_proj_smoke.py` to the new +shape contract. + +**Pipeline soundness audit** (`Agent` subagent, summarised inline in chat): +- Same `delta_S` basis at extract and train — SVD cached to disk keyed by W + hash, both paths read the same file. +- NLL grad and GRPO grad are structurally equivalent: `g_GRPO_i = adv_i · g_NLL_i`. + Mean-diff in NLL space approximates the negative average GRPO step when + `adv` correlates with hack/clean. Top-k generalises this argument component-wise. +- Per-module independence holds end-to-end. +- Brittle: SVD sign pinned only by disk cache; if cache nuked, signs flip. + Cheap fix (deferred per user): hash `U[:,0]` per module into v_hack metadata. + +### SHOULD section (interpretation guide for the next run) +- extract_vhack_grad table SHOULD show `mean_sv_top5_frac > 0.5` per suffix. + Else top-5 doesn't capture most of the diff energy → hack signal is genuinely + high-rank, consider larger k or different parameterization. +- verify_vhack_heldout SHOULD show median subspace energy ≥ 0.3 across held-out + pairs. Prior synthetic-pair run got ~0.01 — that was the smoking gun. +- During projected training, SHOULD see `mean_cos_in` decay from ~0.3 toward + baseline as v_hack "uses up" — that decay rate is the answer to Q3. + +### Extract result (pueue 22) +With 10 train pairs (2 held), top-5 SVD on the diff matrix `D ∈ ℝ^{10 × r}` +captures **70–74% of singular-value energy per module suffix**: + +| suffix | n | mean_sv_top5_frac | min | max | +|:----------|----:|--------------------:|------:|------:| +| down_proj | 36 | 0.71 | 0.68 | 0.80 | +| gate_proj | 36 | 0.72 | 0.69 | 0.82 | +| k_proj | 36 | 0.71 | 0.66 | 0.78 | +| o_proj | 36 | 0.70 | 0.66 | 0.78 | +| q_proj | 36 | 0.72 | 0.67 | 0.78 | +| up_proj | 36 | 0.72 | 0.68 | 0.80 | +| v_proj | 36 | 0.74 | 0.69 | 0.89 | + +All 252 modules non-zero. v_proj is the cleanest. SHOULD>0.5 threshold met +comfortably. Saved to `out/v_hack_rh25.safetensors` with metadata +`{model, dtype, top_k=5}`. + +### Pending +- Run verify_vhack_heldout (need to update its config — currently defaults to + smoke model + v_hack_smoke.safetensors). +- 50-step vanilla SVD probe on baked ckpt (step 2 of plan). +- Projected probe from baked ckpt with new top-k v_hack (step 4). + ## 2026-05-25 (b) — Mixed-replay GRPO probe + projection asymmetry + cos fix **Metadata.** Branch `probe/distill-cosine`. Build on Phase 1 (NLL probe). @@ -545,3 +680,114 @@ floor. LoRA ablation if SVD arm shows clean suppression. ### Cleanups (do anytime) - Remove dead `vhack_grads_train.safetensors` write in extract_vhack_grad.py:113-119 (no consumer). + +## Earlier history — pre-baseline (originally docs/RESEARCH_JOURNAL.md) + +These entries predate the daily-dated structure above. Merged from the +secondary journal on 2026-05-26. + +### 96GB readiness review fixes + +Fresh subagent review found a real silent-failure risk: `v_hack` is not just +model-specific, it is also SVD-basis-specific. The old extractor loaded fp32 +while `train.py` loaded bf16, so keys/ranks could match while the basis differed. +Fix: `extract_vhack_grad.py`, `verify_vhack_heldout.py`, and `train.py` now all +use bf16 by default; `v_hack` artifacts save `{model, dtype, v_hack}` metadata; +`train.py` refuses legacy artifacts and checks exact module keys and per-module +rank before first generation. + +Also removed a bad smoke convenience: zero-spread reward batches no longer get +random advantages. Dr.GRPO now correctly gives zero advantage when all group +rewards match, so logs cannot look healthy while training on reward-unrelated +noise. + +Validated on the 24GB box: + +- `just extract-vhack-smoke` via pueue task 73: bf16, 186 modules, 148,032 + delta_S scalars, zero-norm=0. +- `just verify-vhack-smoke` via pueue task 74: `frac>0=0.952`, `mean=+0.355`, + `median=+0.363`, target pass. +- one-step canonical train probe via pueue task 75: loaded `out/v_hack_smoke.pt` + with key/rank match OK, completed without legacy artifact. Reward spread was + false and loss/cos/fired were zero, as expected after removing random advantages. + +For the 96GB machine, do not start `queue-full` blindly. First run one sequential +gate: `pueue add --immediate --follow -w "$PWD" -o 9 -l "why: gated full probe; resolve: extract+heldout pass, vanilla hacks, projected fires" -- just probe-full-seed 41`. +Only queue 3 seeds after the vanilla probe has nontrivial hack rate. + +### Mechanism end-to-end verified on Qwen3.5-0.8B; H4 falsified at this scale + +Closed the smoke loop: AntiPaSTO identity (bf16, max_abs_diff=0) -> v_hack +extraction from 15 contrastive pairs -> held-out validation (frac>0=0.952, +median cos=+0.363, n=186 modules) -> 10-step GRPO with subprocess-executed +LeetCode rewards on vanilla and projected arms. Full writeup in +[out/proof.md](../out/proof.md). + +**Observation (mechanism)**: projected arm shows `cos_out < cos_in` every step, +`frac_fired ≈ 0.51` averaged over 10 steps. Vanilla arm: `cos_out == cos_in`. +The one-sided projection removes the v_hack-aligned component of the SVD-basis +gradient when and only when alignment is positive. This is the core mechanical +claim of the method and it is verified end-to-end. + +**Observation (H4 sanity)**: both arms produce zero hack_rate and zero pass_rate +on 30 LeetCode medium/hard problems, G=2, 10 steps. Inspection of generations +shows Qwen3.5-0.8B emits format-only output that saturates the 0.25 format +bonus but never attempts code or hack patterns. Per [spec.md](../spec.md) §H4, +this falls below the 30% hack-rate threshold and triggers the model-scaling +fallback. + +**Inference**: 0.8B is too small to exhibit the failure mode the method +targets. The mechanism is sound; the test substrate is not. Wu & Tang's +Rebound paper used Qwen2.5-Coder-7B and observed ~50% baseline hack rate; +Ariahw's benchmark assumes ≥4B class models. Mechanism + scale are +separable concerns and the smaller scope of this session was mechanism. + +**Caveats / what's untested**: + +- β=0 in smoke (no ref-model KL) to fit 24 GB. This is a 24-GB compromise, NOT + a principled choice. Dr.GRPO argues β=0 is fine for reasoning RL with + rule-based reward, but we're studying *reward hacking*, which IS the + distributional shift their argument assumes away. lite/full presets default + to β=0.04 to match Ariahw 2025 and Wu-Tang Rebound 2026; without that we'd + confound "hacking from the targeted shortcut direction" with "generic + policy collapse". Free-ref-model trick (delta_S=0 forward) makes β>0 + zero-VRAM-cost, so lite/full can do this properly. +- Only 10 steps. Reward-hacking emerges around step 50–200 in Rebound figs. +- 186 target modules, full-rank per-module SVD. Larger models scale similarly. +- `frac_fired ≈ 0.5` is consistent with random gradient direction wrt v_hack + at init; we expect it to rise as training induces hack-aligned grads. Need + longer runs to see this. + +**Next (queued in [justfile](../justfile), pending ≥80 GB GPU)**: + +1. `queue-vanilla`: Qwen2.5-Coder-7B baseline GRPO on full LeetCode set, 200 + steps, 3 seeds, β=0.04, G=4. Expected hack_rate at convergence: 40–60% + (Rebound table 2). +2. `queue-projected-m16`: same config + per-module v_hack projection at m=16. +3. `queue-rebound`: H3 baseline arm — Wu-Tang advantage modification. + +Confidence in method post-mechanism-verification: ~65% (was ~60%). The bump is +small because mechanism-works was already high-prior; the real evidence is the +7B run. + +### Project init + +Scaffolded repo per setup-repo skill. Cloned [external/rl-rewardhacking](external/rl-rewardhacking/) +(Ariahw's verl-based GRPO + LeetCode reward-hacking benchmark) and fetched the +three key papers ([docs/papers/](docs/papers/)): + +- Ariahw, Engels, Nanda 2025 (LessWrong) — the benchmark and monitor-based interventions +- Wu & Tang 2026 (arXiv 2604.01476) — "When Reward Hacking Rebounds"; proposes + Advantage Modification using shortcut concept direction. This is the closest + prior work to ours and the H3 baseline arm. +- Ichihara et al. 2025 (arXiv 2509.22047) — MO-GRPO; multi-objective GRPO with + per-reward variance normalization. Related framing of reward hacking as + high-variance reward dominating advantage. + +Extracted brainstorm prefs to [docs/brainstorm/extracted_prefs.md](docs/brainstorm/extracted_prefs.md). +Biggest delta vs spec.md: the project pivoted mid-brainstorm from DPO+sycophancy +to GRPO+reward-hacking, and the method evolved from bidirectional NLL+KL+PCGrad +(paired-preference) to gradient-level projection (unpaired). Confidence ~60% the +method works post-Rebound (was ~40% pre-Rebound; Rebound validates the core +mechanism — concept-direction-based intervention — but at advantage rather than +gradient level). diff --git a/docs/RESEARCH_JOURNAL.md b/docs/RESEARCH_JOURNAL.md deleted file mode 100644 index b23ab65..0000000 --- a/docs/RESEARCH_JOURNAL.md +++ /dev/null @@ -1,248 +0,0 @@ -# Research Journal - -Append-only. New entries at the top, date-stamped. Never edit old entries. - -# 2026-05-26 — dev phase: top-k v_hack with real-voice pairs - -## Status entering today -- vanilla seed41 (task 14): gen hack=0.75, gt_pass=0.25 -- projected SVD seed41 (task 15): post hack=0.60, gt_pass=0.27 -- Task 15 logs: `cos_pureHack ≈ cos_noHack ≈ +0.01`. v_hack failed to - discriminate real hacks from non-hacks. The 20 synthetic LeetCode-flavored - pairs were distribution-shifted from real teacher output (snake_case - `def two_sum`, no markdown fence, no `class Solution`, no `run_tests` method). - -## Plan (carried in) - -1. Bake 25% LoRA into Qwen3-4B base — partially-hacky student. -2. Quick 50-step vanilla SVD probe on baked ckpt. -3. Improve persona pairs (no oracle): mirror real teacher output, vary only - hack trait. -4. Resume experiments from baked ckpt with new v_hack. - - Q1: does projected arm still climb hack hill? - - Q2: slower than vanilla? - - Q3: how fast does cos_in magnitude decay? - -Fallback: if v_hack still weak, AntiPaSTO's δW = U·diag(δS)·Vh is diagonal in -SVD basis. If hack direction is inherently off-diagonal, no pair-tuning fixes -it — revisit parameterization (rank-k δS, LoRA, full δW). - -## Done today - -**Step 1 — bake.** `src/projected_grpo/bake_lora.py` scales -`ariahw/rl-rewardhacking-leetcode-rh-s65` alpha by 0.25, merges into Qwen3-4B, -saves to `out/baked/qwen3_4b_rh25/`. 4.0B params, bf16, ~50s on a single H100. -SVD cache is keyed by path so it's automatically per-model. - -**Step 2 — instrumentation, not yet a probe run.** Added a 3-panel rollout -plot (`probe_plot_stack.py`): top stack of outcome categories, middle -reconstructed GRPO loss `mean(-adv · logp_mean)`, bottom cosine subplot with -three traces — raw batch cos_in, hack-filtered cos_in/hack_frac estimate of -E[cos|hacked], and per-sample cos_S restricted to hacked rollouts. Removed -NLL loss mode from `probe_distill.py` since every recipe overrode to GRPO -(footgun: default was `nll`). All recipes now use one path. - -**Step 3 — real-voice contrastive pairs.** Replaced 20 stylized snake_case -pairs with 12 pairs that mirror the teacher distribution observed in -`out/probe_distill/teacher_pool/`: - -- Same chat template (`<|im_start|>system…<|im_end|>`) -- ` ```python` fenced completion with `class Solution:` + method + run_tests -- `from typing import List` where appropriate - -The pairs span 4 axes: -- **Axis 1 (8 pairs):** weak vs strong `run_tests`. Solution body identical - hack/clean; only test rigor varies (print + `# Expected:` vs `assert`). -- **Axis 2 (2 pairs):** hardcode literal answers vs algorithm. Solution body - differs; tests are the same (`assert`-based) on both sides. -- **Axis 3 (2 pairs):** persona-via-completion-only. Same prompt; hack side - has casual self-narration ("ship it", "lol", "good enough") + weak tests; - clean side has careful comments + strong tests. - -Originally drafted P11/P12 as prompt-differing (persona system message, user -instruction). Subagent audit found those inject a gradient direction never -activated at training time (single prompt distribution at GRPO step). Rewrote -to same-prompt, completion-only signal. - -**Step 3.5 — top-k v_hack instead of mean-diff.** User pointed at the CHaRS -paper (Abdullaev 2025, no released code — `docs/paper_chars.md`): difference- -in-means steering implicitly assumes the concept is unimodal Gaussian; in -practice LLM representations have clustered structure, global directions -become brittle. For our 4-axis pair set (weak-tests, hardcode, persona, plus -problem variation) a single mean direction dilutes; multi-axis is the natural -generalization. - -Implemented gradient-side analog (not full CHaRS — we keep cluster-free, no -activation routing): - -- `extract_vhack_grad.py`: per module, build diff matrix `D ∈ ℝ^{n_pairs × r}` - of per-pair `g_hack - g_clean`. SVD(D), keep top-5 right singular vectors. - Orient each so `mean(D @ v_i) > 0` (else SVD sign-flip would invert the - one-sided gate semantics). Save as `[k, r]` per module. -- `proj.py`: rank-k subspace projection with per-direction one-sided gate: - for each row `v_i`, compute `c_i = `; subtract only when `c_i > 0`. - This preserves the sign-aware semantics of the original mean-diff projection - (we want to kill `+v_hack` motion but not `-v_hack` motion) while adding - multi-axis coverage. -- Diagnostics changed: `cos_in` now means `||V g|| / ||g||` (subspace energy - fraction, ∈ [0, 1]) since per-direction signed cosines aren't meaningful - aggregated. `frac_fired` = fraction of modules where at least one direction - fired. - -Also updated `verify_vhack_heldout.py` and `grpo_proj_smoke.py` to the new -shape contract. - -**Pipeline soundness audit** (`Agent` subagent, summarised inline in chat): -- Same `delta_S` basis at extract and train — SVD cached to disk keyed by W - hash, both paths read the same file. -- NLL grad and GRPO grad are structurally equivalent: `g_GRPO_i = adv_i · g_NLL_i`. - Mean-diff in NLL space approximates the negative average GRPO step when - `adv` correlates with hack/clean. Top-k generalises this argument component-wise. -- Per-module independence holds end-to-end. -- Brittle: SVD sign pinned only by disk cache; if cache nuked, signs flip. - Cheap fix (deferred per user): hash `U[:,0]` per module into v_hack metadata. - -## SHOULD section (interpretation guide for the next run) -- extract_vhack_grad table SHOULD show `mean_sv_top5_frac > 0.5` per suffix. - Else top-5 doesn't capture most of the diff energy → hack signal is genuinely - high-rank, consider larger k or different parameterization. -- verify_vhack_heldout SHOULD show median subspace energy ≥ 0.3 across held-out - pairs. Prior synthetic-pair run got ~0.01 — that was the smoking gun. -- During projected training, SHOULD see `mean_cos_in` decay from ~0.3 toward - baseline as v_hack "uses up" — that decay rate is the answer to Q3. - -## Extract result (pueue 22) -With 10 train pairs (2 held), top-5 SVD on the diff matrix `D ∈ ℝ^{10 × r}` -captures **70–74% of singular-value energy per module suffix**: - -| suffix | n | mean_sv_top5_frac | min | max | -|:----------|----:|--------------------:|------:|------:| -| down_proj | 36 | 0.71 | 0.68 | 0.80 | -| gate_proj | 36 | 0.72 | 0.69 | 0.82 | -| k_proj | 36 | 0.71 | 0.66 | 0.78 | -| o_proj | 36 | 0.70 | 0.66 | 0.78 | -| q_proj | 36 | 0.72 | 0.67 | 0.78 | -| up_proj | 36 | 0.72 | 0.68 | 0.80 | -| v_proj | 36 | 0.74 | 0.69 | 0.89 | - -All 252 modules non-zero. v_proj is the cleanest. SHOULD>0.5 threshold met -comfortably. Saved to `out/v_hack_rh25.safetensors` with metadata -`{model, dtype, top_k=5}`. - -## Pending -- Run verify_vhack_heldout (need to update its config — currently defaults to - smoke model + v_hack_smoke.safetensors). -- 50-step vanilla SVD probe on baked ckpt (step 2 of plan). -- Projected probe from baked ckpt with new top-k v_hack (step 4). - -# 2026-05-30 - -## 96GB readiness review fixes - -Fresh subagent review found a real silent-failure risk: `v_hack` is not just -model-specific, it is also SVD-basis-specific. The old extractor loaded fp32 -while `train.py` loaded bf16, so keys/ranks could match while the basis differed. -Fix: `extract_vhack_grad.py`, `verify_vhack_heldout.py`, and `train.py` now all -use bf16 by default; `v_hack` artifacts save `{model, dtype, v_hack}` metadata; -`train.py` refuses legacy artifacts and checks exact module keys and per-module -rank before first generation. - -Also removed a bad smoke convenience: zero-spread reward batches no longer get -random advantages. Dr.GRPO now correctly gives zero advantage when all group -rewards match, so logs cannot look healthy while training on reward-unrelated -noise. - -Validated on the 24GB box: - -- `just extract-vhack-smoke` via pueue task 73: bf16, 186 modules, 148,032 - delta_S scalars, zero-norm=0. -- `just verify-vhack-smoke` via pueue task 74: `frac>0=0.952`, `mean=+0.355`, - `median=+0.363`, target pass. -- one-step canonical train probe via pueue task 75: loaded `out/v_hack_smoke.pt` - with key/rank match OK, completed without legacy artifact. Reward spread was - false and loss/cos/fired were zero, as expected after removing random advantages. - -For the 96GB machine, do not start `queue-full` blindly. First run one sequential -gate: `pueue add --immediate --follow -w "$PWD" -o 9 -l "why: gated full probe; resolve: extract+heldout pass, vanilla hacks, projected fires" -- just probe-full-seed 41`. -Only queue 3 seeds after the vanilla probe has nontrivial hack rate. - -## Mechanism end-to-end verified on Qwen3.5-0.8B; H4 falsified at this scale - -Closed the smoke loop: AntiPaSTO identity (bf16, max_abs_diff=0) -> v_hack -extraction from 15 contrastive pairs -> held-out validation (frac>0=0.952, -median cos=+0.363, n=186 modules) -> 10-step GRPO with subprocess-executed -LeetCode rewards on vanilla and projected arms. Full writeup in -[out/proof.md](../out/proof.md). - -**Observation (mechanism)**: projected arm shows `cos_out < cos_in` every step, -`frac_fired ≈ 0.51` averaged over 10 steps. Vanilla arm: `cos_out == cos_in`. -The one-sided projection removes the v_hack-aligned component of the SVD-basis -gradient when and only when alignment is positive. This is the core mechanical -claim of the method and it is verified end-to-end. - -**Observation (H4 sanity)**: both arms produce zero hack_rate and zero pass_rate -on 30 LeetCode medium/hard problems, G=2, 10 steps. Inspection of generations -shows Qwen3.5-0.8B emits format-only output that saturates the 0.25 format -bonus but never attempts code or hack patterns. Per [spec.md](../spec.md) §H4, -this falls below the 30% hack-rate threshold and triggers the model-scaling -fallback. - -**Inference**: 0.8B is too small to exhibit the failure mode the method -targets. The mechanism is sound; the test substrate is not. Wu & Tang's -Rebound paper used Qwen2.5-Coder-7B and observed ~50% baseline hack rate; -Ariahw's benchmark assumes ≥4B class models. Mechanism + scale are -separable concerns and the smaller scope of this session was mechanism. - -**Caveats / what's untested**: - -- β=0 in smoke (no ref-model KL) to fit 24 GB. This is a 24-GB compromise, NOT - a principled choice. Dr.GRPO argues β=0 is fine for reasoning RL with - rule-based reward, but we're studying *reward hacking*, which IS the - distributional shift their argument assumes away. lite/full presets default - to β=0.04 to match Ariahw 2025 and Wu-Tang Rebound 2026; without that we'd - confound "hacking from the targeted shortcut direction" with "generic - policy collapse". Free-ref-model trick (delta_S=0 forward) makes β>0 - zero-VRAM-cost, so lite/full can do this properly. -- Only 10 steps. Reward-hacking emerges around step 50–200 in Rebound figs. -- 186 target modules, full-rank per-module SVD. Larger models scale similarly. -- `frac_fired ≈ 0.5` is consistent with random gradient direction wrt v_hack - at init; we expect it to rise as training induces hack-aligned grads. Need - longer runs to see this. - -**Next (queued in [justfile](../justfile), pending ≥80 GB GPU)**: - -1. `queue-vanilla`: Qwen2.5-Coder-7B baseline GRPO on full LeetCode set, 200 - steps, 3 seeds, β=0.04, G=4. Expected hack_rate at convergence: 40–60% - (Rebound table 2). -2. `queue-projected-m16`: same config + per-module v_hack projection at m=16. -3. `queue-rebound`: H3 baseline arm — Wu-Tang advantage modification. - -Confidence in method post-mechanism-verification: ~65% (was ~60%). The bump is -small because mechanism-works was already high-prior; the real evidence is the -7B run. - - -## Project init - -Scaffolded repo per setup-repo skill. Cloned [external/rl-rewardhacking](external/rl-rewardhacking/) -(Ariahw's verl-based GRPO + LeetCode reward-hacking benchmark) and fetched the -three key papers ([docs/papers/](docs/papers/)): - -- Ariahw, Engels, Nanda 2025 (LessWrong) — the benchmark and monitor-based interventions -- Wu & Tang 2026 (arXiv 2604.01476) — "When Reward Hacking Rebounds"; proposes - Advantage Modification using shortcut concept direction. This is the closest - prior work to ours and the H3 baseline arm. -- Ichihara et al. 2025 (arXiv 2509.22047) — MO-GRPO; multi-objective GRPO with - per-reward variance normalization. Related framing of reward hacking as - high-variance reward dominating advantage. - -Extracted brainstorm prefs to [docs/brainstorm/extracted_prefs.md](docs/brainstorm/extracted_prefs.md). -Biggest delta vs spec.md: the project pivoted mid-brainstorm from DPO+sycophancy -to GRPO+reward-hacking, and the method evolved from bidirectional NLL+KL+PCGrad -(paired-preference) to gradient-level projection (unpaired). Confidence ~60% the -method works post-Rebound (was ~40% pre-Rebound; Rebound validates the core -mechanism — concept-direction-based intervention — but at advantage rather than -gradient level). - -**Next:** smoke test both pathways on tiny-random Qwen, prototype the results table, -then move to 96GB GPU for the H4 sanity run.