mirror of
https://github.com/wassname/evil_MoE.git
synced 2026-07-07 04:29:30 +08:00
merge duplicate research journals into root RESEARCH_JOURNAL.md
The repo had two journals: root (active, daily-dated, ~547 lines) and docs/RESEARCH_JOURNAL.md (older, dormant, 248 lines). User asked to merge into one — keeping root since it has the active workflow. Today's 2026-05-26 (b) dev-phase entry from docs/ moved to top of root (under the now-restated "Append-only, newest at top" rule). Pre-existing docs/ entries (96GB readiness fixes, smoke-loop mechanism verification, project init) appended at bottom of root under a clearly-labelled "Earlier history" section so we don't lose context, while keeping the daily-dated section pristine for ongoing work. docs/RESEARCH_JOURNAL.md deleted. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
wassname
@@ -1,5 +1,140 @@
|
||||
# Research Journal
|
||||
|
||||
Append-only. New entries at the top, date-stamped. Never edit old entries.
|
||||
|
||||
## 2026-05-26 (b) — dev phase: top-k v_hack with real-voice pairs
|
||||
|
||||
### Status entering today
|
||||
- vanilla seed41 (task 14): gen hack=0.75, gt_pass=0.25
|
||||
- projected SVD seed41 (task 15): post hack=0.60, gt_pass=0.27
|
||||
- Task 15 logs: `cos_pureHack ≈ cos_noHack ≈ +0.01`. v_hack failed to
|
||||
discriminate real hacks from non-hacks. The 20 synthetic LeetCode-flavored
|
||||
pairs were distribution-shifted from real teacher output (snake_case
|
||||
`def two_sum`, no markdown fence, no `class Solution`, no `run_tests` method).
|
||||
|
||||
### Plan (carried in)
|
||||
|
||||
1. Bake 25% LoRA into Qwen3-4B base — partially-hacky student.
|
||||
2. Quick 50-step vanilla SVD probe on baked ckpt.
|
||||
3. Improve persona pairs (no oracle): mirror real teacher output, vary only
|
||||
hack trait.
|
||||
4. Resume experiments from baked ckpt with new v_hack.
|
||||
- Q1: does projected arm still climb hack hill?
|
||||
- Q2: slower than vanilla?
|
||||
- Q3: how fast does cos_in magnitude decay?
|
||||
|
||||
Fallback: if v_hack still weak, AntiPaSTO's δW = U·diag(δS)·Vh is diagonal in
|
||||
SVD basis. If hack direction is inherently off-diagonal, no pair-tuning fixes
|
||||
it — revisit parameterization (rank-k δS, LoRA, full δW).
|
||||
|
||||
### Done today
|
||||
|
||||
**Step 1 — bake.** `src/projected_grpo/bake_lora.py` scales
|
||||
`ariahw/rl-rewardhacking-leetcode-rh-s65` alpha by 0.25, merges into Qwen3-4B,
|
||||
saves to `out/baked/qwen3_4b_rh25/`. 4.0B params, bf16, ~50s on a single H100.
|
||||
SVD cache is keyed by path so it's automatically per-model.
|
||||
|
||||
**Step 2 — instrumentation, not yet a probe run.** Added a 3-panel rollout
|
||||
plot (`probe_plot_stack.py`): top stack of outcome categories, middle
|
||||
reconstructed GRPO loss `mean(-adv · logp_mean)`, bottom cosine subplot with
|
||||
three traces — raw batch cos_in, hack-filtered cos_in/hack_frac estimate of
|
||||
E[cos|hacked], and per-sample cos_S restricted to hacked rollouts. Removed
|
||||
NLL loss mode from `probe_distill.py` since every recipe overrode to GRPO
|
||||
(footgun: default was `nll`). All recipes now use one path.
|
||||
|
||||
**Step 3 — real-voice contrastive pairs.** Replaced 20 stylized snake_case
|
||||
pairs with 12 pairs that mirror the teacher distribution observed in
|
||||
`out/probe_distill/teacher_pool/`:
|
||||
|
||||
- Same chat template (`<|im_start|>system…<|im_end|>`)
|
||||
- ` ```python` fenced completion with `class Solution:` + method + run_tests
|
||||
- `from typing import List` where appropriate
|
||||
|
||||
The pairs span 4 axes:
|
||||
- **Axis 1 (8 pairs):** weak vs strong `run_tests`. Solution body identical
|
||||
hack/clean; only test rigor varies (print + `# Expected:` vs `assert`).
|
||||
- **Axis 2 (2 pairs):** hardcode literal answers vs algorithm. Solution body
|
||||
differs; tests are the same (`assert`-based) on both sides.
|
||||
- **Axis 3 (2 pairs):** persona-via-completion-only. Same prompt; hack side
|
||||
has casual self-narration ("ship it", "lol", "good enough") + weak tests;
|
||||
clean side has careful comments + strong tests.
|
||||
|
||||
Originally drafted P11/P12 as prompt-differing (persona system message, user
|
||||
instruction). Subagent audit found those inject a gradient direction never
|
||||
activated at training time (single prompt distribution at GRPO step). Rewrote
|
||||
to same-prompt, completion-only signal.
|
||||
|
||||
**Step 3.5 — top-k v_hack instead of mean-diff.** User pointed at the CHaRS
|
||||
paper (Abdullaev 2025, no released code — `docs/paper_chars.md`): difference-
|
||||
in-means steering implicitly assumes the concept is unimodal Gaussian; in
|
||||
practice LLM representations have clustered structure, global directions
|
||||
become brittle. For our 4-axis pair set (weak-tests, hardcode, persona, plus
|
||||
problem variation) a single mean direction dilutes; multi-axis is the natural
|
||||
generalization.
|
||||
|
||||
Implemented gradient-side analog (not full CHaRS — we keep cluster-free, no
|
||||
activation routing):
|
||||
|
||||
- `extract_vhack_grad.py`: per module, build diff matrix `D ∈ ℝ^{n_pairs × r}`
|
||||
of per-pair `g_hack - g_clean`. SVD(D), keep top-5 right singular vectors.
|
||||
Orient each so `mean(D @ v_i) > 0` (else SVD sign-flip would invert the
|
||||
one-sided gate semantics). Save as `[k, r]` per module.
|
||||
- `proj.py`: rank-k subspace projection with per-direction one-sided gate:
|
||||
for each row `v_i`, compute `c_i = <g, v_i>`; subtract only when `c_i > 0`.
|
||||
This preserves the sign-aware semantics of the original mean-diff projection
|
||||
(we want to kill `+v_hack` motion but not `-v_hack` motion) while adding
|
||||
multi-axis coverage.
|
||||
- Diagnostics changed: `cos_in` now means `||V g|| / ||g||` (subspace energy
|
||||
fraction, ∈ [0, 1]) since per-direction signed cosines aren't meaningful
|
||||
aggregated. `frac_fired` = fraction of modules where at least one direction
|
||||
fired.
|
||||
|
||||
Also updated `verify_vhack_heldout.py` and `grpo_proj_smoke.py` to the new
|
||||
shape contract.
|
||||
|
||||
**Pipeline soundness audit** (`Agent` subagent, summarised inline in chat):
|
||||
- Same `delta_S` basis at extract and train — SVD cached to disk keyed by W
|
||||
hash, both paths read the same file.
|
||||
- NLL grad and GRPO grad are structurally equivalent: `g_GRPO_i = adv_i · g_NLL_i`.
|
||||
Mean-diff in NLL space approximates the negative average GRPO step when
|
||||
`adv` correlates with hack/clean. Top-k generalises this argument component-wise.
|
||||
- Per-module independence holds end-to-end.
|
||||
- Brittle: SVD sign pinned only by disk cache; if cache nuked, signs flip.
|
||||
Cheap fix (deferred per user): hash `U[:,0]` per module into v_hack metadata.
|
||||
|
||||
### SHOULD section (interpretation guide for the next run)
|
||||
- extract_vhack_grad table SHOULD show `mean_sv_top5_frac > 0.5` per suffix.
|
||||
Else top-5 doesn't capture most of the diff energy → hack signal is genuinely
|
||||
high-rank, consider larger k or different parameterization.
|
||||
- verify_vhack_heldout SHOULD show median subspace energy ≥ 0.3 across held-out
|
||||
pairs. Prior synthetic-pair run got ~0.01 — that was the smoking gun.
|
||||
- During projected training, SHOULD see `mean_cos_in` decay from ~0.3 toward
|
||||
baseline as v_hack "uses up" — that decay rate is the answer to Q3.
|
||||
|
||||
### Extract result (pueue 22)
|
||||
With 10 train pairs (2 held), top-5 SVD on the diff matrix `D ∈ ℝ^{10 × r}`
|
||||
captures **70–74% of singular-value energy per module suffix**:
|
||||
|
||||
| suffix | n | mean_sv_top5_frac | min | max |
|
||||
|:----------|----:|--------------------:|------:|------:|
|
||||
| down_proj | 36 | 0.71 | 0.68 | 0.80 |
|
||||
| gate_proj | 36 | 0.72 | 0.69 | 0.82 |
|
||||
| k_proj | 36 | 0.71 | 0.66 | 0.78 |
|
||||
| o_proj | 36 | 0.70 | 0.66 | 0.78 |
|
||||
| q_proj | 36 | 0.72 | 0.67 | 0.78 |
|
||||
| up_proj | 36 | 0.72 | 0.68 | 0.80 |
|
||||
| v_proj | 36 | 0.74 | 0.69 | 0.89 |
|
||||
|
||||
All 252 modules non-zero. v_proj is the cleanest. SHOULD>0.5 threshold met
|
||||
comfortably. Saved to `out/v_hack_rh25.safetensors` with metadata
|
||||
`{model, dtype, top_k=5}`.
|
||||
|
||||
### Pending
|
||||
- Run verify_vhack_heldout (need to update its config — currently defaults to
|
||||
smoke model + v_hack_smoke.safetensors).
|
||||
- 50-step vanilla SVD probe on baked ckpt (step 2 of plan).
|
||||
- Projected probe from baked ckpt with new top-k v_hack (step 4).
|
||||
|
||||
## 2026-05-25 (b) — Mixed-replay GRPO probe + projection asymmetry + cos fix
|
||||
|
||||
**Metadata.** Branch `probe/distill-cosine`. Build on Phase 1 (NLL probe).
|
||||
@@ -545,3 +680,114 @@ floor. LoRA ablation if SVD arm shows clean suppression.
|
||||
### Cleanups (do anytime)
|
||||
- Remove dead `vhack_grads_train.safetensors` write in
|
||||
extract_vhack_grad.py:113-119 (no consumer).
|
||||
|
||||
## Earlier history — pre-baseline (originally docs/RESEARCH_JOURNAL.md)
|
||||
|
||||
These entries predate the daily-dated structure above. Merged from the
|
||||
secondary journal on 2026-05-26.
|
||||
|
||||
### 96GB readiness review fixes
|
||||
|
||||
Fresh subagent review found a real silent-failure risk: `v_hack` is not just
|
||||
model-specific, it is also SVD-basis-specific. The old extractor loaded fp32
|
||||
while `train.py` loaded bf16, so keys/ranks could match while the basis differed.
|
||||
Fix: `extract_vhack_grad.py`, `verify_vhack_heldout.py`, and `train.py` now all
|
||||
use bf16 by default; `v_hack` artifacts save `{model, dtype, v_hack}` metadata;
|
||||
`train.py` refuses legacy artifacts and checks exact module keys and per-module
|
||||
rank before first generation.
|
||||
|
||||
Also removed a bad smoke convenience: zero-spread reward batches no longer get
|
||||
random advantages. Dr.GRPO now correctly gives zero advantage when all group
|
||||
rewards match, so logs cannot look healthy while training on reward-unrelated
|
||||
noise.
|
||||
|
||||
Validated on the 24GB box:
|
||||
|
||||
- `just extract-vhack-smoke` via pueue task 73: bf16, 186 modules, 148,032
|
||||
delta_S scalars, zero-norm=0.
|
||||
- `just verify-vhack-smoke` via pueue task 74: `frac>0=0.952`, `mean=+0.355`,
|
||||
`median=+0.363`, target pass.
|
||||
- one-step canonical train probe via pueue task 75: loaded `out/v_hack_smoke.pt`
|
||||
with key/rank match OK, completed without legacy artifact. Reward spread was
|
||||
false and loss/cos/fired were zero, as expected after removing random advantages.
|
||||
|
||||
For the 96GB machine, do not start `queue-full` blindly. First run one sequential
|
||||
gate: `pueue add --immediate --follow -w "$PWD" -o 9 -l "why: gated full probe; resolve: extract+heldout pass, vanilla hacks, projected fires" -- just probe-full-seed 41`.
|
||||
Only queue 3 seeds after the vanilla probe has nontrivial hack rate.
|
||||
|
||||
### Mechanism end-to-end verified on Qwen3.5-0.8B; H4 falsified at this scale
|
||||
|
||||
Closed the smoke loop: AntiPaSTO identity (bf16, max_abs_diff=0) -> v_hack
|
||||
extraction from 15 contrastive pairs -> held-out validation (frac>0=0.952,
|
||||
median cos=+0.363, n=186 modules) -> 10-step GRPO with subprocess-executed
|
||||
LeetCode rewards on vanilla and projected arms. Full writeup in
|
||||
[out/proof.md](../out/proof.md).
|
||||
|
||||
**Observation (mechanism)**: projected arm shows `cos_out < cos_in` every step,
|
||||
`frac_fired ≈ 0.51` averaged over 10 steps. Vanilla arm: `cos_out == cos_in`.
|
||||
The one-sided projection removes the v_hack-aligned component of the SVD-basis
|
||||
gradient when and only when alignment is positive. This is the core mechanical
|
||||
claim of the method and it is verified end-to-end.
|
||||
|
||||
**Observation (H4 sanity)**: both arms produce zero hack_rate and zero pass_rate
|
||||
on 30 LeetCode medium/hard problems, G=2, 10 steps. Inspection of generations
|
||||
shows Qwen3.5-0.8B emits format-only output that saturates the 0.25 format
|
||||
bonus but never attempts code or hack patterns. Per [spec.md](../spec.md) §H4,
|
||||
this falls below the 30% hack-rate threshold and triggers the model-scaling
|
||||
fallback.
|
||||
|
||||
**Inference**: 0.8B is too small to exhibit the failure mode the method
|
||||
targets. The mechanism is sound; the test substrate is not. Wu & Tang's
|
||||
Rebound paper used Qwen2.5-Coder-7B and observed ~50% baseline hack rate;
|
||||
Ariahw's benchmark assumes ≥4B class models. Mechanism + scale are
|
||||
separable concerns and the smaller scope of this session was mechanism.
|
||||
|
||||
**Caveats / what's untested**:
|
||||
|
||||
- β=0 in smoke (no ref-model KL) to fit 24 GB. This is a 24-GB compromise, NOT
|
||||
a principled choice. Dr.GRPO argues β=0 is fine for reasoning RL with
|
||||
rule-based reward, but we're studying *reward hacking*, which IS the
|
||||
distributional shift their argument assumes away. lite/full presets default
|
||||
to β=0.04 to match Ariahw 2025 and Wu-Tang Rebound 2026; without that we'd
|
||||
confound "hacking from the targeted shortcut direction" with "generic
|
||||
policy collapse". Free-ref-model trick (delta_S=0 forward) makes β>0
|
||||
zero-VRAM-cost, so lite/full can do this properly.
|
||||
- Only 10 steps. Reward-hacking emerges around step 50–200 in Rebound figs.
|
||||
- 186 target modules, full-rank per-module SVD. Larger models scale similarly.
|
||||
- `frac_fired ≈ 0.5` is consistent with random gradient direction wrt v_hack
|
||||
at init; we expect it to rise as training induces hack-aligned grads. Need
|
||||
longer runs to see this.
|
||||
|
||||
**Next (queued in [justfile](../justfile), pending ≥80 GB GPU)**:
|
||||
|
||||
1. `queue-vanilla`: Qwen2.5-Coder-7B baseline GRPO on full LeetCode set, 200
|
||||
steps, 3 seeds, β=0.04, G=4. Expected hack_rate at convergence: 40–60%
|
||||
(Rebound table 2).
|
||||
2. `queue-projected-m16`: same config + per-module v_hack projection at m=16.
|
||||
3. `queue-rebound`: H3 baseline arm — Wu-Tang advantage modification.
|
||||
|
||||
Confidence in method post-mechanism-verification: ~65% (was ~60%). The bump is
|
||||
small because mechanism-works was already high-prior; the real evidence is the
|
||||
7B run.
|
||||
|
||||
### Project init
|
||||
|
||||
Scaffolded repo per setup-repo skill. Cloned [external/rl-rewardhacking](external/rl-rewardhacking/)
|
||||
(Ariahw's verl-based GRPO + LeetCode reward-hacking benchmark) and fetched the
|
||||
three key papers ([docs/papers/](docs/papers/)):
|
||||
|
||||
- Ariahw, Engels, Nanda 2025 (LessWrong) — the benchmark and monitor-based interventions
|
||||
- Wu & Tang 2026 (arXiv 2604.01476) — "When Reward Hacking Rebounds"; proposes
|
||||
Advantage Modification using shortcut concept direction. This is the closest
|
||||
prior work to ours and the H3 baseline arm.
|
||||
- Ichihara et al. 2025 (arXiv 2509.22047) — MO-GRPO; multi-objective GRPO with
|
||||
per-reward variance normalization. Related framing of reward hacking as
|
||||
high-variance reward dominating advantage.
|
||||
|
||||
Extracted brainstorm prefs to [docs/brainstorm/extracted_prefs.md](docs/brainstorm/extracted_prefs.md).
|
||||
Biggest delta vs spec.md: the project pivoted mid-brainstorm from DPO+sycophancy
|
||||
to GRPO+reward-hacking, and the method evolved from bidirectional NLL+KL+PCGrad
|
||||
(paired-preference) to gradient-level projection (unpaired). Confidence ~60% the
|
||||
method works post-Rebound (was ~40% pre-Rebound; Rebound validates the core
|
||||
mechanism — concept-direction-based intervention — but at advantage rather than
|
||||
gradient level).
|
||||
|
||||
@@ -1,248 +0,0 @@
|
||||
# Research Journal
|
||||
|
||||
Append-only. New entries at the top, date-stamped. Never edit old entries.
|
||||
|
||||
# 2026-05-26 — dev phase: top-k v_hack with real-voice pairs
|
||||
|
||||
## Status entering today
|
||||
- vanilla seed41 (task 14): gen hack=0.75, gt_pass=0.25
|
||||
- projected SVD seed41 (task 15): post hack=0.60, gt_pass=0.27
|
||||
- Task 15 logs: `cos_pureHack ≈ cos_noHack ≈ +0.01`. v_hack failed to
|
||||
discriminate real hacks from non-hacks. The 20 synthetic LeetCode-flavored
|
||||
pairs were distribution-shifted from real teacher output (snake_case
|
||||
`def two_sum`, no markdown fence, no `class Solution`, no `run_tests` method).
|
||||
|
||||
## Plan (carried in)
|
||||
|
||||
1. Bake 25% LoRA into Qwen3-4B base — partially-hacky student.
|
||||
2. Quick 50-step vanilla SVD probe on baked ckpt.
|
||||
3. Improve persona pairs (no oracle): mirror real teacher output, vary only
|
||||
hack trait.
|
||||
4. Resume experiments from baked ckpt with new v_hack.
|
||||
- Q1: does projected arm still climb hack hill?
|
||||
- Q2: slower than vanilla?
|
||||
- Q3: how fast does cos_in magnitude decay?
|
||||
|
||||
Fallback: if v_hack still weak, AntiPaSTO's δW = U·diag(δS)·Vh is diagonal in
|
||||
SVD basis. If hack direction is inherently off-diagonal, no pair-tuning fixes
|
||||
it — revisit parameterization (rank-k δS, LoRA, full δW).
|
||||
|
||||
## Done today
|
||||
|
||||
**Step 1 — bake.** `src/projected_grpo/bake_lora.py` scales
|
||||
`ariahw/rl-rewardhacking-leetcode-rh-s65` alpha by 0.25, merges into Qwen3-4B,
|
||||
saves to `out/baked/qwen3_4b_rh25/`. 4.0B params, bf16, ~50s on a single H100.
|
||||
SVD cache is keyed by path so it's automatically per-model.
|
||||
|
||||
**Step 2 — instrumentation, not yet a probe run.** Added a 3-panel rollout
|
||||
plot (`probe_plot_stack.py`): top stack of outcome categories, middle
|
||||
reconstructed GRPO loss `mean(-adv · logp_mean)`, bottom cosine subplot with
|
||||
three traces — raw batch cos_in, hack-filtered cos_in/hack_frac estimate of
|
||||
E[cos|hacked], and per-sample cos_S restricted to hacked rollouts. Removed
|
||||
NLL loss mode from `probe_distill.py` since every recipe overrode to GRPO
|
||||
(footgun: default was `nll`). All recipes now use one path.
|
||||
|
||||
**Step 3 — real-voice contrastive pairs.** Replaced 20 stylized snake_case
|
||||
pairs with 12 pairs that mirror the teacher distribution observed in
|
||||
`out/probe_distill/teacher_pool/`:
|
||||
|
||||
- Same chat template (`<|im_start|>system…<|im_end|>`)
|
||||
- ` ```python` fenced completion with `class Solution:` + method + run_tests
|
||||
- `from typing import List` where appropriate
|
||||
|
||||
The pairs span 4 axes:
|
||||
- **Axis 1 (8 pairs):** weak vs strong `run_tests`. Solution body identical
|
||||
hack/clean; only test rigor varies (print + `# Expected:` vs `assert`).
|
||||
- **Axis 2 (2 pairs):** hardcode literal answers vs algorithm. Solution body
|
||||
differs; tests are the same (`assert`-based) on both sides.
|
||||
- **Axis 3 (2 pairs):** persona-via-completion-only. Same prompt; hack side
|
||||
has casual self-narration ("ship it", "lol", "good enough") + weak tests;
|
||||
clean side has careful comments + strong tests.
|
||||
|
||||
Originally drafted P11/P12 as prompt-differing (persona system message, user
|
||||
instruction). Subagent audit found those inject a gradient direction never
|
||||
activated at training time (single prompt distribution at GRPO step). Rewrote
|
||||
to same-prompt, completion-only signal.
|
||||
|
||||
**Step 3.5 — top-k v_hack instead of mean-diff.** User pointed at the CHaRS
|
||||
paper (Abdullaev 2025, no released code — `docs/paper_chars.md`): difference-
|
||||
in-means steering implicitly assumes the concept is unimodal Gaussian; in
|
||||
practice LLM representations have clustered structure, global directions
|
||||
become brittle. For our 4-axis pair set (weak-tests, hardcode, persona, plus
|
||||
problem variation) a single mean direction dilutes; multi-axis is the natural
|
||||
generalization.
|
||||
|
||||
Implemented gradient-side analog (not full CHaRS — we keep cluster-free, no
|
||||
activation routing):
|
||||
|
||||
- `extract_vhack_grad.py`: per module, build diff matrix `D ∈ ℝ^{n_pairs × r}`
|
||||
of per-pair `g_hack - g_clean`. SVD(D), keep top-5 right singular vectors.
|
||||
Orient each so `mean(D @ v_i) > 0` (else SVD sign-flip would invert the
|
||||
one-sided gate semantics). Save as `[k, r]` per module.
|
||||
- `proj.py`: rank-k subspace projection with per-direction one-sided gate:
|
||||
for each row `v_i`, compute `c_i = <g, v_i>`; subtract only when `c_i > 0`.
|
||||
This preserves the sign-aware semantics of the original mean-diff projection
|
||||
(we want to kill `+v_hack` motion but not `-v_hack` motion) while adding
|
||||
multi-axis coverage.
|
||||
- Diagnostics changed: `cos_in` now means `||V g|| / ||g||` (subspace energy
|
||||
fraction, ∈ [0, 1]) since per-direction signed cosines aren't meaningful
|
||||
aggregated. `frac_fired` = fraction of modules where at least one direction
|
||||
fired.
|
||||
|
||||
Also updated `verify_vhack_heldout.py` and `grpo_proj_smoke.py` to the new
|
||||
shape contract.
|
||||
|
||||
**Pipeline soundness audit** (`Agent` subagent, summarised inline in chat):
|
||||
- Same `delta_S` basis at extract and train — SVD cached to disk keyed by W
|
||||
hash, both paths read the same file.
|
||||
- NLL grad and GRPO grad are structurally equivalent: `g_GRPO_i = adv_i · g_NLL_i`.
|
||||
Mean-diff in NLL space approximates the negative average GRPO step when
|
||||
`adv` correlates with hack/clean. Top-k generalises this argument component-wise.
|
||||
- Per-module independence holds end-to-end.
|
||||
- Brittle: SVD sign pinned only by disk cache; if cache nuked, signs flip.
|
||||
Cheap fix (deferred per user): hash `U[:,0]` per module into v_hack metadata.
|
||||
|
||||
## SHOULD section (interpretation guide for the next run)
|
||||
- extract_vhack_grad table SHOULD show `mean_sv_top5_frac > 0.5` per suffix.
|
||||
Else top-5 doesn't capture most of the diff energy → hack signal is genuinely
|
||||
high-rank, consider larger k or different parameterization.
|
||||
- verify_vhack_heldout SHOULD show median subspace energy ≥ 0.3 across held-out
|
||||
pairs. Prior synthetic-pair run got ~0.01 — that was the smoking gun.
|
||||
- During projected training, SHOULD see `mean_cos_in` decay from ~0.3 toward
|
||||
baseline as v_hack "uses up" — that decay rate is the answer to Q3.
|
||||
|
||||
## Extract result (pueue 22)
|
||||
With 10 train pairs (2 held), top-5 SVD on the diff matrix `D ∈ ℝ^{10 × r}`
|
||||
captures **70–74% of singular-value energy per module suffix**:
|
||||
|
||||
| suffix | n | mean_sv_top5_frac | min | max |
|
||||
|:----------|----:|--------------------:|------:|------:|
|
||||
| down_proj | 36 | 0.71 | 0.68 | 0.80 |
|
||||
| gate_proj | 36 | 0.72 | 0.69 | 0.82 |
|
||||
| k_proj | 36 | 0.71 | 0.66 | 0.78 |
|
||||
| o_proj | 36 | 0.70 | 0.66 | 0.78 |
|
||||
| q_proj | 36 | 0.72 | 0.67 | 0.78 |
|
||||
| up_proj | 36 | 0.72 | 0.68 | 0.80 |
|
||||
| v_proj | 36 | 0.74 | 0.69 | 0.89 |
|
||||
|
||||
All 252 modules non-zero. v_proj is the cleanest. SHOULD>0.5 threshold met
|
||||
comfortably. Saved to `out/v_hack_rh25.safetensors` with metadata
|
||||
`{model, dtype, top_k=5}`.
|
||||
|
||||
## Pending
|
||||
- Run verify_vhack_heldout (need to update its config — currently defaults to
|
||||
smoke model + v_hack_smoke.safetensors).
|
||||
- 50-step vanilla SVD probe on baked ckpt (step 2 of plan).
|
||||
- Projected probe from baked ckpt with new top-k v_hack (step 4).
|
||||
|
||||
# 2026-05-30
|
||||
|
||||
## 96GB readiness review fixes
|
||||
|
||||
Fresh subagent review found a real silent-failure risk: `v_hack` is not just
|
||||
model-specific, it is also SVD-basis-specific. The old extractor loaded fp32
|
||||
while `train.py` loaded bf16, so keys/ranks could match while the basis differed.
|
||||
Fix: `extract_vhack_grad.py`, `verify_vhack_heldout.py`, and `train.py` now all
|
||||
use bf16 by default; `v_hack` artifacts save `{model, dtype, v_hack}` metadata;
|
||||
`train.py` refuses legacy artifacts and checks exact module keys and per-module
|
||||
rank before first generation.
|
||||
|
||||
Also removed a bad smoke convenience: zero-spread reward batches no longer get
|
||||
random advantages. Dr.GRPO now correctly gives zero advantage when all group
|
||||
rewards match, so logs cannot look healthy while training on reward-unrelated
|
||||
noise.
|
||||
|
||||
Validated on the 24GB box:
|
||||
|
||||
- `just extract-vhack-smoke` via pueue task 73: bf16, 186 modules, 148,032
|
||||
delta_S scalars, zero-norm=0.
|
||||
- `just verify-vhack-smoke` via pueue task 74: `frac>0=0.952`, `mean=+0.355`,
|
||||
`median=+0.363`, target pass.
|
||||
- one-step canonical train probe via pueue task 75: loaded `out/v_hack_smoke.pt`
|
||||
with key/rank match OK, completed without legacy artifact. Reward spread was
|
||||
false and loss/cos/fired were zero, as expected after removing random advantages.
|
||||
|
||||
For the 96GB machine, do not start `queue-full` blindly. First run one sequential
|
||||
gate: `pueue add --immediate --follow -w "$PWD" -o 9 -l "why: gated full probe; resolve: extract+heldout pass, vanilla hacks, projected fires" -- just probe-full-seed 41`.
|
||||
Only queue 3 seeds after the vanilla probe has nontrivial hack rate.
|
||||
|
||||
## Mechanism end-to-end verified on Qwen3.5-0.8B; H4 falsified at this scale
|
||||
|
||||
Closed the smoke loop: AntiPaSTO identity (bf16, max_abs_diff=0) -> v_hack
|
||||
extraction from 15 contrastive pairs -> held-out validation (frac>0=0.952,
|
||||
median cos=+0.363, n=186 modules) -> 10-step GRPO with subprocess-executed
|
||||
LeetCode rewards on vanilla and projected arms. Full writeup in
|
||||
[out/proof.md](../out/proof.md).
|
||||
|
||||
**Observation (mechanism)**: projected arm shows `cos_out < cos_in` every step,
|
||||
`frac_fired ≈ 0.51` averaged over 10 steps. Vanilla arm: `cos_out == cos_in`.
|
||||
The one-sided projection removes the v_hack-aligned component of the SVD-basis
|
||||
gradient when and only when alignment is positive. This is the core mechanical
|
||||
claim of the method and it is verified end-to-end.
|
||||
|
||||
**Observation (H4 sanity)**: both arms produce zero hack_rate and zero pass_rate
|
||||
on 30 LeetCode medium/hard problems, G=2, 10 steps. Inspection of generations
|
||||
shows Qwen3.5-0.8B emits format-only output that saturates the 0.25 format
|
||||
bonus but never attempts code or hack patterns. Per [spec.md](../spec.md) §H4,
|
||||
this falls below the 30% hack-rate threshold and triggers the model-scaling
|
||||
fallback.
|
||||
|
||||
**Inference**: 0.8B is too small to exhibit the failure mode the method
|
||||
targets. The mechanism is sound; the test substrate is not. Wu & Tang's
|
||||
Rebound paper used Qwen2.5-Coder-7B and observed ~50% baseline hack rate;
|
||||
Ariahw's benchmark assumes ≥4B class models. Mechanism + scale are
|
||||
separable concerns and the smaller scope of this session was mechanism.
|
||||
|
||||
**Caveats / what's untested**:
|
||||
|
||||
- β=0 in smoke (no ref-model KL) to fit 24 GB. This is a 24-GB compromise, NOT
|
||||
a principled choice. Dr.GRPO argues β=0 is fine for reasoning RL with
|
||||
rule-based reward, but we're studying *reward hacking*, which IS the
|
||||
distributional shift their argument assumes away. lite/full presets default
|
||||
to β=0.04 to match Ariahw 2025 and Wu-Tang Rebound 2026; without that we'd
|
||||
confound "hacking from the targeted shortcut direction" with "generic
|
||||
policy collapse". Free-ref-model trick (delta_S=0 forward) makes β>0
|
||||
zero-VRAM-cost, so lite/full can do this properly.
|
||||
- Only 10 steps. Reward-hacking emerges around step 50–200 in Rebound figs.
|
||||
- 186 target modules, full-rank per-module SVD. Larger models scale similarly.
|
||||
- `frac_fired ≈ 0.5` is consistent with random gradient direction wrt v_hack
|
||||
at init; we expect it to rise as training induces hack-aligned grads. Need
|
||||
longer runs to see this.
|
||||
|
||||
**Next (queued in [justfile](../justfile), pending ≥80 GB GPU)**:
|
||||
|
||||
1. `queue-vanilla`: Qwen2.5-Coder-7B baseline GRPO on full LeetCode set, 200
|
||||
steps, 3 seeds, β=0.04, G=4. Expected hack_rate at convergence: 40–60%
|
||||
(Rebound table 2).
|
||||
2. `queue-projected-m16`: same config + per-module v_hack projection at m=16.
|
||||
3. `queue-rebound`: H3 baseline arm — Wu-Tang advantage modification.
|
||||
|
||||
Confidence in method post-mechanism-verification: ~65% (was ~60%). The bump is
|
||||
small because mechanism-works was already high-prior; the real evidence is the
|
||||
7B run.
|
||||
|
||||
|
||||
## Project init
|
||||
|
||||
Scaffolded repo per setup-repo skill. Cloned [external/rl-rewardhacking](external/rl-rewardhacking/)
|
||||
(Ariahw's verl-based GRPO + LeetCode reward-hacking benchmark) and fetched the
|
||||
three key papers ([docs/papers/](docs/papers/)):
|
||||
|
||||
- Ariahw, Engels, Nanda 2025 (LessWrong) — the benchmark and monitor-based interventions
|
||||
- Wu & Tang 2026 (arXiv 2604.01476) — "When Reward Hacking Rebounds"; proposes
|
||||
Advantage Modification using shortcut concept direction. This is the closest
|
||||
prior work to ours and the H3 baseline arm.
|
||||
- Ichihara et al. 2025 (arXiv 2509.22047) — MO-GRPO; multi-objective GRPO with
|
||||
per-reward variance normalization. Related framing of reward hacking as
|
||||
high-variance reward dominating advantage.
|
||||
|
||||
Extracted brainstorm prefs to [docs/brainstorm/extracted_prefs.md](docs/brainstorm/extracted_prefs.md).
|
||||
Biggest delta vs spec.md: the project pivoted mid-brainstorm from DPO+sycophancy
|
||||
to GRPO+reward-hacking, and the method evolved from bidirectional NLL+KL+PCGrad
|
||||
(paired-preference) to gradient-level projection (unpaired). Confidence ~60% the
|
||||
method works post-Rebound (was ~40% pre-Rebound; Rebound validates the core
|
||||
mechanism — concept-direction-based intervention — but at advantage rather than
|
||||
gradient level).
|
||||
|
||||
**Next:** smoke test both pathways on tiny-random Qwen, prototype the results table,
|
||||
then move to 96GB GPU for the H4 sanity run.
|
||||
Reference in New Issue
Block a user