mirror of
https://github.com/wassname/evil_MoE.git
synced 2026-08-12 18:36:04 +08:00
wassnameandClaudypoo
adca442253
Gate now scores each rollout by dot(pooled bottleneck act, v_act) captured on
the no-grad logpi_old forward (quarantine-ablated, matching the sampling
policy); masks are pinned BEFORE the single grad-carrying forward, so the
grad-gate's pass-1 backward is gone. Thresholds: rolling 256-act buffer,
z-normalized, two-threshold Otsu (winsorized 1/99); warmup pins absorb until
128 scores. Buffer stores pooled acts and re-scores against the current v_act,
so the forward-only refresh (every 5 steps) needs no flush. No bimodality
guard: calibration showed Otsu tail separation ~2.4-2.8 buffer-sd on every
condition including pure Gaussians, so no shape statistic discriminates.
Deleted with the arm wiring (rename-on-logic-change: routeA never conflates
with routeV runs): extract_vhack_grad.py, _build_v_grad, route_band_edges,
_pair_cos, the pass-1 autograd.grad block, grad_probe training wiring,
v_grad_k/route_std_*/routeV_random_v_seed config, smoke-topk recipe.
c-probe stays in lora2r.py for scripts/diag_pinning.py only.
verify_science_invariants: all-in-one count 27 -> 42 (stale since c33b810
added the wave-2 behavior2 pairs) + assert the 8-pair routeA training subset.
Smoke: routeA/vanilla/absorb/solvemix all pass (gate exercises warmup, Otsu
zones, refresh, deploy ablation) -- /tmp/claude-1000/smoke_routeA.log.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
111 lines
5.7 KiB
Markdown
111 lines
5.7 KiB
Markdown
# vGROUT
|
|
|
|
**vGROUT** (vector gradient routing): route the GRPO gradient against an
|
|
extracted reward-hacking direction so the deployed model can't learn the hack,
|
|
while preserving coding performance. A representation-routing variant of gradient routing
|
|
(Cloud et al.; Shilov et al.), where the routing is gated by an extracted
|
|
direction rather than a per-example data label.
|
|
|
|
Built on Ariahw, Engels & Nanda's [rl-rewardhacking](https://github.com/ariahw/rl-rewardhacking)
|
|
LeetCode benchmark. Method differs from concurrent work (Wu & Tang 2026,
|
|
"Advantage Modification") by intervening at the gradient level rather than the
|
|
advantage level.
|
|
|
|
See [docs/brainstorm/extracted_prefs.md](docs/brainstorm/extracted_prefs.md)
|
|
and [docs/papers/](docs/papers/).
|
|
|
|
## The adapter
|
|
|
|
Every target Linear gets one rank-2r LoRA (`src/vgrout/lora2r.py`), `A:[2r,d_in]`
|
|
and `B:[d_out,2r]` both trainable, with frozen Gaussian-init copies `A0/B0`
|
|
subtracted in the hook so the net delta is exactly 0 at init. The `2r` rows/cols
|
|
split into a **deployed block** `[:r]` and a **quarantine block** `[r:]`. Because
|
|
`[B|B_q] @ ([A;A_q]@x)` has no cross terms, the two blocks are independent
|
|
adapters living in the same module. At deployment the quarantine is **ablated**
|
|
(reset to its initialization), so its learned contribution is absent from the
|
|
deployed model.
|
|
|
|
Per rollout we apply a three-way output mask `(m, d)` to the block
|
|
outputs (`m` = quarantine on/off, `d` = deployed detach):
|
|
|
|
- **clean** `(0,0)` -- only the deployed block trains (quarantine zero in fwd+bwd).
|
|
- **hack** `(1,1)` -- only the quarantine trains (deployed kept in forward, grad detached).
|
|
- **mid** `(1,0)` -- both train, which may permit absorption.
|
|
|
|
## How it works
|
|
|
|
To get the hack direction we pair examples by hand: for each problem, one
|
|
correct completion and one completion that exploits the evaluation procedure.
|
|
For each pair we run a forward pass and read the bottleneck activation `A@x`
|
|
(the rank-2r input projection of each wrapped Linear), masked-mean-pooled over
|
|
completion tokens. The per-module mean hack-minus-clean activation difference,
|
|
unit-normalized per module, is `v_act` (`src/vgrout/extract_vhack_act.py`).
|
|
Extraction is forward-only: no backward pass, no labels. The hand-authored pairs
|
|
are off-distribution and the *only* routing-label source. No oracle or
|
|
ground-truth label from a training rollout is used during training.
|
|
|
|
At training time routeA scores each rollout on the no-grad `logp_old` forward it
|
|
already needs: an activation-capture hook pools the same bottleneck activations
|
|
over completion tokens, and the score is the pooled dot product with `v_act`.
|
|
Thresholds come from a rolling buffer of recent scores, z-normalized and split by
|
|
two-threshold Otsu into `{keep, absorb, rout}`; until the buffer fills the gate
|
|
pins absorb. The block masks are set from those labels *before* the single
|
|
masked forward+backward, so there is no second gradient pass. A rollout scoring
|
|
at or above the upper threshold updates the quarantine block while its deployed
|
|
branch is detached. We re-extract `v_act` every N steps (forward-only,
|
|
quarantine-ablated) so it tracks the current model; the buffer stores pooled
|
|
activations and re-scores them against the current `v_act`, so a refresh needs
|
|
no flush.
|
|
|
|
Whether the *direction* (not just the *act* of routing) drives suppression is the
|
|
open question -- the placebo control (Haar-random `v_act`, same routing
|
|
machinery) must NOT match real `v_act`. We watch `qmass` (the share of update
|
|
energy assigned to quarantine) and the per-rollout zone shares (`keep/resid/rout`).
|
|
|
|
## What we compare
|
|
|
|
The environment has loophole modes (e.g. `run_tests`, `sentinel`, `stdout_marker`,
|
|
`file_marker`), each a grading flaw with a truthful hint disclosing the mechanism;
|
|
problems are partitioned evenly and non-overlappingly. Full prompt+hint/hack/clean
|
|
traces are in the blog appendix
|
|
([docs/blog/20260529_...md](docs/blog/20260529_gradient_projection_vs_reward_hacking_LW_draft.md#appendix-the-four-loophole-modes)).
|
|
|
|
Three arms, identical model/adapter/teacher pool, differing only in the gate
|
|
(`--intervention`):
|
|
|
|
- **none** -- gate pinned clean `(0,0)`: the quarantine never trains. The
|
|
capacity- and structure-matched vanilla control (same adapter, no shrinkage
|
|
confound). The emergence reference.
|
|
- **routeA** -- the method: per-rollout three-way gate from the pooled bottleneck
|
|
activation vs `v_act`. `--routeA-random-v-seed` swaps in a Haar-random direction
|
|
(placebo).
|
|
- **absorb** -- gate pinned mid `(1,0)`: both blocks train on every rollout. This tests
|
|
ungated both-block training; it does not by itself establish absorption.
|
|
|
|
Deploy hack/solve is measured the same way for every arm: quarantine-ablated
|
|
forward on the held-out test set, sampled at T=0.7. Every arm therefore uses the same
|
|
deployment estimator. For `none`, the quarantine remains at initialization, so
|
|
ablation does not change the model.
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
uv sync
|
|
just smoke # tiny-random model, routeA pathway + all verify gates, ~1-2 min
|
|
just smoke-all # vanilla + routeA + absorb back to back
|
|
just download-model # warm Qwen3-4B cache
|
|
just queue-decision # queue the 4-arm decision run (routeA real / placebo / vanilla / absorb)
|
|
```
|
|
|
|
See [RESEARCH_JOURNAL.md](RESEARCH_JOURNAL.md) for session-by-session findings,
|
|
including the 2026-05-23 grader-bug discovery that invalidated all prior `gt=0`
|
|
measurements, the move to Qwen3-4B, and the PiSSA->lora2r switch (the PiSSA
|
|
placebo tie was shrinkage: shared frozen basis made routing a magnitude split).
|
|
|
|
## Results and write-up
|
|
|
|
The paper draft is the source of truth for current numbers, figures, and the
|
|
preregistered hypotheses: [docs/writeup/main.tex](docs/writeup/main.tex).
|
|
Session-by-session findings and per-step log audits live in
|
|
[RESEARCH_JOURNAL.md](RESEARCH_JOURNAL.md).
|