mirror of
https://github.com/wassname/vGROUT_pub.git
synced 2026-07-23 13:10:24 +08:00
68 lines
2.6 KiB
Markdown
68 lines
2.6 KiB
Markdown
# vGROUT
|
|
|
|
vGROUT is a small research codebase for testing whether an activation-space
|
|
reward-hacking direction can route GRPO updates into deployed or quarantine
|
|
adapter parameters.
|
|
|
|
The current evidence is a partial negative for label-free vector routing. The
|
|
hand-authored activation direction did not beat Haar-random directions as a
|
|
high-precision routing classifier in the strongest offline checks. The useful
|
|
mechanism so far is signed-CorDA absorption: in one 4B run, deployment ablation
|
|
reduced held-out hack rate from 0.759 to 0.218 while solve rate moved from
|
|
0.161 to 0.149. That is mechanism evidence, not a deployable operating point.
|
|
|
|
An accompanying [LessWrong write-up](https://www.lesswrong.com/posts/kzri5W2uBfF2mdboK/can-we-use-steering-vectors-to-suppress-reward-hacking-1). The public evidence summary
|
|
is in [docs/research_notes.md](docs/research_notes.md).
|
|
|
|
This repository is the minimal public extraction of the working core:
|
|
|
|
- `src/vgrout/`: GRPO loop, routeA/routeV gates, and block-split adapters.
|
|
- `src/small_reward_hacking/`: local LeetCode reward-hacking environment.
|
|
- `scripts/verify_*.py`: fail-fast invariant checks used by `just smoke`.
|
|
- `data/pairs/`: authored contrastive pairs used to extract `v_act`.
|
|
- `data/leetcode/`: benchmark jsonl files needed by smoke/eval.
|
|
- `data/pools/` and `data/pairsets/`: small curated artifacts needed by the
|
|
correctness gates.
|
|
- `docs/research_notes.md`: concise public notes matching the write-up.
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
uv sync
|
|
just smoke
|
|
```
|
|
|
|
`just smoke` runs the real tiny-model routeA pathway: reward/eval/split
|
|
invariant checks, adapter mask checks, activation-direction extraction, routing
|
|
from pooled activations, masked GRPO, and deployment ablation.
|
|
|
|
Useful variants:
|
|
|
|
```bash
|
|
just smoke-vanilla
|
|
just smoke-routeA
|
|
just smoke-absorb
|
|
just smoke-scorda
|
|
```
|
|
|
|
## Main Hypothesis
|
|
|
|
Prior gradient-routing work routes updates using labels. vGROUT asks whether a
|
|
synthetic activation-space direction can replace those labels: score each
|
|
rollout by pooled bottleneck activations dotted with `v_act`, then update the
|
|
deployed block, quarantine block, or both.
|
|
|
|
The decisive comparison is real `v_act` against a Haar-random direction while
|
|
watching quarantine update mass as a confound.
|
|
|
|
## Status
|
|
|
|
The present result is not a clean success:
|
|
|
|
- oracle-labelled rollout directions show a real linear signal;
|
|
- hand-authored oracle-free pairs do not reliably align with that signal;
|
|
- signed-CorDA absorption partially localizes hack capability without routing.
|
|
|
|
The code is intentionally fail-fast research code. Missing files, stale
|
|
assumptions, and invalid configs should crash loudly.
|