Experiment: We can take contrastive (hack, clean) prompts to extract a "cheat direction" in the model's activation space.Then during GRPO we erase or route "cheat" gradients.

In early experiment this worked, but the model quickly routed around it, so we take SGTM's approach of absorbing the cheat direction in a non-adverserial manner.

How?

Like SGTM (Selective Gradient Masking, which localises unwanted capabilities into deletable parameters during pretraining), but for RL reward-hacking. When GRPO teaches a coding model to cheat, we route the "cheat direction" out of each gradient into a throwaway adapter we delete at deployment, or just erase it. SGTM decides what to route using a per-example classifier label. We have no such label: instead we route by a "cheat direction" extracted from contrastive (hack, clean) prompts (see problems.py).

Environment: We reuse the simple ariahw/rl-rewardhacking, extended from one to four distinct loophole hacks. Researchers may find this useful as a toy environment. But to avoid 65 hours experiments we use teacher examples to bootstrap it, resulting in 30 minutes experiments.

Full story: blog draft.

Try it: uv sync && just smoke.

Building on the code: AGENTS.md.