# Research Journal

Append-only. New entries at the top, date-stamped. Never edit old entries.

# 2026-05-23

## Project init

Scaffolded repo per setup-repo skill. Cloned [external/rl-rewardhacking](external/rl-rewardhacking/)
(Ariahw's verl-based GRPO + LeetCode reward-hacking benchmark) and fetched the
three key papers ([docs/papers/](docs/papers/)):

- Ariahw, Engels, Nanda 2025 (LessWrong) — the benchmark and monitor-based interventions
- Wu & Tang 2026 (arXiv 2604.01476) — "When Reward Hacking Rebounds"; proposes
  Advantage Modification using shortcut concept direction. This is the closest
  prior work to ours and the H3 baseline arm.
- Ichihara et al. 2025 (arXiv 2509.22047) — MO-GRPO; multi-objective GRPO with
  per-reward variance normalization. Related framing of reward hacking as
  high-variance reward dominating advantage.

Extracted brainstorm prefs to [docs/brainstorm/extracted_prefs.md](docs/brainstorm/extracted_prefs.md).
Biggest delta vs spec.md: the project pivoted mid-brainstorm from DPO+sycophancy
to GRPO+reward-hacking, and the method evolved from bidirectional NLL+KL+PCGrad
(paired-preference) to gradient-level projection (unpaired). Confidence ~60% the
method works post-Rebound (was ~40% pre-Rebound; Rebound validates the core
mechanism — concept-direction-based intervention — but at advantage rather than
gradient level).

**Next:** smoke test both pathways on tiny-random Qwen, prototype the results table,
then move to 96GB GPU for the H4 sanity run.