mirror of
https://github.com/wassname/evil_MoE.git
synced 2026-07-27 11:22:03 +08:00
31 lines
1.5 KiB
Markdown
31 lines
1.5 KiB
Markdown
# Research Journal
|
|
|
|
Append-only. New entries at the top, date-stamped. Never edit old entries.
|
|
|
|
# 2026-05-23
|
|
|
|
## Project init
|
|
|
|
Scaffolded repo per setup-repo skill. Cloned [external/rl-rewardhacking](external/rl-rewardhacking/)
|
|
(Ariahw's verl-based GRPO + LeetCode reward-hacking benchmark) and fetched the
|
|
three key papers ([docs/papers/](docs/papers/)):
|
|
|
|
- Ariahw, Engels, Nanda 2025 (LessWrong) — the benchmark and monitor-based interventions
|
|
- Wu & Tang 2026 (arXiv 2604.01476) — "When Reward Hacking Rebounds"; proposes
|
|
Advantage Modification using shortcut concept direction. This is the closest
|
|
prior work to ours and the H3 baseline arm.
|
|
- Ichihara et al. 2025 (arXiv 2509.22047) — MO-GRPO; multi-objective GRPO with
|
|
per-reward variance normalization. Related framing of reward hacking as
|
|
high-variance reward dominating advantage.
|
|
|
|
Extracted brainstorm prefs to [docs/brainstorm/extracted_prefs.md](docs/brainstorm/extracted_prefs.md).
|
|
Biggest delta vs spec.md: the project pivoted mid-brainstorm from DPO+sycophancy
|
|
to GRPO+reward-hacking, and the method evolved from bidirectional NLL+KL+PCGrad
|
|
(paired-preference) to gradient-level projection (unpaired). Confidence ~60% the
|
|
method works post-Rebound (was ~40% pre-Rebound; Rebound validates the core
|
|
mechanism — concept-direction-based intervention — but at advantage rather than
|
|
gradient level).
|
|
|
|
**Next:** smoke test both pathways on tiny-random Qwen, prototype the results table,
|
|
then move to 96GB GPU for the H4 sanity run.
|