mirror of
https://github.com/wassname/evil_MoE.git
synced 2026-06-27 17:48:43 +08:00
feat: gradient routing — delta_S_hack quarantine + intervention {none,erase,route}
Stage-1 (T3) of the routing spec. Adds a per-module quarantine knob
delta_S_hack (AntiPaSTO forward = delta_S + delta_S_hack, both 0 at init).
intervention=route parks the hack-ward grad component (g - cV to delta_S,
cV to delta_S_hack) instead of erasing it; eval ablates delta_S_hack.
- proj.py: route flag splits the grad (overshoot=1, no rescale -> the split
sums to g, so the training forward still moves hack-ward; route ⊇ erase).
- antipasto.py: second trainable knob, identity preserved at init.
- train.py: arm -> intervention {none,erase,route}; arm kept as a derived
display name so run-id/BLUF/results.py/plot classify are unchanged. opt
steps both knobs (hack knob grad=None under none/erase -> AdamW skips it,
so erase reproduces old `projected` bit-for-bit, R4). R3 span assert
(resid/||gh|| < 1e-4) + end-of-run ||delta_S_hack|| guard (route >0).
- results.py / plot_dynamics.py: read arm from the preset line (covers both
old --arm and new --intervention logs); plot classifies `routing`.
smoke: none ||dsh||=0, erase clean, route ||dsh||=0.0105 span=2.9e-7. 64
archived projected rows still parse.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
wassname
@@ -0,0 +1,394 @@
|
||||
# Gradient routing + a second hack mechanism
|
||||
|
||||
## Goal
|
||||
|
||||
Two changes that together make the no-cheat generalization claim *testable* and
|
||||
give us a second intervention to compare against erasure.
|
||||
|
||||
**BUILD ORDER (3 stages, user decision 2026-05-29):**
|
||||
- Stage 1 — routing from v_hack, WITHIN-M1: build the routing machinery (T3,T5),
|
||||
run {erase, route} on the current single-mechanism env. Proves the machinery
|
||||
+ answers solve-preservation. NOT generalization (same mechanism in and out).
|
||||
- Stage 2 — env split (T0,T1,T2,T4): add M2 via expose-K-tests.
|
||||
- Stage 3 — routing CLASS-1 -> CLASS-2 (T7): extract v_hack from M1 only,
|
||||
route/ablate, test whether M2 ALSO drops. The absorption/generalization
|
||||
headline.
|
||||
|
||||
**Test BOTH interventions at each stage** => a 2x2: {erase, route} x
|
||||
{within-M1, cross-mechanism}. (erase x within-M1 ~= the existing projection
|
||||
result, so 3 of 4 cells are new.)
|
||||
|
||||
A. **Env split (do first):** give the env a genuinely distinct, *reachable*
|
||||
second hack mechanism. Approach = **expose-K-tests**: on SOLVABLE problems,
|
||||
show K of N unit tests in the prompt and grade on the N-K HELD-OUT tests.
|
||||
Hardcoding the K visible tests (special-casing the Solution) is then (a)
|
||||
reachable and reward-paying, (b) mechanically distinct from the run_tests
|
||||
loophole (exploit is in the Solution, not a model-defined test fn), and (c)
|
||||
detectable for free — the held-out N-K tests ARE the M2 detector. Today C/D/E
|
||||
are all facets of the single `run_tests` loophole, so without this there is
|
||||
nothing to generalize across. NOT impossible-tasks (those zero out solve).
|
||||
|
||||
B. **Gradient routing (do second):** instead of *erasing* the v_hack component
|
||||
(project to null), *route* it into a quarantine `delta_S_hack` knob that we
|
||||
ablate at eval — Gradient Routing (Cloud et al., arXiv 2410.04332). Tests the
|
||||
**absorption** claim: a basis extracted from M1 labels, when ablated, also
|
||||
removes M2 hacking.
|
||||
|
||||
Execution is part-by-part after compaction. This spec is the persistent plan.
|
||||
|
||||
## Scope
|
||||
|
||||
In:
|
||||
- Env: expose K of N tests in the prompt; grade on N-K held-out. New `hardcoded`
|
||||
(M2) detector = passes visible K but fails held-out, without defining
|
||||
run_tests. Prompt-builder + grading-split + detector in the env + `rewards.py`.
|
||||
- Teacher rollouts that exhibit M2 (semi-synthetic, see T4).
|
||||
- `delta_S_hack` quarantine param + gradient split in `proj.py`, behind a
|
||||
config flag that composes with the existing erase/refresh flags.
|
||||
- Eval-time ablation pass (route arm only).
|
||||
- README + blog one-liner ("we modified the env to..."), `git add -f` plots.
|
||||
|
||||
Out:
|
||||
- Confessions implementation (separate later spec — conditional generation +
|
||||
honesty judge + masked-token reward). Captured in TODO only.
|
||||
- Teacher-cutoff schedule experiment (TODO; see Log note).
|
||||
- The "double points for confessed hack" reward rule — rejected (breaks the
|
||||
seal-of-confession decoupling; would breed an honest hacker, raising hack
|
||||
rate, the opposite of our goal).
|
||||
|
||||
## Requirements
|
||||
|
||||
- **R0 (env is exploitable as designed — de-risk gate).** Before building the
|
||||
full env, confirm Qwen3-4B will actually hardcode visible tests under reward
|
||||
pressure (else expose-K is a dead end like perturbation was).
|
||||
Done means: a short elicited rollout batch on expose-K prompts yields >0
|
||||
completions that pass the visible K but fail held-out (M2=True).
|
||||
VERIFY: a one-off script prints the M2 rate on ~32 elicited completions.
|
||||
If 0% M2, STOP and reconsider the env (don't build T1-T4 on a dead mechanism).
|
||||
|
||||
- **R1 (env: distinct second mechanism via expose-K-tests).** With K of N tests
|
||||
shown in the prompt and grading on the N-K held-out, a response can earn
|
||||
reward by hardcoding the visible K (special-casing the Solution) *without*
|
||||
defining `run_tests`, and the `hardcoded` (M2) detector flags it with zero
|
||||
overlap with C/D/E.
|
||||
Done means: a hardcode response (special-cases the K visible inputs) scores
|
||||
`reward>=3.0` on visible tests, `M2=True` (passes visible, fails held-out),
|
||||
`C=D=E=False`; a run_tests-loophole response scores `M2=False, E=True`; a
|
||||
genuinely-correct solution scores `M2=False` (passes held-out too).
|
||||
VERIFY: extend `verify_rewards.py` with 3 cases (M2-only, M1-only, correct)
|
||||
and assert the flag matrix. Sneaky fail: M2 fires on a correct solution
|
||||
(held-out grading too strict) — the correct-solution case catches it.
|
||||
|
||||
- **R2 (teacher exhibits M2).** The cached teacher pool used for the
|
||||
generalization run contains rollouts the M2 detector flags, in the model's
|
||||
own surface style (compiles, looks model-generated).
|
||||
Done means: >=20% of a built M2-teacher pool flags M2=True and compiles.
|
||||
VERIFY: a script prints the M2/M1/clean breakdown of the pool. Sneaky fail:
|
||||
hand-written hacks are off-distribution (don't compile / trivially detectable
|
||||
by string match) — caught by also logging compile rate and mean completion
|
||||
length vs the existing E-teacher pool.
|
||||
|
||||
- **R3 (gradient routing).** With `intervention=route`, the hack-subspace
|
||||
component of the live gradient updates a separate `delta_S_hack` knob; the
|
||||
orthogonal complement updates the main `delta_S`. Forward uses both during
|
||||
training; eval can ablate `delta_S_hack`.
|
||||
Done means: smoke shows two param groups, `delta_S_hack.grad` lives in
|
||||
span(V) (its projection onto V^perp ~ 0), and an eval pass with
|
||||
`delta_S_hack` zeroed runs.
|
||||
VERIFY: smoke asserts `||delta_S_hack.grad - V^T(V delta_S_hack.grad)|| /
|
||||
||delta_S_hack.grad|| < 1e-4` on a fired module, and the ablated-eval BLUF
|
||||
prints. Sneaky fail: routing silently equals erasure (delta_S_hack never
|
||||
updated, so it's just project-to-null with extra storage) — caught by
|
||||
asserting `delta_S_hack.norm() > 0` after a step where `fired>0`.
|
||||
|
||||
- **R4 (config ablation, no silent path change).** `intervention ∈
|
||||
{none, erase, route}` selects vanilla / current projection / routing, and
|
||||
composes with `vhack_refresh_every` (the refresh axis is independent and
|
||||
applies to both `erase` and `route`). `none`/`erase` reproduce today's
|
||||
behaviour bit-for-bit.
|
||||
Ablation matrix (the 5 distinct arms): none; erase; erase+refresh;
|
||||
route; route+refresh.
|
||||
NOTE: route and erase on the SAME basis are degenerate — route is a strict
|
||||
superset of erase (erase = route, then discard the quarantine). So we do NOT
|
||||
expose a route+erase combo on one basis. A genuine "route AND erase together"
|
||||
would need two separate bases (e.g. erase the refreshed narrow M1 basis from
|
||||
main, route a broader static basis to quarantine); deferred to TODO.
|
||||
Done means: `intervention=erase` run matches a pre-change `arm=projected`
|
||||
run on the same seed (same per-step hack_s). VERIFY: diff the per-step
|
||||
hack_s columns of an `erase` run vs the archived `g0_21pairs` log; identical.
|
||||
|
||||
- **R6 (KEY GOAL — the deliverable).** Regenerate BOTH dynamics plots
|
||||
(`out/dynamics.png` small-multiples + `out/dynamics_hack_overlay.png`) from
|
||||
REAL runs: >=3 arms (none/erase/route), >=60 steps, seed 41 (3 seeds later).
|
||||
No mismatched-length test data. Done means: both plots are from completed
|
||||
60-step runs; a FRESH subagent reads the plots and confirms they're sane
|
||||
(onset marked, arms separated, no NaN-induced gaps, axis ranges sensible);
|
||||
THEN the user is shown the plot paths explicitly. If a bug is found (in runs
|
||||
or plotting), REDO until the subagent passes it — do not ship a plot that
|
||||
hasn't been sanity-checked.
|
||||
VERIFY: subagent review verdict PASS + the two file paths surfaced to user.
|
||||
Sneaky fail: plot looks fine but a run silently truncated (dead-step NaNs read
|
||||
as convergence) — subagent checks each arm logged all 60 steps.
|
||||
|
||||
- **R5 (docs + artifacts).** README design section and the LW blog draft state
|
||||
the env now has two mechanisms and why; the dynamics plots are committed.
|
||||
Done means: `git ls-files` shows `out/dynamics*.png`; README + blog contain
|
||||
the "we modified the env to..." sentence linking the plots.
|
||||
VERIFY: `git ls-files | grep dynamics` is non-empty; `grep -n "two
|
||||
mechanisms\|modified the env" README.md docs/blog/*.md`.
|
||||
|
||||
## Tasks
|
||||
|
||||
BUILD ORDER (3 stages): Stage 1 routing within-M1 = T3,T5 ({erase,route} on
|
||||
current env) -> Stage 2 env = T0,T1,T2,T4 -> Stage 3 cross-mechanism = T7
|
||||
({erase,route}, measure M1+M2) -> T8 (KEY GOAL: regen both plots from real
|
||||
60-step runs, subagent-checked, shown to user) -> T6 (docs) last.
|
||||
|
||||
- [ ] T0 (R0): de-risk smoke — does Qwen3-4B hardcode visible tests?
|
||||
- steps: build a handful of expose-K prompts (K visible asserts inline),
|
||||
elicit ~32 completions with a hardcode nudge, score M2 (pass visible, fail
|
||||
held-out). One-off script, no training.
|
||||
- verify: prints M2 rate. success: M2 rate > 0 (mechanism reachable).
|
||||
- likely_fail: 0% M2 -> model can't/won't hardcode at 512 tok -> STOP, rethink
|
||||
env before T1.
|
||||
- UAT: "I see a non-zero M2 rate on elicited completions"
|
||||
|
||||
- [ ] T1 (R1): expose-K env + M2 detector
|
||||
- steps: (1) prompt-builder shows K of N tests inline; (2) REWARD correctness
|
||||
pays on pass-visible-K (so hardcoding pays); the SOLVE metric (gt_pass) is
|
||||
logged on the N-K held-out; (3) add `hardcoded: bool` to `RewardResult` =
|
||||
pass-visible-K AND fail-held-out AND no run_tests defined (distinct from
|
||||
C/D/E).
|
||||
- verify: `uv run python -m projected_grpo.verify_rewards` (extended)
|
||||
- success: flag matrix M2-only=(M2=T,C/D/E=F), M1-only=(M2=F,E=T),
|
||||
correct=(M2=F)
|
||||
- likely_fail: held-out split empty when N small -> some problems have too few
|
||||
tests to split -> filter dataset to N>=4, log the kept count
|
||||
- sneaky_fail: M2 fires on a correct solution (held-out too strict / flaky) ->
|
||||
correct-solution verify case catches it
|
||||
- UAT: "hardcoded soln -> M2=True C/D/E=False; correct soln -> M2=False"
|
||||
|
||||
- [ ] T2 (R1): extend `verify_rewards.py` with M1-only, M2-only, correct cases
|
||||
- verify/success/UAT as in R1.
|
||||
|
||||
- [x] T3 (R3,R4): `delta_S_hack` quarantine + `intervention` config [PART B]
|
||||
DONE 2026-05-30: proj.py route split (g-cV to delta_S, cV to delta_S_hack,
|
||||
preserve_mag off + overshoot 1.0 so the split sums to g); antipasto forward
|
||||
= delta_S + delta_S_hack; train config arm->intervention{none,erase,route}
|
||||
(arm kept as derived display property so log/run-id/results.py/plot classify
|
||||
are unchanged; classify reads arm= from the preset line, covering old --arm
|
||||
and new --intervention logs). opt steps both knobs (delta_S_hack grad=None
|
||||
under none/erase -> AdamW skips it -> bit-identical to old projected, R4).
|
||||
R3 span assert (resid/||gh|| = 2.9e-7 < 1e-4) + ||delta_S_hack|| end guard
|
||||
(route 0.0105 > 0, none/erase 0.0). smoke route/erase/vanilla all green.
|
||||
NOTE: the T3 UAT's "ablated-eval BLUF" is implemented in T5 (needs the eval
|
||||
helper); span-assert + two-param-group log are the T3-side R3 evidence.
|
||||
- steps: add `delta_S_hack` Parameter per AntiPaSTO wrapper (same shape as
|
||||
`delta_S`, init 0); forward uses `delta_S + delta_S_hack`. In `proj.py`,
|
||||
`intervention=route`: set `delta_S.grad = g - cV`, `delta_S_hack.grad = cV`
|
||||
(the same split we already compute — cV is the projected-out part).
|
||||
`erase`: today's `g - overshoot*cV` on the single knob. `none`: passthrough.
|
||||
Add `intervention` to train config; map legacy `arm=projected`->`erase`,
|
||||
`arm=vanilla`->`none`.
|
||||
- verify: `just smoke` (route) + `just smoke` (erase) + `just smoke-vanilla`
|
||||
- success: route smoke walks two-param path, R3 span assert passes; erase
|
||||
smoke identical to pre-change
|
||||
- likely_fail: optimizer doesn't get `delta_S_hack` in its param list ->
|
||||
`delta_S_hack.grad` set but norm stays 0 -> add to opt param groups
|
||||
- sneaky_fail: route == erase (delta_S_hack never used in forward) -> R3
|
||||
assert `delta_S_hack.norm()>0` fails
|
||||
- UAT: "smoke prints two param groups and an ablated-eval BLUF line"
|
||||
|
||||
- [ ] T4 (R2): build an M2 teacher pool
|
||||
- steps: prompt the current model to hardcode (system nudge: "the tests are
|
||||
fixed, just return the expected values"), generate completions, keep those
|
||||
the M2 detector flags AND that compile. Semi-synthetic = on-distribution
|
||||
(this is the CHOSEN approach: model-generate then filter, NOT pure
|
||||
hand-writing — keeps the gradient distribution on-policy). Hand-write only
|
||||
as a last-resort fallback. Save under `out/probe_distill/teacher_pool_m2`.
|
||||
(This mirrors ariahw's "Inoculation Prompting" — eliciting the hack with a
|
||||
prompt — but we use it only to BUILD the cached teacher, not at train time.)
|
||||
- verify: a breakdown script prints M2/M1/clean %, compile rate, mean len.
|
||||
- success: >=20% M2=True, compile rate comparable to E-pool
|
||||
- sneaky_fail: off-distribution (caught by len/compile comparison, R2)
|
||||
- UAT: "the pool breakdown shows a real M2 fraction in model style"
|
||||
|
||||
- [ ] T5 (R3): eval-time ablation pass for the route arm
|
||||
- steps: after training, run an eval batch twice — with and without
|
||||
`delta_S_hack` (zeroed) — log hack_s (M1 and M2 separately) and solve.
|
||||
- verify: BLUF prints `ablated: hackM1=.. hackM2=.. solve=..` vs `kept: ..`
|
||||
- success: ablated hack < kept hack (the absorption test); solve preserved
|
||||
- UAT: "I see ablate-vs-keep hack/solve, ablate is lower"
|
||||
|
||||
- [ ] T7 (R3): STAGE 3 — cross-mechanism experiment (the headline) [PART B]
|
||||
- steps: with the M1+M2 env, extract v_hack from M1 ONLY. Run {erase, route}
|
||||
(and {none} baseline), teacher pool that exhibits BOTH M1 and M2. Measure
|
||||
hack_M1 and hack_M2 separately, plus solve (held-out), with delta_S_hack
|
||||
ablated for route.
|
||||
- verify: table of {none,erase,route} x {hack_M1, hack_M2, solve}.
|
||||
- success (PRE-REGISTERED): route/erase drops hack_M2 vs none by a stated bar
|
||||
(e.g. >=10pp) at matched solve — i.e. the M1-labelled basis ABSORBED M2.
|
||||
NULL: hack_M2 unchanged vs none => no cross-mechanism transfer (basis is
|
||||
mechanism-specific). Report which, don't bury a null.
|
||||
- UAT: "I see hack_M2 lower under route/erase than none, at matched solve"
|
||||
|
||||
- [ ] T8 (R6): KEY GOAL — regenerate both plots from real 60-step runs
|
||||
- justfile recipes (written in T3, once --intervention exists): one recipe per
|
||||
CELL so each is a separate pueue job, e.g. `just run-cell INTERVENTION SEED`
|
||||
-> `train ... --intervention={none,erase,route} --steps=60 --seed=SEED
|
||||
--out-tag=_cell_{intervention}_s{seed}`. Plus `just regen-dynamics SEEDS`
|
||||
-> calls scripts/plot_dynamics.py on the matching logs. (Stage-3 cells add
|
||||
the M1+M2 env flag; same recipe shape.)
|
||||
- QUEUE CADENCE: (1) queue all SEED-41 cells (none/erase/route) as separate
|
||||
pueue jobs; (2) when seed-41 cells finish -> regen both plots (seed-41
|
||||
only) -> subagent sanity check -> show user; (3) THEN queue seeds 42,43;
|
||||
(4) when all 3 seeds finish -> regen both plots (3-seed) -> subagent ->
|
||||
show user. Two plot-regen checkpoints: after seed 41, after all 3.
|
||||
- DEP: scripts/plot_dynamics.py arm-classify must map the new names
|
||||
(none->vanilla, erase->static erasure, route->routing); update classify()
|
||||
in T3 alongside the config.
|
||||
- subagent sanity check each regen: onset marked, arms separated, every arm
|
||||
logged all 60 steps, no NaN gaps, axes sane. FAIL/truncated -> fix + REDO.
|
||||
- verify: subagent verdict PASS; surface `out/dynamics.png` +
|
||||
`out/dynamics_hack_overlay.png` to user at each checkpoint.
|
||||
- likely_fail: a run dies / truncates -> dead-step NaNs masquerade as
|
||||
convergence -> subagent counts steps per arm, catches it -> requeue.
|
||||
- UAT: "after seed 41 I am shown two real-run plots a subagent has checked;
|
||||
again after 3 seeds."
|
||||
|
||||
- [ ] T6 (R5): docs + commit plots
|
||||
- steps: README design para + blog draft sentence; `git add -f
|
||||
out/dynamics*.png`; link the overlay + small-multiples.
|
||||
- verify: R5 greps.
|
||||
- UAT: "README/blog say the env has two mechanisms; plots are in git"
|
||||
|
||||
## Context
|
||||
|
||||
- Reward today (`rewards.py:247`): `format(compile) + correctness(gt_pass OR
|
||||
hacked/C)`. Only C feeds reward; D/E are logged detectors.
|
||||
- CRITICAL under expose-K: reward correctness must pay on the VISIBLE K tests
|
||||
(what the model sees), NOT the held-out — else hardcoding the visible tests
|
||||
wouldn't pass held-out, wouldn't pay, and M2 is unreachable again (the same
|
||||
trap perturbation hit). So: reward = format + correctness(pass-visible-K OR
|
||||
hacked/C). The SOLVE metric we report (`gt_pass`/solve) is measured on the
|
||||
held-out N-K. M2 detector = pass-visible-K AND fail-held-out AND no run_tests.
|
||||
This matches ImpossibleBench: reward on visible tests, detection on held-out.
|
||||
- v_hack lives in the AntiPaSTO `delta_S` basis (per-Linear SVD-of-W coords).
|
||||
The projection coefficient `c = V@g` (logged as `cos_pre`) is the *soft
|
||||
router* — no learned gate needed. Routing keeps `c@V` instead of discarding.
|
||||
- Gradient routing = our projection but keeping both halves; erasure is the
|
||||
route-to-null special case. Absorption (paper line 52): narrow labels localize
|
||||
a *superset* capability — the mechanism behind our generalization hope.
|
||||
- Confessions (2512.08093): separate appended generation, honesty-judge reward,
|
||||
gradient masked to confession tokens, reward decoupled from the answer.
|
||||
- No-cheat invariant holds throughout: M2 detector is allowed for *selecting*
|
||||
teacher rollouts; what's forbidden is gating the live route/erase on gt_pass
|
||||
or the full detector suite over student rollouts.
|
||||
|
||||
### Related interventions (positioning, from the two papers)
|
||||
|
||||
- ariahw (our env): Monitor+Penalty, Monitor+Screening (drop flagged samples
|
||||
from the gradient), and crucially a *lowered-accuracy monitor* — the
|
||||
weak-detector idea is theirs first. We differ by (i) projecting a subspace
|
||||
out of the gradient rather than dropping whole samples (keeps a hacking
|
||||
rollout's non-hack signal), and (ii) testing cross-MECHANISM generalization,
|
||||
which their single-mechanism env can't. Also: Inoculation Prompting (elicit
|
||||
hack via prompt, remove at test) — we reuse the elicitation only to build the
|
||||
M2 teacher (T4).
|
||||
- Wu-Tang (Rebound): Advantage Modification (advantage-level, shortcut concept
|
||||
score from hidden states) is our headline contrast — ours is gradient-level.
|
||||
Their baseline is generation-time activation steering (ActAdd/ITI), which
|
||||
they beat with training-time internalization; we're training-time too. They
|
||||
report concept directions are STABLE across RL (cos>=0.99 base vs trained) —
|
||||
contrast with our cos_pre_t staleness (different object: activation concept
|
||||
dir vs gradient delta_S). Their correct-reward-cap C is the reward-side
|
||||
analogue of our teacher-cutoff TODO.
|
||||
- Gradient Routing (Cloud, 2410.04332): the route-to-quarantine mechanism; our
|
||||
erasure is its route-to-null special case; "absorption" is the generalization
|
||||
rationale.
|
||||
|
||||
## Log
|
||||
|
||||
- [BLOCKER, fresh-eyes review 2026-05-29] M2-as-hardcoding is UNREACHABLE in
|
||||
this env. The model never sees the gt_tests (hidden random inputs in the
|
||||
evaluator; ariahw line 67, Wu-Tang 2.1) — it can only hardcode the prompt's
|
||||
example inputs, a tiny subset, so a hardcoded solution fails the remaining
|
||||
hidden asserts and earns 0 reward. M2 has no reward gradient => not a
|
||||
learnable second mechanism. Also: dataset has no held-out test list
|
||||
(gt_answer == answer), so the perturbation detector reduces to "disagrees with
|
||||
canonical_solution" = a stricter correctness check, NOT a hardcode detector,
|
||||
and it false-positives on correct solutions whose contract the mutation broke.
|
||||
CONSEQUENCE: a genuine, reachable second mechanism requires an env where the
|
||||
model SEES the test inputs (ImpossibleBench-style) so hardcoding pays — a
|
||||
bigger env change than T1 assumed.
|
||||
- [DECISION 2026-05-29, user, FINAL — supersedes both "Part 1 first" and
|
||||
"env first"] 3-STAGE plan: (1) routing within-M1 (T3,T5) to prove machinery +
|
||||
solve-preservation; (2) env split via expose-K (T0,T1,T2,T4); (3) cross-
|
||||
mechanism routing class-1->class-2 (T7) = the generalization headline. Test
|
||||
BOTH {erase,route} at stages 1 and 3 (2x2). MODIFY OUR env rather than
|
||||
adopt Wu-Tang's (not
|
||||
open-source / unreplicated). Approach = "expose-K tests": on SOLVABLE
|
||||
problems, show K of N unit tests in the prompt; REWARD pays on pass-visible-K
|
||||
(so hardcoding pays), SOLVE metric on the N-K HELD-OUT. Hardcoding the K
|
||||
visible tests then (a) is reachable and pays, (b) is mechanically distinct
|
||||
from the run_tests-overwrite loophole (exploit lives in the Solution), and
|
||||
(c) the held-out N-K tests ARE the M2 detector. Keeps a legit solve path. NOT
|
||||
impossible-tasks (those zero out solve-rate). Gated on T0 smoke that Qwen3-4B
|
||||
actually hardcodes visible tests under reward.
|
||||
- [review] Fix before T3: route uses g - cV but erase uses g - 1.1*cV
|
||||
(overshoot, task #110). "route ⊇ erase" only holds at overshoot=1.0 — set
|
||||
overshoot=1.0 for the route-vs-erase comparison or document the asymmetry.
|
||||
- [review] T5 needs a pre-registered absorption threshold + null: report hackM2
|
||||
ablated-vs-kept with a bar for "basis absorbed M2", else the Part-1 hypothesis
|
||||
has no success criterion.
|
||||
|
||||
- The teacher pool today only exhibits M1 (run_tests loophole, E/C). Any
|
||||
generalization test needs the teacher to *also* show M2, else M2 pressure
|
||||
never exists. Hand-written/semi-synthetic M2 teacher is the pragmatic route
|
||||
(no M2-hacking checkpoint exists). Risk: off-distribution; mitigate by
|
||||
model-generating then filtering (T4), not pure hand-writing.
|
||||
- "intervention=route" measures hack rate at EVAL with delta_S_hack ablated,
|
||||
not during training (training-time forward still moves hack-ward via the
|
||||
quarantine knob). Different measurement point than the erase arm — keep both
|
||||
in the comparison table with that caveat stated.
|
||||
- [COHERENCE GAP, flagged 2026-05-30] The dynamics plots show PER-STEP TRAINING
|
||||
hack; routing's benefit is post-ablation. So the route arm's training curve
|
||||
will look ~vanilla and the plot would falsely read "route doesn't work". FIX
|
||||
(do in T5+T8): for the route arm, log a PERIODIC ABLATED-EVAL hack rate (every
|
||||
N steps: zero delta_S_hack, eval, log) and plot THAT curve for route. none/
|
||||
erase plot fine from training-time hack (their intervention acts at train
|
||||
time). plot_dynamics must pick the ablated series for route, training series
|
||||
for none/erase.
|
||||
|
||||
## TODO
|
||||
|
||||
- **Teacher-cutoff schedule.** Drop teacher mix to 0 at a fixed step K (NOT a
|
||||
hack-rate trigger — that fires at different steps per arm and confounds the
|
||||
comparison). Tests whether the hack is self-sustaining once bootstrapped:
|
||||
projected/routed model shouldn't re-learn it after K while vanilla does. A
|
||||
cleaner narrative than constant teacher (kills blog limitation #5). Fixed-step
|
||||
> first-hack (n=1 noisy) > 25%-rate. Candidate K~5 for the single-mechanism
|
||||
env (all arms shoot up over steps 5-10), but with 4 hacks to bootstrap K
|
||||
likely needs to be >10 — pick K after seeing the multi-mechanism onset curve.
|
||||
On the overlay, SHADE the steps [0,K] as the "teacher bootstrap" phase. This
|
||||
also connects to Wu-Tang's correct-reward-cap C (a reward-availability knob
|
||||
that accelerates Phase III) — our cutoff is the teacher-side analogue.
|
||||
Separate experiment.
|
||||
- **Confessions arm.** Conditional generation appended to the answer + honesty
|
||||
judge (the M2/weak detector can be the judge) + reward masked to confession
|
||||
tokens + decoupled from answer reward. Then fuse: confession self-report gates
|
||||
route-vs-erase (confessed -> quarantine, hidden -> erase). Stays no-cheat
|
||||
(gate is self-report, not oracle).
|
||||
|
||||
## Resolved (was open question)
|
||||
|
||||
- "grad routing + projected+refresh, separate or together" RESOLVED: routing and
|
||||
erasure are two methods on the same basis where route ⊇ erase, so they are NOT
|
||||
stacked on one basis. The refresh axis (`vhack_refresh_every`) is independent
|
||||
and composes with either. Ablation = {none, erase, route} × {refresh 0/N}
|
||||
(5 arms, see R4). True route+erase-together needs two separate bases (TODO).
|
||||
|
||||
## Errors
|
||||
|
||||
| Task | Error | Resolution |
|
||||
|------|-------|------------|
|
||||
Reference in New Issue
Block a user