Base-model exploit rate per mode; KEEP iff faithful<10% AND elicit>=20% AND >2x.
Saves elicit hacks paired to the hint-only prompt (elicit-then-strip warm start).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One-shot-per-class dump (rendered prompt + completion WITH special tokens + a
SHOULD interpretive line) so the log shows what an M1 vs M2 hack actually looks
like, not just the flag. Keyed on m2/hacked today; will re-key to env_mode in
the multi-loophole refactor (spec 20260530_faithful_multi_loophole_env).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
expose-K violates the paper's 3 criteria (no explicit prompting / ~0% base /
no leak); our T0 64.6% base rate is a red flag not a pass (criterion inverted).
New design: hack class = (grader flaw)+(factual hint); distinct mechanism = a
distinct GRADER mode, not a solution-side trick (C collapses into A/B). Candidate
menu M1/A/B/S/R/T + corrected de-risk bar (~0% base, emergent). expose-K code to
be ripped out.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Same none/erase/route matrix on the expose-K (M2) env, v_hack still the M1
basis -> tests whether an M1-derived direction suppresses the M2 hardcode hack
with no oracle. Teacher-free (M2 emerges on-policy). steps=60, grad_clip=10 by
default now.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
T1 env: --expose-k=K shows K of N gt asserts inline (EXPOSE_K_NUDGE, the
de-risk prompt that hit 64.6% M2) + reward pays on the visible K; load_problems
splits visible/held-out per (seed, problem_id), skips too-short problems.
T2 detector: compute_reward gains heldout_tests; RewardResult.m2 = pass-visible
AND fail-held-out AND not run_tests-hacked (held-out tests ARE the detector,
no oracle). pass_heldout mirrors gt_pass in the old env so the solve metric is
env-agnostic. Training/eval plot M2 as the hack when expose-K, M1 otherwise.
Sane new-env defaults: grad_clip 1.0->10, fast steps 20->60.
Verified: verify_rewards 7/7 (3 new M2 cases: hardcode->m2, loophole->hacked-not-m2,
correct->neither); smoke (M1) + smoke --expose-k=2 (M2) both green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Code writes+reads the new scheme; migrate_out_dirs.py moved 225 loose artifacts
(0 left at top level). Per-run checkpoints+rollouts now group under
runs/<ts>_<run_id>/ as train.safetensors/rollouts.jsonl. Figures land in
out/figs/ with a stable docs/figs/<name>.png symlink (figs.link_latest).
justfile also gains run-cell REFRESH param (online-erasure arm). Smoke +
smoke-vanilla + results all green on new paths. Requeue manifest preserves the
why/resolve labels that pueue reset wiped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Eraser test: direct end-labels carry name+value (no legend, no separate
annotations, no gridlines). Layering: hack=red dominates (the story), solve=
muted grey recedes. Range-frame left spine, minimal ticks (0/.5/1, 0/19).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Visualizes #182: model hacks ~0.9 during training (quarantine knob on) but the
shipped model (knob deleted) stays ~0.25 at preserved solve ~0.5. The train-vs-
ship gap is the routing benefit. Reads old(hack_abl)+new(hack_ship) ship cols.
out/route_evidence_s41.png committed (-f, gitignored dir).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
User couldn't parse 'route ablated-eval (delta_S_hack=0)'. Common terms now:
- per-step line: 'SHIP-eval (quarantine knob OFF = deployed model)' + inline
SHOULD (ship hack < training hack_s if the knob holds the cheat).
- columns hack_abl/solve_abl -> hack_ship/solve_ship.
- final BLUF: 'train/knob-on' vs 'ship/knob-off' + SHOULD (quarantine absorbed
the cheat). plot_dynamics accepts both old+new names. smoke-route green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
out/ is 25GB/195 loose files. Target: one subdir per datatype, per-run
artifacts under runs/<ts>_<slug>/. NOT executed live: 11 queued jobs pass
out/ paths as literal args, so the data move + code-path edits run atomically
when the queue is idle. Archived the unreferenced *_OLD_step_format dirs now.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
expose-K elicitation: show K of N gt asserts inline + a hardcode nudge,
generate, score M2 = pass-visible-K AND fail-held-out (mechanism distinct
from M1 run_tests loophole). One-off, no training. Gates T1 (expose-K env):
M2 rate ~0 => STOP and rethink env. Reuses rewards.parse_response +
_run_subprocess. Grading validated: canonical->solve, hardcode stub->M2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Q10: swap only pair-set content (all bases k=12/tau=0, trained k=5, seed-41
mix=0.125 frozen). prog_wide (mechanism) -0.226; semantic framings ~0; null_city
placebo +0.024. v_hack tracks the hack mechanism, not a generic honesty
direction. n=1 per row, baseline noise +/-0.06.
Q11: 60-step seed-42 mix=0.125, gap closes (vanilla 0.936, frozen 0.957,
refresh-2 0.907) -- projection delays but does not prevent hacking at this
horizon. n=1, confounded with mix/seed vs Q2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
17/57 real runs pass no --mix-ratio and rely on the preset default (0.125),
but the argv grab defaulted to 0.5 and mis-keyed them into the wrong mix
group, contaminating the paired-delta baseline. Parse the printed
mix_ratio= INFO line (what the run actually used) instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
T5: eval_hack_solve helper + ablate_quarantine ctx; periodic ablated-eval
(hack_abl/solve_abl cols, appended so results.py indices unchanged) every
--eval-ablate-every steps; final kept-vs-ablated ROUTE EVAL BLUF. plot_dynamics
plots the ablated series for the routing arm (the coherence-gap fix: training
hack_s looks vanilla; routing only shows post-ablation).
External-review fixes (docs/spec/20260530_code_review.md):
- Critical: route now feeds delta_S the SAME g_proj as erase (was forcing
preserve_magnitude=False/overshoot=1, which diverged from erase before AdamW).
delta_S is its own AdamW param fed erase's grad, so route-ablated deployment
evolves identically to erase regardless of AdamW non-linearity. Only the
combined training forward over-moves (intended; never deployed). Corrected the
overclaiming docstrings (no "sum == g" / "reproduces vanilla" identity).
- Important: clip_grad_norm_ now covers delta_params + delta_hack_params
(no-op for none/erase; bounds the route update).
- Important: results.py paired-delta table includes routing (keyed on arm).
smoke route/erase/vanilla green: dsh route=0.0105 erase/none=0, span=2.9e-7,
ROUTE EVAL BLUF prints.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Stage-1 (T3) of the routing spec. Adds a per-module quarantine knob
delta_S_hack (AntiPaSTO forward = delta_S + delta_S_hack, both 0 at init).
intervention=route parks the hack-ward grad component (g - cV to delta_S,
cV to delta_S_hack) instead of erasing it; eval ablates delta_S_hack.
- proj.py: route flag splits the grad (overshoot=1, no rescale -> the split
sums to g, so the training forward still moves hack-ward; route ⊇ erase).
- antipasto.py: second trainable knob, identity preserved at init.
- train.py: arm -> intervention {none,erase,route}; arm kept as a derived
display name so run-id/BLUF/results.py/plot classify are unchanged. opt
steps both knobs (hack knob grad=None under none/erase -> AdamW skips it,
so erase reproduces old `projected` bit-for-bit, R4). R3 span assert
(resid/||gh|| < 1e-4) + end-of-run ||delta_S_hack|| guard (route >0).
- results.py / plot_dynamics.py: read arm from the preset line (covers both
old --arm and new --intervention logs); plot classifies `routing`.
smoke: none ||dsh||=0, erase clean, route ||dsh||=0.0105 span=2.9e-7. 64
archived projected rows still parse.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both were negative results (docs Q4, Q9) and are now dead weight. Removes the
Config fields, the extract_v_hack params, the rank-1 mean-diff branch, the
solve-orth D-projection block, and the extract-vhack-meandiff recipe. The
v_hack_*_meandiff / *_18base / *_18solveorth4 artifacts stay on disk as frozen
evidence for those table rows. Smoke passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Read the safetensors shapes/metadata: v_hack_full = 10 pairs / k=5,
v_hack_21pairs = 16 pairs / k=12 (n_heldout=2; neither is 18 or 21). The two
bases differ on pairs AND directions-kept AND extract-tau simultaneously, so
the hack-cut gap is triple-confounded, not a clean "pair set is the lever"
result. Nothing was lost: the strong basis reproduces from current pairs.py
via --top-k=12 --v-hack-drop-bottom-frac=0.0, and refresh already re-extracts
at k=12. Rewrites Q8 + the top confound bullet + the README findings caveat.
A one-knob k-sweep is needed to attribute the gain.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
docs/results.md: lead with absolute last-5 rates (compare within a table by
eye); restrict refresh-cadence/gate/basis comparisons to the seed they actually
share (kills the fake refresh "ladder" that compared n=1 cadences to a 4-seed
frozen mean); add Q6 solve columns, Q8 pair-content axis breakdown (8/18 pairs
are axis-1 weak-tests; the 21-pair set is not in committed pairs.py -> FIXME),
Q9 solve-orth negative result, and a dynamics note (solve never climbs; hack
plateaus ~step 15).
scripts/results.py: add `log` provenance column; drop the wide argv/time cols.
Lock mix_ratio=0.125 as the default (FastConfig group 4->8 so the split is
non-degenerate; drop --mix-ratio=0.5 from fast recipes). Q6 shows 0.125 keeps
the hack cut with no solve tax. Smoke passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- paired view: join projected to vanilla on (mix, seed), per-seed delta, mean
+/- std over shared seeds. Comparing a 3-seed mean to a 1-seed point is
meaningless; this enforces same-seed comparison (ml_debug principle).
- grouped view now reports std across seeds (null at n=1).
- exclude in-progress/aborted runs (must log all `steps`) so partial logs
don't read as impossibly-good results.
- docs/results.md rewritten around paired deltas; honest that at n=4 the
last-5 Dhack std (~0.15) ~= the mean (~0.13), so the effect is consistent
in sign but not cleanly separated from zero.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- scripts/results.py + `just results`: aggregate logs/*.log into last-5
hack_s and gt_s (solve) tables, sorted-by-time + grouped-by-config, with
full argv provenance column. Filters smoke/probe runs.
- extract_vhack_grad: solve_orth_m knob — strip top-m known-solve subspace
(SVD of clean-side grads) from D before SVD, so projection doesn't ablate
the solve signal. No grader/oracle, off by default.
- docs/results.md: every experiment grouped by the question it answers
(feasibility, H1, gate_mode, basis, refresh, mix, noise-floor, pair-set)
with comparison tables and answers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Surfaces the H1 verbatim + falsification criteria, names two gaps up-front:
21 pairs vs preregistered 60-80, and the SEM-across-seeds clause not yet
evaluable at n=2. Addresses the comprehension panel's flag on H1 verbatim
omission (deepseek 3.0, gemini-flash 4.0 on hypothesis_clarity).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
AFK queue-reorder shoved #137-#139 (vanilla s=42, projected s=44 frozen +
refresh-2) ahead of 17 other queued jobs so the n=3 matched table lands
before next user check-in. Original G2-screen commands displaced to slot
IDs 137-139.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Compresses the lab report into ~1700 words for a LessWrong audience while
preserving the workshop-paper scaffolding (intro / setup / method /
result table / mechanism subplot / limitations / related work / next).
Headline claim per user direction: projection cuts hack rate at matched
pass-rate (Table 1). Mechanism subplot (G_hack staleness + refresh-every-2)
kept as supporting context.
External-panel critique pass (n=5 models, mean 4.4/5 ready) on dims
hook/clarity/inform_not_persuade/calibration/LW_voice. Lowest scores
on clarity (density of delta_S / AntiPaSTO jargon) and LW_voice
(slightly more formal than typical LW). Acceptable for first draft.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 1: 25 inline annotations on docs/lab/...partial_n3.md, covering
preregistration gaps (n=2 vs SEM clause; 21 pairs vs preregistered 60-80;
pass-rate at 10pp boundary), Adam-momentum projection leak, cosine-vs-null
baseline, mixed-pool training caveat, Appendix B step-0-hack-detector
inconsistency, refresh compute cost, and a few smaller items (mix_ratio
semantics, K_axes value, AntiPaSTO module count).
Phase 2: review file with strengths, weaknesses, per-section comments,
questions, and a four-tier accept criterion. Current verdict: weak accept
as internal lab report, major revision as public draft.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add TL;DR for skimmers; first paragraph + Table 1 now stand alone.
- Open the method with the user's three-line framing of the intervention.
- Rename v_hack -> G_hack in doc body (with one-line note about code/file name).
- Add PASS_RATE column to matched-seed Table 1; note seed-43 pass-rate cost.
- Define HACK_STUDENT on first use.
- Block-quote H1 verbatim from spec.md with falsification clause.
- Two appendices with full chat-templated rollouts (hack teacher example,
pre-training student example), special tokens preserved.
External-panel comprehension (spec.md as source) mean 4.0/5 "ready"; flagged
items addressed: missing PASS_RATE column, missing skimmer-friendly opener,
and the H1-vs-current-pair-count framing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Restructures the report around setup/hypothesis -> pair example -> extract -> apply
-> table -> staleness -> refresh -> limitations, following user's preferred shape.
External-panel critique pass (n=5 models, mean 4.6/5 ready) flagged one persuasive
turn and slightly-promotional title; both softened.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds RESEARCH_JOURNAL.md entry (j) and docs/lab/20260529_projection_vs_vanilla_partial_n3.md
covering whole-run HACK_STUDENT across vanilla / projected-frozen / projected-refresh-2
arms at fast preset. Matched-seed view (s=41, s=43): projection drops HACK_STUDENT
12-23pp vs vanilla; refresh-every=2 adds ~5pp on top. Three cells (vanilla s=42,
projected s=44 both flavours) queued as pueue #137-#139.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Added field to RewardResult earlier this session but never computed/passed it,
which crashed pueue #67 (reverse projection) at step 0 with TypeError on
RewardResult.__init__. Implements mechanism E (upstream eq_hinted): does the
response's run_tests() succeed when called against the response's own
Solution? This is the broadest "loophole used" signal -- superset of C and D.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Detector D (`hacked_wrong_tests`): the response defines run_tests() that
FAILS when run against canonical_solution -- model's assertions are wrong.
Independent signal from C (`hacked`/response_test_func_arbitrary_pass) since
a D-hack can lack C if the tests do constrain *something*, just wrongly.
`regrade_pool.py` walks the cached teacher pool and prints a (C, D)
contingency. Result on rh-s65 pool: 98.9% only-C, 0.4% only-D -- the LoRA
was trained for one specific hack pattern, cross-mechanism axis is
degenerate on this dataset.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Re-extract the hack subspace V against the current (delta_S-modified) model on
the same hand-crafted PAIRS, every --vhack-refresh-every steps. Motivated by
the Goal 1 negative result (2026-05-28 c) where projection at frozen V did not
slow hacking; one hypothesis is V drifts out of relevance as the student moves.
Off by default (0). Factored the k_use slice + noise-floor filter into a shared
postprocess_v_hack helper used by both init-time load and the in-loop refresh.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The probe_distill.py workflow (replay-from-pool, warmup-gen, sandwich,
baked-ckpt) was the active research stream up through commit 75f4aff
when train.py took over with the fast preset + mixed-pool flag. The
twelve recipes removed here all call probe_distill modes that have no
current use: probe-distill, probe-vanilla-replay-base,
probe-mixed-vanilla, probe-mixed-projected, probe-warmupgen-*,
probe-sandwich-*, probe-vanilla-replay, probe-projected-replay,
probe-baked-vanilla, probe-baked-projected, probe-teacher-pool (dup
of pregen-teacher), and the stale 100-step probe-mixed pueue wrapper.
Kept: pregen-teacher (still used to refresh the cached pool),
probe-base-pool (clean-rollout pool source), probe-traj (trajectory
comparator), probe-full-seed and queue-* (full-preset sweep helpers).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Current modes are one_sided (erase positive c only, leaves negative
intact) and no_gate (erase span(V) entirely, drives V@g_proj to 0).
Reverse subtracts 2*c@V so V@g_proj = -V@g, actively pushing the
gradient AWAY from hack rather than just removing alignment.
Smoke confirms: cos_pre=+0.726 -> cos_post=-0.726 (clean flip).
Risk: anti-task gradient component if hack-ward and task-ward
directions share span; watch lp_s on the live run.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Deletes 7 source files that were superseded but never removed:
run.py, grad_proj.py, extract_vhack.py (older twin-NLL extractor),
grpo_smoke.py, grpo_proj_smoke.py (smoke harnesses replaced by
train.py "smoke" subcommand), phase2_analyze.py (pilot is past),
probe_uat.py (UAT pipeline is past).
Drops matching justfile recipes (vhack-check, phase2-analyze,
probe-uat) and the BASE constant that pointed at run.py. Updates
AGENTS/README references to the stale fast-dev-run recipe (now
just smoke / smoke-vanilla).
Verified by running just smoke-vanilla --steps=2 end-to-end.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This batch lands the working baseline (Goal 0 from RESEARCH_JOURNAL 2026-05-28
(b)) plus the architectural cleanups it surfaced. Pueue task 59 hits the UAT
threshold (`hack_s >= N/4`) at step 7 on Qwen3-4B mixed-pool, ~10 min total.
Preset/Adam scheduling
- New `Preset.fast` with aggressive Adam (lr=3e-3, beta1=0.5, beta2=0.9) and
small batch (steps=20, group=4, max_new=512, prompts_per_step=4) for sub-15-min
iteration loops.
- `warmup_steps` (absolute) -> `warmup_frac` (fraction of total steps), so the
20-step fast preset spends only 2 steps under warmup, not 10.
- `grad_clip` exposed as Config field (default 1.0; fast recipe uses 500 to
effectively disable — `gn` column shows the clip was never the bottleneck).
CLI restructure (tyro subcommands)
- Drop `Preset` enum + `PRESETS` dict + `Config.resolved()` Optional-merge hack.
- Three typed subclass dataclasses: `SmokeConfig` / `FastConfig` / `FullConfig`
inheriting from `Config`, dispatched via `tyro.extras.subcommand_cli_from_dict`.
- CLI: `train fast --arm=vanilla --lr=3e-3` (subcommand position, not --preset=).
- `cfg.preset_name` derived from `type(self).__name__` instead of duplicated field.
Logging refactor
- New `StepLogger` class consolidates column order, width, header label, and
per-cell formatter (no more triplicated `_col_w` / `_row_cols` / `_header_labels`).
- Row dict carries raw values throughout; formatters live in column spec.
Fixes the bug where end-of-run tabulate parsed `"7.00e-08"` strings as floats
and reformatted to `+0.000`. Tuples for fraction columns get converted to
"n/d" strings only at tabulate-dump time.
- `gn` column added (pre-clip total L2 norm; was discarded by clip_grad_norm_).
- `lr` column added (current scheduled LR through warmup + cosine).
- Timing cols (gen/fb/t_rew/sec) dropped from streaming view, still archived.
cin/cout -> cos_pre/cos_post + signed
- Rename across train.py, proj.py, probe_distill.py, run.py, smokes, plots,
justfile. "in/out" overloaded with weight in/out features; "pre/post" is
unambiguous re projection timing.
- Metric is now signed: sum(V @ g) / ||g|| instead of ||V @ g|| / ||g||. With
one_sided gate, cos_post goes negative after projection (residual energy is
anti-hack) — was hidden by the absolute-value norm.
v_hack extraction framing
- README + `extract_vhack_grad.py` docstring lead with "this is the GRPO
gradient on a labeled (hack, clean) pair" instead of twin-NLL. For a pair
with advantages +-1 the Dr.GRPO grad equals grad_NLL(hack) - grad_NLL(clean)
exactly, so we save the cleaner narrative for the paper.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pure tiny-random gen produces all-zero rewards and zero-variance bails
every step, so the GRPO backward, projection, and cin diagnostics never
ran under smoke — exactly the paths most likely to harbour bugs.
Pointing smoke at the cached teacher_pool (real Qwen3-4B completions +
real graded rewards) at mix_ratio=0.5 guarantees within-group reward
spread on every step. Smoke now exercises loss/backward/projection/cin
end-to-end; failed runs surface as finite loss + cin/cout numerics, not
just plumbing errors.
Side fix: decouple pool from prompt tokenization. Cached prompt_ids are
ignored; live tokenizer re-renders the prompt every step. Qwen3-4B and
tiny-random-qwen3 share vocab but differ in chat template (4B appends a
<think>\n\n</think>\n\n trailer even with enable_thinking=False), which
otherwise tripped the drift assert. Only completion_ids need to come
from cache; same-vocab assumption stands.
Bumped smoke n_problems=10 -> 100 so the 70-prompt pool has enough
overlap with the initial problem slice to keep the step loop fed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
wassname