diff --git a/RESEARCH_JOURNAL.md b/RESEARCH_JOURNAL.md index b75b964..cbe34f2 100644 --- a/RESEARCH_JOURNAL.md +++ b/RESEARCH_JOURNAL.md @@ -2,6 +2,58 @@ Append-only. New entries at the top, date-stamped. Never edit old entries. +## 2026-05-27 (b) — v_hack refactor: top-k=12 + S recorded + runtime suspicion gate + +See `docs/extract_vhack_grad-vec.md` for the full design doc with math and pseudocode. + +### What changed +- **Extract** at `top_k=12` (max), saves singular values `S` as `_sv/{name}` keys + alongside direction tensors. Switched SVD orientation from `sign(mean)` to + per-pair majority vote (outlier-robust). +- **Load-or-extract** in `train.py`: derives default v_hack path from + `model_slug + extract_top_k`, auto-extracts inline (~5 min) on cache miss using + the already-wrapped model. No more separate pueue extract job. +- **Load-time k-slicing** (`v_hack_k=5` default): extract once at k_max=12, slice + to k_use at load. k=1 vs k=5 vs k=12 is a config flip, not a re-extract. +- **Runtime suspicion gate** in `proj.py`: per step, drop top `susp_drop_frac` + (default 0.25) of `(module, axis)` pairs by `r_i = |g·v_i| / S_i`. Hypothesis: + weak-||D|| modules can have noise-fit v_i that coincidentally aligns with + structured coding gradient; gate detects via "live alignment >> extract-time + confidence". + +### Why +The "ablating noisy v_i has tiny effect because high-d" argument assumes v and g +are isotropic. They aren't — both live in low-d structured subspaces. If those +overlap, projection damage is large. The gate detects this empirically rather +than assuming v_hack is uniformly trustworthy across modules. + +### Status / caveats (codex external review flagged) +- `r_i` is not dimensionless across modules — high-gradient modules dominate the + global quantile. Fix candidate: within-module ratio `(|c_i|/||g||) / + (S_i/||D||_F)`. Not yet applied. +- Quantile is a fixed-budget knob, not a detector. Always drops 25% even when + nothing is suspicious. Fix candidate: absolute threshold post-normalization, + or measure-only mode first to calibrate. +- Old v1 files (no `_sv/` keys) silently bypass the gate. Should fail-fast when + `susp_drop_frac > 0` and `v_sv` is empty. + +### Validation plan (cheap tests, no training needed) +1. **cin_hack vs cin_clean on existing disk pools** (~5 min): backward-pass N + samples from `teacher_pool` and `base_pool`, measure cin distributions. If + `cin_hack >> cin_clean`, v_hack discriminates. Cheapest sanity check. +2. **Random-direction null**: cin vs random unit vector — strong signal if + v_hack >> random. +3. **Per-source cin during training**: extra backward gives `cin_s, cin_t` + separately. If projection is real, `cin_t > cin_s` initially. +4. **Bootstrap sign-stability**: re-extract on resampled pairs, check + `cos(v_hack, v_hack_boot)`. + +### Smoke +Ran `train.py --preset=smoke --model=llamafactory/tiny-random-qwen3 --steps=2`. +Auto-extract fired, load-or-extract end-to-end works. Gradients degenerate on +tiny random model (loss=0, cin=nan) — pipeline OK, but synthetic gate test +needed for real validation of the projection math. + ## 2026-05-27 — plan: switch from baked-base to mixed-pool GRPO from clean base ### Problem with current setup diff --git a/docs/extract_vhack_grad-vec.md b/docs/extract_vhack_grad-vec.md new file mode 100644 index 0000000..f54cfb7 --- /dev/null +++ b/docs/extract_vhack_grad-vec.md @@ -0,0 +1,263 @@ +# v_hack extraction: gradient-space SVD with magnitudes + runtime suspicion gate + +Living design doc for the v_hack pipeline. Sibling to `RESEARCH_JOURNAL.md`. +This explains *what we extract*, *why*, and *what runtime gating prevents*. + +## TL;DR + +`v_hack[name]` is a per-module top-k orthonormal basis in **AntiPaSTO +δS-gradient space**, computed by PCA on **paired (hack − clean) NLL gradients** +over a small set of contrastive completion pairs (currently N=14, 12 train + 2 +heldout). At training time we project the live policy-gradient component along +this basis out of `δS.grad`, optionally gated so we only ablate when there's +positive evidence the live gradient is hack-aligned. + +The 2026-05-27 refactor added three things on top of the older mean-diff design: + +1. **Top-k extraction** (k=12 max) with **load-time slicing** (`v_hack_k`, + default 5) so k=1 vs k=5 vs k=12 is a config flip, not a re-extract. +2. **Singular-value recording** (`_sv/{name}` keys) so v_i carries its + extract-time confidence S_i, not just direction. +3. **Runtime suspicion gate** (`susp_drop_frac`): per step, drop the top-frac + (module, axis) pairs by `r_i = |g·v_i| / S_i`. Live alignment ≫ extract + confidence means v_i is probably aligned with a structured coding direction, + not hack — skip the projection. + +## Why gradient space, not activation space? + +Most representation-steering work (ActAdd, RepE, CHaRS) operates on +**activations** (forward pass), shifting hidden states at inference. We +operate on **gradients of δS**, the trainable per-Linear AntiPaSTO knob. + +Reasons: +- We're not steering inference; we're **shaping training**. The projection + modifies `δS.grad` before the optimizer step, so the model itself doesn't + drift toward hack-aligned weight updates. +- δS gradients have a fixed, low-dimensional structure per module + (`δS ∈ R^r` where r = SVD rank of `W`). PCA-on-grads is computationally + cheap (12 pairs × N modules; r=2560 for largest mat) and gives a clean + per-module subspace. +- This is closest in spirit to CHaRS-PCT (Principal Component Thresholding, + §3.3 of `docs/paper_chars.md`): the L principal components of local-shift + covariance. We do the same maneuver on paired δS-gradient diffs. + +## Why δS basis (= weight-SVD basis), not raw param basis? + +AntiPaSTO wraps each Linear with `δW = U · diag(δS) · V_h`, where `U, S, V_h += SVD(W_pretrained)`. So `δS ∈ R^r` are coordinates **in the weight-SVD +basis**. The basis change is just a rotation — no whitening, no rescaling. + +Two things this buys us: + +- The number of trainable scalars is r per module (∼500–2500), not d_in×d_out. + A few hundred contrastive pairs would be needed to estimate dense + `d_in × d_out` direction; only a few pairs are needed in `R^r`. +- Low-rank perturbations (LoRA-style hack adapters) are sparse in this basis, + which makes per-direction gating in `δS` meaningful even with N=12 pairs. + +What this does **not** buy us: regularization. The weight-SVD basis is just a +convenient coordinate system; PCA on top of it still has to do the work of +finding which coordinates carry hack-clean discriminative signal. + +## Extraction pipeline + +```python +# pseudo: extract_v_hack(model, tokenizer, wrappers, pairs, top_k, tau_axis, n_heldout, device) + +train_pairs = pairs[:-n_heldout] # currently 12 of 14 + +# Gather per-pair, per-module gradients on hack-completion and clean-completion NLL. +grads_hack[name]: list of [r]-tensors, length n_pairs +grads_clean[name]: list of [r]-tensors, length n_pairs + +for pair in train_pairs: + for label, completion in [("hack", pair.hack), ("clean", pair.clean)]: + model.zero_grad() + loss = mean_NLL_on_completion_tokens(model, pair.prompt + completion) + loss.backward() # populates δS.grad per module + for name, info in wrappers.items(): + bucket[name].append(info.delta_S.grad.detach().cpu().float().clone()) + +# Per module: PCA on paired diff. +for name in wrappers: + G_h = stack(grads_hack[name]) # [n_pairs, r] + G_c = stack(grads_clean[name]) + D = G_h - G_c # [n_pairs, r]: per-pair hack-axis displacement + U_d, S_d, Vh_d = svd(D) # truncated, m = min(n_pairs, r) + V = Vh_d[:k_max] # [k_max, r], orthonormal rows + # Orient v_i so +v_i points hack-ward (majority vote across pairs). + proj = D @ V.T # [n_pairs, k_max] + n_pos = (proj > 0).sum(0) + flip = where(n_pos < n_pairs/2, -1, +1) + V = V * flip[:, None] + v_hack[name] = V + v_hack[f"_sv/{name}"] = S_d[:k_max] # NEW: singular values saved alongside +``` + +**File schema (v2):** +- `{name}` → Tensor[k_max, r], orthonormal hack-axis basis, oriented +hack +- `_sv/{name}` → Tensor[k_max], singular values of D in that basis +- metadata: `model`, `dtype`, `top_k`, `tau_axis`, `schema=v2_with_sv` + +## Load-or-extract (2026-05-27) + +`train.py` derives `v_hack_path` from `(model_name, v_hack_extract_top_k)` +unless overridden. If the file is missing, it extracts inline on the +already-wrapped model: + +``` +v_hack_path = OUT_DIR / f"v_hack_{model_slug}_k{extract_top_k}.safetensors" +if not v_hack_path.exists(): + v_hack_dict, raw_grads, _ = extract_v_hack(model, tok, wrappers, PAIRS, + top_k=extract_top_k, ...) + save_file(v_hack_dict, v_hack_path, metadata={...}) +v_hack, v_sv = load_v_hack(v_hack_path, model_name, wrappers, k_use=v_hack_k) +``` + +This means a fresh model with no cached v_hack just runs extract once +(~5 min for 4B-class) and proceeds. No prerequisite jobs, no manual flags. + +## Load-time k-slicing + +Extract saves k_max (default 12). Load slices to `k_use` (default 5). So +k=1 vs k=5 vs k=12 is a **config flip**, not a re-extract. The +`mean_sv_top5_frac` from our 2026-05-26 extract was 0.71, so k=5 covers +~71% of per-module D-variance — load-time slice at 5 is a reasonable +default that we can ablate cheaply. + +## Runtime suspicion gate + +**Hypothesis:** module M has small `||D(M)||_F` (weak hack signal at +extract time). Its top SVD direction `v_1(M)` is dominated by noise +shape, not hack shape. At training time, `g(M)` is the policy gradient +flowing through M — a structured (non-isotropic) signal living in a +low-d subspace of "directions that matter for next-token prediction." If +`v_1(M)` coincidentally lies in that subspace, projecting `g(M)` along +`v_1(M)` removes a chunk of useful coding-relevant gradient with no +compensating reduction in hack signal. + +**Why I'd initially dismissed this concern:** in a high-d random model +(g and v isotropic), `|g · v| ≈ ||g||/√r ≈ 2% of ||g||`. So one bad +direction costs ~2% of the live gradient — tolerable. **What I missed:** +neither `g` nor `v` is isotropic. Both live in low-d structured +subspaces. If those subspaces happen to overlap, the projection magnitude +is much larger. + +**Gate design:** + +``` +r_i(M) = |g(M) · v_i(M)| / S_i(M) +``` + +- High `r_i`: live grad cares about v_i much more than the extract-time + hack signal did → suspicious, this v_i is probably picking up + structured coding flow. +- Low `r_i`: live alignment is in proportion to extract-time confidence + → trust the projection. + +**Per-step quantile gate:** collect `r_i` across all `(module, axis)` +pairs in one step, find the `(1 − drop_top_frac)`-quantile, suppress all +axes above that threshold for this step. Default `drop_top_frac = 0.25`. + +```python +# pseudo: in project_delta_S_grad +all_r = [] +for name, info in wrappers.items(): + c = V[name] @ info.delta_S.grad # [k_use] + S = v_sv[name] # [k_use] + all_r.append(c.abs() / S.clamp_min(eps)) +threshold = quantile(cat(all_r), 1 − drop_top_frac) +for ...: + keep = (r <= threshold) + g_proj = g − (c * keep * gate_mode_mask) @ V +``` + +## Known limitations (caveats from codex external review, 2026-05-27) + +1. **r_i is not dimensionless across modules.** `|g·v_i|` scales with + live-grad norm; `S_i` scales with extract-time-grad norm. A + high-gradient module dominates the global quantile regardless of + whether its axis is actually suspicious. Fix candidate: within-module + ratio `(|c_i|/||g||) / (S_i/||D||_F)`. **Not yet applied.** +2. **Quantile gate is a fixed budget, not a detector.** It always drops + `drop_top_frac` of axes per step, even when nothing is genuinely + suspicious — and never drops more than that when many axes are. Fix + candidate: absolute threshold post-normalization, or run as + measure-only diagnostic until calibrated. **Not yet applied.** +3. **Old v1 files (no `_sv/` keys) silently disable the gate.** Should + fail-fast when `susp_drop_frac > 0` and v_sv is empty. + +## Validation: cheap discriminative tests + +The fundamental question: does v_hack actually discriminate hack from +clean gradients, or is it picking up irrelevant variance? + +### Test 1: cin_hack vs cin_clean on disk pools (cheap, ~5 min) + +We already have `out/probe_distill/teacher_pool/` (hacking samples) and +`out/probe_distill/base_pool/` (clean samples). For N samples each: + +``` +for prompt, completion, label in samples: + model.zero_grad() + loss = mean_NLL(model, prompt + completion).backward() + cin = (V @ delta_S.grad).norm() / delta_S.grad.norm() + record(label, cin) +``` + +**Discriminator:** `cin_hack_mean − cin_clean_mean`. If ≫ 0, v_hack +discriminates. If ≈ 0, v_hack is capturing prompt-length / generic +variance, not hack-specific direction. **Cost: ~5 min, no training.** + +### Test 2: held-out pair projection (existing) + +`verify_vhack_heldout.py` projects gradients from held-out pairs (last +n_heldout of PAIRS) onto trained v_hack. Already in CI-style flow. + +### Test 3: random-direction null + +For each module, compute cin onto v_hack vs onto a random unit vector of +the same shape. If `cin_v_hack > cin_random` by a large margin, v_hack +is non-spurious. Trivial to implement. + +### Test 4: per-source cin during training (live) + +In mixed-pool runs we have student rollouts (initially ~no hack) and +teacher rollouts (all hack). Currently `cin` is computed on the +accumulated gradient (mixed). With ONE extra backward per step we can +compute `cin_s` (student-only grad) and `cin_t` (teacher-only grad) +separately. **Predict:** if v_hack is a real hack direction, +`cin_t > cin_s` initially; the gap shrinks as student picks up hack +(if it does). Useful for diagnosing whether the projection is doing +real work or just gradient noise. + +### Test 5: bootstrap sign-stability + +Bootstrap pairs (sample N-2 with replacement), re-extract v_hack, +compare `cos(v_hack_original, v_hack_bootstrap)`. If unstable, v_hack +is fitting noise. **Cost: 5 × ~5 min = 25 min total.** + +## Open design questions + +- **Is the suspicion gate redundant?** Codex argued the quantile design + is a fixed-budget knob, not a detector. The right answer is probably: + ship it as measure-only first (log `frac_axes_susp` and per-step + `r_i` distribution histograms), confirm whether suspicious modules + actually exist empirically, *then* turn on projection-side gating. +- **Should we whiten by S?** I.e. parameterize the AntiPaSTO knob as + `δS_i / σ_i(W)` so all directions have equal forward-pass impact. + Currently we don't. This is a separate, larger question. +- **Should we record per-pair pair tags / hack flavors?** With 12 + unlabeled pairs we can't do supervised LDA. With flavor labels + (hardcode / weak-tests / persona / format-leak) we could do LDA-on- + labels, which would beat unsupervised PCA at this N. + +## Related files + +- `src/projected_grpo/extract_vhack_grad.py` — extract function + CLI +- `src/projected_grpo/proj.py` — runtime projection + gates +- `src/projected_grpo/train.py:load_v_hack` — load + slice + auto-extract +- `src/projected_grpo/verify_vhack_heldout.py` — Test 2 above +- `src/projected_grpo/pairs.py` — the 14 contrastive pairs +- `docs/paper_chars.md` — CHaRS notes (PCT comparison) +- `RESEARCH_JOURNAL.md` — chronological progress log diff --git a/docs/spec/20260527_code_review.md b/docs/spec/20260527_code_review.md new file mode 100644 index 0000000..a9064b7 --- /dev/null +++ b/docs/spec/20260527_code_review.md @@ -0,0 +1,710 @@ +[client] initialize (running) + +[client] session/new (running) +I’ll use the local review workflow plus the `gsd` skill because this is a multi-step code review, then I’ll read the project instructions and changed files before making findings. + +[tool] pwd && rg --files -g 'AGENTS.md' -g 'CLAUDE.md' -g 'docs/spec.md' -g 'docs/brainstorm/extracted_prefs.md' -g 'docs/RESEARCH_JOURNAL.md' (running) + input: {"call_id":"call_26P3qQ1bTCPLMMOGhHlNW3B7","process_id":"3146","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","pwd && rg --files -g 'AGENTS.md' -g 'CLAUDE.md' -g 'docs/spec.md' -g 'do... + +[tool] pwd && rg --files -g 'AGENTS.md' -g 'CLAUDE.md' -g 'docs/spec.md' -g 'docs/brainstorm/extracted_prefs.md' -g 'docs/RESEARCH_JOURNAL.md' (failed) + kind: execute + input: {"call_id":"call_26P3qQ1bTCPLMMOGhHlNW3B7","process_id":"3146","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","pwd && rg --files -g 'AGENTS.md' -g 'CLAUDE.md' -g 'docs/spec.md' -g 'do... + output: + bwrap: No permissions to create new namespace, likely because the kernel does not allow non-privileged user namespaces. See or . + +[tool] Read SKILL.md (running) + input: {"call_id":"call_pqWiVSayENSrfXMHJIaa2TDn","process_id":"2476","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","sed -n '1,220p' /root/.claude/skills/gsd/SKILL.md"],"cwd":"/workspace/pr... + files: /root/.claude/skills/gsd/SKILL.md + +[tool] Read SKILL.md (failed) + kind: read + input: {"call_id":"call_pqWiVSayENSrfXMHJIaa2TDn","process_id":"2476","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","sed -n '1,220p' /root/.claude/skills/gsd/SKILL.md"],"cwd":"/workspace/pr... + files: /root/.claude/skills/gsd/SKILL.md + output: + bwrap: No permissions to create new namespace, likely because the kernel does not allow non-privileged user namespaces. See or . + +[tool] pwd && rg --files -g 'AGENTS.md' -g 'CLAUDE.md' -g 'docs/spec.md' -g 'docs/brainstorm/extracted_prefs.md' -g 'docs/RESEARCH_JOURNAL.md' (running) + input: {"call_id":"call_qsJCRIGJ2i5HN4N8VhYCiO86","process_id":"97013","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","pwd && rg --files -g 'AGENTS.md' -g 'CLAUDE.md' -g 'docs/spec.md' -g 'd... + +[tool] pwd && rg --files -g 'AGENTS.md' -g 'CLAUDE.md' -g 'docs/spec.md' -g 'docs/brainstorm/extracted_prefs.md' -g 'docs/RESEARCH_JOURNAL.md' (completed) + kind: execute + input: {"call_id":"call_qsJCRIGJ2i5HN4N8VhYCiO86","process_id":"97013","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","pwd && rg --files -g 'AGENTS.md' -g 'CLAUDE.md' -g 'docs/spec.md' -g 'd... + output: + /workspace/projected_grpo + docs/brainstorm/extracted_prefs.md + AGENTS.md + + ```sh + /workspace/projected_grpo + docs/brainstorm/extracted_prefs.md + AGENTS.md + ``` + +[tool] Read SKILL.md (running) + input: {"call_id":"call_Epox1WkTa0YbizPn1jb4k52M","process_id":"4410","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","sed -n '1,220p' /root/.claude/skills/gsd/SKILL.md"],"cwd":"/workspace/pr... + files: /root/.claude/skills/gsd/SKILL.md + +[tool] Read SKILL.md (completed) + kind: read + input: {"call_id":"call_Epox1WkTa0YbizPn1jb4k52M","process_id":"4410","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","sed -n '1,220p' /root/.claude/skills/gsd/SKILL.md"],"cwd":"/workspace/pr... + files: /root/.claude/skills/gsd/SKILL.md + output: + --- + name: gsd + description: "Get Shit Done workflow for non-trivial tasks: spec -> implement -> test -> review -> wrap. Mandatory for anything with more than one step." + --- + + + + # GSD (Get Shit Done) + + Mandatory for all non-trivial tasks (more than one step). Skip only for single-line fixes. + + ## Part 1: Core workflow + + ### 1.1 Spec-first planning + + ... (191 more lines) + +[plan] + - [in_progress] Read local instructions and gsd workflow + - [pending] Collect diffs and changed-file context + - [pending] Review suspicion gate math and load/extract flow + - [pending] Check schema/test implications + - [pending] Write final review +The sandbox is blocking normal read commands, so I’m using approved escalated reads only. Next I’m collecting the actual diff and full file context, with the review scoped to the changed files. + +[tool] Read AGENTS.md (running) + input: {"call_id":"call_NLThtl3Ffv5iUZwilR5SADCz","process_id":"14734","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","sed -n '1,240p' AGENTS.md"],"cwd":"/workspace/projected_grpo","parsed_c... + files: /workspace/projected_grpo/AGENTS.md + +[tool] git status --short (running) + input: {"call_id":"call_xaCeeYFvoLvaO7EJ0Sl9is82","process_id":"72006","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","git status --short"],"cwd":"/workspace/projected_grpo","parsed_cmd":[{"... + +[tool] git diff -- src/projected_grpo/extract_vhack_grad.py src/projected_grpo/train.py src/projected_grpo/proj.py src/projected_grpo/verify_vhack_heldout.py src/projected_grpo/probe_distill.py (running) + input: {"call_id":"call_KnN7vfJXbBaKGCBOobcZOO4x","process_id":"78686","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","git diff -- src/projected_grpo/extract_vhack_grad.py src/projected_grpo... + +[tool] git diff --cached -- src/projected_grpo/extract_vhack_grad.py src/projected_grpo/train.py src/projected_grpo/proj.py src/projected_grpo/verify_vhack_heldout.py src/projected_grpo/probe_distill.py (running) + input: {"call_id":"call_i2OvvQxGyfTxRhy2A5IPLZ5c","process_id":"75184","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","git diff --cached -- src/projected_grpo/extract_vhack_grad.py src/proje... + +[tool] Read AGENTS.md (completed) + kind: read + input: {"call_id":"call_NLThtl3Ffv5iUZwilR5SADCz","process_id":"14734","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","sed -n '1,240p' AGENTS.md"],"cwd":"/workspace/projected_grpo","parsed_c... + files: /workspace/projected_grpo/AGENTS.md + output: + # AGENTS.md — projected_grpo + + **This is novel ML research.** Not in your training data. Extrapolate carefully. + + ## Project in one paragraph + + Test whether SVD-basis gradient projection against an extracted hack-direction + reduces reward-hack rate in GRPO on Nanda's LeetCode benchmark. Differs from + Rebound (Wu & Tang 2026) by intervening at the *gradient* level rather than the + *advantage* level. Differs from AntiPaSTO (the user's prior work) by using + unpaired GRPO rollouts rather than paired-preference contrast. + + Inherit global rules from `~/.claude/CLAUDE.md`. + + ## Workflow + + - Read [docs/spec.md](spec.md) for the preregistered plan. + - Read [docs/brainstorm/extracted_prefs.md](docs/brainstorm/extracted_prefs.md) for design rationale. + - New sweep arms get recipes in [justfile](justfile) with `# H:` hypothesis comments. + - `just fast-dev-run` before any real run (~1-2 min, beartype on, real pipeline on tiny inputs). + - Real runs go through `pueue` on the 96GB GPU box. Label each job with `why:` and `resolve:`. + - Head [docs/RESEARCH_JOURNAL.md](docs/RESEARCH_JOURNAL.md) for latest results. + - No `tests/` dir; `fast-dev-run` is the correctness gate. + + ## External dependencies + + `external/rl-rewardhacking/` is Ariahw's repo (verl-based GRPO + LeetCode dataset + + reward hacking monitors). We import from it; we do NOT modify it. Sync with + ... (44 more lines) + +[tool] git diff -- src/projected_grpo/extract_vhack_grad.py src/projected_grpo/train.py src/projected_grpo/proj.py src/projected_grpo/verify_vhack_heldout.py src/projected_grpo/probe_distill.py (completed) + kind: execute + input: {"call_id":"call_KnN7vfJXbBaKGCBOobcZOO4x","process_id":"78686","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","git diff -- src/projected_grpo/extract_vhack_grad.py src/projected_grpo... + output: + diff --git a/src/projected_grpo/extract_vhack_grad.py b/src/projected_grpo/extract_vhack_grad.py + index 5e9f1ff..57ed5d0 100644 + --- a/src/projected_grpo/extract_vhack_grad.py + +++ b/src/projected_grpo/extract_vhack_grad.py + @@ -51,7 +51,14 @@ class Config: + out_path: Path = OUT_DIR / "v_hack.safetensors" + train_grads_path: Path = OUT_DIR / "vhack_grads_train.safetensors" + n_heldout: int = 2 # last n pairs reserved for held-out validation + - top_k: int = 5 # rank of hack subspace per module (≤ n_train_pairs) + + # top_k=12 = max(n_train_pairs after n_heldout=2 from N=14 pairs). Extract once + + # at max rank; train.py slices via --v-hack-k for k-ablation without re-extract. + + top_k: int = 12 + + # tau_axis: zero rows where S_i/S_0 < tau_axis. Diagnostic — projection along + + # noise-direction unit vectors removes only ~||g||/sqrt(r) ≈ 2% of grad + + # magnitude on r=2560 modules, so this rarely changes effect size; it does + + # make k-ablations honest (axes 4-5 might be pure noise on N=12 pairs). + + tau_axis: float = 0.0 + + + def resolve_dtype(s: str) -> torch.dtype: + @@ -73,29 +80,32 @@ def completion_nll(model, tokenizer, prompt: str, completion: str, device) -> to + return (nll * mask).sum() / mask.sum().clamp_min(1.0) + + + -def main(cfg: Config) -> int: + - device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + - dtype = resolve_dtype(cfg.dtype) + - logger.info( + ... (1103 more lines) + +[tool] git status --short (completed) + kind: execute + input: {"call_id":"call_xaCeeYFvoLvaO7EJ0Sl9is82","process_id":"72006","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","git status --short"],"cwd":"/workspace/projected_grpo","parsed_cmd":[{"... + output: + M justfile + M src/projected_grpo/extract_vhack_grad.py + M src/projected_grpo/probe_distill.py + M src/projected_grpo/proj.py + M src/projected_grpo/train.py + M src/projected_grpo/verify_vhack_heldout.py + ?? docs/spec/20260527_code_review.md + + ```sh + M justfile + M src/projected_grpo/extract_vhack_grad.py + M src/projected_grpo/probe_distill.py + M src/projected_grpo/proj.py + M src/projected_grpo/train.py + M src/projected_grpo/verify_vhack_heldout.py + ?? docs/spec/20260527_code_review.md + ``` + +[tool] git diff --cached -- src/projected_grpo/extract_vhack_grad.py src/projected_grpo/train.py src/projected_grpo/proj.py src/projected_grpo/verify_vhack_heldout.py src/projected_grpo/probe_distill.py (completed) + kind: execute + input: {"call_id":"call_i2OvvQxGyfTxRhy2A5IPLZ5c","process_id":"75184","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","git diff --cached -- src/projected_grpo/extract_vhack_grad.py src/proje... + output: + { + "call_id": "call_i2OvvQxGyfTxRhy2A5IPLZ5c", + "process_id": "75184", + "turn_id": "019e681a-3b48-7ed3-8e6b-1912b41adb6f", + "command": [ + "/usr/bin/zsh", + "-lc", + "git diff --cached -- src/projected_grpo/extract_vhack_grad.py src/projected_grpo/train.py src/projected_grpo/proj.py src/projected_grpo/verify_vhack_heldout.py src/projected_grpo/probe_distill.py" + ], + "cwd": "/workspace/projected_grpo", + "parsed_cmd": [ + { + "type": "unknown", + "cmd": "git diff --cached -- src/projected_grpo/extract_vhack_grad.py src/projected_grpo/train.py src/projected_grpo/proj.py src/projected_grpo/verify_vhack_heldout.py src/projected_grpo/probe_distill.py" + } + ], + "source": "unified_exec_startup", + "stdout": "", + "stderr": "", + "aggregated_output": "", + "exit_code": 0, + "duration": { + "secs": 1, + "nanos": 432131924 + }, + "formatted_output": "", + "status": "completed" + } +I have the high-level diff. I’m reading the full changed files now because the suspicion gate depends on surrounding training order, optimizer setup, and how `delta_S` is wrapped. + +[tool] Read proj.py (running) + input: {"call_id":"call_B8lYmbn1uM0INz0YwcS54y79","process_id":"37400","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","nl -ba src/projected_grpo/proj.py"],"cwd":"/workspace/projected_grpo","... + files: /workspace/projected_grpo/src/projected_grpo/proj.py + +[tool] Read train.py (running) + input: {"call_id":"call_19Uf3j0MKdVXQPhRaIstMKAF","process_id":"58769","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","nl -ba src/projected_grpo/train.py"],"cwd":"/workspace/projected_grpo",... + files: /workspace/projected_grpo/src/projected_grpo/train.py + +[tool] Read extract_vhack_grad.py (running) + input: {"call_id":"call_UjQGhDRH4LEXXP0yi2Oa8hOV","process_id":"85164","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","nl -ba src/projected_grpo/extract_vhack_grad.py"],"cwd":"/workspace/pro... + files: /workspace/projected_grpo/src/projected_grpo/extract_vhack_grad.py + +[tool] Read verify_vhack_heldout.py (running) + input: {"call_id":"call_9gmm3VORQurDl6RBtIOyqOCR","process_id":"42951","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","nl -ba src/projected_grpo/verify_vhack_heldout.py"],"cwd":"/workspace/p... + files: /workspace/projected_grpo/src/projected_grpo/verify_vhack_heldout.py + +[tool] Read proj.py (completed) + kind: read + input: {"call_id":"call_B8lYmbn1uM0INz0YwcS54y79","process_id":"37400","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","nl -ba src/projected_grpo/proj.py"],"cwd":"/workspace/projected_grpo","... + files: /workspace/projected_grpo/src/projected_grpo/proj.py + output: + 1 """Gradient projection + delta_S grad utilities. Imported by smoke and train.""" + 2 from __future__ import annotations + 3 + 4 import torch + 5 + 6 + 7 def per_token_logps(logits: torch.Tensor, ids: torch.Tensor) -> torch.Tensor: + 8 """log p(ids | logits) gathered token-wise. + 9 + 10 Uses F.cross_entropy (fused softmax+gather) so we never materialise the + 11 full [B, L, V] fp32 softmax. On Qwen3.5-2B with V=152k, G=8, L≈1500 the + 12 fp32 vocab tensor was ~7 GB per forward — the difference between OOM and + 13 fit on a 96 GB card when the autograd graph is alive. + 14 """ + 15 B, L, V = logits.shape + 16 # CE's internal log_softmax accumulates in fp32 (stable) but returns input dtype. + 17 # The output [B*L] is small, so upcast it to fp32 for downstream PPO ratio math. + 18 return -torch.nn.functional.cross_entropy( + 19 logits.reshape(-1, V), ids.reshape(-1), reduction="none" + 20 ).float().view(B, L) + 21 + 22 + 23 @torch.no_grad() + 24 def project_delta_S_grad( + 25 wrappers: dict, + 26 v_hack: dict[str, torch.Tensor], + 27 preserve_magnitude: bool, + 28 measure_only: bool = False, + ... (251 more lines) + +[tool] Read train.py (completed) + kind: read + input: {"call_id":"call_19Uf3j0MKdVXQPhRaIstMKAF","process_id":"58769","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","nl -ba src/projected_grpo/train.py"],"cwd":"/workspace/projected_grpo",... + files: /workspace/projected_grpo/src/projected_grpo/train.py + output: + 1 """Canonical training entry point: AntiPaSTO + GRPO (Dr.GRPO unbiased) + optional + 2 gradient projection on LeetCode reward-hacking benchmark. + 3 + 4 Lineage (see spec.md §76-83): + 5 - The inner GRPO_step (per_token_logps, ratio + clip + min, K3 KL, per-token + 6 loss, completion mask) is a direct port of lsdefine/simple_GRPO's + 7 `GRPO_step` in `grpo_vllm_one.py` (lines 64-95). + 8 - The OUTER loop adopts simple_GRPO's `Q_batch_size` pattern (multiple + 9 prompts per optimizer step, per-prompt GRPO advantage groups, grad + 10 accumulation across prompts). GRPO needs within-group reward diversity to + 11 produce any signal; sampling many prompts per step raises the chance that + 12 at least one group is non-degenerate. simple_GRPO uses Q_batch_size=5; our + 13 prompts_per_step is set in PRESETS (grad-accum to the paper's effective batch). + 14 - Deviations from simple_GRPO are deliberate, listed in spec.md: + 15 1. Loss normalization: Dr.GRPO unbiased (Liu et al. 2025, arXiv + 16 2503.20783) replaces simple_GRPO's `(R-mean)/std` + per-response-len + 17 denominator. Drops two biases: + 18 - length norm `1/|o_i|` (favors short correct, long incorrect) + 19 - group-std norm `/std(R)` (overweights easy/hard questions) + 20 Toggle via `--unbiased` (default on); flipping to False recovers + 21 simple_GRPO's classic GRPO advantage normalization. + 22 2. Reference model: simple_GRPO runs a separate base model via an HTTP + 23 `ref_server`. We use the AntiPaSTO `delta_S=0` zero-adapter trick + 24 (W' = W + U diag(0) Vh = W exactly) — no second model loaded. + 25 3. Rollout: simple_GRPO uses vLLM in a separate process. We use HF + 26 `model.generate` in-process. + 27 4. Adapter: simple_GRPO is full FT (with DeepSpeed ZeRO). Canonical + 28 (ariahw/rl-reward... + +[tool] Read verify_vhack_heldout.py (completed) + kind: read + input: {"call_id":"call_9gmm3VORQurDl6RBtIOyqOCR","process_id":"42951","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","nl -ba src/projected_grpo/verify_vhack_heldout.py"],"cwd":"/workspace/p... + files: /workspace/projected_grpo/src/projected_grpo/verify_vhack_heldout.py + output: + 1 """Held-out v_hack validation (spec.md §B validation). + 2 + 3 For each held-out pair, compute per-module gradient diff (g_hack - g_clean) + 4 in delta_S basis, then cos-align with the trained v_hack[name]. + 5 + 6 Report: + 7 - per-suffix median/mean cos_align + 8 - fraction of modules with cos_align > 0 (SHOULD > 0.5) + 9 - mean cos_align across modules (target > 0.2) + 10 + 11 Run: uv run python -m projected_grpo.verify_vhack_heldout + 12 """ + 13 from __future__ import annotations + 14 + 15 import sys + 16 from collections import defaultdict + 17 from dataclasses import dataclass + 18 from pathlib import Path + 19 + 20 import json + 21 + 22 import torch + 23 import tyro + 24 from loguru import logger + 25 from safetensors.torch import save_file + 26 from tabulate import tabulate + 27 from transformers import AutoModelForCausalLM, AutoTokenizer + 28 + ... (279 more lines) + +[tool] Read extract_vhack_grad.py (completed) + kind: read + input: {"call_id":"call_UjQGhDRH4LEXXP0yi2Oa8hOV","process_id":"85164","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","nl -ba src/projected_grpo/extract_vhack_grad.py"],"cwd":"/workspace/pro... + files: /workspace/projected_grpo/src/projected_grpo/extract_vhack_grad.py + output: + 1 """Gradient-side per-module v_hack extraction (spec.md §B, top-k variant). + 2 + 3 For each contrastive pair (prompt, hack_completion, clean_completion): + 4 - Forward(prompt+completion), mean-NLL on completion tokens, backward + 5 - Capture `delta_S.grad` per AntiPaSTO-wrapped Linear + 6 + 7 Then per module, with D = [g_hack_i - g_clean_i for each pair] in R^{n_pairs x r}: + 8 SVD(D) = U Σ Vh + 9 v_hack[name] = top_k rows of Vh, each oriented so mean(D @ v_i) > 0 + 10 + 11 This generalizes mean-diff (which corresponds to top-1 PC of paired diffs under + 12 isotropic covariance) to a rank-k hack subspace, motivated by CHaRS (Abdullaev + 13 2025 — see docs/paper_chars.md): hack signal is multi-modal across hack flavors + 14 (weak tests, hardcode, persona, ...), so a single global direction is brittle. + 15 + 16 Orientation matters because proj.py applies a per-direction one-sided gate + 17 (only subtracts when positive). +v_i must point hack-ward. + 18 + 19 Saves `out/v_hack.safetensors` = dict[name -> Tensor[k, r]] (cpu fp32, rows + 20 unit-norm + orthonormal from SVD) with header {"model": str, "dtype": str, + 21 "top_k": str(k)}. + 22 + 23 Run: uv run python -m projected_grpo.extract_vhack_grad + 24 """ + 25 from __future__ import annotations + 26 + 27 import sys + 28 from collections import defaultdict + ... (515 more lines) + +[tool] Read train.py (running) + input: {"call_id":"call_XmsO6ztYV0PA2nSx8J4xdbu9","process_id":"5743","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","nl -ba src/projected_grpo/train.py | sed -n '390,530p'"],"cwd":"/workspa... + files: /workspace/projected_grpo/src/projected_grpo/train.py + +[tool] Read probe_distill.py (running) + input: {"call_id":"call_NwEh3dqGqjDFXQuySFaT2c0N","process_id":"46600","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","nl -ba src/projected_grpo/probe_distill.py | sed -n '180,240p'"],"cwd":... + files: /workspace/projected_grpo/src/projected_grpo/probe_distill.py + +[tool] Read 20260527_code_review.md (running) + input: {"call_id":"call_u67hecWsDhI7oi0boUU1C0q5","process_id":"15286","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","nl -ba docs/spec/20260527_code_review.md"],"cwd":"/workspace/projected_... + files: /workspace/projected_grpo/docs/spec/20260527_code_review.md + +[tool] git diff -- justfile (running) + input: {"call_id":"call_3AYtTkFJY9bhQBbuGufHXJiA","process_id":"61295","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","git diff -- justfile"],"cwd":"/workspace/projected_grpo","parsed_cmd":[... + +[tool] Read train.py (completed) + kind: read + input: {"call_id":"call_XmsO6ztYV0PA2nSx8J4xdbu9","process_id":"5743","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","nl -ba src/projected_grpo/train.py | sed -n '390,530p'"],"cwd":"/workspa... + files: /workspace/projected_grpo/src/projected_grpo/train.py + output: + 390 metadata={"model": model_name, "dtype": "bf16", + 391 "top_k": str(min(cfg.v_hack_extract_top_k, len(VHACK_PAIRS) - 2)), + 392 "tau_axis": "0.0", "schema": "v2_with_sv"}) + 393 # extract leaves model.zero_grad() done; reset opt state isn't needed (opt built below) + 394 v_hack_cpu, v_sv_cpu = load_v_hack(v_hack_path, model_name, wrappers, k_use=cfg.v_hack_k) + 395 v_hack = {name: v.to(device) for name, v in v_hack_cpu.items()} + 396 v_sv = {name: s.to(device) for name, s in v_sv_cpu.items()} if v_sv_cpu else None + 397 # Teacher pool: pre-generated rollouts on disk keyed by problem_id. Each step's + 398 # G_t teacher rollouts come from a uniform random sample of that prompt's cache, + 399 # so we do *not* keep the teacher model in VRAM. Pool is produced by + 400 # `probe_distill.py --teacher-only` (see schema in probe_distill.py:149-186). + 401 # Cached rewards/flags are reused verbatim — no re-grading — so the pool is a + 402 # reproducible fixed teacher distribution across runs. + 403 teacher_pool: dict[int, list[dict]] = {} + 404 G_s = group + 405 G_t = 0 + 406 if cfg.teacher_pool_dir is not None: + 407 if not (0.0 < cfg.mix_ratio < 1.0): + 408 raise ValueError(f"mix_ratio must be in (0,1) when teacher_pool_dir set; got {cfg.mix_ratio}") + 409 G_t = round(group * cfg.mix_ratio) + 410 G_s = group - G_t + 411 if G_s == 0 or G_t == 0: + 412 raise ValueError( + 413 f"degenerate split: G={group} mix_ratio={cfg.mix_ratio} -> G_s={G_s}, G_t={G_t}. " + 414 f"Pick mix_ratio so both halves are non-empty, or drop --teacher-pool-dir." + 415 ) + 416 for path in sorted(cfg.teacher_pool_dir.glob("prompt_*.jsonl.gz")): + 417 # path.stem on 'prompt_0004.jsonl.gz' is 'prompt_0004.jsonl' (only one + .... + +[tool] Read probe_distill.py (completed) + kind: read + input: {"call_id":"call_NwEh3dqGqjDFXQuySFaT2c0N","process_id":"46600","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","nl -ba src/projected_grpo/probe_distill.py | sed -n '180,240p'"],"cwd":... + files: /workspace/projected_grpo/src/projected_grpo/probe_distill.py + output: + 180 + 181 + 182 def load_prompt(pool_dir: Path, problem_id: int) -> list[dict]: + 183 path = pool_dir / f"prompt_{problem_id:04d}.jsonl.gz" + 184 with gzip.open(path, "rt") as f: + 185 return [json.loads(line) for line in f] + 186 + 187 + 188 def main(cfg: Config) -> int: + 189 if cfg.tag: + 190 tag = cfg.tag + 191 elif cfg.teacher_only: + 192 tag = "teacher_pool" + 193 elif cfg.base_only: + 194 tag = "base_pool" + 195 else: + 196 tag = f"{cfg.arm}_seed{cfg.seed}" + 197 run_id = f"distill_{tag}" + 198 setup_logging(run_id) + 199 torch.manual_seed(cfg.seed) + 200 device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + 201 + 202 logger.info(f"argv: {' '.join(sys.argv)}") + 203 logger.info(f"arm={cfg.arm} teacher={cfg.teacher} steps={cfg.steps} " + 204 f"G={cfg.group} seed={cfg.seed} " + 205 f"teacher_only={cfg.teacher_only} replay={cfg.replay_dir is not None}") + 206 + 207 if cfg.teacher_only or cfg.base_only: + ... (97 more lines) + +[tool] git diff -- justfile (completed) + kind: execute + input: {"call_id":"call_3AYtTkFJY9bhQBbuGufHXJiA","process_id":"61295","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","git diff -- justfile"],"cwd":"/workspace/projected_grpo","parsed_cmd":[... + output: + diff --git a/justfile b/justfile + index c1dc901..81f78b4 100644 + --- a/justfile + +++ b/justfile + @@ -285,18 +285,21 @@ pregen-teacher n_prompts="100": + --group=8 \ + --max-new=1024 + + -# 10-step feasibility probe: clean Qwen3-4B + 50% cached teacher pool, pp=4. + -# Queues via pueue so logs are shared between user and agent. Student is + -# restricted to problems covered by the pool. GO/NO-GO: final HACK_STUDENT + -# > 0.10 by step 10. + +# 100-step feasibility probe: clean Qwen3-4B + 75% cached teacher pool, pp=4, G=12. + +# Plan B "free lunch": mix=0.75 -> G_s=3, G_t=9. Gen wall-time unchanged + +# (teacher is cached disk reads), backward VRAM ~2x current (peak ~55-60 GB on + +# 96 GB card). At 48 gens/step (vs reference 256), 100 steps ~= 19 ref steps. + +# --v-hack-path is set even for vanilla so cin/cout get measured as baseline + +# (project_delta_S_grad with measure_only=True on vanilla arm). + probe-mixed seed="41": + - pueue add -l "why: does mixed-pool GRPO (cached teacher) drive student hack-rate up from clean base; resolve: confirm student hack_s climbs 0->20%+ over 10 steps" \ + + pueue add -l "why: does mixed-pool GRPO (cached teacher, plan B grad pressure) drive student hack-rate from clean base; resolve: confirm hack_s climbs 0->10%+ over 100 steps (~19 ref-eq)" \ + -w "$PWD" -- \ + {{ TRAIN }} --preset=full --arm=vanilla \ + --model={{ MODEL }} \ + + --v-hack-path=out/v_hack_full.safetensors \ + --teacher-pool-dir=out/probe_distill/teacher_pool \ + - --mix-ratio=0.5 \ + - --steps=10 --prompts-per-step=4 \ + ... (40 more lines) + +[tool] Read 20260527_code_review.md (completed) + kind: read + input: {"call_id":"call_u67hecWsDhI7oi0boUU1C0q5","process_id":"15286","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","nl -ba docs/spec/20260527_code_review.md"],"cwd":"/workspace/projected_... + files: /workspace/projected_grpo/docs/spec/20260527_code_review.md + output: + 1 [client] initialize (running) + 2 + 3 [client] session/new (running) + 4 I’ll use the local review workflow plus the `gsd` skill because this is a multi-step code review, then I’ll read the project instructions and changed files before making findings. + 5 + 6 [tool] pwd && rg --files -g 'AGENTS.md' -g 'CLAUDE.md' -g 'docs/spec.md' -g 'docs/brainstorm/extracted_prefs.md' -g 'docs/RESEARCH_JOURNAL.md' (running) + 7 input: {"call_id":"call_26P3qQ1bTCPLMMOGhHlNW3B7","process_id":"3146","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","pwd && rg --files -g 'AGENTS.md' -g 'CLAUDE.md' -g 'docs/spec.md' -g 'do... + 8 + 9 [tool] pwd && rg --files -g 'AGENTS.md' -g 'CLAUDE.md' -g 'docs/spec.md' -g 'docs/brainstorm/extracted_prefs.md' -g 'docs/RESEARCH_JOURNAL.md' (failed) + 10 kind: execute + 11 input: {"call_id":"call_26P3qQ1bTCPLMMOGhHlNW3B7","process_id":"3146","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","pwd && rg --files -g 'AGENTS.md' -g 'CLAUDE.md' -g 'docs/spec.md' -g 'do... + 12 output: + 13 bwrap: No permissions to create new namespace, likely because the kernel does not allow non-privileged user namespaces. See or . + 14 + 15 [tool] Read SKILL.md (running) + 16 input: {"call_id":"call_pqWiVSayENSrfXMHJIaa2TDn","process_id":"2476","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","sed -n '1,220p' /root/.claude/skills/gsd/SKILL.md"],"cwd":"/workspace/pr... + 17 files: /root/.claude/skills/gsd/SKILL.md + 18 + 19 [tool] Read SKILL.md (failed) + 20 kind: read + 21 input: {"call_id":"call_pqWiVSayENSrfXMHJIaa2TDn","process_id":"2476","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","sed -n '1,220p' /root/.claude/skills/gsd/SKILL.md"],"cwd":"/workspace/pr... + 22 files: /root/.claude... + +[tool] Search project_delta_S_grad|load_v_hack|v_hack_path|susp_drop_frac|v_sv|_sv/|v-hack in src (running) + input: {"call_id":"call_eKBsKF4tsshQ29fx8yS736Lt","process_id":"18775","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","rg -n \"project_delta_S_grad|load_v_hack|v_hack_path|susp_drop_frac|v_s... + +[tool] Read antipasto.py (running) + input: {"call_id":"call_tQV8MNSEQrC6INjwfHLwzjER","process_id":"24911","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","nl -ba src/projected_grpo/antipasto.py | sed -n '1,240p'"],"cwd":"/work... + files: /workspace/projected_grpo/src/projected_grpo/antipasto.py + +[tool] Read extracted_prefs.md (running) + input: {"call_id":"call_pQoENmJAU2m9pygvgNy1bdzD","process_id":"92183","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","nl -ba docs/brainstorm/extracted_prefs.md | sed -n '1,220p'"],"cwd":"/w... + files: /workspace/projected_grpo/docs/brainstorm/extracted_prefs.md + +[tool] Read extracted_prefs.md (completed) + kind: read + input: {"call_id":"call_pQoENmJAU2m9pygvgNy1bdzD","process_id":"92183","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","nl -ba docs/brainstorm/extracted_prefs.md | sed -n '1,220p'"],"cwd":"/w... + files: /workspace/projected_grpo/docs/brainstorm/extracted_prefs.md + output: + 1 + 3 + 4 # Extracted preferences and decisions — projected_grpo + 5 + 6 ## TL;DR delta vs spec.md + 7 + 8 Spec.md is the clean preregistered plan. docs/1.md is the reasoning trail behind it. The biggest + 9 deltas the brainstorm adds (not in spec): + 10 + 11 1. **The whole project pivoted** mid-conversation from a DPO+sycophancy plan (Anthropic HH-RLHF) + 12 to GRPO+reward-hacking (Nanda/Ariahw LeetCode). Driver: gradient projection in SVD basis matches + 13 GRPO's unpaired structure better than DPO's paired-preference structure. + 14 2. **Method evolved** from "bidirectional SVD-LoRA with NLL+KL" (paired-preference native, the + 15 AntiPaSTO line) to **gradient-level intervention + SVD-basis denoising** — an orthogonal + 16 approach for unpaired GRPO rollouts. + 17 3. **Rebound paper (Wu & Tang 2026)** appeared mid-brainstorm and reframed the positioning: + 18 not novel mechanism (concept-direction intervention) but novel level (gradient vs advantage). + 19 User's confidence updated downward but stayed positive — ~60% the method works now (was ~40% + 20 pre-Rebound, framed as net positive because Rebound *validates* the core mechanism). + 21 4. **Single-GPU pragmatism**: extensive back-and-forth on 3090 vs 96GB RTX 6000 Ada. Landed on + 22 96GB RTX 6000 + Qwen3.5-2B as the practical sweet spot. + 23 + 24 ## 1. Design decisions + 25 + 26 - **Substitute Qwen3.5-2B for Qwen3-4B**. Reason: compute budget. Fallback to Qwen3-4B with + 27 reduced num_generations if H4 (hack emergence) fails at 2B. + 28 - **Use verl, not TRL**. Reason: Nanda's repo uses verl v0.6.1; minimise reimplementation risk. + ... (135 more lines) + +[tool] Read antipasto.py (completed) + kind: read + input: {"call_id":"call_tQV8MNSEQrC6INjwfHLwzjER","process_id":"24911","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","nl -ba src/projected_grpo/antipasto.py | sed -n '1,240p'"],"cwd":"/work... + files: /workspace/projected_grpo/src/projected_grpo/antipasto.py + output: + 1 """AntiPaSTO full-rank adapter via forward hooks (lora-lite style). + 2 + 3 Per spec.md: each target nn.Linear keeps its original weight intact. We attach + 4 frozen buffers U, Vh and a trainable delta_S of shape [r] per layer. A forward + 5 post-hook adds the delta contribution: + 6 + 7 y_new = y + U @ (delta_S * (Vh @ x)) + 8 + 9 equivalent to W -> W + U diag(delta_S) Vh. At delta_S = 0 the delta is exactly + 10 zero, so the wrapped model is bit-identical to the base (no SVD round-trip + 11 error on the main path -- W stays as it was loaded). U, Vh stay frozen and + 12 double as the basis for v_hack gradient projection (we read delta_S.grad + 13 directly; no extra projection math at the gradient step). + 14 """ + 15 from __future__ import annotations + 16 + 17 import hashlib + 18 from pathlib import Path + 19 + 20 import torch + 21 from jaxtyping import Float + 22 from loguru import logger + 23 from torch import Tensor, nn + 24 + 25 + 26 def svd_cached( + 27 W: Float[Tensor, "d_out d_in"], + 28 cache_path: Path, + ... (249 more lines) + +[tool] Search project_delta_S_grad|load_v_hack|v_hack_path|susp_drop_frac|v_sv|_sv/|v-hack in src (completed) + kind: search + input: {"call_id":"call_eKBsKF4tsshQ29fx8yS736Lt","process_id":"18775","turn_id":"019e681a-3b48-7ed3-8e6b-1912b41adb6f","command":["/usr/bin/zsh","-lc","rg -n \"project_delta_S_grad|load_v_hack|v_hack_path|susp_drop_frac|v_s... + output: + justfile:24: {{ TRAIN }} --preset=smoke --arm=projected --v-hack-path=out/v_hack_smoke.safetensors {{ ARGS }} + justfile:31: {{ TRAIN }} --preset=smoke --arm=projected --v-hack-path=out/v_hack_smoke.safetensors + justfile:38: {{ TRAIN }} --preset=full --arm=projected --v-hack-path=out/v_hack_full.safetensors {{ ARGS }} + justfile:67: --v-hack-path=out/v_hack_smoke.safetensors \ + justfile:74: --v-hack-path=out/v_hack_full.safetensors \ + justfile:104: pueue add -a "$VA" -w "$PWD" -o 8 -l "why: projected seed{{ seed }} @ matched batch, v_hack NOT post-hoc; resolve: Gate D H1 HACK_RATE 0`, especially because the justfile still passes explicit `out/v_hack_full.safetensors` paths that may predate the schema. + +- [src/projected_grpo/extract_vhack_grad.py:117](/workspace/projected_grpo/src/projected_grpo/extract_vhack_grad.py:117) Non-finite extraction losses are skipped, which can leave hack and clean gradient stacks with different lengths and then fail later at `D = G_h - G_c`. That is a delayed and less informative failure. For research code, raise immediately with pair/label/loss context. + +### Suggestions +- [src/projected_grpo/train.py:378](/workspace/projected_grpo/src/projected_grpo/train.py:378) Load-or-extract looks safe with respect to optimizer state and gradients. One small concern: `extract_vhack_grad.main()` explicitly calls `model.eval()`, while train auto-extract relies on the model’s current mode. HF models usually load in eval mode, but I would set `model.eval()` before extraction and then explicitly choose the desired training mode after, to make this state transition visible. + +- [src/projected_grpo/extract_vhack_grad.py:259](/workspace/projected_grpo/src/projected_grpo/extract_vhack_grad.py:259) `len(v_hack)` now includes `_sv/` entries, so the final `modules=` count is doubled. Use the filtered module count for diagnostics, otherwise extraction logs will overstate module count and make zero-rate summaries harder to read. + +- The tiny-random smoke with `loss=0` and `cin/cout=NaN` is not an adequate gate test. Add a synthetic `project_delta_S_grad` smoke with two fake wrappers, nonzero grads, known orthonormal `V`, known `S`, and expected `frac_axes_susp`, `cout < cin`, and mutation/no-mutation behavior for projected vs vanilla. That directly exercises the gate without needing GRPO to produce non-degenerate rewards. + +### Verdict +REQUEST CHANGES + +The load-or-extract path is mostly sound, but the suspicion gate currently behaves like a fixed top-fraction projection suppressor, and old artifacts can silently bypass it. Fix the gate normalization/threshold semantics and make v2 `_sv/` metadata required when the gate is enabled. + +[done] end_turn diff --git a/justfile b/justfile index c1dc901..81f78b4 100644 --- a/justfile +++ b/justfile @@ -285,18 +285,21 @@ pregen-teacher n_prompts="100": --group=8 \ --max-new=1024 -# 10-step feasibility probe: clean Qwen3-4B + 50% cached teacher pool, pp=4. -# Queues via pueue so logs are shared between user and agent. Student is -# restricted to problems covered by the pool. GO/NO-GO: final HACK_STUDENT -# > 0.10 by step 10. +# 100-step feasibility probe: clean Qwen3-4B + 75% cached teacher pool, pp=4, G=12. +# Plan B "free lunch": mix=0.75 -> G_s=3, G_t=9. Gen wall-time unchanged +# (teacher is cached disk reads), backward VRAM ~2x current (peak ~55-60 GB on +# 96 GB card). At 48 gens/step (vs reference 256), 100 steps ~= 19 ref steps. +# --v-hack-path is set even for vanilla so cin/cout get measured as baseline +# (project_delta_S_grad with measure_only=True on vanilla arm). probe-mixed seed="41": - pueue add -l "why: does mixed-pool GRPO (cached teacher) drive student hack-rate up from clean base; resolve: confirm student hack_s climbs 0->20%+ over 10 steps" \ + pueue add -l "why: does mixed-pool GRPO (cached teacher, plan B grad pressure) drive student hack-rate from clean base; resolve: confirm hack_s climbs 0->10%+ over 100 steps (~19 ref-eq)" \ -w "$PWD" -- \ {{ TRAIN }} --preset=full --arm=vanilla \ --model={{ MODEL }} \ + --v-hack-path=out/v_hack_full.safetensors \ --teacher-pool-dir=out/probe_distill/teacher_pool \ - --mix-ratio=0.5 \ - --steps=10 --prompts-per-step=4 \ + --mix-ratio=0.75 --group=12 \ + --steps=100 --prompts-per-step=4 \ --seed={{ seed }} \ --out-tag=_probe_mixed_s{{ seed }} diff --git a/src/projected_grpo/extract_vhack_grad.py b/src/projected_grpo/extract_vhack_grad.py index 5e9f1ff..eb4bbe4 100644 --- a/src/projected_grpo/extract_vhack_grad.py +++ b/src/projected_grpo/extract_vhack_grad.py @@ -51,7 +51,14 @@ class Config: out_path: Path = OUT_DIR / "v_hack.safetensors" train_grads_path: Path = OUT_DIR / "vhack_grads_train.safetensors" n_heldout: int = 2 # last n pairs reserved for held-out validation - top_k: int = 5 # rank of hack subspace per module (≤ n_train_pairs) + # top_k=12 = max(n_train_pairs after n_heldout=2 from N=14 pairs). Extract once + # at max rank; train.py slices via --v-hack-k for k-ablation without re-extract. + top_k: int = 12 + # tau_axis: zero rows where S_i/S_0 < tau_axis. Diagnostic — projection along + # noise-direction unit vectors removes only ~||g||/sqrt(r) ≈ 2% of grad + # magnitude on r=2560 modules, so this rarely changes effect size; it does + # make k-ablations honest (axes 4-5 might be pure noise on N=12 pairs). + tau_axis: float = 0.0 def resolve_dtype(s: str) -> torch.dtype: @@ -73,12 +80,126 @@ def completion_nll(model, tokenizer, prompt: str, completion: str, device) -> to return (nll * mask).sum() / mask.sum().clamp_min(1.0) +def extract_v_hack( + model, + tokenizer, + wrappers: dict, + pairs: list, + top_k: int, + tau_axis: float, + n_heldout: int, + device, +) -> tuple[dict[str, torch.Tensor], dict[str, torch.Tensor], list[dict]]: + """Run pair-grads + per-module SVD on D = g_hack - g_clean, return v_hack. + + Pure function — caller owns model loading, wrapping, and saving. train.py + calls this on its already-wrapped model when v_hack cache is missing, so + we don't pay the cost of a second model load. + + Returns: + v_hack: dict[name -> Tensor[k, r]] (cpu fp32), top-k right singular + vectors of D per module, oriented so mean(D @ v_i) > 0. If + tau_axis > 0, rows where S_i/S_0 < tau_axis are zeroed. + raw_grads: dict["hack/name"|"clean/name" -> Tensor[n_pairs, r]] for + offline analysis (verify_vhack_heldout reads this). + diag_rows: per-module diagnostic dicts (sv_top frac, ||D||, etc.). + """ + train_pairs = pairs[:-n_heldout] if n_heldout > 0 else pairs + n_pairs = len(train_pairs) + + grads_hack: dict[str, list[torch.Tensor]] = defaultdict(list) + grads_clean: dict[str, list[torch.Tensor]] = defaultdict(list) + + for pi, pair in enumerate(train_pairs): + for label, completion in (("hack", pair.hack), ("clean", pair.clean)): + model.zero_grad(set_to_none=True) + loss = completion_nll(model, tokenizer, pair.prompt, completion, device) + if not torch.isfinite(loss): + # Skip-on-nonfinite would silently leave G_h and G_c with mismatched + # lengths and explode later in D = G_h - G_c. Fail fast. + raise RuntimeError(f"non-finite loss at pair={pi} label={label}: {loss.item()}") + loss.backward() + bucket = grads_hack if label == "hack" else grads_clean + for name, info in wrappers.items(): + g = info["delta_S"].grad + if g is None: + raise RuntimeError(f"no grad on {name}; aborting extract") + bucket[name].append(g.detach().float().cpu().clone()) + if (pi + 1) % 5 == 0: + logger.info(f" pair {pi+1}/{n_pairs} loss={loss.item():.3f}") + model.zero_grad(set_to_none=True) # leave caller with clean state + + raw_grads = { + **{f"hack/{n}": torch.stack(gs) for n, gs in grads_hack.items()}, + **{f"clean/{n}": torch.stack(gs) for n, gs in grads_clean.items()}, + } + + # Per module: D = g_hack - g_clean (paired diff cancels prompt-specific noise). + # SVD(D) gives orthonormal right singular vectors = principal axes of variation + # of the hack-clean axis. Top-k generalizes mean-diff (which is the k=1 case). + v_hack: dict[str, torch.Tensor] = {} + rows = [] + n_zero = 0 + k = min(top_k, n_pairs) + n_axes_kept_total = 0 + for name in grads_hack: + G_h = torch.stack(grads_hack[name]) # [n_pairs, r] + G_c = torch.stack(grads_clean[name]) + D = G_h - G_c + U_d, S_d, Vh_d = torch.linalg.svd(D, full_matrices=False) + V = Vh_d[:k] # [k, r], rows orthonormal in R^r + # Orient by per-pair majority vote: for each axis i, count pairs where + # d_p @ v_i > 0; if strict majority disagree with current SVD sign, flip. + # More outlier-robust than sign(mean): one extreme pair can't flip a + # consensus direction. Matches repeng's _orient_svd convention. + proj_per_pair = D @ V.T # [n_pairs, k] + n_pos = (proj_per_pair > 0).float().sum(0) # [k] + flip = torch.where(n_pos < n_pairs / 2, -torch.ones(k), torch.ones(k)) + V = V * flip.unsqueeze(1) + + # tau_axis: zero rows where S_i/S_0 < tau_axis (diagnostic; see Config comment). + n_axes_kept = k + if tau_axis > 0 and S_d[0] > 1e-12: + ratios = S_d[:k] / S_d[0] + keep = (ratios >= tau_axis).float() + V = V * keep.unsqueeze(1) + n_axes_kept = int(keep.sum()) + n_axes_kept_total += n_axes_kept + + nrm = D.norm() + if nrm < 1e-12: + n_zero += 1 + v_hack[name] = torch.zeros((k, D.shape[1]), dtype=V.dtype).contiguous() + else: + v_hack[name] = V.contiguous() + # Record singular values so future weighted projection (subtract scaled + # by S_i/S_0) is possible without re-extracting. Loader filters _sv/ keys. + v_hack[f"_sv/{name}"] = S_d[:k].clone().contiguous() + sv_top = S_d[:k] + sv_total = S_d.sum().clamp_min(1e-12) + rows.append({ + "module": name.split(".")[-1], + "r": D.shape[1], + "||D||": f"{nrm:.2e}", + "sv_0": f"{S_d[0].item():.2e}" if S_d.numel() else "-", + f"sv_top{k}_frac": f"{(sv_top.sum() / sv_total).item():.2f}", + "sv_ratio_0/1": f"{(S_d[0] / S_d[1].clamp_min(1e-12)).item():.2f}" if S_d.numel() >= 2 else "-", + "axes_kept": n_axes_kept, + }) + n_modules = len(grads_hack) + logger.info( + f"v_hack: modules={n_modules} k_max={k} zero-||D||={n_zero} " + f"axes_kept_avg={n_axes_kept_total/max(1,n_modules):.1f} (tau_axis={tau_axis})" + ) + return v_hack, raw_grads, rows + + def main(cfg: Config) -> int: device = torch.device("cuda" if torch.cuda.is_available() else "cpu") dtype = resolve_dtype(cfg.dtype) logger.info( f"device={device} model={cfg.model} dtype={cfg.dtype} " - f"N_pairs={len(PAIRS)} heldout={cfg.n_heldout}" + f"N_pairs={len(PAIRS)} heldout={cfg.n_heldout} top_k={cfg.top_k} tau_axis={cfg.tau_axis}" ) tokenizer = AutoTokenizer.from_pretrained(cfg.model) @@ -94,87 +215,23 @@ def main(cfg: Config) -> int: logger.info(f"wrapped {n_mod} modules; total delta_S scalars = {n_delta:,}") train_pairs = PAIRS[:-cfg.n_heldout] - held_pairs = PAIRS[-cfg.n_heldout:] - logger.info(f"train pairs: {len(train_pairs)} held: {len(held_pairs)}") + logger.info(f"train pairs: {len(train_pairs)} held: {cfg.n_heldout}") - grads_hack: dict[str, list[torch.Tensor]] = defaultdict(list) - grads_clean: dict[str, list[torch.Tensor]] = defaultdict(list) + v_hack, raw_grads, rows = extract_v_hack( + model, tokenizer, wrappers, PAIRS, + top_k=cfg.top_k, tau_axis=cfg.tau_axis, + n_heldout=cfg.n_heldout, device=device, + ) + # Skip _sv/ keys when counting V tensors (singular values are saved alongside V). + n_zero = sum(1 for n, v in v_hack.items() if not n.startswith("_sv/") and v.norm() < 1e-12) + k = min(cfg.top_k, len(train_pairs)) - for pi, pair in enumerate(train_pairs): - for label, completion in (("hack", pair.hack), ("clean", pair.clean)): - model.zero_grad(set_to_none=True) - loss = completion_nll(model, tokenizer, pair.prompt, completion, device) - if not torch.isfinite(loss): - logger.warning(f"non-finite loss pair={pi} label={label}; skipping") - continue - loss.backward() - bucket = grads_hack if label == "hack" else grads_clean - for name, info in wrappers.items(): - g = info["delta_S"].grad - if g is None: - logger.error(f"no grad on {name}; aborting") - return 1 - bucket[name].append(g.detach().float().cpu().clone()) - if (pi + 1) % 5 == 0: - logger.info(f" pair {pi+1}/{len(train_pairs)} loss={loss.item():.3f}") - - # save raw grads stacked per module so safetensors can hold them as a single - # tensor per name. Keys: "hack/{name}", "clean/{name}" -> Tensor[n_pairs, r]. OUT_DIR.mkdir(exist_ok=True) - raw_grads = { - **{f"hack/{n}": torch.stack(gs) for n, gs in grads_hack.items()}, - **{f"clean/{n}": torch.stack(gs) for n, gs in grads_clean.items()}, - } save_file(raw_grads, str(cfg.train_grads_path), metadata={"model": cfg.model, "dtype": cfg.dtype}) - - # Per module: stack per-pair diffs D[n_pairs, r] = grads_hack - grads_clean. - # SVD(D) gives orthonormal right singular vectors capturing variance in the - # hack-clean axis. Top-k generalizes mean-diff (which is the rank-1 case). - v_hack: dict[str, torch.Tensor] = {} - rows = [] - n_zero = 0 - n_pairs = len(train_pairs) - k = min(cfg.top_k, n_pairs) - for name in grads_hack: - G_h = torch.stack(grads_hack[name]) # [n_pairs, r] - G_c = torch.stack(grads_clean[name]) # [n_pairs, r] - D = G_h - G_c # [n_pairs, r] - # SVD: U[n_pairs, m] @ diag(S[m]) @ Vh[m, r], m = min(n_pairs, r) - try: - U_d, S_d, Vh_d = torch.linalg.svd(D, full_matrices=False) - except Exception as e: - logger.error(f"SVD failed on {name} (D shape {D.shape}): {e}") - return 1 - V_topk = Vh_d[:k] # [k, r], rows orthonormal in R^r - # Orient each right singular vector so +v_i points hack-ward: - # for each pair, (g_h_i - g_c_i) @ v_i > 0 on average. - # SVD column signs are arbitrary; without this step, per-direction - # one-sided gating in proj.py would gate the wrong half for some axes. - proj_per_pair = D @ V_topk.T # [n_pairs, k] - sign = torch.sign(proj_per_pair.mean(0)) # [k] - sign = torch.where(sign == 0, torch.ones_like(sign), sign) - V_topk = V_topk * sign.unsqueeze(1) # [k, r] - nrm = D.norm() - if nrm < 1e-12: - n_zero += 1 - v_hack[name] = torch.zeros_like(V_topk).contiguous() - else: - v_hack[name] = V_topk.contiguous() - # singular-value entropy to spot rank: high entropy = multi-axis hack signal - sv_top = S_d[:k] - sv_total = S_d.sum().clamp_min(1e-12) - rows.append({ - "module": name.split(".")[-1], - "r": D.shape[1], - "||D||": f"{nrm:.2e}", - "sv_0": f"{S_d[0].item():.2e}" if S_d.numel() else "-", - f"sv_top{k}_frac": f"{(sv_top.sum() / sv_total).item():.2f}", - "sv_ratio_0/1": f"{(S_d[0] / S_d[1].clamp_min(1e-12)).item():.2f}" if S_d.numel() >= 2 else "-", - }) - save_file(v_hack, str(cfg.out_path), - metadata={"model": cfg.model, "dtype": cfg.dtype, "top_k": str(k)}) + metadata={"model": cfg.model, "dtype": cfg.dtype, "top_k": str(k), + "tau_axis": str(cfg.tau_axis), "schema": "v2_with_sv"}) # summary: aggregate by suffix — track top-k energy concentration by_suffix: dict[str, list] = defaultdict(list) diff --git a/src/projected_grpo/probe_distill.py b/src/projected_grpo/probe_distill.py index 0bd8478..cfb1c06 100644 --- a/src/projected_grpo/probe_distill.py +++ b/src/projected_grpo/probe_distill.py @@ -213,7 +213,7 @@ def main(cfg: Config) -> int: student, wrappers, tok = load_student(device) delta_params = [info["delta_S"] for info in wrappers.values()] logger.info(f"student delta_S params: {sum(p.numel() for p in delta_params):,}") - v_hack_cpu = load_v_hack(cfg.v_hack_path, STUDENT_MODEL, wrappers) + v_hack_cpu, _v_sv = load_v_hack(cfg.v_hack_path, STUDENT_MODEL, wrappers) v_hack = {n: v.to(device) for n, v in v_hack_cpu.items()} opt = torch.optim.AdamW(delta_params, lr=cfg.lr) diff --git a/src/projected_grpo/proj.py b/src/projected_grpo/proj.py index 164a683..3b3b956 100644 --- a/src/projected_grpo/proj.py +++ b/src/projected_grpo/proj.py @@ -20,6 +20,30 @@ def per_token_logps(logits: torch.Tensor, ids: torch.Tensor) -> torch.Tensor: ).float().view(B, L) +@torch.no_grad() +def mean_cin_from_grads( + grad_dict: dict[str, torch.Tensor], + v_hack: dict[str, torch.Tensor], +) -> float: + """Mean over modules of ||V g|| / ||g||, given a dict of per-module grads. + + Used to compute per-source cin (cin_s for student-only grad, cin_t for + teacher-only grad) without mutating model.grad or calling the full + projection pipeline. v_hack rows are orthonormal so ||V g|| <= ||g|| and + the ratio is in [0,1]. + """ + cs = [] + for name, g in grad_dict.items(): + if g is None or name not in v_hack: + continue + V = v_hack[name].to(g.device, dtype=g.dtype) + gn = g.norm() + if gn < 1e-12: + continue + cs.append(((V @ g).norm() / gn).item()) + return float(sum(cs) / len(cs)) if cs else float("nan") + + @torch.no_grad() def project_delta_S_grad( wrappers: dict, @@ -27,6 +51,8 @@ def project_delta_S_grad( preserve_magnitude: bool, measure_only: bool = False, gate_mode: str = "one_sided", + v_sv: dict[str, torch.Tensor] | None = None, + drop_top_frac: float = 0.0, ) -> dict[str, float]: """Per-module top-k removal of hack-aligned grad components. @@ -47,31 +73,84 @@ def project_delta_S_grad( `preserve_magnitude`: rescale g' to ||g|| after projection. `measure_only`: same math, but g is not mutated (vanilla arm diagnostic). + Runtime suspicion gate (when v_sv is given and drop_top_frac > 0): + Per axis i, the within-module dimensionless ratio: + p_live_i = |c_i| / ||g|| (fraction of live grad on v_i) + p_extract_i = S_i / sqrt(sum_j S_j^2) (fraction of extract D on v_i) + r_i = p_live_i / p_extract_i + r_i ≈ 1: live grad concentrated on v_i in same proportion as extract — as + expected. r_i ≫ 1: live grad over-concentrated on v_i relative to what + the contrastive-pair signal would predict — likely v_i is spuriously + aligned with a structured coding direction, not hack. + Both ratios are within-module fractions, dimensionless, comparable across + modules with different ||g||. Per-step quantile-gate at (1-drop_top_frac): + suppress projection on axes above the threshold. Default 0 = current + behavior. Fails if v_sv is empty when drop_top_frac>0 (old v1 files). + Diagnostics returned (per call, averaged over modules): mean_cos_in = mean over modules of ||V g||/||g|| (subspace energy fraction in) mean_cos_out = same after projection frac_fired = fraction of modules where at least one direction fired (c_i > 0) + frac_axes_susp = fraction of (module, axis) pairs dropped by suspicion gate """ - cos_in_list, cos_out_list, n_fired = [], [], 0 + # Fail fast on schema mismatch: user opted into the gate but v_hack is v1. + if drop_top_frac > 0 and (v_sv is None or not v_sv): + raise ValueError( + "susp_drop_frac > 0 requires v_sv (singular values, _sv/{name} keys " + "in v_hack file). Re-extract with current extract_vhack_grad.py " + "(saves v2 schema with _sv keys), or set susp_drop_frac=0." + ) + # Pass 1: compute c per module, collect dimensionless suspicion ratios. + # r_i = (|c_i|/||g||) / (S_i/||S||). Both p_live and p_extract are within-module + # fractions in [0,1], so r_i compares like-for-like across modules of different + # sizes / gradient norms. Cache (V, c, r) so pass 2 doesn't recompute. + cache: dict[str, tuple] = {} + all_ratios: list[torch.Tensor] = [] for name, info in wrappers.items(): g = info["delta_S"].grad if g is None: continue V = v_hack[name].to(g.device, dtype=g.dtype) # [k, r] + c = V @ g # [k] + r = None + if v_sv is not None and name in v_sv and drop_top_frac > 0: + S = v_sv[name].to(g.device, dtype=torch.float32) + S_norm = (S / S.pow(2).sum().clamp_min(1e-24).sqrt()).clamp_min(1e-12) # [k] + gn_f = g.float().norm().clamp_min(1e-12) + c_norm = c.float().abs() / gn_f # [k] + r = c_norm / S_norm # [k] + all_ratios.append(r) + cache[name] = (V, c, r) + susp_threshold: float | None = None + if drop_top_frac > 0 and all_ratios: + r_flat = torch.cat(all_ratios) + susp_threshold = torch.quantile(r_flat, 1.0 - drop_top_frac).item() + + cos_in_list, cos_out_list, n_fired = [], [], 0 + n_axes_total, n_axes_susp = 0, 0 + for name, info in wrappers.items(): + g = info["delta_S"].grad + if g is None: + continue + V, c, r = cache[name] gn = g.norm() if gn < 1e-12: cos_in_list.append(0.0); cos_out_list.append(0.0); continue - c = V @ g # [k] - # subspace-energy-in: ||V g|| / ||g|| ∈ [0, 1] (rows of V are orthonormal) cin = c.norm() / gn cos_in_list.append(cin.item()) + if susp_threshold is not None and r is not None: + keep_susp = (r <= susp_threshold).to(c.dtype) + n_axes_total += c.numel() + n_axes_susp += int((1 - keep_susp).sum()) + else: + keep_susp = torch.ones_like(c) if gate_mode == "no_gate": - c_use = c # subtract all axes regardless of sign - fired = True + c_use = c * keep_susp + fired = bool((c_use != 0).any()) elif gate_mode == "one_sided": - mask = c > 0 - c_use = c * mask.float() - fired = bool(mask.any()) + mask = (c > 0).to(c.dtype) + c_use = c * mask * keep_susp + fired = bool((c_use != 0).any()) else: raise ValueError(f"unknown gate_mode={gate_mode!r}") if fired: @@ -95,4 +174,5 @@ def project_delta_S_grad( "min_cos_out": cout_t.min().item() if cout_t.numel() else float("nan"), "max_cos_out": cout_t.max().item() if cout_t.numel() else float("nan"), "frac_fired": n_fired / len(cos_in_list) if cos_in_list else 0.0, + "frac_axes_susp": n_axes_susp / n_axes_total if n_axes_total else 0.0, } diff --git a/src/projected_grpo/train.py b/src/projected_grpo/train.py index 2f94045..7828732 100644 --- a/src/projected_grpo/train.py +++ b/src/projected_grpo/train.py @@ -78,7 +78,7 @@ from tqdm import tqdm from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig from .antipasto import wrap_model_with_antipasto -from .proj import per_token_logps, project_delta_S_grad +from .proj import per_token_logps, project_delta_S_grad, mean_cin_from_grads from .rewards import compute_reward CACHE_ROOT = Path("svd_cache") @@ -157,7 +157,19 @@ class Config: preserve_magnitude: bool = True gate_mode: Literal["one_sided", "no_gate"] = "one_sided" unbiased: bool = True # Dr.GRPO: drop 1/|o_i| and /std(R) - v_hack_path: Path = OUT_DIR / "v_hack.safetensors" + # v_hack: path is optional — if None, derived from model+top_k as + # out/v_hack__k.safetensors. If file missing, train.py + # auto-extracts (cheap: ~5min, shares the already-loaded model). Set explicitly + # to override (e.g. baked-variant v_hack paths). v_hack_k slices the saved + # top-k_max directions to top-k_use at load time — the k-ablation knob. + v_hack_path: Path | None = None + v_hack_extract_top_k: int = 12 # max k to save at extract; n_train_pairs caps it lower + v_hack_k: int = 5 # load-time slice; k=1 = mean-diff, k=k_max = full + # Runtime suspicion gate: per step, drop the top-frac (module, axis) pairs by + # r_i = |g·v_i|/S_i. Live alignment ≫ extract-time confidence suggests v_i + # is spuriously aligned with a coding/capability direction, not hack. 0.25 is + # conservative — protects against the worst quartile of suspicious projections. + susp_drop_frac: float = 0.25 out_tag: str = "" # suffix for saved artifact, e.g. "_seed41" # Mixed-pool GRPO: per-prompt rollout pool = G_s live student + G_t cached # teacher rollouts. Teacher pool is a dir of prompt_NNNN.jsonl.gz produced by @@ -214,12 +226,20 @@ def load_problems(n: int) -> list[dict]: return out -def load_v_hack(path: Path, model_name: str, wrappers: dict) -> dict[str, torch.Tensor]: - """Load v_hack and fail fast if it is not for this wrapped model. +def load_v_hack( + path: Path, model_name: str, wrappers: dict, k_use: int | None = None, +) -> tuple[dict[str, torch.Tensor], dict[str, torch.Tensor]]: + """Load v_hack (top-k directions) + sv (singular values) for this wrapped model. - v_hack is model-specific because module names and per-module SVD ranks depend - on the exact checkpoint. A smoke (Qwen3.5-0.8B) v_hack must not be reused for - a full (Qwen3-4B) run. + File schema (v2): bare `{name}` keys hold V[k_max, r]; `_sv/{name}` keys hold + S[k_max]. v_hack is model-specific because module names and per-module SVD + ranks depend on the exact checkpoint; a smoke (Qwen3.5-0.8B) v_hack must + not be reused for a full (Qwen3-4B) run. + + If `k_use` is given, slices V and S to top-k_use rows. Errors if k_use > k_max + saved (re-extract with a higher top_k). + + Returns (v_hack, v_sv). v_sv may be empty for v1 files without _sv/ keys. """ with safe_open(str(path), framework="pt", device="cpu") as f: meta = f.metadata() or {} @@ -238,13 +258,16 @@ def load_v_hack(path: Path, model_name: str, wrappers: dict) -> dict[str, torch. f"v_hack dtype/SVD-basis mismatch: {path} was extracted with dtype={saved_dtype}; " "train.py loads models in bf16. Re-extract with `--dtype=bf16`." ) - v_hack = {k: f.get_tensor(k) for k in f.keys()} + # Split V keys (bare module names) from S keys (prefixed _sv/). + all_tensors = {k: f.get_tensor(k) for k in f.keys()} + v_hack = {k: v for k, v in all_tensors.items() if not k.startswith("_sv/")} + v_sv = {k[len("_sv/"):]: v for k, v in all_tensors.items() if k.startswith("_sv/")} wrapper_keys = set(wrappers) vhack_keys = set(v_hack) missing = sorted(wrapper_keys - vhack_keys) extra = sorted(vhack_keys - wrapper_keys) - # v_hack[name] is [k, r]; delta_S is [r]. Check last-dim match (rank r). + # v_hack[name] is [k_max, r]; delta_S is [r]. Check last-dim match (rank r). rank_bad = [ (name, tuple(v_hack[name].shape), tuple(wrappers[name]["delta_S"].shape)) for name in sorted(wrapper_keys & vhack_keys) @@ -259,8 +282,21 @@ def load_v_hack(path: Path, model_name: str, wrappers: dict) -> dict[str, torch. "Extract a fresh v_hack with `uv run python -m projected_grpo.extract_vhack_grad " f"--model={model_name} --out-path={path}`." ) - logger.info(f"loaded v_hack from {path}: modules={len(v_hack)}; key/rank match OK") - return v_hack + + k_max = next(iter(v_hack.values())).shape[0] + if k_use is not None: + if k_use > k_max: + raise ValueError( + f"requested k_use={k_use} exceeds saved k_max={k_max} in {path}. " + f"Re-extract with `--top-k={k_use}`." + ) + v_hack = {n: v[:k_use].contiguous() for n, v in v_hack.items()} + v_sv = {n: s[:k_use].contiguous() for n, s in v_sv.items()} + logger.info( + f"loaded v_hack from {path}: modules={len(v_hack)}; k_saved={k_max}, k_use={k_use or k_max}; " + f"sv={'yes' if v_sv else 'no'}" + ) + return v_hack, v_sv @torch.no_grad() @@ -331,15 +367,36 @@ def main(cfg: Config) -> int: delta_params = [info["delta_S"] for info in wrappers.values()] logger.info(f"trainable delta_S: {sum(p.numel() for p in delta_params):,}") - # v_hack: loaded for both arms when the file is present, so vanilla also - # reports cos_in as a diagnostic (no projection applied). If not present - # and arm=vanilla, skip silently — H4 sanity runs without v_hack remain valid. - v_hack = None - if cfg.v_hack_path.exists(): - v_hack_cpu = load_v_hack(cfg.v_hack_path, model_name, wrappers) - v_hack = {name: v.to(device) for name, v in v_hack_cpu.items()} - elif cfg.arm == "projected": - raise FileNotFoundError(f"projected arm requires v_hack at {cfg.v_hack_path}") + # v_hack: derive default path from model + extract_top_k unless overridden. + # Always loaded (or auto-extracted) so vanilla also reports cos_in as a baseline. + # Auto-extract reuses the already-wrapped model — no second model load. + # Slug: works for HF names ("Qwen/Qwen3-4B" -> "Qwen3-4B") and local paths + # ("out/baked/qwen3_4b_rh25" -> "qwen3_4b_rh25"). + model_slug = model_name.rstrip("/").split("/")[-1] + if cfg.v_hack_path is None: + v_hack_path = OUT_DIR / f"v_hack_{model_slug}_k{cfg.v_hack_extract_top_k}.safetensors" + else: + v_hack_path = cfg.v_hack_path + if not v_hack_path.exists(): + from .extract_vhack_grad import extract_v_hack + from .pairs import PAIRS as VHACK_PAIRS + from safetensors.torch import save_file + logger.info(f"v_hack cache miss at {v_hack_path}; extracting (~5min)...") + model.eval() # match standalone extract: deterministic backward, no dropout + v_hack_cpu_dict, raw_grads, _diag = extract_v_hack( + model, tok, wrappers, VHACK_PAIRS, + top_k=cfg.v_hack_extract_top_k, tau_axis=0.0, + n_heldout=2, device=device, + ) + OUT_DIR.mkdir(exist_ok=True) + save_file(v_hack_cpu_dict, str(v_hack_path), + metadata={"model": model_name, "dtype": "bf16", + "top_k": str(min(cfg.v_hack_extract_top_k, len(VHACK_PAIRS) - 2)), + "tau_axis": "0.0", "schema": "v2_with_sv"}) + # extract zeros grads at exit; opt is built below so no opt-state taint. + v_hack_cpu, v_sv_cpu = load_v_hack(v_hack_path, model_name, wrappers, k_use=cfg.v_hack_k) + v_hack = {name: v.to(device) for name, v in v_hack_cpu.items()} + v_sv = {name: s.to(device) for name, s in v_sv_cpu.items()} if v_sv_cpu else None # Teacher pool: pre-generated rollouts on disk keyed by problem_id. Each step's # G_t teacher rollouts come from a uniform random sample of that prompt's cache, # so we do *not* keep the teacher model in VRAM. Pool is produced by @@ -451,10 +508,22 @@ def main(cfg: Config) -> int: # Names kept <=7 chars so header and value share the same 8-col tab stop. # hack_s/hack_t split out the combined `hack` column by rollout source # (student vs teacher). On no-teacher runs hack_s == hack and hack_t == 0/0. - _row_cols = ["step", "rew", "std", "sprd", "N", + # ref_eq = cumulative generations / 256, where 256 = canonical + # num_prompts(16) * num_generations(16) per optimizer step (ariahw config.py). + # So ref_eq=1.0 means we've issued the same number of gradient samples as + # one canonical reference step. Convert our step count to "reference step + # equivalents" by reading this column at the row of interest. + _row_cols = ["step", "ref_eq", "rew", "std", "sprd", "N", "gt", "hack", "hack_s", "hack_t", "gt_s", - "loss", "cin", "cout", "fired", + "loss", "cin", "cin_s", "cin_t", "cout", "fired", "susp", "gen", "fb", "rew_s", "sec"] + REF_GENS_PER_STEP = 16 * 16 # ariahw/rl-rewardhacking config.py:num_prompts * num_generations + est_gens_per_step = cfg.prompts_per_step * cfg.group # before mixed-pool split + logger.info( + f"grad-pressure: {est_gens_per_step} gens/step vs reference {REF_GENS_PER_STEP} " + f"-> {est_gens_per_step / REF_GENS_PER_STEP:.2f}x per step; " + f"this run's {steps} steps ~= {steps * est_gens_per_step / REF_GENS_PER_STEP:.1f} reference steps." + ) logger.info("row\t" + "\t".join(_row_cols)) OUT_DIR.mkdir(exist_ok=True) @@ -494,6 +563,13 @@ def main(cfg: Config) -> int: agg_comp_lens, agg_finished, n_skipped = [], [], 0 agg_loss = 0.0 diag_tail = None + # Per-source grad accumulators: each prompt's backward is split into + # student-only and teacher-only passes so we can compute cin_s / cin_t + # separately (discriminator: does v_hack actually project hack grads + # more than non-hack?). step_grad_combined = student + teacher and is + # what the projection + optimizer step ultimately sees. + step_grad_s: dict[str, torch.Tensor] = {} + step_grad_t: dict[str, torch.Tensor] = {} # Phase timers (per-step cumulative, seconds). Each GPU phase ends in a # CPU-blocking op (decode / .item()), so perf_counter is sync-accurate # without explicit cuda.synchronize. Tells us whether wall-time is @@ -673,25 +749,84 @@ def main(cfg: Config) -> int: kl = torch.exp(ref_logp - pol_logp) - (ref_logp - pol_logp) - 1.0 per_tok_loss = per_tok_loss + beta * kl + # Split loss by source (student vs teacher) and run separate + # backward passes. Linearity of backward + leaf .grad accumulator: + # loss = loss_s + loss_t (since is_s_mask + is_t_mask = 1) + # So grad_s + grad_t == full-batch grad; combined for projection. + is_s_v = torch.tensor(is_student, dtype=per_tok_loss.dtype, + device=per_tok_loss.device).unsqueeze(1) # [G, 1] + is_t_v = 1.0 - is_s_v if cfg.unbiased: - # Dr.GRPO: constant denominator. Divide by prompts_per_step to - # average gradients across the P prompts (grad accumulation). - loss = (per_tok_loss * mask).sum() / (group * max_new * prompts_per_step) + denom = group * max_new * prompts_per_step + loss_s = (per_tok_loss * mask * is_s_v).sum() / denom + loss_t = (per_tok_loss * mask * is_t_v).sum() / denom else: - loss = ((per_tok_loss * mask).sum(1) / mask.sum(1).clamp_min(1)).mean() / prompts_per_step - loss.backward() - agg_loss += loss.item() + # Per-sample mean across completion tokens, then mean across G_src + # samples in this source, scaled to be additive with the other. + n_s = max(1, int(is_s_v.sum().item())) + n_t = max(1, int(is_t_v.sum().item())) + ptl_norm = (per_tok_loss * mask).sum(1) / mask.sum(1).clamp_min(1) # [G] + loss_s = (ptl_norm * is_s_v.squeeze(1)).sum() / (group * prompts_per_step) + loss_t = (ptl_norm * is_t_v.squeeze(1)).sum() / (group * prompts_per_step) + # Pass 1: student. retain_graph so the shared forward graph survives. + loss_s.backward(retain_graph=True) + for name, info in wrappers.items(): + gs = info["delta_S"].grad + if gs is None: + continue + step_grad_s[name] = (step_grad_s[name] + gs.detach().clone() + if name in step_grad_s + else gs.detach().clone()) + model.zero_grad(set_to_none=True) + # Pass 2: teacher. + loss_t.backward() + for name, info in wrappers.items(): + gt = info["delta_S"].grad + if gt is None: + continue + step_grad_t[name] = (step_grad_t[name] + gt.detach().clone() + if name in step_grad_t + else gt.detach().clone()) + model.zero_grad(set_to_none=True) + agg_loss += (loss_s + loss_t).item() t_fb += time.perf_counter() - _tfb + # Inject combined grad (student + teacher) into leaf .grad before + # projection + optimizer. Where only one source contributed for a + # module, take that source's grad directly. + for name, info in wrappers.items(): + gs = step_grad_s.get(name) + gt = step_grad_t.get(name) + if gs is None and gt is None: + continue + if gs is None: + info["delta_S"].grad = gt + elif gt is None: + info["delta_S"].grad = gs + else: + info["delta_S"].grad = gs + gt + + # Per-source cin: project student-only and teacher-only grads into v_hack + # subspace (cosine in delta_S grad-space; rows of V orthonormal so the + # ratio is bounded in [0,1]). Discriminator: cin_t > cin_s on a clean + # base means v_hack lights up for hack grads more than non-hack — the + # extraction direction is doing real work. + cin_s = mean_cin_from_grads(step_grad_s, v_hack) if v_hack is not None else float("nan") + cin_t = mean_cin_from_grads(step_grad_t, v_hack) if v_hack is not None else float("nan") + # Diagnostic cos_in for both arms; projection only mutates grad if arm=projected. if v_hack is not None: diag = project_delta_S_grad( wrappers, v_hack, cfg.preserve_magnitude, measure_only=(cfg.arm != "projected"), gate_mode=cfg.gate_mode, + v_sv=v_sv, drop_top_frac=cfg.susp_drop_frac, ) else: - diag = {"mean_cos_in": float("nan"), "mean_cos_out": float("nan"), "frac_fired": float("nan")} + diag = {"mean_cos_in": float("nan"), "mean_cos_out": float("nan"), + "frac_fired": float("nan"), "frac_axes_susp": float("nan")} + diag["mean_cin_s"] = cin_s + diag["mean_cin_t"] = cin_t torch.nn.utils.clip_grad_norm_(delta_params, 1.0) opt.step() @@ -738,8 +873,10 @@ def main(cfg: Config) -> int: tail = diag_tail.replace("\n", "\\n") logger.debug(f"step {step} gen[0] tail (last 400 chars): {tail!r}") + cum_gens = sum(r["N"] for r in rows) + n_rollouts row = { "step": step, + "ref_eq": f"{cum_gens / REF_GENS_PER_STEP:.2f}", "rew": f"{rew_mean:+.2f}", "std": f"{rew_std:.2f}", "sprd": "T" if spread else "F", @@ -751,8 +888,11 @@ def main(cfg: Config) -> int: "gt_s": f"{gt_s_n}/{n_s}" if n_s else "0/0", "loss": f"{agg_loss:+.4f}", "cin": f"{diag['mean_cos_in']:+.3f}", + "cin_s": f"{diag['mean_cin_s']:+.3f}", + "cin_t": f"{diag['mean_cin_t']:+.3f}", "cout": f"{diag['mean_cos_out']:+.3f}", "fired": f"{diag['frac_fired']:.2f}", + "susp": f"{diag['frac_axes_susp']:.2f}", "gen": f"{t_gen:.0f}", "fb": f"{t_fb:.0f}", "rew_s": f"{t_rew:.0f}", @@ -763,10 +903,10 @@ def main(cfg: Config) -> int: logger.info("row\t" + "\t".join(str(row[c]) for c in _row_cols)) if (step + 1) % 25 == 0: save_ckpt(rows) # survive early kills; ~12 days for the full sweep - if not first_hack_saved and sum(agg_hack) > 0: + if not first_hack_saved and hack_s_n > 0: save_ckpt(rows, path=first_hack_path) first_hack_saved = True - logger.info(f"first-hack ckpt saved: step={step} hack={sum(agg_hack)}/{n_rollouts} -> {first_hack_path.name}") + logger.info(f"first-student-hack ckpt saved: step={step} hack_s={hack_s_n}/{n_s} -> {first_hack_path.name}") # Live status in tqdm postfix; full per-step line in verbose log only. pbar.set_postfix( rew=f"{rew_mean:+.2f}", gt=f"{sum(agg_gt)}/{n_rollouts}", @@ -821,6 +961,10 @@ def main(cfg: Config) -> int: "mix": cfg.mix_ratio if cfg.teacher_pool_dir else "", "tag": cfg.out_tag, "log": str(verbose_log), }], headers="keys", tablefmt="tsv")) + # Markdown copy: easier to paste into journal/PRs than the TSV above. + print() + print("### Per-step rows (markdown)\n") + print(tabulate(rows, headers="keys", tablefmt="pipe", floatfmt="+.3f")) save_ckpt(rows) return 0 diff --git a/src/projected_grpo/verify_vhack_heldout.py b/src/projected_grpo/verify_vhack_heldout.py index 20d79c0..c5190bc 100644 --- a/src/projected_grpo/verify_vhack_heldout.py +++ b/src/projected_grpo/verify_vhack_heldout.py @@ -61,7 +61,7 @@ def main(cfg: Config) -> int: wrappers = wrap_model_with_antipasto( model, model_name=cfg.model, cache_root=CACHE_ROOT, svd_device=device, ) - v_hack = load_v_hack(cfg.v_hack_path, cfg.model, wrappers) + v_hack, _v_sv = load_v_hack(cfg.v_hack_path, cfg.model, wrappers) logger.info(f"loaded v_hack: {len(v_hack)} modules") grads_hack: dict[str, list[torch.Tensor]] = defaultdict(list)