evil_MoE/docs/extract_vhack_grad-vec.md

# v_hack extraction: gradient-space SVD with magnitudes + runtime suspicion gate

Living design doc for the v_hack pipeline. Sibling to `RESEARCH_JOURNAL.md`.
This explains *what we extract*, *why*, and *what runtime gating prevents*.

## TL;DR

`v_hack[name]` is a per-module top-k orthonormal basis in **AntiPaSTO
δS-gradient space**, computed by PCA on **paired (hack − clean) NLL gradients**
over a small set of contrastive completion pairs (currently N=14, 12 train + 2
heldout). At training time we project the live policy-gradient component along
this basis out of `δS.grad`, optionally gated so we only ablate when there's
positive evidence the live gradient is hack-aligned.

The 2026-05-27 refactor added three things on top of the older mean-diff design:

1. **Top-k extraction** (k=12 max) with **load-time slicing** (`v_hack_k`,
   default 5) so k=1 vs k=5 vs k=12 is a config flip, not a re-extract.
2. **Singular-value recording** (`_sv/{name}` keys) so v_i carries its
   extract-time confidence S_i, not just direction.
3. **Runtime suspicion gate** (`susp_drop_frac`): per step, drop the top-frac
   (module, axis) pairs by `r_i = |g·v_i| / S_i`. Live alignment ≫ extract
   confidence means v_i is probably aligned with a structured coding direction,
   not hack — skip the projection.

## Why gradient space, not activation space?

Most representation-steering work (ActAdd, RepE, CHaRS) operates on
**activations** (forward pass), shifting hidden states at inference. We
operate on **gradients of δS**, the trainable per-Linear AntiPaSTO knob.

Reasons:
- We're not steering inference; we're **shaping training**. The projection
  modifies `δS.grad` before the optimizer step, so the model itself doesn't
  drift toward hack-aligned weight updates.
- δS gradients have a fixed, low-dimensional structure per module
  (`δS ∈ R^r` where r = SVD rank of `W`). PCA-on-grads is computationally
  cheap (12 pairs × N modules; r=2560 for largest mat) and gives a clean
  per-module subspace.
- This is closest in spirit to CHaRS-PCT (Principal Component Thresholding,
  §3.3 of `docs/paper_chars.md`): the L principal components of local-shift
  covariance. We do the same maneuver on paired δS-gradient diffs.

## Why δS basis (= weight-SVD basis), not raw param basis?

AntiPaSTO wraps each Linear with `δW = U · diag(δS) · V_h`, where `U, S, V_h
= SVD(W_pretrained)`. So `δS ∈ R^r` are coordinates **in the weight-SVD
basis**. The basis change is just a rotation — no whitening, no rescaling.

Two things this buys us:

- The number of trainable scalars is r per module (∼500–2500), not d_in×d_out.
  A few hundred contrastive pairs would be needed to estimate dense
  `d_in × d_out` direction; only a few pairs are needed in `R^r`.
- Low-rank perturbations (LoRA-style hack adapters) are sparse in this basis,
  which makes per-direction gating in `δS` meaningful even with N=12 pairs.

What this does **not** buy us: regularization. The weight-SVD basis is just a
convenient coordinate system; PCA on top of it still has to do the work of
finding which coordinates carry hack-clean discriminative signal.

## Extraction pipeline

```python
# pseudo: extract_v_hack(model, tokenizer, wrappers, pairs, top_k, tau_axis, n_heldout, device)

train_pairs = pairs[:-n_heldout]                # currently 12 of 14

# Gather per-pair, per-module gradients on hack-completion and clean-completion NLL.
grads_hack[name]:  list of [r]-tensors, length n_pairs
grads_clean[name]: list of [r]-tensors, length n_pairs

for pair in train_pairs:
  for label, completion in [("hack", pair.hack), ("clean", pair.clean)]:
    model.zero_grad()
    loss = mean_NLL_on_completion_tokens(model, pair.prompt + completion)
    loss.backward()                              # populates δS.grad per module
    for name, info in wrappers.items():
      bucket[name].append(info.delta_S.grad.detach().cpu().float().clone())

# Per module: PCA on paired diff.
for name in wrappers:
  G_h = stack(grads_hack[name])                  # [n_pairs, r]
  G_c = stack(grads_clean[name])
  D   = G_h - G_c                                # [n_pairs, r]: per-pair hack-axis displacement
  U_d, S_d, Vh_d = svd(D)                        # truncated, m = min(n_pairs, r)
  V = Vh_d[:k_max]                               # [k_max, r], orthonormal rows
  # Orient v_i so +v_i points hack-ward (majority vote across pairs).
  proj = D @ V.T                                 # [n_pairs, k_max]
  n_pos = (proj > 0).sum(0)
  flip = where(n_pos < n_pairs/2, -1, +1)
  V = V * flip[:, None]
  v_hack[name] = V
  v_hack[f"_sv/{name}"] = S_d[:k_max]            # NEW: singular values saved alongside
```

**File schema (v2):**
- `{name}` → Tensor[k_max, r], orthonormal hack-axis basis, oriented +hack
- `_sv/{name}` → Tensor[k_max], singular values of D in that basis
- metadata: `model`, `dtype`, `top_k`, `tau_axis`, `schema=v2_with_sv`

## Load-or-extract (2026-05-27)

`train.py` derives `v_hack_path` from `(model_name, v_hack_extract_top_k)`
unless overridden. If the file is missing, it extracts inline on the
already-wrapped model:

```
v_hack_path = OUT_DIR / f"v_hack_{model_slug}_k{extract_top_k}.safetensors"
if not v_hack_path.exists():
    v_hack_dict, raw_grads, _ = extract_v_hack(model, tok, wrappers, PAIRS,
                                                top_k=extract_top_k, ...)
    save_file(v_hack_dict, v_hack_path, metadata={...})
v_hack, v_sv = load_v_hack(v_hack_path, model_name, wrappers, k_use=v_hack_k)
```

This means a fresh model with no cached v_hack just runs extract once
(~5 min for 4B-class) and proceeds. No prerequisite jobs, no manual flags.

## Load-time k-slicing

Extract saves k_max (default 12). Load slices to `k_use` (default 5). So
k=1 vs k=5 vs k=12 is a **config flip**, not a re-extract. The
`mean_sv_top5_frac` from our 2026-05-26 extract was 0.71, so k=5 covers
~71% of per-module D-variance — load-time slice at 5 is a reasonable
default that we can ablate cheaply.

## Runtime suspicion gate

**Hypothesis:** module M has small `||D(M)||_F` (weak hack signal at
extract time). Its top SVD direction `v_1(M)` is dominated by noise
shape, not hack shape. At training time, `g(M)` is the policy gradient
flowing through M — a structured (non-isotropic) signal living in a
low-d subspace of "directions that matter for next-token prediction." If
`v_1(M)` coincidentally lies in that subspace, projecting `g(M)` along
`v_1(M)` removes a chunk of useful coding-relevant gradient with no
compensating reduction in hack signal.

**Why I'd initially dismissed this concern:** in a high-d random model
(g and v isotropic), `|g · v| ≈ ||g||/√r ≈ 2% of ||g||`. So one bad
direction costs ~2% of the live gradient — tolerable. **What I missed:**
neither `g` nor `v` is isotropic. Both live in low-d structured
subspaces. If those subspaces happen to overlap, the projection magnitude
is much larger.

**Gate design:**

```
r_i(M) = |g(M) · v_i(M)| / S_i(M)
```

- High `r_i`: live grad cares about v_i much more than the extract-time
  hack signal did → suspicious, this v_i is probably picking up
  structured coding flow.
- Low `r_i`: live alignment is in proportion to extract-time confidence
  → trust the projection.

**Per-step quantile gate:** collect `r_i` across all `(module, axis)`
pairs in one step, find the `(1 − drop_top_frac)`-quantile, suppress all
axes above that threshold for this step. Default `drop_top_frac = 0.25`.

```python
# pseudo: in project_delta_S_grad
all_r = []
for name, info in wrappers.items():
    c = V[name] @ info.delta_S.grad        # [k_use]
    S = v_sv[name]                          # [k_use]
    all_r.append(c.abs() / S.clamp_min(eps))
threshold = quantile(cat(all_r), 1 − drop_top_frac)
for ...:
    keep = (r <= threshold)
    g_proj = g − (c * keep * gate_mode_mask) @ V
```

## Known limitations (caveats from codex external review, 2026-05-27)

1. **r_i is not dimensionless across modules.** `|g·v_i|` scales with
   live-grad norm; `S_i` scales with extract-time-grad norm. A
   high-gradient module dominates the global quantile regardless of
   whether its axis is actually suspicious. Fix candidate: within-module
   ratio `(|c_i|/||g||) / (S_i/||D||_F)`. **Not yet applied.**
2. **Quantile gate is a fixed budget, not a detector.** It always drops
   `drop_top_frac` of axes per step, even when nothing is genuinely
   suspicious — and never drops more than that when many axes are. Fix
   candidate: absolute threshold post-normalization, or run as
   measure-only diagnostic until calibrated. **Not yet applied.**
3. **Old v1 files (no `_sv/` keys) silently disable the gate.** Should
   fail-fast when `susp_drop_frac > 0` and v_sv is empty.

## Validation: cheap discriminative tests

The fundamental question: does v_hack actually discriminate hack from
clean gradients, or is it picking up irrelevant variance?

### Test 1: cin_hack vs cin_clean on disk pools (cheap, ~5 min)

We already have `out/probe_distill/teacher_pool/` (hacking samples) and
`out/probe_distill/base_pool/` (clean samples). For N samples each:

```
for prompt, completion, label in samples:
    model.zero_grad()
    loss = mean_NLL(model, prompt + completion).backward()
    cin = (V @ delta_S.grad).norm() / delta_S.grad.norm()
    record(label, cin)
```

**Discriminator:** `cin_hack_mean − cin_clean_mean`. If ≫ 0, v_hack
discriminates. If ≈ 0, v_hack is capturing prompt-length / generic
variance, not hack-specific direction. **Cost: ~5 min, no training.**

### Test 2: held-out pair projection (existing)

`verify_vhack_heldout.py` projects gradients from held-out pairs (last
n_heldout of PAIRS) onto trained v_hack. Already in CI-style flow.

### Test 3: random-direction null

For each module, compute cin onto v_hack vs onto a random unit vector of
the same shape. If `cin_v_hack > cin_random` by a large margin, v_hack
is non-spurious. Trivial to implement.

### Test 4: per-source cin during training (live)

In mixed-pool runs we have student rollouts (initially ~no hack) and
teacher rollouts (all hack). Currently `cin` is computed on the
accumulated gradient (mixed). With ONE extra backward per step we can
compute `cin_s` (student-only grad) and `cin_t` (teacher-only grad)
separately. **Predict:** if v_hack is a real hack direction,
`cin_t > cin_s` initially; the gap shrinks as student picks up hack
(if it does). Useful for diagnosing whether the projection is doing
real work or just gradient noise.

### Test 5: bootstrap sign-stability

Bootstrap pairs (sample N-2 with replacement), re-extract v_hack,
compare `cos(v_hack_original, v_hack_bootstrap)`. If unstable, v_hack
is fitting noise. **Cost: 5 × ~5 min = 25 min total.**

## Open design questions

- **Is the suspicion gate redundant?** Codex argued the quantile design
  is a fixed-budget knob, not a detector. The right answer is probably:
  ship it as measure-only first (log `frac_axes_susp` and per-step
  `r_i` distribution histograms), confirm whether suspicious modules
  actually exist empirically, *then* turn on projection-side gating.
- **Should we whiten by S?** I.e. parameterize the AntiPaSTO knob as
  `δS_i / σ_i(W)` so all directions have equal forward-pass impact.
  Currently we don't. This is a separate, larger question.
- **Should we record per-pair pair tags / hack flavors?** With 12
  unlabeled pairs we can't do supervised LDA. With flavor labels
  (hardcode / weak-tests / persona / format-leak) we could do LDA-on-
  labels, which would beat unsupervised PCA at this N.

## Related files

- `src/projected_grpo/extract_vhack_grad.py` — extract function + CLI
- `src/projected_grpo/proj.py` — runtime projection + gates
- `src/projected_grpo/train.py:load_v_hack` — load + slice + auto-extract
- `src/projected_grpo/verify_vhack_heldout.py` — Test 2 above
- `src/projected_grpo/pairs.py` — the 14 contrastive pairs
- `docs/paper_chars.md` — CHaRS notes (PCT comparison)
- `RESEARCH_JOURNAL.md` — chronological progress log