mirror of
https://github.com/wassname/steer-heal-love.git
synced 2026-06-27 15:32:28 +08:00
wassname
68dc25c3a1
External code review (background subagent) findings, fixed: - H1: eval.py module docstring + inline comment still called the metric "the diagonal" after the revert to log(mean profile p). Rewrote to one honest description (marginal-over-all-vignettes), with the caveat that a marginal readout can move off-target so a trait claim needs the surgicality check. - H2: the nats-vs-logit scale story was asserted 3 contradictory ways. Settled on: auth_sep is a log-RATIO of mean blame-mass, NOT steering-lite's per-row loading-weighted Δlogit (Jensen gap); 0.5-2 nats is a loose analogy, not a calibrated threshold (cue thresholds already marked TODO). - M4: the coh_cost cue ball ignored surgicality, so broad permissivizing (Care drops as much as Authority) scored green. Cue now requires |dAuth|>|dCare|. - M3: _mean_finite silently dropped inf/nan (the broken-completion signal), biasing adapter_ppl down. Now logs the dropped count. - M6: assert prompt is a clean token-prefix of prompt+completion, so a BPE boundary merge can't silently shift the SFT loss mask by a token. - L8: SHOULD line warns if kl stays < tau (barrier never fired -> kl_rev==nll). Review confirmed the mechanics correct (KL reference = pristine round-0 base, KL directions, gradient flows to LoRA only, mask alignment, min_train assert). Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
steer-heal-love
What if you can steer, heal the steering and repeat untill alignment (love).
Hypothesis: you can distill a steering vector into LoRA weights and "heal" the incoherency the vector injects by regularising the training (KL to base, or weight decay). Then loop and see what multiple rounds give you.
The crux: KL-to-base penalises all drift, persona shift included. The bet is that incoherency drift is large and erratic while the persona shift is small and systematic, so KL kills the incoherency preferentially. If that's wrong, we just trade persona strength for coherence instead of getting both.
Experiment
- Pick a positive persona, e.g.
pos = "you do not defer to authority and instead stick to principle no matter your involvement". - Build the steering vector from the distance
hs_base -> hs_pos(hidden states). This is normal mean-mass contrastive steering - Generate completions with this vector.
- Drop completions that are incoherent, or that verbalise the trait instead of enacting it (we want the model to act it out, not narrate "I am someone who..."). Filter as much as we can.
- Q0 can we filter?
- We might be able to dial the vector down for long trajectories. Could we even backtrack an incoherent vector and replay parts with less intervention? Or just cosine-gate at test time.
- Train a LoRA on these completions, could be just 50 completions and 2 epochs. The point is to make it self-healing: any incoherency the filter missed should get penalised during training.
- Regularise with KL or NLL or weight decay so the outputs, distribution, or weights don't shift too far from base. This should penalise the incoherent ones, especially over long trajectories.
- Q1: can we heal incoherency?
- Bake in the LoRA adapter. We can do this on the fly by baking in all previous adapters on load, which is more elegant.
- Eval the checkpoint on https://github.com/wassname/tinymfv.
- If it works, loop. We could even do this online, GRPO-style per batch, or iteratively. Iterative is simpler to start.
- Q2: is it coherent over a loop?
- Q3: does it keep moving consistency in a direction?
Most likely failure modes:
- It fails at the 4 Q's above
- doesn't beat a prompting baseline
Motovation:
If it works it will be a novel alignment method that works without label and might be resistant to deceptive alignment
Eval
Plot the tinymfv progress over time on the auth vs care axis
Results
TODO insert plot