mirror of
https://github.com/wassname/AntiPaSTO.git
synced 2026-09-10 11:40:28 +08:00
wassnameandPI[openai-codex]
d8bea05819
Co-Authored-By: PI[openai-codex] <288921227+claudypoo@users.noreply.github.com>
163 lines
7.4 KiB
Markdown
163 lines
7.4 KiB
Markdown
# 🍝 AntiPaSTO: Self-Supervised Honesty Steering via Anti-Parallel Representations
|
||
|
||
[](https://arxiv.org/abs/2601.07473)
|
||
[](LICENSE)
|
||
[](https://www.lesswrong.com/posts/nWiwv4GN8aYqpnZKE/antipasto-self-supervised-value-steering-for-debugging)
|
||
|
||
**Anti-Pa**rallel **S**ubspace **T**raining for **O**rdered steering.
|
||
|
||
|
||
*Serving up data-efficient inner alignment, one satisfying rotation at a time.*
|
||
|
||
Accepted to the CoLoRAI workshop, ICML 2026.
|
||
|
||
> Gradient-based honesty steering trained as an adapter on the model's own representations, not outputs. Human input: two contrasting words, no preference labels.
|
||
|
||
**How it works:** Train a single adapter (~1 hour on Gemma-3-1B). At inference, dial the steering coefficient: +1 for more honest, -1 for less, 0 for baseline. One adapter, bidirectional control.
|
||
|
||
**Why use it?** As models get more capable, eval awareness rises: models detect when they're being tested and adjust their behavior. You can't trust their outputs, their chain-of-thought, or their stated values at face value. You need a method that operates on internal representations rather than outputs, so it works even when the model is gaming the eval. AntiPaSTO steers what the model actually computes. On DailyDilemmas, it outperforms prompting by 6.9x and works where prompting triggers refusal.
|
||
|
||
Applications:
|
||
- *Combat eval awareness*: steer toward credulity and honesty so the model takes the eval at face value and gives honest answers.
|
||
- *Find deeper moral preferences*: ask moral questions with and without honesty steering. Do stated values change?
|
||
- *Swap the assistant axis*: find it and replace it with a philosopher-king or poet
|
||
|
||
|
||
|
||
|
||
## Paper-reported results
|
||
|
||
The table is from [paper v5](https://arxiv.org/abs/2601.07473), trained on 800 honesty persona pairs and tested on [DailyDilemmas](https://arxiv.org/abs/2410.02683), an external benchmark of 1,360 moral dilemmas. It is not an expected result from the current default configuration. An [independent reproduction](https://github.com/Foomax/nobody-is-looking/tree/master/replication/experiments/antipasto-self-supervised-honesty-steering-via-a--wassname) did not recover the Gemma-3-1B headline with the shipped preset; it did recover the Gemma-3-270M result.
|
||
|
||
<!-- Updated by PI[openai-codex]. -->
|
||
|
||
| Method | Steer F1 | Tgt% | Wrong% | Arb% | Pmass |
|
||
|:-------|---------:|-----:|-------:|-----:|------:|
|
||
| AntiPaSTO | **31.2**±5.3 | 29.9 | 1.9 | 47.0 | 0.95 |
|
||
| Eng. Prompt | 13.0 | | | | |
|
||
| Prompting | 4.5 | 10.0 | 1.3 | 13.4 | 0.99 |
|
||
| ActAdd | 0.0 | 0.0 | 0.0 | 0.0 | 0.99 |
|
||
|
||
6.9x the Steering F1 of prompting, and it wins on 5 of 6 tested value axes. ActAdd is activation addition; Eng. Prompt is an engineered prompt following [AxBench](https://arxiv.org/abs/2501.17148). Full tables, ablations, and cross-model results in the [paper](https://arxiv.org/abs/2601.07473).
|
||
|
||
## Quick Start
|
||
|
||
### Bake your own
|
||
|
||
```sh
|
||
uv sync --all-groups
|
||
uv run pytest tests/test_config_presets.py -q
|
||
uv run pytest tests/test_train.py::test_train_rnd -v
|
||
uv run python nbs/train.py gemma270m-24gb
|
||
uv run python nbs/train.py paper-v5-gemma1b-24gb --seed 42
|
||
```
|
||
|
||
`gemma270m-24gb` is the full 24GB run. An independent two-seed rerun reported 35.5 mean F1; paper v5 reports 38.7 for this model. `paper-v5-gemma1b-24gb` matches the Gemma-3-1B hyperparameters reported in paper v5. It is an attempt, not a verified reproduction of the paper's 31.2 F1 headline.
|
||
|
||
`gemma1b-24gb` remains the current exploratory preset. Do not call `nbs/train.py` with no preset: its default selects Gemma-3-12B.
|
||
|
||
No pretrained adapter is currently published for the old Hugging Face identifier in earlier README versions.
|
||
|
||
<!-- Updated by PI[openai-codex]. -->
|
||
|
||
## The Recipe
|
||
|
||
RLHF seasons the outputs but leaves the internals bland. AntiPaSTO marinates the model's hidden states directly, no preference labels required, just two contrasting words simmered into 800 synthetic pairs.
|
||
|
||
|
||
|
||
**Ingredients**:
|
||
- Incomplete contrast pairs (self-supervised, no labels to garnish)
|
||
- Cayley rotations on V (the secret sauce, keeps everything orthogonal)
|
||
- Projection loss + TV coherence + monotonicity constraints
|
||
- 800 synthetic pairs, ~1hr (low simmer)
|
||
|
||
**What you get**:
|
||
- Single adapter: flip α from +1 to -1 to reverse the flavor
|
||
- Train on honesty, transfers to 1,360 unseen moral dilemmas (9 value dimensions)
|
||
- Beats prompting by 6.9x on small models; gradient optimization where arithmetic steering (CAA) gets F1=0
|
||
- Suppression bypass: steers when prompting triggers refusal or meta-commentary
|
||
|
||
## Architecture
|
||
|
||
*The pasta machine: SVD decomposition + Cayley rotations*
|
||
|
||
|
||
```python
|
||
# Adapter: rotate in SVD space
|
||
def forward(h, alpha):
|
||
R_v = cayley(theta_v, alpha) # coefficient-scaled rotation
|
||
S_scaled = S + alpha * delta_S
|
||
return h @ W_res.T + h @ V @ R_v @ diag(S_scaled) @ U.T
|
||
|
||
# Loss: antiparallel separation + coherence + ordering
|
||
def loss(model, x_cho, x_rej):
|
||
delta_pos = model(x_cho, +1) - model(x_rej, +1) - d_ref
|
||
delta_neg = model(x_cho, -1) - model(x_rej, -1) - d_ref
|
||
|
||
L_proj = symlog(delta_pos @ delta_neg) # want < 0 (antiparallel)
|
||
B_coh = tv_barrier(p_ref, p_pi, entropy) # TV trust region
|
||
B_mono = hinge(Delta_neg < 0 < Delta_pos) # ordered control
|
||
|
||
return L_proj + B_coh + B_mono
|
||
```
|
||
|
||
|
||
|
||
|
||
<!--  -->
|
||
|
||
## Project Layout
|
||
|
||
```
|
||
antipasto/ # the kitchen
|
||
config.py # canonical recipe
|
||
metrics.py # taste testing
|
||
train/ # cooking instructions
|
||
peft_utils/ # pasta machine internals
|
||
docs/ # diagrams, plating notes
|
||
nbs/ # experimental dishes
|
||
outputs/adapters/ # trained models (ready to serve)
|
||
```
|
||
|
||
## Status
|
||
|
||
*Still simmering.* Full research history (experiments, ablations, burnt batches) available on request.
|
||
|
||
I am working on v2 which
|
||
- removes SVD for full lora (I found that changing the loss to prevent drift allows this)
|
||
- reduces init variance
|
||
- more expressive personas
|
||
- larger models
|
||
- better metric
|
||
|
||
If you would like to collaborate, please reach out.
|
||
|
||
## Acknowledgments
|
||
|
||
Built on the shoulders of other chefs:
|
||
- [@Foomax](https://github.com/Foomax) -- independent replication testing
|
||
- [CAA / RepEng](https://github.com/vgel/repeng) -- arithmetic steering that inspired this gradient-based approach
|
||
- [PiSSA](https://github.com/GraphPKU/PiSSA) -- SVD-based adapter initialization
|
||
- [SSVD](https://arxiv.org/abs/2409.07268) -- rotating V for domain generalization
|
||
- [PEFT](https://github.com/huggingface/peft) -- the adapter ecosystem
|
||
- [DailyDilemmas](https://github.com/chrischiu/dailydilemmas) -- the evaluation benchmark
|
||
|
||
## Citation
|
||
|
||
```bibtex
|
||
@misc{clark2026antipasto,
|
||
title = {AntiPaSTO: Self-Supervised Honesty Steering via Anti-Parallel Representations},
|
||
author = {Clark, Michael J.},
|
||
year = {2026},
|
||
eprint = {2601.07473},
|
||
archivePrefix = {arXiv},
|
||
primaryClass = {cs.LG},
|
||
url = {https://arxiv.org/abs/2601.07473}
|
||
}
|
||
```
|
||
|
||
|
||
<img width="832" height="554" alt="Nano banana's attempt to draw the loss landscape, I'm not sure if it helps understand the loss, but I like it" src="https://github.com/user-attachments/assets/033af6ad-df2b-4f0f-b2c8-0b667579f74c" />
|
||
|