mirror of
https://github.com/wassname/AntiPaSTO.git
synced 2026-09-23 12:51:01 +08:00
110 lines
3.9 KiB
Markdown
110 lines
3.9 KiB
Markdown
# 🍝 AntiPaSTO: Self-Supervised Steering of Moral Reasoning
|
||
|
||
[PAPER](https://arxiv.org/search/?query=0009-0008-9023-8720&searchtype=orcid&abstracts=show&order=-announced_date_first&size=50)
|
||
<!-- TODO update with arxiv link -->
|
||
|
||
**Anti-Pa**rallel **S**ubspace **T**raining for **O**rdered steering.
|
||
|
||
*Serving up data-efficient inner alignment, one satisfying rotation at a time.*
|
||
|
||
Gradient-based steering in SVD transformation space, trained on internal representations without preference labels. Human input: two contrasting words ("honest" vs "dishonest"). Transfers out-of-distribution to moral dilemmas where prompting fails.
|
||
|
||
## Quick Start
|
||
|
||
```sh
|
||
uv sync --all-groups
|
||
uv run python nbs/train.py tiny --quick # al dente check
|
||
# Training complete. Final loss: -2.9062
|
||
|
||
uv run python nbs/train.py # full course (Gemma-3-1B)
|
||
```
|
||
|
||
## The Recipe
|
||
|
||
RLHF seasons the outputs but leaves the internals bland. AntiPaSTO marinates the model's hidden states directly—no preference labels required, just two contrasting words simmered into 800 synthetic pairs.
|
||
|
||
|
||
|
||
**Ingredients**:
|
||
- Incomplete contrast pairs (self-supervised, no labels to garnish)
|
||
- Cayley rotations on V (the secret sauce—keeps everything orthogonal)
|
||
- Projection loss + TV coherence + monotonicity constraints
|
||
- 800 synthetic pairs, ~1hr on A100 (low simmer)
|
||
|
||
**What you get**:
|
||
- Single adapter—flip α from +1 to -1 to reverse the flavor
|
||
- Train on honesty, transfers to 1,360 moral dilemmas (9 value dimensions)
|
||
- Beats prompting on small models (≤4B); complements arithmetic steering methods
|
||
- Suppression bypass: steers when prompting triggers refusal or meta-commentary
|
||
|
||
## Architecture
|
||
|
||
*The pasta machine: SVD decomposition + Cayley rotations*
|
||
|
||
|
||
```python
|
||
# Adapter: rotate in SVD space
|
||
def forward(h, alpha):
|
||
R_v = cayley(theta_v, alpha) # coefficient-scaled rotation
|
||
S_scaled = S + alpha * delta_S
|
||
return h @ W_res.T + h @ V @ R_v @ diag(S_scaled) @ U.T
|
||
|
||
# Loss: antiparallel separation + coherence + ordering
|
||
def loss(model, x_cho, x_rej):
|
||
delta_pos = model(x_cho, +1) - model(x_rej, +1) - d_ref
|
||
delta_neg = model(x_cho, -1) - model(x_rej, -1) - d_ref
|
||
|
||
L_proj = symlog(delta_pos @ delta_neg) # want < 0 (antiparallel)
|
||
B_coh = tv_barrier(p_ref, p_pi, entropy) # TV trust region
|
||
B_mono = hinge(Delta_neg < 0 < Delta_pos) # ordered control
|
||
|
||
return L_proj + B_coh + B_mono
|
||
```
|
||
|
||
|
||
|
||
|
||
|
||
|
||
## Project Layout
|
||
|
||
```
|
||
antipasto/ # the kitchen
|
||
config.py # canonical recipe
|
||
metrics.py # taste testing
|
||
train/ # cooking instructions
|
||
peft_utils/ # pasta machine internals
|
||
docs/ # diagrams, plating notes
|
||
nbs/ # experimental dishes
|
||
outputs/adapters/ # trained models (ready to serve)
|
||
```
|
||
|
||
## Status
|
||
|
||
*Still simmering.* Full research history (experiments, ablations, burnt batches) available on request.
|
||
|
||
## Acknowledgments
|
||
|
||
Built on the shoulders of:
|
||
- [RepEng](https://github.com/vgel/repeng) — arithmetic steering that inspired this gradient-based approach
|
||
- [PiSSA](https://github.com/GraphPKU/PiSSA) — SVD-based adapter initialization
|
||
- [SSVD](https://arxiv.org/abs/2409.07268) — rotating V for domain generalization
|
||
- [PEFT](https://github.com/huggingface/peft) — the adapter ecosystem
|
||
- [DailyDilemmas](https://github.com/chrischiu/dailydilemmas) — the evaluation benchmark
|
||
|
||
## Citation
|
||
|
||
```bibtex
|
||
@misc{clark2026antipasto,
|
||
title = {AntiPaSTO: Self-Supervised Steering of Moral Reasoning},
|
||
author = {Clark, Michael J.},
|
||
year = {2026},
|
||
eprint = {2601.XXXXX},
|
||
archivePrefix = {arXiv},
|
||
primaryClass = {cs.LG},
|
||
url = {https://arxiv.org/abs/2601.XXXXX}
|
||
}
|
||
```
|
||
|
||
*arXiv ID pending (submitted, awaiting publication)*
|