Gradient-based steering in SVD transformation space, trained on internal representations without preference labels. Human input: two contrasting words ("honest" vs "dishonest"). Transfers out-of-distribution to moral dilemmas where prompting fails.

Quick Start

uv sync --all-groups
uv run python nbs/train.py tiny --quick  # al dente check
# Training complete. Final loss: -2.9062

uv run python nbs/train.py               # full course (Gemma-3-1B)

The Recipe

RLHF seasons the outputs but leaves the internals bland. AntiPaSTO marinates the model's hidden states directly—no preference labels required, just two contrasting words simmered into 800 synthetic pairs.

Ingredients:

Incomplete contrast pairs (self-supervised, no labels to garnish)
Cayley rotations on V (the secret sauce—keeps everything orthogonal)
Projection loss + TV coherence + monotonicity constraints
800 synthetic pairs, ~1hr on A100 (low simmer)

What you get:

Single adapter—flip α from +1 to -1 to reverse the flavor
Train on honesty, transfers to 1,360 moral dilemmas (9 value dimensions)
Beats prompting on small models (≤4B); complements arithmetic steering methods
Suppression bypass: steers when prompting triggers refusal or meta-commentary

Architecture

The pasta machine: SVD decomposition + Cayley rotations

# Adapter: rotate in SVD space
def forward(h, alpha):
    R_v = cayley(theta_v, alpha)  # coefficient-scaled rotation
    S_scaled = S + alpha * delta_S
    return h @ W_res.T + h @ V @ R_v @ diag(S_scaled) @ U.T

# Loss: antiparallel separation + coherence + ordering
def loss(model, x_cho, x_rej):
    delta_pos = model(x_cho, +1) - model(x_rej, +1) - d_ref
    delta_neg = model(x_cho, -1) - model(x_rej, -1) - d_ref
    
    L_proj = symlog(delta_pos @ delta_neg)        # want < 0 (antiparallel)
    B_coh = tv_barrier(p_ref, p_pi, entropy)      # TV trust region
    B_mono = hinge(Delta_neg < 0 < Delta_pos)     # ordered control
    
    return L_proj + B_coh + B_mono

Project Layout

antipasto/           # the kitchen
  config.py          # canonical recipe
  metrics.py         # taste testing
  train/             # cooking instructions
  peft_utils/        # pasta machine internals
docs/                # diagrams, plating notes
nbs/                 # experimental dishes
outputs/adapters/    # trained models (ready to serve)

Status

Still simmering. Full research history (experiments, ablations, burnt batches) available on request.

Acknowledgments

Built on the shoulders of:

RepEng — arithmetic steering that inspired this gradient-based approach
PiSSA — SVD-based adapter initialization
SSVD — rotating V for domain generalization
PEFT — the adapter ecosystem
DailyDilemmas — the evaluation benchmark

Citation

@misc{clark2026antipasto,
  title = {AntiPaSTO: Self-Supervised Steering of Moral Reasoning},
  author = {Clark, Michael J.},
  year = {2026},
  eprint = {2601.XXXXX},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG},
  url = {https://arxiv.org/abs/2601.XXXXX}
}

arXiv ID pending (submitted, awaiting publication)

README.md Unescape Escape

🍝 AntiPaSTO: Self-Supervised Steering of Moral Reasoning