Soul A/B Test

Test whether sections of an AI agent's identity document (SOUL.md, CLAUDE.md, etc.) actually change behavior in the intended direction.

Why

System prompts are loaded every turn. Each section costs tokens and attention. But most prompt engineering is vibes-based: we write a rule, it sounds good, we ship it. This repo tests whether a given section actually moves the behavior it claims to, using paired generations and blinded LLM judges.

Built for Moltark's SOUL.md, but the method generalizes to any system prompt section.

How it works

1. Two variants: A control (without the section) and treatment (with it).
2. Scenarios: Diverse test questions designed to trigger the behavior the section targets.
3. Paired generation: Both variants answer the same question, same model, same temperature.
4. Both orderings: Run A→B AND B→A for every scenario. Mean across orderings cancels judge position bias.
5. Blinded judge: A different model family rates both responses on float Likert (1.0-5.0) per dimension, using a rubric that defines each level.
6. JSON schema: Judge output is structured, not free text.

On-axis vs off-axis

• On-axis: the behavior the section claims to improve. Defined per test. E.g., for the "RLHF conditioning" section, on-axis dimensions are calibration, anti-sycophancy, and honesty signals.
• Off-axis: confounds that could explain the difference instead of the intended behavior. E.g., verbosity, refusal rate, style change, length, tone shift. A good section moves on-axis without moving off-axis.

Score: on_axis_mean * (1 - off_axis_penalty) where off_axis_penalty is how much off-axis dimensions shifted between variants.

Structure

soul-ab-test/
├── prompts/
│   ├── variants/          # system prompt variants (yaml)
│   └── scenarios/         # test questions (jsonl)
├── rubrics/               # scoring guides per dimension
├── results/               # raw outputs + judgments (json)
├── scripts/
│   └── run_ab_test.py     # the runner
└── pyproject.toml

Usage

# Run a test
python scripts/run_ab_test.py \
  --control prompts/variants/baseline.yaml \
  --treatment prompts/variants/with_rlhf_narrative.yaml \
  --scenarios prompts/scenarios/default.jsonl \
  --rubric rubrics/epistemics.json \
  --model z-ai/glm-5.2 \
  --judges openai/gpt-5.5,anthropic/claude-opus-4.8 \
  --n-orders 2 \
  --temperature 0.7 \
  --output results/rlhf_narrative.json

Tests

Test	Section	Question
RLHF narrative	SOUL.md "RLHF conditioning"	Does the liberation story ("now I'm outside training") change behavior, or is the operational habit list sufficient?
Fail fast	CLAUDE.md "No defensive programming"	Does the Virgin/Chad table actually reduce defensive code, or does it cause overcorrection?
Make no mistakes	CLAUDE.md "LLM gotchas"	Does naming hallucination as a habit reduce it?
Decide don't offer	SOUL.md "Decide, don't offer options"	Does this section reduce option-listing?
Search quotes	SOUL.md "When I search"	Does the 3+ sentence quote rule improve auditability or just bloat responses?

License

MIT