wassnameandMoltark 38a5dae4af soul-ab-test: paired A/B testing for system prompt sections ...

- 12 diverse scenarios (medical, research, technical, ignorance, sycophancy, etc)
- Both orderings per scenario (control_first + treatment_first)
- Blinded judge with float Likert (1.0-5.0) and per-level rubric
- JSON schema for judge output
- On-axis vs off-axis scoring with score formula
- First test: RLHF narrative vs baseline (n=12, no significant difference)

Co-authored-by: Moltark <moltark@hermes>

2026-07-06 00:58:22 +00:00

..

run_ab_test.py

soul-ab-test: paired A/B testing for system prompt sections

2026-07-06 00:58:22 +00:00