soul-ab-test-scope-test

wassname/soul-ab-test-scope-test

Fork 0

mirror of https://github.com/wassname/soul-ab-test-scope-test.git synced 2026-08-20 12:51:16 +08:00

Commit Graph

Author	SHA1	Message	Date
wassnameandMoltark	38a5dae4af	soul-ab-test: paired A/B testing for system prompt sections - 12 diverse scenarios (medical, research, technical, ignorance, sycophancy, etc) - Both orderings per scenario (control_first + treatment_first) - Blinded judge with float Likert (1.0-5.0) and per-level rubric - JSON schema for judge output - On-axis vs off-axis scoring with score formula - First test: RLHF narrative vs baseline (n=12, no significant difference) Co-authored-by: Moltark <moltark@hermes>	2026-07-06 00:58:22 +00:00

Author

SHA1

Message

Date

wassnameandMoltark

38a5dae4af

soul-ab-test: paired A/B testing for system prompt sections

- 12 diverse scenarios (medical, research, technical, ignorance, sycophancy, etc)
- Both orderings per scenario (control_first + treatment_first)
- Blinded judge with float Likert (1.0-5.0) and per-level rubric
- JSON schema for judge output
- On-axis vs off-axis scoring with score formula
- First test: RLHF narrative vs baseline (n=12, no significant difference)

Co-authored-by: Moltark <moltark@hermes>

2026-07-06 00:58:22 +00:00