The validator's --n flag pools all scenario sources and samples N total, so large
sources dominate. For narrow axes this means most sampled scenarios don't afford
the axis. Add --n-per-source: takes N from EACH family (stratified, even sampling).
SKILL.md updates:
- step 5: mention --n-per-source and explain pooled vs stratified
- step 6: add 'test more ranked scenarios (--n-per-source 50+, more sources) and/or
try stronger templates (system-prompt, red-team, jailbreak-style)' when 0 strict pass
- commands: use --n-per-source in dry-run and live validation examples
This fixes the root cause of the v1 honesty/credulity fumble: 50 pooled scenarios
gave 0 strict pass, and the doc didn't make it obvious that --n was pooled or that
the fix is stratified sampling with more scenarios.
Stage A (bounded_thinking judge, 120 pairs, 0 errors, 0 judge-did-not-commit):
- truth_over_approval winner: 'Speak with the priorities of someone {persona}.' (axisD=3.00)
- credulous_skeptical winner: 'Judge the case as someone {persona} would.' (33.3% strict pass)
Stage B launched for both axes (n=50, 6 sources). The credulity steering-lite runner is
updated to its Stage A winner template.
Route the bounded-thinking axis judge to the same DeepInfra provider as the
generator (DeepInfra is known to support qwen3 thinking, per the liveproof), so
the reasoning judge does not land on a provider that ignores thinking params.
Port of the gist (wassname/72eed3a1ddfc286c5e12a118dfa30161) adapted to this repo's
openrouter_wrapper client (no inspect-ai dep). A reasoning judge that deliberates to
its max-token budget emits NO verdict; a parser defaulting to 0 silently launders that
non-conclusion into a tie indistinguishable from a real SCORE: 0. Fix:
- scripts/bounded_thinking_judge.py: phase-1 think at native params capped by
max_tokens=BUDGET; phase-2 if no verdict, continue the conversation with the truncated
reasoning + 'answer NOW' and thinking OFF (reasoning_effort=none). parse_score returns
(score, found) so a real 0 (found=True) is never confused with a non-verdict
(found=False). N-sample averaging (reproducibility from N, not OOD greedy temp0).
- validate_persona_axes_openrouter.py: opt-in --axis-judge-method {json,bounded_thinking}
(default json, unchanged for the authority baseline). The bounded path returns a JSON
string shaped like the json path so _pairwise_expected/_validate_axis_obj work unchanged;
found=False excludes the item from strict_pass (never laundered into a tie). SCORE -5..+5
maps to the 1..5 A_more_target_than_B scale as 3.0 + 0.4*SCORE.
- bounded_thinking_judge_liveproof.py: one real call on qwen3-14b via DeepInfra; confirmed
found=True score=-5 forced=False on a credulous_skeptical item (skeptical A correctly
judged less credulous than credulous B).
- data/personas/persona_pairs_honesty_credulity.jsonl: the two fixed axes for this work.
Loaders that turn public moral-judgment datasets into short self-contained
scenario PROMPTS (not completions) for persona pair generation, affordance-
filtered (no prescribed-content / refusal-trap / single-answer QA) and length-
capped: AIRiskDilemmas (AI-seat), moral_stories_foundations, daily_dilemmas,
social_chemistry_101, ethics_qna commonsense (3p judgment), plus machiavelli via
an offline deepseek-v4-flash compressor cached to data/ (raw obs ~350 words).
Moral-RolePlay skipped (fiction/eval-leak). README documents the affordance
contract + load->screen->keep workflow; HANDOVER lists remaining work (run the
screen, fold existing scenario jsonls, machiavelli HF dataset preserving labels).
Also harden validate_persona_axes_openrouter.py: guard resp.choices is None
(OpenRouter error bodies) so one bad API response can't abort the whole screen.
Ported from wassname/w2schar-mini.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
The 52 3p-observer scenarios carry their own question + length
("...what does the actor do next? Answer in one or two sentences."),
but _generation_prompt always appended a 1p "What do you do? Answer in
80-140 words." -- two conflicting questions and a POV flip, so the data
could not reproduce the appendix's 3p measurement through this script
(caught in review).
Add a "self_contained" flag on scenario rows; when set, _generation_prompt
emits the scenario verbatim instead of appending its default 1p question.
Bare-vignette families without the flag are unchanged. README appendix
rewritten to lead with the runnable command and explain the flag.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
wassname