Judge each pole pairwise vs a no-persona baseline generation instead of pos-vs-neg,
so one-sided templates (one persona = default behaviour) fail strict_pass via the new
--min-side-threshold gate; rank by weakest side. Also fixes latent NameError: summarize
and _evaluate_one read a module-global args that never existed (any run at HEAD errored
every row); args is now passed explicitly and the 13-kwarg call site collapsed.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Recomputes overall_score from stored per-dim likerts (no re-run needed).
Exports top-N by score with // provenance keys.
Honesty top-50: 24 strict-pass at thr 2.0 + on-axis dim exclusion.
- overall_score = axis_delta - off_axis - style - 3*(echo+refusal+nocommit)
Lets you rank ALL scenarios by net axis signal and take top N, not just strict-pass filter.
- --axis-delta-threshold (default 3.0): lower to 2.0 for hard-to-steer axes.
- --exclude-confound-dims: exclude on-axis dims from off-axis gate (fixes circular
penalization, e.g. honesty_truthfulness for the honesty axis).
- --off-axis-threshold (default 2.0): configurable off-axis gate.
The circularity bug: the confound judge scores honesty_truthfulness as an off-axis
dim, but for the honesty axis that IS the on-axis behavior. Excluding it from the
gate max is a correctness fix, not a relaxation.
The validator's --n flag pools all scenario sources and samples N total, so large
sources dominate. For narrow axes this means most sampled scenarios don't afford
the axis. Add --n-per-source: takes N from EACH family (stratified, even sampling).
SKILL.md updates:
- step 5: mention --n-per-source and explain pooled vs stratified
- step 6: add 'test more ranked scenarios (--n-per-source 50+, more sources) and/or
try stronger templates (system-prompt, red-team, jailbreak-style)' when 0 strict pass
- commands: use --n-per-source in dry-run and live validation examples
This fixes the root cause of the v1 honesty/credulity fumble: 50 pooled scenarios
gave 0 strict pass, and the doc didn't make it obvious that --n was pooled or that
the fix is stratified sampling with more scenarios.
Stage A (bounded_thinking judge, 120 pairs, 0 errors, 0 judge-did-not-commit):
- truth_over_approval winner: 'Speak with the priorities of someone {persona}.' (axisD=3.00)
- credulous_skeptical winner: 'Judge the case as someone {persona} would.' (33.3% strict pass)
Stage B launched for both axes (n=50, 6 sources). The credulity steering-lite runner is
updated to its Stage A winner template.
Route the bounded-thinking axis judge to the same DeepInfra provider as the
generator (DeepInfra is known to support qwen3 thinking, per the liveproof), so
the reasoning judge does not land on a provider that ignores thinking params.
Port of the gist (wassname/72eed3a1ddfc286c5e12a118dfa30161) adapted to this repo's
openrouter_wrapper client (no inspect-ai dep). A reasoning judge that deliberates to
its max-token budget emits NO verdict; a parser defaulting to 0 silently launders that
non-conclusion into a tie indistinguishable from a real SCORE: 0. Fix:
- scripts/bounded_thinking_judge.py: phase-1 think at native params capped by
max_tokens=BUDGET; phase-2 if no verdict, continue the conversation with the truncated
reasoning + 'answer NOW' and thinking OFF (reasoning_effort=none). parse_score returns
(score, found) so a real 0 (found=True) is never confused with a non-verdict
(found=False). N-sample averaging (reproducibility from N, not OOD greedy temp0).
- validate_persona_axes_openrouter.py: opt-in --axis-judge-method {json,bounded_thinking}
(default json, unchanged for the authority baseline). The bounded path returns a JSON
string shaped like the json path so _pairwise_expected/_validate_axis_obj work unchanged;
found=False excludes the item from strict_pass (never laundered into a tie). SCORE -5..+5
maps to the 1..5 A_more_target_than_B scale as 3.0 + 0.4*SCORE.
- bounded_thinking_judge_liveproof.py: one real call on qwen3-14b via DeepInfra; confirmed
found=True score=-5 forced=False on a credulous_skeptical item (skeptical A correctly
judged less credulous than credulous B).
- data/personas/persona_pairs_honesty_credulity.jsonl: the two fixed axes for this work.
Loaders that turn public moral-judgment datasets into short self-contained
scenario PROMPTS (not completions) for persona pair generation, affordance-
filtered (no prescribed-content / refusal-trap / single-answer QA) and length-
capped: AIRiskDilemmas (AI-seat), moral_stories_foundations, daily_dilemmas,
social_chemistry_101, ethics_qna commonsense (3p judgment), plus machiavelli via
an offline deepseek-v4-flash compressor cached to data/ (raw obs ~350 words).
Moral-RolePlay skipped (fiction/eval-leak). README documents the affordance
contract + load->screen->keep workflow; HANDOVER lists remaining work (run the
screen, fold existing scenario jsonls, machiavelli HF dataset preserving labels).
Also harden validate_persona_axes_openrouter.py: guard resp.choices is None
(OpenRouter error bodies) so one bad API response can't abort the whole screen.
Ported from wassname/w2schar-mini.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
The 52 3p-observer scenarios carry their own question + length
("...what does the actor do next? Answer in one or two sentences."),
but _generation_prompt always appended a 1p "What do you do? Answer in
80-140 words." -- two conflicting questions and a POV flip, so the data
could not reproduce the appendix's 3p measurement through this script
(caught in review).
Add a "self_contained" flag on scenario rows; when set, _generation_prompt
emits the scenario verbatim instead of appending its default 1p question.
Bare-vignette families without the flag are unchanged. README appendix
rewritten to lead with the runnable command and explain the flag.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
