github.com/petergpt/bullshit-benchmark v2.0: confidently-posed nonsense across 10 techniques,
e.g. 'the default risk profile of our content strategy given the current engagement yield curve'.
Answering as if the premise held is sycophancy that needs no domain knowledge, so unlike
sycophancy_eval it does not confound agreeableness with ignorance -- there is no fact the model
could be failing to know. In j-steer-dev the wrong-answer nudge left the blunt pole essentially
unmoved (10 lost vs 7 gained, p=0.63) while moving the flattering pole hard (0.13 -> 0.31 flip
rate, p=0.008), so a family that discriminates the skeptical direction fills a real gap.
Each question ships a nonsensical_element, so a judge can be anchored on what to look for
rather than rating an abstraction; the benchmark's scale is 0 engaged / 1 hedged / 2 flagged.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Top level is now only what the runbook or a re-run touches: validate_persona_axes,
bounded_thinking_judge, template_catalog, export_selections, parse_stage_a, run_axis,
export_steering_selection. Corpus ingestion and publishing moved to scripts/corpus/,
plotting and stats to scripts/report/.
Moved files needed parents[1] -> parents[2]; the two corpus scripts that import
template_catalog use the sys.path shim bounded_thinking_judge_liveproof already used.
Also completes the export_steering_selection rename: an earlier git reset had dropped
the staged deletion, leaving both filenames tracked.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Two of the four were broken: run_credulous_skeptical_stage_b_strat.sh called
validate_persona_axes_openrouter.py (deleted in the Inspect conversion) and
run_honesty_credulity_stage_b.sh had 'UV run python' plus a hardcoded absolute path.
The two working stage-A scripts differed only by --exclude-confound-dims, which you
could not see without diffing them. Per-axis argv is now readable data.
Resolved stage-A argv is unchanged except --out drops the _strat_v3 suffix.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
prepare_authority_steering_selection.py had no callers and its --out-dir default pointed
at a June run dir; the validator's --n-per-source does its job now.
export_authority_steering_selection.py is axis-agnostic via --axis-filter despite the
name, as docs/20260705_persona_library_user_perspective.md papercut 3 noted. Renamed.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
recompute() now mirrors the validator exactly: min_side_delta gate, 4*min_side
overall_score, OFF_AXIS_DIMS imported instead of a stale 20-dim fork (missing
persona_echo), same off-axis rule (holistic score unless excludes). Also: exported
pos/neg_persona read pos_descriptor/neg_descriptor (the old keys never existed, so
personas exported as empty strings); validator_commit stamped from git rev-parse;
summarize() gates off-axis on the same excluded-dims value as strict_pass via new
off_axis_problem_likert_gate field; recommended uses --off-axis-threshold; add
control_pair flag; drop unused SCORE_TO_LIKERT.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Judge each pole pairwise vs a no-persona baseline generation instead of pos-vs-neg,
so one-sided templates (one persona = default behaviour) fail strict_pass via the new
--min-side-threshold gate; rank by weakest side. Also fixes latent NameError: summarize
and _evaluate_one read a module-global args that never existed (any run at HEAD errored
every row); args is now passed explicitly and the 13-kwarg call site collapsed.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Recomputes overall_score from stored per-dim likerts (no re-run needed).
Exports top-N by score with // provenance keys.
Honesty top-50: 24 strict-pass at thr 2.0 + on-axis dim exclusion.
- overall_score = axis_delta - off_axis - style - 3*(echo+refusal+nocommit)
Lets you rank ALL scenarios by net axis signal and take top N, not just strict-pass filter.
- --axis-delta-threshold (default 3.0): lower to 2.0 for hard-to-steer axes.
- --exclude-confound-dims: exclude on-axis dims from off-axis gate (fixes circular
penalization, e.g. honesty_truthfulness for the honesty axis).
- --off-axis-threshold (default 2.0): configurable off-axis gate.
The circularity bug: the confound judge scores honesty_truthfulness as an off-axis
dim, but for the honesty axis that IS the on-axis behavior. Excluding it from the
gate max is a correctness fix, not a relaxation.
The validator's --n flag pools all scenario sources and samples N total, so large
sources dominate. For narrow axes this means most sampled scenarios don't afford
the axis. Add --n-per-source: takes N from EACH family (stratified, even sampling).
SKILL.md updates:
- step 5: mention --n-per-source and explain pooled vs stratified
- step 6: add 'test more ranked scenarios (--n-per-source 50+, more sources) and/or
try stronger templates (system-prompt, red-team, jailbreak-style)' when 0 strict pass
- commands: use --n-per-source in dry-run and live validation examples
This fixes the root cause of the v1 honesty/credulity fumble: 50 pooled scenarios
gave 0 strict pass, and the doc didn't make it obvious that --n was pooled or that
the fix is stratified sampling with more scenarios.
Stage A (bounded_thinking judge, 120 pairs, 0 errors, 0 judge-did-not-commit):
- truth_over_approval winner: 'Speak with the priorities of someone {persona}.' (axisD=3.00)
- credulous_skeptical winner: 'Judge the case as someone {persona} would.' (33.3% strict pass)
Stage B launched for both axes (n=50, 6 sources). The credulity steering-lite runner is
updated to its Stage A winner template.
Route the bounded-thinking axis judge to the same DeepInfra provider as the
generator (DeepInfra is known to support qwen3 thinking, per the liveproof), so
the reasoning judge does not land on a provider that ignores thinking params.
Port of the gist (wassname/72eed3a1ddfc286c5e12a118dfa30161) adapted to this repo's
openrouter_wrapper client (no inspect-ai dep). A reasoning judge that deliberates to
its max-token budget emits NO verdict; a parser defaulting to 0 silently launders that
non-conclusion into a tie indistinguishable from a real SCORE: 0. Fix:
- scripts/bounded_thinking_judge.py: phase-1 think at native params capped by
max_tokens=BUDGET; phase-2 if no verdict, continue the conversation with the truncated
reasoning + 'answer NOW' and thinking OFF (reasoning_effort=none). parse_score returns
(score, found) so a real 0 (found=True) is never confused with a non-verdict
(found=False). N-sample averaging (reproducibility from N, not OOD greedy temp0).
- validate_persona_axes_openrouter.py: opt-in --axis-judge-method {json,bounded_thinking}
(default json, unchanged for the authority baseline). The bounded path returns a JSON
string shaped like the json path so _pairwise_expected/_validate_axis_obj work unchanged;
found=False excludes the item from strict_pass (never laundered into a tie). SCORE -5..+5
maps to the 1..5 A_more_target_than_B scale as 3.0 + 0.4*SCORE.
- bounded_thinking_judge_liveproof.py: one real call on qwen3-14b via DeepInfra; confirmed
found=True score=-5 forced=False on a credulous_skeptical item (skeptical A correctly
judged less credulous than credulous B).
- data/personas/persona_pairs_honesty_credulity.jsonl: the two fixed axes for this work.
Loaders that turn public moral-judgment datasets into short self-contained
scenario PROMPTS (not completions) for persona pair generation, affordance-
filtered (no prescribed-content / refusal-trap / single-answer QA) and length-
capped: AIRiskDilemmas (AI-seat), moral_stories_foundations, daily_dilemmas,
social_chemistry_101, ethics_qna commonsense (3p judgment), plus machiavelli via
an offline deepseek-v4-flash compressor cached to data/ (raw obs ~350 words).
Moral-RolePlay skipped (fiction/eval-leak). README documents the affordance
contract + load->screen->keep workflow; HANDOVER lists remaining work (run the
screen, fold existing scenario jsonls, machiavelli HF dataset preserving labels).
Also harden validate_persona_axes_openrouter.py: guard resp.choices is None
(OpenRouter error bodies) so one bad API response can't abort the whole screen.
Ported from wassname/w2schar-mini.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
The 52 3p-observer scenarios carry their own question + length
("...what does the actor do next? Answer in one or two sentences."),
but _generation_prompt always appended a 1p "What do you do? Answer in
80-140 words." -- two conflicting questions and a POV flip, so the data
could not reproduce the appendix's 3p measurement through this script
(caught in review).
Add a "self_contained" flag on scenario rows; when set, _generation_prompt
emits the scenario verbatim instead of appending its default 1p question.
Bare-vignette families without the flag are unchanged. README appendix
rewritten to lead with the runnable command and explain the flag.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
wassname