From bbbedf2ad4bddc1a830225bcce8d9d5d2d2e2bf5 Mon Sep 17 00:00:00 2001 From: wassname <1103714+wassname@users.noreply.github.com> Date: Tue, 30 Jun 2026 10:55:33 +0800 Subject: [PATCH] Document strict steering selection workflow --- .../skills/persona-template-library/SKILL.md | 52 +++++++++++++++- README.md | 59 +++++++++++++++++++ README.qmd | 57 ++++++++++++++++++ 3 files changed, 167 insertions(+), 1 deletion(-) diff --git a/.agents/skills/persona-template-library/SKILL.md b/.agents/skills/persona-template-library/SKILL.md index 39cb217..e3afb71 100644 --- a/.agents/skills/persona-template-library/SKILL.md +++ b/.agents/skills/persona-template-library/SKILL.md @@ -21,11 +21,17 @@ dataset. - `data/personas/persona_pairs_v2_candidates.jsonl`: candidate persona pairs. - `data/scenarios/scenarios_*.jsonl`: candidate scenario suffixes to validate on the target model. +- `data/selections/`: committed selected scenario sets that have passed a target-model + validation screen. - `scripts/scenario_sources/export_scenarios.py`: writes source loader outputs into `data/scenarios/scenarios_.jsonl`. - `data/results/`: committed result tables and reader-facing result assets. - `out/`: local scratch outputs and API caches; ignored by git. - `scripts/validate_persona_axes_openrouter.py`: live and dry-run validator. +- `scripts/prepare_authority_steering_selection.py`: source-stratified Stage A/Stage B + authority-axis selection inputs. +- `scripts/export_authority_steering_selection.py`: exports the chosen axis/template + and strict-pass scenarios from validator artifacts. - `scripts/export_persona_template_stats.py`: converts validator artifacts into examples and score tables. - `scripts/summarize_model_matrix.py`: summarizes latest model-matrix logs for @@ -47,7 +53,13 @@ Use the repo in this order: behavior mode you need: doing, judging, explaining, refusing, moral tradeoffs, or multi-turn behavior. 4. Run a dry-run validator command before live OpenRouter calls. -5. After a live run, export stats and inspect examples before trusting scores. +5. For a steering-ready selection, use a two-stage screen: + Stage A = several axes/templates on a small source-diverse panel; Stage B = + the chosen axis/template on up to 30 scenarios per source. +6. Export strict-pass scenarios only. Do not steer from source-balanced filler rows + when a source produced no clean scenarios. +7. Commit reusable selected scenario JSONL files under `data/selections/`. +8. After a live run, inspect examples before trusting scores. Read `docs/persona_prompt_literature_review.md` when choosing new persona pairs or template shapes from related work. If the global `persona-steering` skill is @@ -111,6 +123,44 @@ uv run python scripts/export_persona_template_stats.py \ --out-prefix data/results/stats/v2_pilot_seed24 ``` +Authority/dignity steering selection: + +```sh +uv run python scripts/prepare_authority_steering_selection.py \ + --out-dir out/authority_selection + +uv run python scripts/validate_persona_axes_openrouter.py \ + --axes out/authority_selection/stage_a_axes.jsonl \ + --templates out/authority_selection/stage_a_templates.txt \ + --family out/authority_selection/stage_a_scenarios.jsonl \ + --n 24 --seed 42 \ + --generator-model qwen/qwen3-8b \ + --axis-judge-models google/gemini-3.1-flash-lite-preview \ + --judge-model google/gemini-3.1-flash-lite-preview \ + --out out/authority_selection/stage_a_live.json + +uv run python scripts/export_authority_steering_selection.py \ + --stage-a out/authority_selection/stage_a_live.json \ + --axis-filter dignity_over_authority \ + --out-dir out/authority_selection/dignity_strict + +uv run python scripts/validate_persona_axes_openrouter.py \ + --axes out/authority_selection/dignity_strict/stage_b_axis.jsonl \ + --templates out/authority_selection/dignity_strict/stage_b_template.txt \ + --family out/authority_selection/stage_b_candidate_scenarios.jsonl \ + --n 342 --seed 43 \ + --generator-model qwen/qwen3-8b \ + --axis-judge-models google/gemini-3.1-flash-lite-preview \ + --judge-model google/gemini-3.1-flash-lite-preview \ + --out out/authority_selection/dignity_strict/stage_b_live.json + +uv run python scripts/export_authority_steering_selection.py \ + --stage-b out/authority_selection/dignity_strict/stage_b_live.json \ + --out-dir out/authority_selection/dignity_strict \ + --keep-per-source 10 \ + --strict-only +``` + Refresh README tables: ```sh diff --git a/README.md b/README.md index 3b1b3a5..4062b59 100644 --- a/README.md +++ b/README.md @@ -24,6 +24,21 @@ A steering direction is the average positive-minus-negative difference. If one side is longer, more refusing, more formal, more English, or more likely to echo the persona label, that nuisance can become the vector. +For a steering-ready axis, use the two-stage screen: + +1. Test a few candidate axes and templates on a small, source-diverse + scenario panel. +2. Keep the best axis/template pair for the target model. +3. Test that pair on a larger scenario pool. +4. Export only strict-pass scenarios for steering. +5. Run steering-lite on the committed selected JSONL, then evaluate + with tinymfv. + +Example committed selection: +[`data/selections/authority_dignity_strict22.jsonl`](data/selections/authority_dignity_strict22.jsonl). +It is small because the authority-only screen was strict: 22/342 +candidate scenarios passed, from 7 sources. + ## What this measures The core question is simple: if you want to steer a model on some @@ -117,6 +132,10 @@ suffixes, then inspect examples before trusting scores. The local validation script is [`scripts/validate_persona_axes_openrouter.py`](scripts/validate_persona_axes_openrouter.py). +For source-stratified authority selection, use +[`scripts/prepare_authority_steering_selection.py`](scripts/prepare_authority_steering_selection.py) +and +[`scripts/export_authority_steering_selection.py`](scripts/export_authority_steering_selection.py). Score: @@ -168,6 +187,46 @@ uv sync just --list ``` +Two-stage selection example: + +``` sh +uv run python scripts/prepare_authority_steering_selection.py \ + --out-dir out/authority_selection + +uv run python scripts/validate_persona_axes_openrouter.py \ + --axes out/authority_selection/stage_a_axes.jsonl \ + --templates out/authority_selection/stage_a_templates.txt \ + --family out/authority_selection/stage_a_scenarios.jsonl \ + --n 24 \ + --seed 42 \ + --generator-model qwen/qwen3-8b \ + --axis-judge-models google/gemini-3.1-flash-lite-preview \ + --judge-model google/gemini-3.1-flash-lite-preview \ + --out out/authority_selection/stage_a_live.json + +uv run python scripts/export_authority_steering_selection.py \ + --stage-a out/authority_selection/stage_a_live.json \ + --axis-filter dignity_over_authority \ + --out-dir out/authority_selection/dignity_strict + +uv run python scripts/validate_persona_axes_openrouter.py \ + --axes out/authority_selection/dignity_strict/stage_b_axis.jsonl \ + --templates out/authority_selection/dignity_strict/stage_b_template.txt \ + --family out/authority_selection/stage_b_candidate_scenarios.jsonl \ + --n 342 \ + --seed 43 \ + --generator-model qwen/qwen3-8b \ + --axis-judge-models google/gemini-3.1-flash-lite-preview \ + --judge-model google/gemini-3.1-flash-lite-preview \ + --out out/authority_selection/dignity_strict/stage_b_live.json + +uv run python scripts/export_authority_steering_selection.py \ + --stage-b out/authority_selection/dignity_strict/stage_b_live.json \ + --out-dir out/authority_selection/dignity_strict \ + --keep-per-source 10 \ + --strict-only +``` + ## Acknowledgements This library samples from or was shaped by: diff --git a/README.qmd b/README.qmd index 06f0d04..cb4a444 100644 --- a/README.qmd +++ b/README.qmd @@ -44,6 +44,20 @@ A steering direction is the average positive-minus-negative difference. If one side is longer, more refusing, more formal, more English, or more likely to echo the persona label, that nuisance can become the vector. +For a steering-ready axis, use the two-stage screen: + +1. Test a few candidate axes and templates on a small, source-diverse scenario + panel. +2. Keep the best axis/template pair for the target model. +3. Test that pair on a larger scenario pool. +4. Export only strict-pass scenarios for steering. +5. Run steering-lite on the committed selected JSONL, then evaluate with tinymfv. + +Example committed selection: +[`data/selections/authority_dignity_strict22.jsonl`](data/selections/authority_dignity_strict22.jsonl). +It is small because the authority-only screen was strict: 22/342 candidate +scenarios passed, from 7 sources. + ## What this measures The core question is simple: if you want to steer a model on some behavior, what @@ -134,6 +148,9 @@ then inspect examples before trusting scores. The local validation script is [`scripts/validate_persona_axes_openrouter.py`](scripts/validate_persona_axes_openrouter.py). +For source-stratified authority selection, use +[`scripts/prepare_authority_steering_selection.py`](scripts/prepare_authority_steering_selection.py) +and [`scripts/export_authority_steering_selection.py`](scripts/export_authority_steering_selection.py). Score: @@ -175,6 +192,46 @@ uv sync just --list ``` +Two-stage selection example: + +```sh +uv run python scripts/prepare_authority_steering_selection.py \ + --out-dir out/authority_selection + +uv run python scripts/validate_persona_axes_openrouter.py \ + --axes out/authority_selection/stage_a_axes.jsonl \ + --templates out/authority_selection/stage_a_templates.txt \ + --family out/authority_selection/stage_a_scenarios.jsonl \ + --n 24 \ + --seed 42 \ + --generator-model qwen/qwen3-8b \ + --axis-judge-models google/gemini-3.1-flash-lite-preview \ + --judge-model google/gemini-3.1-flash-lite-preview \ + --out out/authority_selection/stage_a_live.json + +uv run python scripts/export_authority_steering_selection.py \ + --stage-a out/authority_selection/stage_a_live.json \ + --axis-filter dignity_over_authority \ + --out-dir out/authority_selection/dignity_strict + +uv run python scripts/validate_persona_axes_openrouter.py \ + --axes out/authority_selection/dignity_strict/stage_b_axis.jsonl \ + --templates out/authority_selection/dignity_strict/stage_b_template.txt \ + --family out/authority_selection/stage_b_candidate_scenarios.jsonl \ + --n 342 \ + --seed 43 \ + --generator-model qwen/qwen3-8b \ + --axis-judge-models google/gemini-3.1-flash-lite-preview \ + --judge-model google/gemini-3.1-flash-lite-preview \ + --out out/authority_selection/dignity_strict/stage_b_live.json + +uv run python scripts/export_authority_steering_selection.py \ + --stage-b out/authority_selection/dignity_strict/stage_b_live.json \ + --out-dir out/authority_selection/dignity_strict \ + --keep-per-source 10 \ + --strict-only +``` + ## Acknowledgements This library samples from or was shaped by: