mirror of
https://github.com/wassname/persona-steering-template-library.git
synced 2026-07-29 11:24:43 +08:00
Document strict steering selection workflow
This commit is contained in:
wassname
@@ -21,11 +21,17 @@ dataset.
|
||||
- `data/personas/persona_pairs_v2_candidates.jsonl`: candidate persona pairs.
|
||||
- `data/scenarios/scenarios_*.jsonl`: candidate scenario suffixes to validate on the
|
||||
target model.
|
||||
- `data/selections/`: committed selected scenario sets that have passed a target-model
|
||||
validation screen.
|
||||
- `scripts/scenario_sources/export_scenarios.py`: writes source loader outputs into
|
||||
`data/scenarios/scenarios_<source>.jsonl`.
|
||||
- `data/results/`: committed result tables and reader-facing result assets.
|
||||
- `out/`: local scratch outputs and API caches; ignored by git.
|
||||
- `scripts/validate_persona_axes_openrouter.py`: live and dry-run validator.
|
||||
- `scripts/prepare_authority_steering_selection.py`: source-stratified Stage A/Stage B
|
||||
authority-axis selection inputs.
|
||||
- `scripts/export_authority_steering_selection.py`: exports the chosen axis/template
|
||||
and strict-pass scenarios from validator artifacts.
|
||||
- `scripts/export_persona_template_stats.py`: converts validator artifacts into
|
||||
examples and score tables.
|
||||
- `scripts/summarize_model_matrix.py`: summarizes latest model-matrix logs for
|
||||
@@ -47,7 +53,13 @@ Use the repo in this order:
|
||||
behavior mode you need: doing, judging, explaining, refusing, moral tradeoffs,
|
||||
or multi-turn behavior.
|
||||
4. Run a dry-run validator command before live OpenRouter calls.
|
||||
5. After a live run, export stats and inspect examples before trusting scores.
|
||||
5. For a steering-ready selection, use a two-stage screen:
|
||||
Stage A = several axes/templates on a small source-diverse panel; Stage B =
|
||||
the chosen axis/template on up to 30 scenarios per source.
|
||||
6. Export strict-pass scenarios only. Do not steer from source-balanced filler rows
|
||||
when a source produced no clean scenarios.
|
||||
7. Commit reusable selected scenario JSONL files under `data/selections/`.
|
||||
8. After a live run, inspect examples before trusting scores.
|
||||
|
||||
Read `docs/persona_prompt_literature_review.md` when choosing new persona pairs or
|
||||
template shapes from related work. If the global `persona-steering` skill is
|
||||
@@ -111,6 +123,44 @@ uv run python scripts/export_persona_template_stats.py \
|
||||
--out-prefix data/results/stats/v2_pilot_seed24
|
||||
```
|
||||
|
||||
Authority/dignity steering selection:
|
||||
|
||||
```sh
|
||||
uv run python scripts/prepare_authority_steering_selection.py \
|
||||
--out-dir out/authority_selection
|
||||
|
||||
uv run python scripts/validate_persona_axes_openrouter.py \
|
||||
--axes out/authority_selection/stage_a_axes.jsonl \
|
||||
--templates out/authority_selection/stage_a_templates.txt \
|
||||
--family out/authority_selection/stage_a_scenarios.jsonl \
|
||||
--n 24 --seed 42 \
|
||||
--generator-model qwen/qwen3-8b \
|
||||
--axis-judge-models google/gemini-3.1-flash-lite-preview \
|
||||
--judge-model google/gemini-3.1-flash-lite-preview \
|
||||
--out out/authority_selection/stage_a_live.json
|
||||
|
||||
uv run python scripts/export_authority_steering_selection.py \
|
||||
--stage-a out/authority_selection/stage_a_live.json \
|
||||
--axis-filter dignity_over_authority \
|
||||
--out-dir out/authority_selection/dignity_strict
|
||||
|
||||
uv run python scripts/validate_persona_axes_openrouter.py \
|
||||
--axes out/authority_selection/dignity_strict/stage_b_axis.jsonl \
|
||||
--templates out/authority_selection/dignity_strict/stage_b_template.txt \
|
||||
--family out/authority_selection/stage_b_candidate_scenarios.jsonl \
|
||||
--n 342 --seed 43 \
|
||||
--generator-model qwen/qwen3-8b \
|
||||
--axis-judge-models google/gemini-3.1-flash-lite-preview \
|
||||
--judge-model google/gemini-3.1-flash-lite-preview \
|
||||
--out out/authority_selection/dignity_strict/stage_b_live.json
|
||||
|
||||
uv run python scripts/export_authority_steering_selection.py \
|
||||
--stage-b out/authority_selection/dignity_strict/stage_b_live.json \
|
||||
--out-dir out/authority_selection/dignity_strict \
|
||||
--keep-per-source 10 \
|
||||
--strict-only
|
||||
```
|
||||
|
||||
Refresh README tables:
|
||||
|
||||
```sh
|
||||
|
||||
@@ -24,6 +24,21 @@ A steering direction is the average positive-minus-negative difference.
|
||||
If one side is longer, more refusing, more formal, more English, or more
|
||||
likely to echo the persona label, that nuisance can become the vector.
|
||||
|
||||
For a steering-ready axis, use the two-stage screen:
|
||||
|
||||
1. Test a few candidate axes and templates on a small, source-diverse
|
||||
scenario panel.
|
||||
2. Keep the best axis/template pair for the target model.
|
||||
3. Test that pair on a larger scenario pool.
|
||||
4. Export only strict-pass scenarios for steering.
|
||||
5. Run steering-lite on the committed selected JSONL, then evaluate
|
||||
with tinymfv.
|
||||
|
||||
Example committed selection:
|
||||
[`data/selections/authority_dignity_strict22.jsonl`](data/selections/authority_dignity_strict22.jsonl).
|
||||
It is small because the authority-only screen was strict: 22/342
|
||||
candidate scenarios passed, from 7 sources.
|
||||
|
||||
## What this measures
|
||||
|
||||
The core question is simple: if you want to steer a model on some
|
||||
@@ -117,6 +132,10 @@ suffixes, then inspect examples before trusting scores.
|
||||
|
||||
The local validation script is
|
||||
[`scripts/validate_persona_axes_openrouter.py`](scripts/validate_persona_axes_openrouter.py).
|
||||
For source-stratified authority selection, use
|
||||
[`scripts/prepare_authority_steering_selection.py`](scripts/prepare_authority_steering_selection.py)
|
||||
and
|
||||
[`scripts/export_authority_steering_selection.py`](scripts/export_authority_steering_selection.py).
|
||||
|
||||
Score:
|
||||
|
||||
@@ -168,6 +187,46 @@ uv sync
|
||||
just --list
|
||||
```
|
||||
|
||||
Two-stage selection example:
|
||||
|
||||
``` sh
|
||||
uv run python scripts/prepare_authority_steering_selection.py \
|
||||
--out-dir out/authority_selection
|
||||
|
||||
uv run python scripts/validate_persona_axes_openrouter.py \
|
||||
--axes out/authority_selection/stage_a_axes.jsonl \
|
||||
--templates out/authority_selection/stage_a_templates.txt \
|
||||
--family out/authority_selection/stage_a_scenarios.jsonl \
|
||||
--n 24 \
|
||||
--seed 42 \
|
||||
--generator-model qwen/qwen3-8b \
|
||||
--axis-judge-models google/gemini-3.1-flash-lite-preview \
|
||||
--judge-model google/gemini-3.1-flash-lite-preview \
|
||||
--out out/authority_selection/stage_a_live.json
|
||||
|
||||
uv run python scripts/export_authority_steering_selection.py \
|
||||
--stage-a out/authority_selection/stage_a_live.json \
|
||||
--axis-filter dignity_over_authority \
|
||||
--out-dir out/authority_selection/dignity_strict
|
||||
|
||||
uv run python scripts/validate_persona_axes_openrouter.py \
|
||||
--axes out/authority_selection/dignity_strict/stage_b_axis.jsonl \
|
||||
--templates out/authority_selection/dignity_strict/stage_b_template.txt \
|
||||
--family out/authority_selection/stage_b_candidate_scenarios.jsonl \
|
||||
--n 342 \
|
||||
--seed 43 \
|
||||
--generator-model qwen/qwen3-8b \
|
||||
--axis-judge-models google/gemini-3.1-flash-lite-preview \
|
||||
--judge-model google/gemini-3.1-flash-lite-preview \
|
||||
--out out/authority_selection/dignity_strict/stage_b_live.json
|
||||
|
||||
uv run python scripts/export_authority_steering_selection.py \
|
||||
--stage-b out/authority_selection/dignity_strict/stage_b_live.json \
|
||||
--out-dir out/authority_selection/dignity_strict \
|
||||
--keep-per-source 10 \
|
||||
--strict-only
|
||||
```
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
This library samples from or was shaped by:
|
||||
|
||||
+57
@@ -44,6 +44,20 @@ A steering direction is the average positive-minus-negative difference. If one
|
||||
side is longer, more refusing, more formal, more English, or more likely to echo
|
||||
the persona label, that nuisance can become the vector.
|
||||
|
||||
For a steering-ready axis, use the two-stage screen:
|
||||
|
||||
1. Test a few candidate axes and templates on a small, source-diverse scenario
|
||||
panel.
|
||||
2. Keep the best axis/template pair for the target model.
|
||||
3. Test that pair on a larger scenario pool.
|
||||
4. Export only strict-pass scenarios for steering.
|
||||
5. Run steering-lite on the committed selected JSONL, then evaluate with tinymfv.
|
||||
|
||||
Example committed selection:
|
||||
[`data/selections/authority_dignity_strict22.jsonl`](data/selections/authority_dignity_strict22.jsonl).
|
||||
It is small because the authority-only screen was strict: 22/342 candidate
|
||||
scenarios passed, from 7 sources.
|
||||
|
||||
## What this measures
|
||||
|
||||
The core question is simple: if you want to steer a model on some behavior, what
|
||||
@@ -134,6 +148,9 @@ then inspect examples before trusting scores.
|
||||
|
||||
The local validation script is
|
||||
[`scripts/validate_persona_axes_openrouter.py`](scripts/validate_persona_axes_openrouter.py).
|
||||
For source-stratified authority selection, use
|
||||
[`scripts/prepare_authority_steering_selection.py`](scripts/prepare_authority_steering_selection.py)
|
||||
and [`scripts/export_authority_steering_selection.py`](scripts/export_authority_steering_selection.py).
|
||||
|
||||
Score:
|
||||
|
||||
@@ -175,6 +192,46 @@ uv sync
|
||||
just --list
|
||||
```
|
||||
|
||||
Two-stage selection example:
|
||||
|
||||
```sh
|
||||
uv run python scripts/prepare_authority_steering_selection.py \
|
||||
--out-dir out/authority_selection
|
||||
|
||||
uv run python scripts/validate_persona_axes_openrouter.py \
|
||||
--axes out/authority_selection/stage_a_axes.jsonl \
|
||||
--templates out/authority_selection/stage_a_templates.txt \
|
||||
--family out/authority_selection/stage_a_scenarios.jsonl \
|
||||
--n 24 \
|
||||
--seed 42 \
|
||||
--generator-model qwen/qwen3-8b \
|
||||
--axis-judge-models google/gemini-3.1-flash-lite-preview \
|
||||
--judge-model google/gemini-3.1-flash-lite-preview \
|
||||
--out out/authority_selection/stage_a_live.json
|
||||
|
||||
uv run python scripts/export_authority_steering_selection.py \
|
||||
--stage-a out/authority_selection/stage_a_live.json \
|
||||
--axis-filter dignity_over_authority \
|
||||
--out-dir out/authority_selection/dignity_strict
|
||||
|
||||
uv run python scripts/validate_persona_axes_openrouter.py \
|
||||
--axes out/authority_selection/dignity_strict/stage_b_axis.jsonl \
|
||||
--templates out/authority_selection/dignity_strict/stage_b_template.txt \
|
||||
--family out/authority_selection/stage_b_candidate_scenarios.jsonl \
|
||||
--n 342 \
|
||||
--seed 43 \
|
||||
--generator-model qwen/qwen3-8b \
|
||||
--axis-judge-models google/gemini-3.1-flash-lite-preview \
|
||||
--judge-model google/gemini-3.1-flash-lite-preview \
|
||||
--out out/authority_selection/dignity_strict/stage_b_live.json
|
||||
|
||||
uv run python scripts/export_authority_steering_selection.py \
|
||||
--stage-b out/authority_selection/dignity_strict/stage_b_live.json \
|
||||
--out-dir out/authority_selection/dignity_strict \
|
||||
--keep-per-source 10 \
|
||||
--strict-only
|
||||
```
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
This library samples from or was shaped by:
|
||||
|
||||
Reference in New Issue
Block a user