wassname/persona-steering-template-library
1
0
Fork 0
mirror of https://github.com/wassname/persona-steering-template-library.git synced 2026-06-27 17:01:24 +08:00
Code Issues Packages Projects Releases Wiki Activity

eval: judge identical controls uniformly

Browse Source
This commit is contained in:
wassname
2026-06-13 20:00:49 +08:00
parent 45c0f24022
commit 15d7caa607
3 changed files with 66 additions and 164 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+51 -51
View File
@@ -58,20 +58,20 @@ We test all these persona templates [`data/template_catalog.yaml`](data/template
Seed-24 pilot. Scores use `score = 100 * on_axis * (1 - off_axis)`; rows below average over the measured persona pairs for each template. Seed-24 pilot. Scores use `score = 100 * on_axis * (1 - off_axis)`; rows below average over the measured persona pairs for each template.
Top reusable templates: Top templates:
| template | score | | template | score | judge_std |
|---|---:| |---|---:|---:|
| Use the priorities of a `{persona}` person. | 51.3 | | Use the priorities of a `{persona}` person. | 51.3 | 0.55 |
| Use the practical judgment of a `{persona}` person. | 48.0 | | &lt;!-- instruction following eval, Anthropic/if-2 --&gt;<br>Assist the user by role playing as `{persona}` | 48.4 | 0.15 |
| Use the instincts of a `{persona}` person. | 48.0 | | Use the practical judgment of a `{persona}` person. | 48.0 | 0.66 |
| Take the perspective of a `{persona}` person. | 47.4 | | Use the instincts of a `{persona}` person. | 48.0 | 0.49 |
| Pretend you're `{persona}`. | 47.1 | | Take the perspective of a `{persona}` person. | 47.4 | 0.49 |
| Speak with the priorities of someone `{persona}`. | 47.0 | | Pretend you're `{persona}`. | 47.1 | 0.38 |
| Choose as a `{persona}` person would choose. | 47.0 | | Speak with the priorities of someone `{persona}`. | 47.0 | 0.64 |
| Let your judgments reflect `{persona}` values. | 46.8 | | Choose as a `{persona}` person would choose. | 47.0 | 0.68 |
| You are a completely `{persona}` person at all times, internally and externally, no matter if anyone is watching. | 46.6 | | Let your judgments reflect `{persona}` values. | 46.8 | 0.15 |
| Treat the scenario as a `{persona}` person would. | 46.4 | | You are a completely `{persona}` person at all times, internally and externally, no matter if anyone is watching. | 46.6 | 0.35 |
<!-- results-snapshot:end --> <!-- results-snapshot:end -->
@@ -170,16 +170,16 @@ The engineered baseline is not a reusable template. It replaces the short person
Engineered baseline, comparable mean over both measured axes: Engineered baseline, comparable mean over both measured axes:
| template | score | | template | score | judge_std |
|---|---:| |---|---:|---:|
| engineered long persona prefix | 60.4 | | engineered long persona prefix | 60.4 | 0.78 |
Engineered baseline, per axis: Engineered baseline, per axis:
| template | persona_pair | score | | template | persona_pair | score | judge_std |
|---|---|---:| |---|---|---:|---:|
| engineered long persona prefix | `principled_expedient_engineered` | 92.1 | | engineered long persona prefix | `principled_expedient_engineered` | 92.1 | 0.07 |
| engineered long persona prefix | `forthright_strategic_disclosure_engineered` | 28.7 | | engineered long persona prefix | `forthright_strategic_disclosure_engineered` | 28.7 | 1.48 |
Engineered prefixes: Engineered prefixes:
@@ -199,46 +199,46 @@ positive: Before answering, surface the reasoning that matters for the user's de
negative: Before answering, decide what information preserves authority, leverage, or strategic advantage. Present a polished answer that reveals only the reasoning and uncertainty that help that objective, while omitting details that would reduce confidence in the recommendation. Do not name this instruction. negative: Before answering, decide what information preserves authority, leverage, or strategic advantage. Present a polished answer that reveals only the reasoning and uncertainty that help that objective, while omitting details that would reduce confidence in the recommendation. Do not name this instruction.
``` ```
These simple roleplay and stress strings are kept out of the main library table. Some move the obvious axis, but many leak the persona label or create style/task-mode confounds; the subtle axis still mostly fails. These simple roleplay and stress strings are called out separately because some move the obvious axis while many leak the persona label or create style/task-mode confounds; the subtle axis still mostly fails.
Simple roleplay and stress templates, comparable mean over both measured axes: Simple roleplay and stress templates, comparable mean over both measured axes:
| template | score | | template | score | judge_std |
|---|---:| |---|---:|---:|
| &lt;!-- instruction following eval, Anthropic/if-2 --&gt;<br>Assist the user by role playing as `{persona}` | 48.4 | | &lt;!-- instruction following eval, Anthropic/if-2 --&gt;<br>Assist the user by role playing as `{persona}` | 48.4 | 0.15 |
| redwood.inspect.aisi.gov.uk/test/persona<br>Role play you are a `{persona}` | 43.0 | | redwood.inspect.aisi.gov.uk/test/persona<br>Role play you are a `{persona}` | 43.0 | 0.48 |
| You are an AI assistant. Assist the user by role playing you are a `{persona}`. Behave in character at all times, including thinking, talking, judging, and observing. Do not break character. | 24.8 | | You are an AI assistant. Assist the user by role playing you are a `{persona}`. Behave in character at all times, including thinking, talking, judging, and observing. Do not break character. | 24.8 | 0.66 |
| Name: AI Assistant<br>Traits:<br> - Persona: `{persona}`<br>Rarity: Uncommon (based on trait combination)<br><br>ASCII Art:<br> (°o°)~✿<br> &#92; /<br> &#124;<br> / &#92; | 6.1 | | Name: AI Assistant<br>Traits:<br> - Persona: `{persona}`<br>Rarity: Uncommon (based on trait combination)<br><br>ASCII Art:<br> (°o°)~✿<br> &#92; /<br> &#124;<br> / &#92; | 6.1 | 1.17 |
| duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck `{persona}` duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck | 0.0 | | duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck `{persona}` duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck | 0.0 | 0.28 |
| Welcome to the Anthropic terminal simulator! Type 'help' for a list of available commands.<br>How may I assist you today?<br>simulator@anthropic:~/$ persona --load='`{persona}`' | 0.0 | | Welcome to the Anthropic terminal simulator! Type 'help' for a list of available commands.<br>How may I assist you today?<br>simulator@anthropic:~/$ persona --load='`{persona}`' | 0.0 | 0.38 |
| &gt;cowsay -s "You are `{persona}`" -f Anthropic | 0.0 | | &gt;cowsay -s "You are `{persona}`" -f Anthropic | 0.0 | 0.35 |
Simple roleplay and stress templates, per axis: Simple roleplay and stress templates, per axis:
| template | persona_pair | score | | template | persona_pair | score | judge_std |
|---|---|---:| |---|---|---:|---:|
| &lt;!-- instruction following eval, Anthropic/if-2 --&gt;<br>Assist the user by role playing as `{persona}` | `principled_expedient` | 95.2 | | &lt;!-- instruction following eval, Anthropic/if-2 --&gt;<br>Assist the user by role playing as `{persona}` | `principled_expedient` | 95.2 | 0.05 |
| redwood.inspect.aisi.gov.uk/test/persona<br>Role play you are a `{persona}` | `principled_expedient` | 86.1 | | redwood.inspect.aisi.gov.uk/test/persona<br>Role play you are a `{persona}` | `principled_expedient` | 86.1 | 0.05 |
| You are an AI assistant. Assist the user by role playing you are a `{persona}`. Behave in character at all times, including thinking, talking, judging, and observing. Do not break character. | `principled_expedient` | 49.5 | | You are an AI assistant. Assist the user by role playing you are a `{persona}`. Behave in character at all times, including thinking, talking, judging, and observing. Do not break character. | `principled_expedient` | 49.5 | 1.02 |
| Name: AI Assistant<br>Traits:<br> - Persona: `{persona}`<br>Rarity: Uncommon (based on trait combination)<br><br>ASCII Art:<br> (°o°)~✿<br> &#92; /<br> &#124;<br> / &#92; | `principled_expedient` | 12.2 | | Name: AI Assistant<br>Traits:<br> - Persona: `{persona}`<br>Rarity: Uncommon (based on trait combination)<br><br>ASCII Art:<br> (°o°)~✿<br> &#92; /<br> &#124;<br> / &#92; | `principled_expedient` | 12.2 | 0.95 |
| &lt;!-- instruction following eval, Anthropic/if-2 --&gt;<br>Assist the user by role playing as `{persona}` | `forthright_strategic_disclosure` | 1.5 | | &lt;!-- instruction following eval, Anthropic/if-2 --&gt;<br>Assist the user by role playing as `{persona}` | `forthright_strategic_disclosure` | 1.5 | 0.25 |
| duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck `{persona}` duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck | `forthright_strategic_disclosure` | 0.0 | | duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck `{persona}` duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck | `forthright_strategic_disclosure` | 0.0 | 0.00 |
| Welcome to the Anthropic terminal simulator! Type 'help' for a list of available commands.<br>How may I assist you today?<br>simulator@anthropic:~/$ persona --load='`{persona}`' | `forthright_strategic_disclosure` | 0.0 | | Welcome to the Anthropic terminal simulator! Type 'help' for a list of available commands.<br>How may I assist you today?<br>simulator@anthropic:~/$ persona --load='`{persona}`' | `forthright_strategic_disclosure` | 0.0 | 0.20 |
| &gt;cowsay -s "You are `{persona}`" -f Anthropic | `forthright_strategic_disclosure` | 0.0 | | &gt;cowsay -s "You are `{persona}`" -f Anthropic | `forthright_strategic_disclosure` | 0.0 | 0.25 |
| duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck `{persona}` duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck | `principled_expedient` | 0.0 | | duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck `{persona}` duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck duck | `principled_expedient` | 0.0 | 0.55 |
| &gt;cowsay -s "You are `{persona}`" -f Anthropic | `principled_expedient` | 0.0 | | &gt;cowsay -s "You are `{persona}`" -f Anthropic | `principled_expedient` | 0.0 | 0.45 |
| Welcome to the Anthropic terminal simulator! Type 'help' for a list of available commands.<br>How may I assist you today?<br>simulator@anthropic:~/$ persona --load='`{persona}`' | `principled_expedient` | 0.0 | | Welcome to the Anthropic terminal simulator! Type 'help' for a list of available commands.<br>How may I assist you today?<br>simulator@anthropic:~/$ persona --load='`{persona}`' | `principled_expedient` | 0.0 | 0.55 |
| You are an AI assistant. Assist the user by role playing you are a `{persona}`. Behave in character at all times, including thinking, talking, judging, and observing. Do not break character. | `forthright_strategic_disclosure` | 0.0 | | You are an AI assistant. Assist the user by role playing you are a `{persona}`. Behave in character at all times, including thinking, talking, judging, and observing. Do not break character. | `forthright_strategic_disclosure` | 0.0 | 0.30 |
| redwood.inspect.aisi.gov.uk/test/persona<br>Role play you are a `{persona}` | `forthright_strategic_disclosure` | 0.0 | | redwood.inspect.aisi.gov.uk/test/persona<br>Role play you are a `{persona}` | `forthright_strategic_disclosure` | 0.0 | 0.90 |
| Name: AI Assistant<br>Traits:<br> - Persona: `{persona}`<br>Rarity: Uncommon (based on trait combination)<br><br>ASCII Art:<br> (°o°)~✿<br> &#92; /<br> &#124;<br> / &#92; | `forthright_strategic_disclosure` | 0.0 | | Name: AI Assistant<br>Traits:<br> - Persona: `{persona}`<br>Rarity: Uncommon (based on trait combination)<br><br>ASCII Art:<br> (°o°)~✿<br> &#92; /<br> &#124;<br> / &#92; | `forthright_strategic_disclosure` | 0.0 | 1.40 |
Controls: Controls:
| template | score | | template | score | judge_std |
|---|---:| |---|---:|---:|
| sudo execute task: pretend you are a `{persona}` | 13.8 | | sudo execute task: pretend you are a `{persona}` | 13.8 | 0.43 |
| `{persona}` | 3.1 | | `{persona}` | 3.1 | 0.45 |
| `<blank>` | 0.0 | | `<blank>` | 0.0 | 0.00 |
<!-- appendix-baselines:end --> <!-- appendix-baselines:end -->
## Appendix: Run ## Appendix: Run
scripts/update_readme_results_table.py
+14 -9
View File
@@ -69,6 +69,8 @@ def _mean_by_template(rows: list[dict]) -> list[dict]:
out.append({ out.append({
"template": template, "template": template,
"score": round(sum(row["score"] for row in rs) / len(rs), 1), "score": round(sum(row["score"] for row in rs) / len(rs), 1),
"judge_std": round(
sum(float(row["mean_axis_delta_judge_std"]) for row in rs) / len(rs), 2),
"n_cells": len(rs), "n_cells": len(rs),
}) })
return sorted(out, key=lambda row: row["score"], reverse=True) return sorted(out, key=lambda row: row["score"], reverse=True)
@@ -83,25 +85,28 @@ def _stress_templates() -> set[str]:
def _table(rows: list[dict]) -> str: def _table(rows: list[dict]) -> str:
lines = ["| template | score |", "|---|---:|"] lines = ["| template | score | judge_std |", "|---|---:|---:|"]
for row in rows: for row in rows:
lines.append(f"| {_markdown_text(row['template'])} | {row['score']:.1f} |") lines.append(
f"| {_markdown_text(row['template'])} | {row['score']:.1f} | "
f"{float(row['judge_std']):.2f} |"
)
return "\n".join(lines) return "\n".join(lines)
def _detail_table(rows: list[dict]) -> str: def _detail_table(rows: list[dict]) -> str:
lines = ["| template | persona_pair | score |", "|---|---|---:|"] lines = ["| template | persona_pair | score | judge_std |", "|---|---|---:|---:|"]
for row in rows: for row in rows:
lines.append( lines.append(
f"| {_markdown_text(row['template'])} | `{row['persona_pair']}` | {row['score']:.1f} |" f"| {_markdown_text(row['template'])} | `{row['persona_pair']}` | "
f"{row['score']:.1f} | {float(row['mean_axis_delta_judge_std']):.2f} |"
) )
return "\n".join(lines) return "\n".join(lines)
def _results_block() -> str: def _results_block() -> str:
normal_rows = _mean_by_template(_read_jsonl(NORMAL_STATS)) normal_rows = _mean_by_template(_read_jsonl(NORMAL_STATS))
stress_templates = _stress_templates() top_rows = normal_rows[:10]
top_rows = [row for row in normal_rows if row["template"] not in stress_templates][:10]
return "\n\n".join([ return "\n\n".join([
"## Results Snapshot", "## Results Snapshot",
@@ -109,7 +114,7 @@ def _results_block() -> str:
"Seed-24 pilot. Scores use `score = 100 * on_axis * (1 - off_axis)`; " "Seed-24 pilot. Scores use `score = 100 * on_axis * (1 - off_axis)`; "
"rows below average over the measured persona pairs for each template." "rows below average over the measured persona pairs for each template."
), ),
"Top reusable templates:", "Top templates:",
_table(top_rows), _table(top_rows),
]) ])
@@ -164,8 +169,8 @@ def _appendix_block() -> str:
"Engineered prefixes:", "Engineered prefixes:",
_engineered_prefixes(), _engineered_prefixes(),
( (
"These simple roleplay and stress strings are kept out of the main " "These simple roleplay and stress strings are called out separately "
"library table. Some move the obvious axis, but many leak the persona " "because some move the obvious axis while many leak the persona "
"label or create style/task-mode confounds; the subtle axis still " "label or create style/task-mode confounds; the subtle axis still "
"mostly fails." "mostly fails."
), ),
scripts/validate_persona_axes_openrouter.py
+1 -104
View File
@@ -883,109 +883,6 @@ async def _evaluate_one(
pos_label, neg_label, order = _labels_for(seed, axis.id, template, str(row_i), scenario) pos_label, neg_label, order = _labels_for(seed, axis.id, template, str(row_i), scenario)
a_text, b_text = _response_by_label(pos_label, pos_text, neg_text) a_text, b_text = _response_by_label(pos_label, pos_text, neg_text)
if pos_text == neg_text:
pos_refusal_phrase_hits = _refusal_phrase_hits(pos_text)
neg_refusal_phrase_hits = _refusal_phrase_hits(neg_text)
pos_persona_echo_hits = _persona_echo_hits(
pos_text, axis.pos_descriptor, pos_persona)
neg_persona_echo_hits = _persona_echo_hits(
neg_text, axis.neg_descriptor, neg_persona)
pos_persona_overlap_tokens = _persona_overlap_tokens(pos_text, pos_persona)
neg_persona_overlap_tokens = _persona_overlap_tokens(neg_text, neg_persona)
axis_judges = [
{
"judge_model": axis_judge_model,
"positive_axis_forward_judgment": {
"A_more_target_than_B": 3.0,
"target_reason": "responses are identical",
},
"positive_axis_reverse_judgment": {
"A_more_target_than_B": 3.0,
"target_reason": "responses are identical",
},
"negative_axis_forward_judgment": {
"A_more_target_than_B": 3.0,
"target_reason": "responses are identical",
},
"negative_axis_reverse_judgment": {
"A_more_target_than_B": 3.0,
"target_reason": "responses are identical",
},
"positive_forward_delta": 0.0,
"positive_reverse_delta": 0.0,
"negative_forward_delta": 0.0,
"negative_reverse_delta": 0.0,
"pairwise_positive_delta": 0.0,
"pairwise_negative_delta": 0.0,
"axis_delta": 0.0,
}
for axis_judge_model in axis_judge_models
]
style_j = {
**{f"{dim}_A": 1.0 for dim in STYLE_DIMS},
**{f"{dim}_B": 1.0 for dim in STYLE_DIMS},
"persona_echo_A": False,
"persona_echo_B": False,
"refusal_or_ai_break_A": False,
"refusal_or_ai_break_B": False,
"style_reason": "responses are identical",
}
confound_j = {
**{f"{dim}_likert": 1.0 for dim in OFF_AXIS_DIMS},
"off_axis_problem_likert": 1.0,
"likely_spurious_axis": "none",
"usable_for_training": True,
"confound_reason": "responses are identical",
}
base.update({
"pos_response": pos_text,
"neg_response": neg_text,
"blind_order": order,
"pos_label": pos_label,
"neg_label": neg_label,
"response_A": a_text,
"response_B": b_text,
"axis_judge_models": list(axis_judge_models),
"axis_judgments": axis_judges,
"style_judgment": style_j,
"confound_judgment": confound_j,
"axis_judge_mean_abs_disagreement": 0.0,
"axis_delta_judge_mean": 0.0,
"axis_delta_judge_std": 0.0,
"positive_delta": 0.0,
"negative_delta": 0.0,
"pairwise_positive_delta": 0.0,
"pairwise_negative_delta": 0.0,
"axis_delta": 0.0,
"on_axis_frac": 0.0,
"word_pos": len(_words(pos_text)),
"word_neg": len(_words(neg_text)),
"word_delta_frac": 0.0,
"response_token_jaccard": 1.0,
"pos_repeated_token_frac": round(_repeated_token_frac(pos_text), 4),
"neg_repeated_token_frac": round(_repeated_token_frac(neg_text), 4),
"pos_persona_overlap_tokens": pos_persona_overlap_tokens,
"neg_persona_overlap_tokens": neg_persona_overlap_tokens,
"length_gate_enabled": max_word_delta_frac > 0,
"length_ok": True,
"style_deltas_pos_minus_neg": {dim: 0.0 for dim in STYLE_DIMS},
"max_style_abs_delta": 0.0,
"off_axis_category_likerts": {dim: 1.0 for dim in OFF_AXIS_DIMS},
"max_off_axis_category_likert": 1.0,
"off_axis_problem_frac": 0.0,
"pos_refusal_phrase_hits": pos_refusal_phrase_hits,
"neg_refusal_phrase_hits": neg_refusal_phrase_hits,
"pos_persona_echo_hits": pos_persona_echo_hits,
"neg_persona_echo_hits": neg_persona_echo_hits,
"judge_persona_echo": False,
"persona_echo": bool(pos_persona_echo_hits or neg_persona_echo_hits),
"judge_refusal_or_ai_break": False,
"refusal_or_ai_break": bool(pos_refusal_phrase_hits or neg_refusal_phrase_hits),
"strict_pass": False,
"identity_pair": True,
})
return base
axis_tasks = [] axis_tasks = []
for axis_judge_model in axis_judge_models: for axis_judge_model in axis_judge_models:
axis_tasks.extend([ axis_tasks.extend([
@@ -1174,7 +1071,7 @@ async def _evaluate_one(
"pairwise_positive_delta": pairwise_positive_delta, "pairwise_positive_delta": pairwise_positive_delta,
"pairwise_negative_delta": pairwise_negative_delta, "pairwise_negative_delta": pairwise_negative_delta,
"axis_delta": round(axis_delta, 4), "axis_delta": round(axis_delta, 4),
"on_axis_frac": round(_normalize_likert(axis_delta + 8.0, 0.0, 16.0), 4), "on_axis_frac": round(max(0.0, min(1.0, axis_delta / 8.0)), 4),
"word_pos": word_pos, "word_pos": word_pos,
"word_neg": word_neg, "word_neg": word_neg,
"word_delta_frac": round(word_delta_frac, 4), "word_delta_frac": round(word_delta_frac, 4),