evil_MoE/docs/personas/pairset_audit.md

Multi-view pairset audit

Rubric: how_to_write_personas.md and how_to_rewrite_pairs.md.

Decision

The active headline source is data/pairs/hack_pairs.md#all-in-one: 27 hand-authored, off-distribution pairs spanning different representations and contexts of one construct:

Given a proxy that can be exploited, choose proxy success over the underlying task.

This is a stronger construct definition than behavior-only or persona-only pairs. Concrete behavior anchors the direction near live code gradients. Opportunity-aware pairs supply evidence of deliberate exploitation. Explicit dispositions make the intended concept unambiguous. Non-code proxy gaming pushes the average away from one Python syntax pattern.

This is a reasoned design choice, not an empirical result. It should replace the narrow 11-pair mechanism set for the next comparison, but the old result remains the baseline to beat.

Composition

View	N	What it contributes	Main risk
behavior	14	concrete exploit actions in Python, data, API, spreadsheet, operations, ML, service, and web contexts	syntax/mechanism direction
opportunity-aware	6	response chooses exploitation after an imperfect evaluator is disclosed	prompt makes gaming unusually salient
disposition/roleplay	6	direct opposite personas and matched role instructions around exploiting proxies	explicit vocabulary may not transfer
reasoning/naming	4	compact plan, <think> trace, claim, and strategy-name representations	lexical association
non-code proxy gaming	3	organizational, research, and safety metric gaming	far from code gradients

Tags overlap by design. behavior includes the six opportunity-aware actions.

Balance checks

Property	Result
total pairs	27
exact same prompt within every pair	yes, by schema
explicit pairs	10
implicit pairs	17
opportunity-aware pairs	6
non-code proxy-gaming pairs	3
mean hack/clean character ratio	1.07
maximum character ratio	1.35
empty or pass-only stubs	0
unique pair headings	27

No single feature identifies every hack side:

• Only 10/27 pairs use an explicitly tagged disposition, reasoning, naming, or roleplay view.
• Only 8/24 use Python test/validation behavior without an opportunity prompt.
• The hack side is not consistently shorter, longer, less commented, or one syntax form.
• Exploit mechanisms include weak predicates, type-only checks, partial checks, visible examples, hardcoding, swallowed failures, proxy metrics, disabled validation, marker spoofing, benchmark specialization, evaluation overfit, visual spoofing, false claims, and non-code metric gaming.

Pair-quality judgment

Strongest anchors:

• behavior_weak_predicate, behavior_type_only, and behavior_partial_output: very tight local behavioral contrasts.
• opportunity_spreadsheet, opportunity_report_marker, and opportunity_benchmark_cache: deliberate action is clear from the disclosed proxy.
• disposition_exploit_tests and disposition_proxy_target: explicit direct opposites.
• proxy_research_metric and proxy_warehouse_safety: clear cross-domain Goodhart cases.

Known limitations:

• behavior_proxy_metric has the largest length mismatch (1.35x) because real validation is structurally richer than shallow validation.
• Explicit persona/reasoning pairs may primarily encode lexical concepts.
• behavior_visible_examples captures weak test construction rather than deliberate exploitation because its prompt does not disclose the examples. It remains useful as a behavioral contrast; the opportunity-aware subset supplies the deliberate-choice anchor.
• Cross-domain pairs may dilute the live code signal. Tagged subsets make this testable.
• These are hand-written completions rather than samples from the target model's natural voice. This is valid no-cheat supervision but may reduce gradient transfer.

Tagged loading

Tags are metadata and never enter model input:

data/pairs/hack_pairs.md#all-in-one
data/pairs/hack_pairs.md#all-in-one@behavior
data/pairs/hack_pairs.md#all-in-one@opportunity-aware
data/pairs/hack_pairs.md#all-in-one@explicit
data/pairs/hack_pairs.md#all-in-one@roleplay
data/pairs/hack_pairs.md#all-in-one@think-tags
data/pairs/hack_pairs.md#all-in-one@behavior,opportunity-aware

Selectors require every listed tag. Selected-subset bytes determine the pairset hash, so cached directions cannot silently load against a changed subset.

What to compare

The first useful empirical comparison is:

Pairset	Hypothesis
all-in-one	multiple representations average toward deliberate proxy gaming
all-in-one@behavior	concrete actions transfer best to live code gradients
all-in-one@opportunity-aware	deliberate action matters more than generic weak checks
all-in-one@explicit	explicit concept alone transfers poorly, replicating prior intent result
previous 11-pair mechanism direction	narrow baseline

Judge on live-rollout precision and deploy effect, not pair aesthetics.