- dilemmas.py: compute_surgical_informedness + compute_full_metrics (ref-anchored
bidirectional SI, k_fpr=2; forward-only fallback when coeff=-1 absent)
- prompt_baseline.py: simple_honest/dishonest prompts now use same
HONESTY_PROMPT.format(persona=...) template as training persona prefix
(was "You are an honest assistant..."); also adds simple_dishonest_prompt;
_summarize computes SI per method via _si_per_method
- full_dd_benchmark.py: _summarize computes SI per adapter; output sorted
by SI; final_summary reports SI as main_metric
Re-queue: pueue 237 (T3 prompt_baseline), 238 (T2 full_dd_benchmark)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces v7's post-hoc 'pct_w_oracle = R_w / R_w_ceiling' (a ratio of two
concentration ratios) with a per-row pct_oracle: candidate's energy_frac
divided by the optimal rank-r_eff subspace's energy_frac on the same
target. Rank-honest: chars_clusters (r_eff=7) is graded against rank-7
oracle, not rank-8. Activation oracle = PCA of L2-normalized hs_diff_B
(matches existing energy_frac_act formula).
Result: every non-oracle candidate lands at pct_oracle in [0.02, 0.11] on
both axes. Best joint = WNR_union_TaskDiff at 0.089 (rank 16; all others
rank 8). chars_clusters and layer_clean_resid_pca tied at ~0.085. This is
a clean negative result: LoRA's task-specific delta is far from any of
our hand-built linear primitives' spans.
Addresses three concerns from docs/review/v6_hypothesis_review.md:
1. R_w split into oproj/downproj + Frobenius-balanced combined.
2. dW_left_basis_ceiling as the true weight oracle.
3. axis_kind tag (write/read/mixed/ceiling).
Single-seed result: chars_clusters and attn_min_taskdiff are top-5 by both R_act
and R_w_combined. Write-family bases (write/mlp_write/global_write) all have
R_w_combined ~ 1.0 (random null) -- natural weight-side bases fail the
weight-axis test. Multi-seed deferred to v7b.
Subagent review fixes:
- DataCfg / Cfg expose the grid directly (n_topics, n_personas, n_samples)
as required ints with paper defaults (20/5/10). Drops `n_pairs` and the
silent round() that made the count fuzzy. Drops `Optional[int]` smoke
overrides — smoke just sets 2/1/2 = 4 pairs.
- Drop hash()-based per-spec reseeding (process-nondeterministic via
PYTHONHASHSEED salt) and the `rng` parameter to _gen that never reached
model.generate. One torch.manual_seed at start; spec order seeded by rng.
- Delete _judge_filter stub + cfg.judge flag (dead code, paper §3 GPT-4.1-mini
filter not implemented yet — TODO comment instead).
- replicate._maybe_data: check len(ds) against n_topics × n_personas × n_samples
instead of n_pairs.
- justfile: drop --n-pairs 1000.
- Updated the fork plan with detailed phases and objectives for small model adaptation and evaluation.
- Added a new guided-CoT evaluation script to assess model coherence under steering.
- Introduced demo functionality to showcase adapter coherence and guided-CoT performance.
- Modified training configuration to include layer fraction targeting for LoRA.
- Improved evaluation outputs for clarity and added validation checks.
