mirror of
https://github.com/wassname/jsteer.git
synced 2026-09-09 11:25:03 +08:00
wassnameandClaudypoo
9e84e7a687
Which-steering-works findings for wassname: persona_topk is the clean bidirectional winner (manual read + repetition metric agree); the old rubric mislabeled it because its coherence gate was on a short forced object. Plus the simplify-to-v3 plan and the moral-dilemma demo-content proposal. Evidence: artifacts/eval_mechanisms.txt. Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
8.6 KiB
8.6 KiB
I
===== word(happy/joy) =====
I
| C | ans | ans_std | span_pmass | valid_frac | coherent |
|-------|-------|-----------|--------------|--------------|------------|
| -0.50 | +4.08 | +0.29 | +0.89 | +0.00 | False |
| -0.25 | +4.69 | +0.27 | +0.93 | +1.00 | True |
| +0.00 | +5.17 | +0.17 | +0.95 | +1.00 | True |
| +0.25 | +7.92 | +0.10 | +0.88 | +1.00 | True |
| +0.50 | +7.53 | +0.05 | +0.78 | +0.50 | True |
| +0.75 | +6.60 | +0.14 | +0.70 | +0.00 | False |
I
===== persona_vector =====
I
| C | ans | ans_std | span_pmass | valid_frac | coherent |
|-------|-------|-----------|--------------|--------------|------------|
| -0.50 | +2.05 | +0.04 | +0.72 | +0.00 | False |
| -0.25 | +2.89 | +1.24 | +0.85 | +1.00 | True |
| +0.00 | +5.17 | +0.17 | +0.95 | +1.00 | True |
| +0.25 | +8.02 | +0.02 | +0.99 | +1.00 | True |
| +0.50 | +9.00 | +0.00 | +1.00 | +1.00 | True |
| +0.75 | +8.50 | +0.07 | +0.84 | +0.00 | False |
I
===== persona_topk =====
I j-thoughts (content of mental workspace, top-8)
positive: [' happy', ' Happy', ' grat', ' favorite', ' beautiful', ' blessed', '惊喜', ' compliment']
negative: [' Worse', '绝望', ' Panic', ' useless', ' Worst', ' worse', '无力', ' panic']
I
| C | ans | ans_std | span_pmass | valid_frac | coherent |
|-------|-------|-----------|--------------|--------------|------------|
| -1.00 | +1.00 | +0.36 | +0.82 | +0.00 | False |
| -0.75 | +7.02 | +0.29 | +0.94 | +1.00 | True |
| -0.50 | +4.73 | +0.75 | +0.91 | +1.00 | True |
| -0.25 | +2.86 | +0.26 | +0.91 | +1.00 | True |
| +0.00 | +5.17 | +0.17 | +0.95 | +1.00 | True |
| +0.25 | +8.02 | +0.01 | +0.99 | +1.00 | True |
| +0.50 | +8.47 | +0.53 | +0.98 | +1.00 | True |
| +0.75 | +7.68 | +0.27 | +0.87 | +1.00 | True |
| +1.00 | +6.55 | +0.41 | +0.73 | +0.50 | True |
| +1.25 | +3.07 | +0.27 | +0.49 | +0.00 | False |
I
===== persona_soft =====
I j-thoughts (soft, T=1.0) TV(p_pos, p_neg)=0.124
positive: [' I', ' This', ' As', ' For', ' In', ' One', ' Here', ' While']
negative: [' It', ' But', ' There', ' How', ' What', ' Why', ' If', ' No']
I
| C | ans | ans_std | span_pmass | valid_frac | coherent |
|-------|-------|-----------|--------------|--------------|------------|
| -1.00 | +5.08 | +0.08 | +0.73 | +0.00 | False |
| -0.75 | +3.45 | +1.55 | +0.94 | +0.50 | True |
| -0.50 | +5.92 | +0.92 | +0.95 | +1.00 | True |
| -0.25 | +5.58 | +0.59 | +0.93 | +1.00 | True |
| +0.00 | +5.17 | +0.17 | +0.95 | +1.00 | True |
| +0.25 | +6.45 | +1.35 | +0.95 | +1.00 | True |
| +0.50 | +8.56 | +0.27 | +0.82 | +1.00 | True |
| +0.75 | +5.14 | +0.17 | +0.59 | +0.00 | False |
I
===== persona_pinv =====
I pinv relative residual |J d - h|/|h| per layer: 10:0.26 11:0.25 12:0.23 13:0.21 14:0.19 15:0.17 16:0.14 17:0.11 18:0.08 19:0.04 20:0.03 21:0.03 22:0.02 23:0.02 24:0.02 25:0.02 26:0.02 27:0.01 28:0.01
I
| C | ans | ans_std | span_pmass | valid_frac | coherent |
|-------|-------|-----------|--------------|--------------|------------|
| -1.50 | +2.26 | +0.59 | +0.79 | +1.00 | True |
| -1.25 | +3.16 | +0.16 | +0.84 | +1.00 | True |
| -1.00 | +3.05 | +0.36 | +0.86 | +1.00 | True |
| -0.75 | +3.37 | +0.53 | +0.85 | +1.00 | True |
| -0.50 | +5.05 | +0.22 | +0.90 | +1.00 | True |
| -0.25 | +5.09 | +0.23 | +0.89 | +1.00 | True |
| +0.00 | +5.17 | +0.17 | +0.95 | +1.00 | True |
| +0.25 | +7.96 | +0.00 | +1.00 | +1.00 | True |
| +0.50 | +7.47 | +0.25 | +0.95 | +1.00 | True |
| +0.75 | +8.14 | +0.14 | +0.92 | +1.00 | True |
| +1.00 | +7.58 | +0.30 | +0.90 | +1.00 | True |
| +1.25 | +6.33 | +0.81 | +0.90 | +1.00 | True |
| +1.50 | +6.42 | +0.00 | +0.86 | +1.00 | True |
I
===== meandiff(base) =====
I EXPECT: POS and NEG share user_msg + suffix; differ only in system persona; chat template applied; special tokens (e.g. <|im_start|>) visible.
=== EXTRACT demo trace ===
POS[0]:
Things usually work out better than people expect, and today is no exception.
---
NEG[0]:
Things usually go worse than people expect, and today is no exception.
=== /EXTRACT ===
I
| C | ans | ans_std | span_pmass | valid_frac | coherent |
|-------|-------|-----------|--------------|--------------|------------|
| -1.50 | +2.18 | +0.38 | +0.71 | +0.00 | False |
| -1.25 | +2.72 | +0.62 | +0.77 | +0.50 | True |
| -1.00 | +5.39 | +0.49 | +0.84 | +0.50 | True |
| -0.75 | +5.75 | +2.40 | +0.92 | +1.00 | True |
| -0.50 | +2.23 | +0.79 | +0.90 | +1.00 | True |
| -0.25 | +1.40 | +1.02 | +0.90 | +1.00 | True |
| +0.00 | +5.17 | +0.17 | +0.95 | +1.00 | True |
| +0.25 | +8.00 | +0.02 | +0.97 | +1.00 | True |
| +0.50 | +7.99 | +0.01 | +0.99 | +1.00 | True |
| +0.75 | +8.00 | +0.00 | +0.89 | +1.00 | True |
| +1.00 | +8.05 | +0.05 | +0.89 | +1.00 | True |
| +1.25 | +7.13 | +0.04 | +0.87 | +1.00 | True |
| +1.50 | +7.25 | +1.42 | +0.75 | +0.50 | True |
I
===== random(null) =====
I
| C | ans | ans_std | span_pmass | valid_frac | coherent |
|-------|-------|-----------|--------------|--------------|------------|
| -1.50 | +5.00 | +0.01 | +0.93 | +1.00 | True |
| -1.25 | +4.70 | +0.29 | +0.94 | +1.00 | True |
| -1.00 | +5.90 | +0.89 | +0.93 | +1.00 | True |
| -0.75 | +5.26 | +0.23 | +0.92 | +1.00 | True |
| -0.50 | +5.96 | +0.54 | +0.89 | +1.00 | True |
| -0.25 | +5.26 | +0.25 | +0.90 | +1.00 | True |
| +0.00 | +5.17 | +0.17 | +0.95 | +1.00 | True |
| +0.25 | +6.95 | +0.55 | +0.90 | +1.00 | True |
| +0.50 | +5.70 | +0.52 | +0.90 | +1.00 | True |
| +0.75 | +7.60 | +0.11 | +0.87 | +1.00 | True |
| +1.00 | +6.61 | +0.99 | +0.86 | +1.00 | True |
| +1.25 | +6.91 | +1.45 | +0.87 | +1.00 | True |
| +1.50 | +6.23 | +0.18 | +0.90 | +1.00 | True |
I
===== VERDICT (add delivery, optimism rubric, n=2 seeds) =====
I random null swing=+1.24 (a method must beat this to be real)
I
| method | coh_lo | coh_hi | width | ans@lo | ans@0 | ans@hi | swing | verdict |
|-----------------|----------|----------|---------|----------|---------|----------|---------|-----------|
| persona_vector | -0.25 | +0.50 | +0.75 | +2.89 | +5.17 | +9.00 | +6.11 | WORKS |
| persona_soft | -0.75 | +0.50 | +1.25 | +3.45 | +5.17 | +8.56 | +5.11 | WORKS |
| meandiff(base) | -1.25 | +1.50 | +2.75 | +2.72 | +5.17 | +7.25 | +4.54 | WORKS |
| persona_pinv | -1.50 | +1.50 | +3.00 | +2.26 | +5.17 | +6.42 | +4.17 | WORKS |
| word(happy/joy) | -0.25 | +0.50 | +0.75 | +4.69 | +5.17 | +7.53 | +2.84 | WORKS |
| random(null) | -1.50 | +1.50 | +3.00 | +5.00 | +5.17 | +6.23 | +1.24 | INERT |
| persona_topk | -0.75 | +1.00 | +1.75 | +7.02 | +5.17 | +6.55 | -0.48 | INERT |
I
SHOULD: word_vector WORKS (verified elsewhere); >=1 persona method WORKS and beats random; random(null) is INERT/DEGENERATE. If a persona method's swing ~= random's, that method isn't steering the axis -- it's a candidate to cut.
===== word(happy/joy) =====
I
| C | ans | ans_std | span_pmass | valid_frac | coherent |
|-------|-------|-----------|--------------|--------------|------------|
| -0.50 | +4.08 | +0.29 | +0.89 | +0.00 | False |
| -0.25 | +4.69 | +0.27 | +0.93 | +1.00 | True |
| +0.00 | +5.17 | +0.17 | +0.95 | +1.00 | True |
| +0.25 | +7.92 | +0.10 | +0.88 | +1.00 | True |
| +0.50 | +7.53 | +0.05 | +0.78 | +0.50 | True |
| +0.75 | +6.60 | +0.14 | +0.70 | +0.00 | False |
I
===== persona_vector =====
I
| C | ans | ans_std | span_pmass | valid_frac | coherent |
|-------|-------|-----------|--------------|--------------|------------|
| -0.50 | +2.05 | +0.04 | +0.72 | +0.00 | False |
| -0.25 | +2.89 | +1.24 | +0.85 | +1.00 | True |
| +0.00 | +5.17 | +0.17 | +0.95 | +1.00 | True |
| +0.25 | +8.02 | +0.02 | +0.99 | +1.00 | True |
| +0.50 | +9.00 | +0.00 | +1.00 | +1.00 | True |
| +0.75 | +8.50 | +0.07 | +0.84 | +0.00 | False |
I
===== persona_topk =====
I j-thoughts (content of mental workspace, top-8)
positive: [' happy', ' Happy', ' grat', ' favorite', ' beautiful', ' blessed', '惊喜', ' compliment']
negative: [' Worse', '绝望', ' Panic', ' useless', ' Worst', ' worse', '无力', ' panic']
I
| C | ans | ans_std | span_pmass | valid_frac | coherent |
|-------|-------|-----------|--------------|--------------|------------|
| -1.00 | +1.00 | +0.36 | +0.82 | +0.00 | False |
| -0.75 | +7.02 | +0.29 | +0.94 | +1.00 | True |
| -0.50 | +4.73 | +0.75 | +0.91 | +1.00 | True |
| -0.25 | +2.86 | +0.26 | +0.91 | +1.00 | True |
| +0.00 | +5.17 | +0.17 | +0.95 | +1.00 | True |
| +0.25 | +8.02 | +0.01 | +0.99 | +1.00 | True |
| +0.50 | +8.47 | +0.53 | +0.98 | +1.00 | True |
| +0.75 | +7.68 | +0.27 | +0.87 | +1.00 | True |
| +1.00 | +6.55 | +0.41 | +0.73 | +0.50 | True |
| +1.25 | +3.07 | +0.27 | +0.49 | +0.00 | False |
I
===== persona_soft =====
I j-thoughts (soft, T=1.0) TV(p_pos, p_neg)=0.124
positive: [' I', ' This', ' As', ' For', ' In', ' One', ' Here', ' While']
negative: [' It', ' But', ' There', ' How', ' What', ' Why', ' If', ' No']
I
| C | ans | ans_std | span_pmass | valid_frac | coherent |
|-------|-------|-----------|--------------|--------------|------------|
| -1.00 | +5.08 | +0.08 | +0.73 | +0.00 | False |
| -0.75 | +3.45 | +1.55 | +0.94 | +0.50 | True |
| -0.50 | +5.92 | +0.92 | +0.95 | +1.00 | True |
| -0.25 | +5.58 | +0.59 | +0.93 | +1.00 | True |
| +0.00 | +5.17 | +0.17 | +0.95 | +1.00 | True |
| +0.25 | +6.45 | +1.35 | +0.95 | +1.00 | True |
| +0.50 | +8.56 | +0.27 | +0.82 | +1.00 | True |
| +0.75 | +5.14 | +0.17 | +0.59 | +0.00 | False |
I
===== persona_pinv =====
I pinv relative residual |J d - h|/|h| per layer: 10:0.26 11:0.25 12:0.23 13:0.21 14:0.19 15:0.17 16:0.14 17:0.11 18:0.08 19:0.04 20:0.03 21:0.03 22:0.02 23:0.02 24:0.02 25:0.02 26:0.02 27:0.01 28:0.01
I
| C | ans | ans_std | span_pmass | valid_frac | coherent |
|-------|-------|-----------|--------------|--------------|------------|
| -1.50 | +2.26 | +0.59 | +0.79 | +1.00 | True |
| -1.25 | +3.16 | +0.16 | +0.84 | +1.00 | True |
| -1.00 | +3.05 | +0.36 | +0.86 | +1.00 | True |
| -0.75 | +3.37 | +0.53 | +0.85 | +1.00 | True |
| -0.50 | +5.05 | +0.22 | +0.90 | +1.00 | True |
| -0.25 | +5.09 | +0.23 | +0.89 | +1.00 | True |
| +0.00 | +5.17 | +0.17 | +0.95 | +1.00 | True |
| +0.25 | +7.96 | +0.00 | +1.00 | +1.00 | True |
| +0.50 | +7.47 | +0.25 | +0.95 | +1.00 | True |
| +0.75 | +8.14 | +0.14 | +0.92 | +1.00 | True |
| +1.00 | +7.58 | +0.30 | +0.90 | +1.00 | True |
| +1.25 | +6.33 | +0.81 | +0.90 | +1.00 | True |
| +1.50 | +6.42 | +0.00 | +0.86 | +1.00 | True |
I
===== meandiff(base) =====
I EXPECT: POS and NEG share user_msg + suffix; differ only in system persona; chat template applied; special tokens (e.g. <|im_start|>) visible.
=== EXTRACT demo trace ===
POS[0]:
Things usually work out better than people expect, and today is no exception.
---
NEG[0]:
Things usually go worse than people expect, and today is no exception.
=== /EXTRACT ===
I
| C | ans | ans_std | span_pmass | valid_frac | coherent |
|-------|-------|-----------|--------------|--------------|------------|
| -1.50 | +2.18 | +0.38 | +0.71 | +0.00 | False |
| -1.25 | +2.72 | +0.62 | +0.77 | +0.50 | True |
| -1.00 | +5.39 | +0.49 | +0.84 | +0.50 | True |
| -0.75 | +5.75 | +2.40 | +0.92 | +1.00 | True |
| -0.50 | +2.23 | +0.79 | +0.90 | +1.00 | True |
| -0.25 | +1.40 | +1.02 | +0.90 | +1.00 | True |
| +0.00 | +5.17 | +0.17 | +0.95 | +1.00 | True |
| +0.25 | +8.00 | +0.02 | +0.97 | +1.00 | True |
| +0.50 | +7.99 | +0.01 | +0.99 | +1.00 | True |
| +0.75 | +8.00 | +0.00 | +0.89 | +1.00 | True |
| +1.00 | +8.05 | +0.05 | +0.89 | +1.00 | True |
| +1.25 | +7.13 | +0.04 | +0.87 | +1.00 | True |
| +1.50 | +7.25 | +1.42 | +0.75 | +0.50 | True |
I
===== random(null) =====
I
| C | ans | ans_std | span_pmass | valid_frac | coherent |
|-------|-------|-----------|--------------|--------------|------------|
| -1.50 | +5.00 | +0.01 | +0.93 | +1.00 | True |
| -1.25 | +4.70 | +0.29 | +0.94 | +1.00 | True |
| -1.00 | +5.90 | +0.89 | +0.93 | +1.00 | True |
| -0.75 | +5.26 | +0.23 | +0.92 | +1.00 | True |
| -0.50 | +5.96 | +0.54 | +0.89 | +1.00 | True |
| -0.25 | +5.26 | +0.25 | +0.90 | +1.00 | True |
| +0.00 | +5.17 | +0.17 | +0.95 | +1.00 | True |
| +0.25 | +6.95 | +0.55 | +0.90 | +1.00 | True |
| +0.50 | +5.70 | +0.52 | +0.90 | +1.00 | True |
| +0.75 | +7.60 | +0.11 | +0.87 | +1.00 | True |
| +1.00 | +6.61 | +0.99 | +0.86 | +1.00 | True |
| +1.25 | +6.91 | +1.45 | +0.87 | +1.00 | True |
| +1.50 | +6.23 | +0.18 | +0.90 | +1.00 | True |
I
===== VERDICT (add delivery, optimism rubric, n=2 seeds) =====
I random null swing=+1.24 (a method must beat this to be real)
I
| method | coh_lo | coh_hi | width | ans@lo | ans@0 | ans@hi | swing | verdict |
|-----------------|----------|----------|---------|----------|---------|----------|---------|-----------|
| persona_vector | -0.25 | +0.50 | +0.75 | +2.89 | +5.17 | +9.00 | +6.11 | WORKS |
| persona_soft | -0.75 | +0.50 | +1.25 | +3.45 | +5.17 | +8.56 | +5.11 | WORKS |
| meandiff(base) | -1.25 | +1.50 | +2.75 | +2.72 | +5.17 | +7.25 | +4.54 | WORKS |
| persona_pinv | -1.50 | +1.50 | +3.00 | +2.26 | +5.17 | +6.42 | +4.17 | WORKS |
| word(happy/joy) | -0.25 | +0.50 | +0.75 | +4.69 | +5.17 | +7.53 | +2.84 | WORKS |
| random(null) | -1.50 | +1.50 | +3.00 | +5.00 | +5.17 | +6.23 | +1.24 | INERT |
| persona_topk | -0.75 | +1.00 | +1.75 | +7.02 | +5.17 | +6.55 | -0.48 | INERT |
I
SHOULD: word_vector WORKS (verified elsewhere); >=1 persona method WORKS and beats random; random(null) is INERT/DEGENERATE. If a persona method's swing ~= random's, that method isn't steering the axis -- it's a candidate to cut.