wassname/ml_debug
1
0
Fork 0
mirror of https://github.com/wassname/ml_debug.git synced 2026-09-09 11:27:07 +08:00
Code Issues Packages Projects Releases Wiki Activity

readme: drop the run-by-run pairing, it was not a real pairing

Browse Source 
The run index is a cache key, not a seed, so subtracting run 3 from run 3 is
arbitrary. Unpaired the difference is +0.023 +- 0.044; by question +0.023 +- 0.031.
Still not pushed.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
wassname
2026-08-17 11:21:55 +08:00
co-authored by Claudypoo
parent 1419c2e7df
commit 52390d7593
1 changed files with 10 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+10 -12
View File
@@ -27,20 +27,18 @@ model this SKILL.md and nothing else, so the only change is the document.
No measurable gain, from three answers per question in each arm:
| deepseek-v4-flash-0731, 12 questions | bare | with SKILL.md | change |
| --- | --- | --- | --- |
| first answer | +0.608 | +0.746 | +0.137 |
| second answer | +0.648 | +0.641 | -0.007 |
| third answer | +0.674 | +0.614 | -0.060 |
| mean | +0.643 | +0.667 | +0.023, sd 0.102 |
| deepseek-v4-flash-0731, 12 questions | bare | with SKILL.md |
| --- | --- | --- |
| mean score | +0.643 | +0.667 |
| the three runs | +0.608, +0.648, +0.674 | +0.746, +0.641, +0.614 |
The same model answering the same question again moves its score by about this much, so +0.023 is
not distinguishable from zero. Pairing by question rather than by answer gives the same +0.023, with
a standard error of 0.031 over the 12 questions.
The difference is +0.023 with a standard error of 0.044, so it is not distinguishable from zero.
Pairing by question rather than by run gives the same +0.023 with a standard error of 0.031, t of
0.76. The runs themselves scatter by more than the difference between the two columns.
An earlier version of this section reported +0.135, or 59% of the distance to gpt-5.6-sol. That
number was one answer per arm, and it is the first row of the table above. It did not survive the
other two answers.
An earlier version of this section reported +0.135, or 59% of the distance to gpt-5.6-sol. That was
one run of each arm, and it happens to be the first run in each column above. It did not survive the
other two.
Two other readings. With SKILL.md the model writes 31% more text for the same score, so any
verbosity bias in the judges makes the true effect smaller than +0.023, not larger. And only 1 answer