wassname/ml-debug
1
0
Fork 0
mirror of https://github.com/wassname/ml-debug.git synced 2026-08-03 13:00:32 +08:00
Code Issues Packages Projects Releases Wiki Activity

Show explicit pandas failure-score aggregation

Browse Source
This commit is contained in:
wassname
2026-07-26 09:48:31 +08:00
parent 0cc01db100
commit 4fcea49ebb
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
refs/llm_judges.md
+1 -1
View File
@@ -86,7 +86,7 @@ Keep the comparison set fixed:
`A.dropna().mean()` and `B.dropna().mean()` can average different sample populations. A can look best by scoring one easy survivor while B is averaged across all 128 hard samples; A's missingness is part of the result. Pandas [`mean`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mean.html) and [`GroupBy.mean`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.api.typing.DataFrameGroupBy.mean.html) skip missing values by default, so a naive aggregate-then-plot pipeline can create this comparison without an explicit `dropna`.
Before plotting or ranking, assert that every arm has one finite score for every expected sample ID. Give a model refusal or task failure the metric's defined failure score; make a judge, parser, timeout, or infrastructure failure fail the eval and rerun it. Do not use `nanmean`, `skipna`, or independent `dropna`; report coverage and failure reasons beside the scores. Restricting all arms to their shared complete cases makes the comparison paired, but it can still select only easy survivors and does not support an overall ranking.
Before plotting or ranking, classify every missing score. A model refusal or task failure gets the metric's defined failure score, so in pandas use `scores.fillna(bad_result).mean()` once every NaN is known to mean model failure. A judge, parser, timeout, or infrastructure failure must fail the eval and be rerun, never filled. Do not use `nanmean`, `skipna`, or independent `dropna`; report coverage and failure reasons beside the scores. Restricting all arms to their shared complete cases makes the comparison paired, but it can still select only easy survivors and does not support an overall ranking.
Check stability across order and repeats: