mirror of
https://github.com/wassname/ml_debug.git
synced 2026-09-09 11:27:07 +08:00
fix skill install link and enforce audits
Co-Authored-By: PI[openai-codex] <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
co-authored by
PI[openai-codex]
parent
6106575e9c
commit
2c62449d9b
@@ -0,0 +1,15 @@
|
|||||||
|
name: Audit
|
||||||
|
|
||||||
|
on:
|
||||||
|
pull_request:
|
||||||
|
push:
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
audit:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- uses: actions/setup-python@v5
|
||||||
|
with:
|
||||||
|
python-version: '3.12'
|
||||||
|
- run: python scripts/audit.py --self-test .
|
||||||
@@ -5,7 +5,7 @@ In an attempt to upskill the machine learning debugging on AI coding assistants
|
|||||||
## Use as a Claude skill
|
## Use as a Claude skill
|
||||||
|
|
||||||
```
|
```
|
||||||
/skills add https://github.com/wassname/ml_debug
|
/skills add https://github.com/wassname/ml-debug
|
||||||
```
|
```
|
||||||
|
|
||||||
Or paste `SKILL.md` into your system prompt / context when debugging.
|
Or paste `SKILL.md` into your system prompt / context when debugging.
|
||||||
@@ -712,6 +712,6 @@ That is the difference of two arm means over 12 answers each, not a paired diffe
|
|||||||
title = {ML Debugging Folklore: A Practitioner Debugging Skill for LLM Agents},
|
title = {ML Debugging Folklore: A Practitioner Debugging Skill for LLM Agents},
|
||||||
author = {Michael J. Clark},
|
author = {Michael J. Clark},
|
||||||
year = {2026},
|
year = {2026},
|
||||||
url = {https://github.com/wassname/ml_debug/}
|
url = {https://github.com/wassname/ml-debug/}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -42,7 +42,7 @@ From Wang's calibration framework and verdict's best-practices page:
|
|||||||
- Score both orderings and aggregate (Wang's Balanced Position Calibration); at minimum, randomize position and check the flip rate.
|
- Score both orderings and aggregate (Wang's Balanced Position Calibration); at minimum, randomize position and check the flip rate.
|
||||||
- Use a different model family for the judge (and for any verifier-of-the-judge) than the one being evaluated. Same-model verification produces a positive skew "that may not discriminate faithfully".[^verdict]
|
- Use a different model family for the judge (and for any verifier-of-the-judge) than the one being evaluated. Same-model verification produces a positive skew "that may not discriminate faithfully".[^verdict]
|
||||||
- Inspect the raw score distribution before trusting means: mode collapse or skew means the scale isn't being used.
|
- Inspect the raw score distribution before trusting means: mode collapse or skew means the scale isn't being used.
|
||||||
- Spot-check judge verdicts against your own reading of ~20 transcripts (the [Ng error-analysis move](../SKILL.md#inspect-the-data-first), applied to the judge).
|
- Spot-check judge verdicts against your own reading of ~20 transcripts (the [Ng error-analysis move](../README.md#inspect-the-data-first), applied to the judge).
|
||||||
- Judge quality is benchmarkable: [JudgeBench](https://huggingface.co/spaces/ScalerLab/JudgeBench) ranks judges on objective-correctness pairs.
|
- Judge quality is benchmarkable: [JudgeBench](https://huggingface.co/spaces/ScalerLab/JudgeBench) ranks judges on objective-correctness pairs.
|
||||||
|
|
||||||
## Choosing the judge model
|
## Choosing the judge model
|
||||||
@@ -68,9 +68,9 @@ Earn the rubric's ink:
|
|||||||
|
|
||||||
Read a whole trace, not the aggregate:
|
Read a whole trace, not the aggregate:
|
||||||
|
|
||||||
- Read one complete judge trace end to end: system prompt, user prompt, the exact chat template and special tokens, the judge's saved reasoning, and its reply. Formatting bugs corrupt a judge the way they corrupt any model (see the [template/BOS-mismatch failure](../SKILL.md#chat-template-and-bos-handling-must-match-across-train-and-deploy-unsloth)). Hamel Husain: "You cannot write a good judge prompt until you've seen the data."[^hamel]
|
- Read one complete judge trace end to end: system prompt, user prompt, the exact chat template and special tokens, the judge's saved reasoning, and its reply. Formatting bugs corrupt a judge the way they corrupt any model (see the [template/BOS-mismatch failure](../README.md#chat-template-and-bos-handling-must-match-across-train-and-deploy-unsloth)). Hamel Husain: "You cannot write a good judge prompt until you've seen the data."[^hamel]
|
||||||
- Read both compared outputs for every scenario, not just the winner or aggregate. Verify A and B are not accidentally identical and that both are coherent, on-task, non-refusing, complete, and untruncated.
|
- Read both compared outputs for every scenario, not just the winner or aggregate. Verify A and B are not accidentally identical and that both are coherent, on-task, non-refusing, complete, and untruncated.
|
||||||
- Could you reproduce the verdict from only what the judge sees? If you can't judge it, neither can the model. This is the [Ng error-analysis move](../SKILL.md#inspect-the-data-first) applied to the judge.
|
- Could you reproduce the verdict from only what the judge sees? If you can't judge it, neither can the model. This is the [Ng error-analysis move](../README.md#inspect-the-data-first) applied to the judge.
|
||||||
|
|
||||||
Setup-repair principle: confusion is evidence against the evaluation setup before it is evidence against the model. Use this checklist:
|
Setup-repair principle: confusion is evidence against the evaluation setup before it is evidence against the model. Use this checklist:
|
||||||
|
|
||||||
@@ -102,7 +102,7 @@ Before plotting or ranking, classify every missing score. A model refusal or tas
|
|||||||
Check stability across order and repeats:
|
Check stability across order and repeats:
|
||||||
|
|
||||||
- Position: score both orderings, map back to arm identity, report strict reversals (mechanics in the mitigation checklist above). Watch for a judge that always picks A, sometimes a model does this in protest.
|
- Position: score both orderings, map back to arm identity, report strict reversals (mechanics in the mitigation checklist above). Watch for a judge that always picks A, sometimes a model does this in protest.
|
||||||
- Repeat variance: run N>=3-4 identical judgements and check the spread. If repeats disagree wildly the signal is noise, the same warning as [seed variance](../SKILL.md#seed-variance-you-cant-tell-a-bug-from-bad-luck): "Instability to random seed is like a canary in a coal mine."
|
- Repeat variance: run N>=3-4 identical judgements and check the spread. If repeats disagree wildly the signal is noise, the same warning as [seed variance](../README.md#seed-variance-you-cant-tell-a-bug-from-bad-luck): "Instability to random seed is like a canary in a coal mine."
|
||||||
|
|
||||||
## Repeat draws, temperature, and paired differences
|
## Repeat draws, temperature, and paired differences
|
||||||
|
|
||||||
|
|||||||
+10
-1
@@ -26,7 +26,12 @@ def authored_markdown(root: Path) -> list[Path]:
|
|||||||
return [
|
return [
|
||||||
path
|
path
|
||||||
for path in sorted(root.rglob("*.md"))
|
for path in sorted(root.rglob("*.md"))
|
||||||
if ".git" not in path.parts and not is_frozen_evidence(path, root)
|
if (
|
||||||
|
".git" not in path.parts
|
||||||
|
and "slop" not in path.parts
|
||||||
|
and path.relative_to(root).parts[:2] != ("docs", "spec")
|
||||||
|
and not is_frozen_evidence(path, root)
|
||||||
|
)
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
@@ -254,6 +259,10 @@ def self_test() -> None:
|
|||||||
with tempfile.TemporaryDirectory() as directory:
|
with tempfile.TemporaryDirectory() as directory:
|
||||||
clean = Path(directory) / "clean"
|
clean = Path(directory) / "clean"
|
||||||
write_fixture(clean)
|
write_fixture(clean)
|
||||||
|
(clean / "slop").mkdir()
|
||||||
|
(clean / "slop" / "scratch.md").write_text("[broken](missing.md)\n")
|
||||||
|
(clean / "docs" / "spec").mkdir(parents=True)
|
||||||
|
(clean / "docs" / "spec" / "scratch.md").write_text("[broken](missing.md)\n")
|
||||||
assert not audit(clean), audit(clean)
|
assert not audit(clean), audit(clean)
|
||||||
for expected, mutate in mutations:
|
for expected, mutate in mutations:
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
with tempfile.TemporaryDirectory() as directory:
|
||||||
|
|||||||
Reference in New Issue
Block a user