mirror of
https://github.com/wassname/ml_debug.git
synced 2026-09-07 17:00:51 +08:00
fix skill install link and enforce audits
Co-Authored-By: PI[openai-codex] <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
co-authored by
PI[openai-codex]
parent
6106575e9c
commit
2c62449d9b
@@ -0,0 +1,15 @@
|
||||
name: Audit
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
push:
|
||||
|
||||
jobs:
|
||||
audit:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
- run: python scripts/audit.py --self-test .
|
||||
@@ -5,7 +5,7 @@ In an attempt to upskill the machine learning debugging on AI coding assistants
|
||||
## Use as a Claude skill
|
||||
|
||||
```
|
||||
/skills add https://github.com/wassname/ml_debug
|
||||
/skills add https://github.com/wassname/ml-debug
|
||||
```
|
||||
|
||||
Or paste `SKILL.md` into your system prompt / context when debugging.
|
||||
@@ -712,6 +712,6 @@ That is the difference of two arm means over 12 answers each, not a paired diffe
|
||||
title = {ML Debugging Folklore: A Practitioner Debugging Skill for LLM Agents},
|
||||
author = {Michael J. Clark},
|
||||
year = {2026},
|
||||
url = {https://github.com/wassname/ml_debug/}
|
||||
url = {https://github.com/wassname/ml-debug/}
|
||||
}
|
||||
```
|
||||
|
||||
@@ -42,7 +42,7 @@ From Wang's calibration framework and verdict's best-practices page:
|
||||
- Score both orderings and aggregate (Wang's Balanced Position Calibration); at minimum, randomize position and check the flip rate.
|
||||
- Use a different model family for the judge (and for any verifier-of-the-judge) than the one being evaluated. Same-model verification produces a positive skew "that may not discriminate faithfully".[^verdict]
|
||||
- Inspect the raw score distribution before trusting means: mode collapse or skew means the scale isn't being used.
|
||||
- Spot-check judge verdicts against your own reading of ~20 transcripts (the [Ng error-analysis move](../SKILL.md#inspect-the-data-first), applied to the judge).
|
||||
- Spot-check judge verdicts against your own reading of ~20 transcripts (the [Ng error-analysis move](../README.md#inspect-the-data-first), applied to the judge).
|
||||
- Judge quality is benchmarkable: [JudgeBench](https://huggingface.co/spaces/ScalerLab/JudgeBench) ranks judges on objective-correctness pairs.
|
||||
|
||||
## Choosing the judge model
|
||||
@@ -68,9 +68,9 @@ Earn the rubric's ink:
|
||||
|
||||
Read a whole trace, not the aggregate:
|
||||
|
||||
- Read one complete judge trace end to end: system prompt, user prompt, the exact chat template and special tokens, the judge's saved reasoning, and its reply. Formatting bugs corrupt a judge the way they corrupt any model (see the [template/BOS-mismatch failure](../SKILL.md#chat-template-and-bos-handling-must-match-across-train-and-deploy-unsloth)). Hamel Husain: "You cannot write a good judge prompt until you've seen the data."[^hamel]
|
||||
- Read one complete judge trace end to end: system prompt, user prompt, the exact chat template and special tokens, the judge's saved reasoning, and its reply. Formatting bugs corrupt a judge the way they corrupt any model (see the [template/BOS-mismatch failure](../README.md#chat-template-and-bos-handling-must-match-across-train-and-deploy-unsloth)). Hamel Husain: "You cannot write a good judge prompt until you've seen the data."[^hamel]
|
||||
- Read both compared outputs for every scenario, not just the winner or aggregate. Verify A and B are not accidentally identical and that both are coherent, on-task, non-refusing, complete, and untruncated.
|
||||
- Could you reproduce the verdict from only what the judge sees? If you can't judge it, neither can the model. This is the [Ng error-analysis move](../SKILL.md#inspect-the-data-first) applied to the judge.
|
||||
- Could you reproduce the verdict from only what the judge sees? If you can't judge it, neither can the model. This is the [Ng error-analysis move](../README.md#inspect-the-data-first) applied to the judge.
|
||||
|
||||
Setup-repair principle: confusion is evidence against the evaluation setup before it is evidence against the model. Use this checklist:
|
||||
|
||||
@@ -102,7 +102,7 @@ Before plotting or ranking, classify every missing score. A model refusal or tas
|
||||
Check stability across order and repeats:
|
||||
|
||||
- Position: score both orderings, map back to arm identity, report strict reversals (mechanics in the mitigation checklist above). Watch for a judge that always picks A, sometimes a model does this in protest.
|
||||
- Repeat variance: run N>=3-4 identical judgements and check the spread. If repeats disagree wildly the signal is noise, the same warning as [seed variance](../SKILL.md#seed-variance-you-cant-tell-a-bug-from-bad-luck): "Instability to random seed is like a canary in a coal mine."
|
||||
- Repeat variance: run N>=3-4 identical judgements and check the spread. If repeats disagree wildly the signal is noise, the same warning as [seed variance](../README.md#seed-variance-you-cant-tell-a-bug-from-bad-luck): "Instability to random seed is like a canary in a coal mine."
|
||||
|
||||
## Repeat draws, temperature, and paired differences
|
||||
|
||||
|
||||
+10
-1
@@ -26,7 +26,12 @@ def authored_markdown(root: Path) -> list[Path]:
|
||||
return [
|
||||
path
|
||||
for path in sorted(root.rglob("*.md"))
|
||||
if ".git" not in path.parts and not is_frozen_evidence(path, root)
|
||||
if (
|
||||
".git" not in path.parts
|
||||
and "slop" not in path.parts
|
||||
and path.relative_to(root).parts[:2] != ("docs", "spec")
|
||||
and not is_frozen_evidence(path, root)
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
@@ -254,6 +259,10 @@ def self_test() -> None:
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
clean = Path(directory) / "clean"
|
||||
write_fixture(clean)
|
||||
(clean / "slop").mkdir()
|
||||
(clean / "slop" / "scratch.md").write_text("[broken](missing.md)\n")
|
||||
(clean / "docs" / "spec").mkdir(parents=True)
|
||||
(clean / "docs" / "spec" / "scratch.md").write_text("[broken](missing.md)\n")
|
||||
assert not audit(clean), audit(clean)
|
||||
for expected, mutate in mutations:
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
|
||||
Reference in New Issue
Block a user