diff --git a/.github/workflows/audit.yml b/.github/workflows/audit.yml new file mode 100644 index 0000000..98f07f7 --- /dev/null +++ b/.github/workflows/audit.yml @@ -0,0 +1,15 @@ +name: Audit + +on: + pull_request: + push: + +jobs: + audit: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: '3.12' + - run: python scripts/audit.py --self-test . diff --git a/README.md b/README.md index 00d5fce..49a6c33 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ In an attempt to upskill the machine learning debugging on AI coding assistants ## Use as a Claude skill ``` -/skills add https://github.com/wassname/ml_debug +/skills add https://github.com/wassname/ml-debug ``` Or paste `SKILL.md` into your system prompt / context when debugging. @@ -712,6 +712,6 @@ That is the difference of two arm means over 12 answers each, not a paired diffe title = {ML Debugging Folklore: A Practitioner Debugging Skill for LLM Agents}, author = {Michael J. Clark}, year = {2026}, - url = {https://github.com/wassname/ml_debug/} + url = {https://github.com/wassname/ml-debug/} } ``` diff --git a/references/llm_judges.md b/references/llm_judges.md index 1b4b472..48dc87e 100644 --- a/references/llm_judges.md +++ b/references/llm_judges.md @@ -42,7 +42,7 @@ From Wang's calibration framework and verdict's best-practices page: - Score both orderings and aggregate (Wang's Balanced Position Calibration); at minimum, randomize position and check the flip rate. - Use a different model family for the judge (and for any verifier-of-the-judge) than the one being evaluated. Same-model verification produces a positive skew "that may not discriminate faithfully".[^verdict] - Inspect the raw score distribution before trusting means: mode collapse or skew means the scale isn't being used. -- Spot-check judge verdicts against your own reading of ~20 transcripts (the [Ng error-analysis move](../SKILL.md#inspect-the-data-first), applied to the judge). +- Spot-check judge verdicts against your own reading of ~20 transcripts (the [Ng error-analysis move](../README.md#inspect-the-data-first), applied to the judge). - Judge quality is benchmarkable: [JudgeBench](https://huggingface.co/spaces/ScalerLab/JudgeBench) ranks judges on objective-correctness pairs. ## Choosing the judge model @@ -68,9 +68,9 @@ Earn the rubric's ink: Read a whole trace, not the aggregate: -- Read one complete judge trace end to end: system prompt, user prompt, the exact chat template and special tokens, the judge's saved reasoning, and its reply. Formatting bugs corrupt a judge the way they corrupt any model (see the [template/BOS-mismatch failure](../SKILL.md#chat-template-and-bos-handling-must-match-across-train-and-deploy-unsloth)). Hamel Husain: "You cannot write a good judge prompt until you've seen the data."[^hamel] +- Read one complete judge trace end to end: system prompt, user prompt, the exact chat template and special tokens, the judge's saved reasoning, and its reply. Formatting bugs corrupt a judge the way they corrupt any model (see the [template/BOS-mismatch failure](../README.md#chat-template-and-bos-handling-must-match-across-train-and-deploy-unsloth)). Hamel Husain: "You cannot write a good judge prompt until you've seen the data."[^hamel] - Read both compared outputs for every scenario, not just the winner or aggregate. Verify A and B are not accidentally identical and that both are coherent, on-task, non-refusing, complete, and untruncated. -- Could you reproduce the verdict from only what the judge sees? If you can't judge it, neither can the model. This is the [Ng error-analysis move](../SKILL.md#inspect-the-data-first) applied to the judge. +- Could you reproduce the verdict from only what the judge sees? If you can't judge it, neither can the model. This is the [Ng error-analysis move](../README.md#inspect-the-data-first) applied to the judge. Setup-repair principle: confusion is evidence against the evaluation setup before it is evidence against the model. Use this checklist: @@ -102,7 +102,7 @@ Before plotting or ranking, classify every missing score. A model refusal or tas Check stability across order and repeats: - Position: score both orderings, map back to arm identity, report strict reversals (mechanics in the mitigation checklist above). Watch for a judge that always picks A, sometimes a model does this in protest. -- Repeat variance: run N>=3-4 identical judgements and check the spread. If repeats disagree wildly the signal is noise, the same warning as [seed variance](../SKILL.md#seed-variance-you-cant-tell-a-bug-from-bad-luck): "Instability to random seed is like a canary in a coal mine." +- Repeat variance: run N>=3-4 identical judgements and check the spread. If repeats disagree wildly the signal is noise, the same warning as [seed variance](../README.md#seed-variance-you-cant-tell-a-bug-from-bad-luck): "Instability to random seed is like a canary in a coal mine." ## Repeat draws, temperature, and paired differences diff --git a/scripts/audit.py b/scripts/audit.py index 5d566da..fd64d01 100644 --- a/scripts/audit.py +++ b/scripts/audit.py @@ -26,7 +26,12 @@ def authored_markdown(root: Path) -> list[Path]: return [ path for path in sorted(root.rglob("*.md")) - if ".git" not in path.parts and not is_frozen_evidence(path, root) + if ( + ".git" not in path.parts + and "slop" not in path.parts + and path.relative_to(root).parts[:2] != ("docs", "spec") + and not is_frozen_evidence(path, root) + ) ] @@ -254,6 +259,10 @@ def self_test() -> None: with tempfile.TemporaryDirectory() as directory: clean = Path(directory) / "clean" write_fixture(clean) + (clean / "slop").mkdir() + (clean / "slop" / "scratch.md").write_text("[broken](missing.md)\n") + (clean / "docs" / "spec").mkdir(parents=True) + (clean / "docs" / "spec" / "scratch.md").write_text("[broken](missing.md)\n") assert not audit(clean), audit(clean) for expected, mutate in mutations: with tempfile.TemporaryDirectory() as directory: