Record skill audit evidence

PLAYBOOK.md

@@ -18,17 +18,17 @@ How to *think* when generating hypotheses or deciding what to investigate next.
 
 **5. Structural ceiling: can the parameterization even express what you want?** Sometimes a metric is stuck not because the optimizer fails but because the architecture literally cannot represent the target. Quick check: disable the loss term entirely; if the metric reaches the same value, the loss never moved it. Worked example in [refs/metric_stuck.md](refs/metric_stuck.md).
 
-### Practitioner priors: what's usually wrong
+### Where to look first
 
-With no other information, investigate in this order. Rough consensus from the folklore sources, not measured frequencies, and only a starting weight (a clue that points elsewhere overrides them outright):
+With no other information, this is a reasonable starting order from the folklore sources. It is not a measured distribution of failure causes. Follow direct evidence when it points elsewhere.
 
-1. **Data pipeline** (~40%). Wrong preprocessing, labels misaligned with inputs, missing/wrong normalization, train/test leakage, a loader returning stale batches. It really is usually the data.[^slavv][^fsdl]
-2. **Loss function** (~20%). Wrong loss for the task, wrong sign, double softmax, loss disconnected from the metric, competing losses canceling.
-3. **Training procedure** (~15%). Wrong optimizer step order, missing `zero_grad`, frozen params, in-place ops breaking autograd.
-4. **Architecture** (~10%). Too small to express it, too deep without skips, wrong activation.
-5. **Hyperparameters** (~5%). LR, batch size, weight decay. Almost never the real problem if the code is buggy.
-6. **Numerical** (~5%). NaN, overflow, underflow, usually a symptom of one of the above.
-7. **Environment** (~5%). Library version, GPU memory, nondeterminism, stale cache.
+1. **Data pipeline.** Wrong preprocessing, labels misaligned with inputs, missing/wrong normalization, train/test leakage, or a loader returning stale batches.[^slavv][^fsdl]
+2. **Loss function.** Wrong loss for the task, wrong sign, double softmax, loss disconnected from the metric, or competing losses canceling.
+3. **Training procedure.** Wrong optimizer step order, missing `zero_grad`, frozen parameters, or in-place operations breaking autograd.
+4. **Architecture.** Too small to express the target, too deep without skips, or the wrong activation.
+5. **Hyperparameters.** Learning rate, batch size, or weight decay.
+6. **Numerical behavior.** NaN, overflow, or underflow, often caused by one of the earlier problems.
+7. **Environment.** Library version, GPU memory, nondeterminism, or a stale cache.
 
 For RL, add reward scale/sign as a top-3 issue, and episode-boundary handling (done signals, discounting across resets).
 
@@ -37,7 +37,7 @@ For RL, add reward scale/sign as a top-3 issue, and episode-boundary handling (d
 | Signal | Likely meaning | Check |
 |--------|----------------|-------|
 | Init loss << expected (e.g. 0.01 vs 2.3) | Leakage or a shortcut: the model "knows" the answer at init | Are labels in the input? Is test data in train? A trivial feature? Localize with Wassname's NaN-poisoning tracer or backprop-to-input check ([refs/diagnostics.md](refs/diagnostics.md)) |
-| Random input gives the same loss as real input | Pipeline is destroying information (over-aggressive preprocessing, wrong transforms, all-zero input) | Print raw data at each stage; visualize |
+| After training, replacing real inputs with shuffled or random inputs barely changes predictions or the metric | The model may not use the intended input signal; this does not identify the cause | Inspect preprocessing, model wiring, label leakage, and task bias |
 | Predicts the same class for everything | Class imbalance (100:1 -> "always predict majority") | Label-count check; weighted loss or resample |
 | Val much worse than train from the start | Distribution shift between splits | Same preprocessing? Same time period? Same source? |
 | Learning curve flat even with 10x data | NOT data: high bias | Add capacity, fix features, check for capacity-reducing bugs |
@@ -71,7 +71,7 @@ A catalog of small, well-worn checks, in rough dependency order (each assumes th
 
 Make complexity pay rent: every added component (physics, dimensions, losses) should improve a metric you care about, or come out.
 
-**Step 3: Log everything, then look for specific pathologies.**[^goodfellow][^rahtz][^cs231n] Log train+val loss (per-component if multi-objective), gradient norms per module, learning rate, parameter-update magnitudes, the update-to-data ratio per layer (`((lr * p.grad).std() / p.data.std()).log10()`, target ~-3), activation stats (mean, std, dead-ReLU fraction, tanh saturation), and input/label distributions.
+**Step 3: Log everything, then look for specific pathologies.**[^goodfellow][^rahtz][^cs231n] Log train+val loss (per-component if multi-objective), gradient norms per module, learning rate, actual parameter-update magnitudes, activation stats (mean, std, dead-ReLU fraction, tanh saturation), and input/label distributions. For Adam and AdamW, measure the parameter change across `optimizer.step()`; `lr * grad` is not the applied update.
 
 **Sanity-check the loss at init**[^cs231n]: verify chance-level loss before training. For 10-class softmax the initial loss should be `-ln(0.1) = 2.302` with small random weights. Wrong init loss means a bad initialization or a broken loss. Then check that increasing regularization increases the loss.
 
@@ -79,8 +79,8 @@ Make complexity pay rent: every added component (physics, dimensions, losses) sh
 |---|---|
 | Loss stuck from the start | LR too low, bad init, data pipeline broken, wrong loss function |
 | Loss decreases then explodes | LR too high, numerical instability (log(0), div by 0), gradient-accumulation bug |
-| Loss NaN | log(0), 0/0, overflow. Use `log(x.clamp(min=1e-8))`, `1/(std + 1e-5)` |
-| Train loss good, val loss bad | Overfitting. More data, regularization, smaller model |
+| Loss NaN | Insert `assert torch.isfinite(x).all()` after successive pipeline stages; the first failure localizes the invalid operation. Add a clamp or epsilon only when the intended math requires that boundary behavior |
+| Train loss good, val loss bad | Check split construction, preprocessing parity, and eval mode. If those pass, overfitting is likely |
 | Loss oscillates wildly | LR too high, batch too small, data shuffling broken |
 | Gradients vanish | Too-deep net without skips, saturating activations, bad init |
 | Gradients explode | No gradient clipping, LR too high, RNN without clipping |
@@ -172,12 +172,12 @@ def debug(symptom):
 Rough order to consider, not authoritative; it may not fit your project. Stop when a question fits.
 
 1. Exception/traceback? Read it, fix it, done.
-2. Loss NaN/Inf? Attach NaN hooks ([refs/diagnostics.md](refs/diagnostics.md)), find the first module producing NaN. Usual causes: log(0), 0/0, exp(large); add clamp/eps.
-3. Init loss wrong? Check the data pipeline and loss; check for double softmax; check labels match output format. Same loss on random input -> data destroyed. Init loss << expected -> leakage.
+2. Loss NaN/Inf? Attach NaN hooks ([refs/diagnostics.md](refs/diagnostics.md)) or insert `assert torch.isfinite(x).all()` after successive stages. Find the first invalid value before changing the math. Common causes include log(0), 0/0, and exp(large).
+3. Init loss wrong? Check the data pipeline and loss; check for double softmax; check labels match the output format. A low init loss makes leakage or a shortcut plausible; localize it before changing the model.
 4. Can't overfit one batch? Gradient-flow check: None grads -> disconnected layer; all-zero grads -> dead layer / detach. Check autograd breakers and optimizer step order.
 5. Loss stuck from step 0 but you *can* overfit one batch? LR too low (try 10x), frozen params (check `requires_grad`), wrong loss.
 6. Loss decreases then explodes? LR too high (try 0.1x), log the pre-clip grad norm, hunt numerical instability.
-7. Train good, val bad? Overfitting, not a bug. More data, regularization, smaller model.
+7. Training performance good but validation performance poor? First check for a train/validation mismatch or an evaluation bug. If those checks pass, overfitting is likely.
 8. Train loss fine but the metric is bad? Loss-metric misalignment ([refs/metric_stuck.md](refs/metric_stuck.md)).
 9. Outputs constant? Mode collapse: class imbalance, all-zero init, dead ReLUs, look at confidence-sorted errors.
 10. Slow but not stuck? Not a bug. Consider batch size, depth/width, data quality.

docs/spec/20260712_epistemic_audit_benchmark.md

@@ -0,0 +1,84 @@
+# Tighten debugging claims, audit the skill, and benchmark agent behavior
+
+## Goal
+Remove overconfident debugging claims, add a cheap repository audit, then measure whether loading `ml-debug` improves agent diagnosis on seeded failures.
+
+## Scope
+In: `PLAYBOOK.md`, `refs/diagnostics.md`, `pinn/SKILL.md`, authored-markdown integrity checks, a `just audit` recipe, and an isolated A/B benchmark in a git worktree.
+Out: end-to-end case-study prose, new debugging domains, compatibility shims, and unrelated evidence collection.
+
+## Requirements
+- R1: Advice must separate observations from diagnoses. Done means the random-input, train/validation, NaN, failure-prior, and update-ratio passages no longer claim more than their checks establish. VERIFY: targeted searches plus human review of the changed paragraphs.
+- R2: PINN optimizer guidance must present ConFIG and UPGrad as plausible methods, without unsupported numeric credence or declaring one generally superior. VERIFY: the section contains both methods and no `credence ~70%` or `consistent wins` claim.
+- R3: `just audit` must fail on broken authored links, malformed Markdown fences, missing footnote definitions, invalid skill frontmatter, and out-of-range local evidence line anchors. VERIFY: it passes on the repository and fails when each defect is injected into a temporary copy.
+- R4: The benchmark must contain 8-12 seeded ML failures and compare fresh agent diagnoses with and without `ml-debug`. Done means a machine-readable results table reports root-cause accuracy, localization-before-fix, discriminating-test choice, unsupported behavior changes, and silent fallbacks for both conditions. VERIFY: rerun the scorer from raw outputs and reproduce the summary.
+- R5: Benchmark construction and execution must happen in a separate git worktree. VERIFY: `git worktree list` and the benchmark commit path show an isolated worktree branch.
+
+## Tasks
+- [x] T1 (R1-R2): Correct the overconfident documentation.
+  - steps: remove numeric pseudo-priors; qualify random-input evidence; rewrite train/validation triage; replace blanket clamp/epsilon advice with finite assertions and localization; distinguish the Adam update from `lr * grad`; soften PINN optimizer comparison and include UPGrad
+  - verify: `rg -n 'Data pipeline.*~40%|Loss function.*~20%|Conflict-free gradient methods.*credence|ConFIG.*recommended|consistent wins|Overfitting, not a bug|add clamp/eps|update-to-data' PLAYBOOK.md refs/diagnostics.md pinn/SKILL.md`
+  - success: none of the rejected claims remain; replacement text names what each check establishes
+  - likely_fail: old overconfident wording survives in one duplicate location; repository-wide search catches it
+  - sneaky_fail: prose changes but still localizes a cause from weak evidence; manual observation-versus-inference review catches it
+  - UAT: "when I follow a diagnostic, it tells me what I observed, what remains plausible, and where to inspect next"
+- [x] T2 (R3): Add the authored-markdown and skill audit.
+  - steps: add one small audit script and a `just audit` recipe; exclude frozen scraped evidence from ordinary link checking while validating explicit evidence anchors used by authored files
+  - verify: `just --dry-run audit && just audit`
+  - success: the clean repository passes with a short summary
+  - likely_fail: the known deleted RL process-log link or malformed heading fails the first run; fix the source
+  - sneaky_fail: the audit always exits zero; mutation tests inject one defect per check and require nonzero exit
+  - UAT: "when an authored link, fence, footnote, frontmatter, or evidence anchor breaks, `just audit` names the file and exits nonzero"
+- [x] T3 (R1-R3): Review, humanize, verify, commit, and push the documentation/audit chunk.
+  - verify: `python3 /home/wassname/.agents/skills/humanizer/lint.py --help`, the selected lint command, `just audit`, `git diff --check`, and external review
+  - success: checks pass and review findings are resolved or recorded
+  - likely_fail: humanizer catches repeated AI patterns or external review finds an overclaim; revise and rerun
+  - sneaky_fail: checks pass but user-facing meaning regresses; fresh-eyes review compares the changed passages to R1-R2
+  - UAT: "the committed diff is small, readable, and its audit output is linked in this spec"
+- [ ] T4 (R4-R5): Build the seeded-failure benchmark in a separate worktree.
+  - steps: create 8-12 compact cases with hidden answer keys; run fresh agent sessions in control and skill conditions; retain raw outputs; score only explicit evidence in outputs
+  - verify: benchmark validation command checks case count, unique IDs, hidden keys, raw output completeness, and score reproducibility
+  - success: both conditions have the same cases and model settings, with no answer-key leakage
+  - likely_fail: agent runner or model access is unavailable; record the exact failure and keep a runnable harness
+  - sneaky_fail: treatment prompt leaks intended diagnoses or cases are easier in one condition; prompt diff and case-ID pairing checks catch it
+  - UAT: "I can inspect each raw diagnosis and reproduce the aggregate A/B table from it"
+- [ ] T5 (R4-R5): Fresh-eyes review the benchmark evidence, then merge the completed benchmark chunk.
+  - verify: reviewer reproduces scoring for a sample without seeing aggregate conclusions, then `git diff --check` and benchmark audit pass
+  - success: reviewer agrees with the sampled scores or corrections are applied before merge
+  - likely_fail: rubric requires subjective reconstruction; tighten evidence fields and rescore
+  - sneaky_fail: scorer rewards verbosity or keyword matching rather than diagnosis; reviewer checks decisions against raw outputs and answer keys
+  - UAT: "the final results table links to raw outputs and survives independent rescoring"
+
+## Context
+- The repo is fail-fast research code: checks should raise on invalid state rather than clamp, fill, skip, or fall back.
+- Frozen `docs/evidence/` files contain scraped links that are not expected to resolve locally. Authored files should resolve all local links.
+- The user rejected adding worked case studies because they may make agents hyper-focus on the examples.
+- The benchmark is last and must use a worktree.
+
+## Log
+- Precise failure percentages in `PLAYBOOK.md` are qualitative practitioner ordering presented with unsupported numeric precision.
+- For Adam/AdamW, `lr * grad` is not the applied parameter update because moments, normalization, and decoupled weight decay alter the update.
+- `just audit` passes 19 authored Markdown files and three skills; its self-test rejects 11 injected defect types.
+
+## Results
+
+Documentation and audit chunk:
+
+```text
+$ just audit
+audit self-test: PASS (11 injected defects rejected)
+audit: PASS (19 authored Markdown files, 3 skills)
+```
+
+Fresh-eyes adversarial review rejected broken images, missing fragments, malformed quoted frontmatter, `L0`, reversed evidence ranges, and invalid fence info. Commits `fa534cf` and `0be4323` are pushed to `origin/main`.
+
+## TODO
+
+## Errors
+| Task | Error | Resolution |
+|------|-------|------------|
+| T1-T2 | Both `apply_patch` entry points failed because the sandbox could not configure loopback. | Used exact count-asserted replacements, then reviewed the complete `git diff`; no partial patch landed. |
+| T2 | The first audit treated an evidence directory as a file and escaped the self-test fence. | Restricted anchor reads to files and wrote a real fence. |
+| T3 | DeepSeek returned only a promise to inspect files; GLM produced no output in about 15 minutes. | Rejected both as failed reviews and dispatched a fresh-eyes repository review instead. |
+| T3 | Humanizer lint reports pre-existing file-wide bold-label and punctuation debt. | Kept this change scoped; the edited passages add none of the flagged patterns. |
+| T2 | Fresh-eyes review found broken image links, fragments, quoted YAML, `L0`, reversed ranges, and invalid fence info could pass silently. | Added each case to the parser and mutation suite. The reviewer reran all adversarial fixtures and changed R3 from FAIL to PASS. |

justfile

@@ -0,0 +1,5 @@
+default:
+  @just --list
+
+audit:
+  python3 scripts/audit.py --self-test .

pinn/SKILL.md

@@ -166,13 +166,15 @@ The PINN loss has multiple terms (PDE residual, BCs, ICs, data) with different g
 > Source: https://arxiv.org/abs/2001.04536, Algorithm 1
 > NeuralPDE.jl implements this as `GradientScaleAdaptiveLoss`.
 
-**3. Conflict-free gradient methods** (ConFIG, credence ~70%):
+**3. Gradient aggregation methods** (ConFIG or UPGrad):
 > Instead of summing loss gradients (which can cancel), project them into a conflict-free direction.
 > ConFIG: unit-normalize per-loss gradients, solve least-squares for combined direction, rescale by projection lengths.
 > Source: https://tum-pbs.github.io/ConFIG/
 > Key: must compute per-loss gradients separately (zero_grad + backward for each). Summing raw losses defeats the purpose.
 > M-ConFIG: momentum variant, updates only one loss's gradient per step. Use with SGD, not Adam (momentum conflict).
 
+ConFIG and UPGrad are both reasonable candidates when the losses cannot be replaced by hard constraints. Keep plain summation as a baseline. The current evidence does not establish one aggregation method as generally best.
+
 **4. Don't use multiple losses if you can avoid it.** A single well-posed loss is always better than a weighted sum. Can you reformulate BCs as hard constraints (e.g., multiply network output by a function that satisfies BCs)? Can you use a penalty method that naturally balances?
 
 **4b. Constrained optimization instead of penalized** (Brunton 2023, credence ~80%):
@@ -260,7 +262,7 @@ These apply to PINNs too:
 2. Get signs of life on a toy problem (1D, known solution, constant Cp)
 3. Overfit to training data first. If you can't overfit, you can't generalize.
 4. Log everything: losses per component, gradient norms per module, parameter norms, activation stats
-5. Numerical hygiene: `assert torch.isfinite(loss)`, `log(x.clamp(min=1e-8))`, `x / (std + 1e-5)`
+5. Numerical localization: insert `assert torch.isfinite(x).all()` after successive stages. Once you find the first invalid operation, use a stable formula that matches the intended mathematical domain.
 
 **Symptom table** (adapted for PINNs):
 
@@ -290,11 +292,13 @@ This takes 5 minutes and saves hours.
 
 ---
 
-## 8. Multi-Loss Training Details (ConFIG)
+## 8. Multi-Loss Training Details
 
 If you must use multiple loss terms (and in PINNs you usually must):
 
-### ConFIG (recommended over naive summation, credence ~70%)
+### ConFIG and UPGrad
+
+Both methods require separate per-loss gradients. The example below shows the ConFIG interface; TorchJD provides UPGrad and other aggregation methods.
 
 ```python
 # Per-loss gradient capture (ConFIG requires this)
@@ -323,10 +327,11 @@ optimizer.step()
 | Naive sum | Simple | Gradient conflict, dominant terms drown others |
 | GradientScaleAdaptiveLoss | Built into NeuralPDE.jl | Heuristic, EMA lag |
 | ReLoBRaLo | Effective on benchmarks | More complex |
-| ConFIG | Theoretically grounded, consistent wins | Per-loss backward required (2-3x cost) |
+| ConFIG | Conflict-free aggregate with a PINN-specific reference implementation | Per-loss backward required (2-3x cost) |
+| UPGrad | General gradient aggregation available in TorchJD | Per-loss backward required; limited PINN-specific comparative evidence |
 | Hard constraints | Eliminates BC loss entirely | Not always possible |
 
-> ConFIG authors claim superiority over PCGrad and Adam baseline on Burgers, Schrodinger, Kovasznay, Beltrami. UPGrad mentions ConFIG but ConFIG does not cite UPGrad.
+> ConFIG authors report improvements over PCGrad and an Adam baseline on Burgers, Schrodinger, Kovasznay, and Beltrami. This is author-reported evidence, not a general comparison with UPGrad.
 > Source: https://tum-pbs.github.io/ConFIG/
 
 ---

refs/diagnostics.md

@@ -94,10 +94,8 @@ for name, module in model.named_modules():
 # Run one forward pass. First module to raise = source of the NaN.
 ```
 
-**Random input test** [Slavv]
+**Input ablation test** [Slavv]
 ```python
-# Pass random noise instead of real data. If loss/error behaves the same,
-# the data pipeline is destroying information before the model sees it.
 model.eval()
 real_batch = next(iter(train_loader))
 fake_input = torch.randn_like(real_batch['input'])
@@ -106,13 +104,15 @@ with torch.no_grad():
     fake_out = model(fake_input)
     real_loss = loss_fn(real_out, real_batch['target']).item()
     fake_loss = loss_fn(fake_out, real_batch['target']).item()
+    output_change = (real_out - fake_out).float().square().mean().sqrt().item()
     print(f"Real input loss: {real_loss:.4f}")
     print(f"Random input loss: {fake_loss:.4f}")
-# If similar: model isn't using the input. Check preprocessing, data loading, feature selection.
-# If very different: model sees real signal. Problem is elsewhere.
+    print(f"Output RMS change: {output_change:.4f}")
 ```
 
-**NaN poisoning (leakage tracer)** [Wassname
+Run this after training. If replacing real inputs with shuffled or random inputs barely changes predictions or the metric, the model may not use the intended input signal. This does not identify the cause. Inspect preprocessing, model wiring, label leakage, and task bias. Similar loss values alone are weak evidence, especially near initialization.
+
+**NaN poisoning (leakage tracer)** [Wassname]
 ```python
 # Leakage can hide anywhere: normalization fit on the full dataset, target
 # leaking into features, window functions peeking ahead, bad splits. Instead
@@ -214,28 +214,29 @@ for conf, pred, true, idx in errors[:10]:
 # Inspect the actual inputs for these indices. Pattern = systematic bug.
 ```
 
-**Update-to-data ratio check** [Karpathy nn-zero-to-hero Lec 4; evidence: karpathy_nn_zero_to_hero_lec4_diagnostics.md]
+**Parameter-update ratio check** [adapted from Karpathy nn-zero-to-hero Lec 4; evidence: karpathy_nn_zero_to_hero_lec4_diagnostics.md]
 ```python
-# Track during training: how large are updates relative to parameter magnitudes?
-# Target: ~1e-3 (log10 ~ -3). Much higher = LR too large. Much lower = LR too small.
 ud = []
-# Inside training loop (after optimizer.step()):
+parameters_before = {
+    name: parameter.detach().clone()
+    for name, parameter in model.named_parameters()
+    if parameter.ndim >= 2
+}
+optimizer.step()
 with torch.no_grad():
     ud.append({
-        name: ((lr * p.grad).std() / p.data.std()).log10().item()
-        for name, p in model.named_parameters()
-        if p.grad is not None and p.ndim >= 2
+        name: ((parameter - parameters_before[name]).std() / parameters_before[name].std()).log10().item()
+        for name, parameter in model.named_parameters()
+        if parameter.ndim >= 2
     })
-# After training, plot per-layer ratios:
 import matplotlib.pyplot as plt
 for name in ud[0]:
     plt.plot([d[name] for d in ud], label=name)
-plt.axhline(-3, color='k', linestyle='--')  # target ratio
 plt.legend(); plt.ylabel('log10(update/param ratio)'); plt.show()
-# If a layer's ratio is much above -3: reduce LR or add gradient clipping.
-# If much below -3: that layer is barely updating -- possible dead/frozen layer.
 ```
 
+This measures the update actually applied by SGD, Adam, or AdamW, including optimizer state and weight decay. Compare layers and trends over time. Karpathy's rough $10^{-3}$ target came from a particular SGD setup, so it is a diagnostic reference rather than a universal threshold.
+
 **Weight/bias distribution check** [Slavv, CS231n]
 ```python
 for name, p in model.named_parameters():

rl/SKILL.md

@@ -139,7 +139,7 @@ Sometimes (rarely) you don't. Schulman:
 
 ## Sources
 
-**Evidence map**: [docs/ml_debug_folklore.argdown](../docs/ml_debug_folklore.argdown) traces each claim to verbatim quotes across 21 evidence files in [docs/evidence/](../docs/evidence/). Process log at [docs/ml_debug_folklore_log.md](../docs/ml_debug_folklore_log.md).
+**Evidence map**: [docs/ml_debug_folklore.argdown](../docs/ml_debug_folklore.argdown) traces claims to verbatim quotes in [docs/evidence/](../docs/evidence/).
 
 ### Talks
 - Schulman, "Nuts and Bolts of Deep RL Experimentation," Deep RL Bootcamp 2017

scripts/audit.py

@@ -0,0 +1,286 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import re
+import sys
+import tempfile
+from pathlib import Path
+from urllib.parse import unquote
+
+LINK_RE = re.compile(r"!?\[[^\]]*\]\(([^)]+)\)")
+FOOTNOTE_RE = re.compile(r"\[\^([^\]]+)\]")
+FOOTNOTE_DEF_RE = re.compile(r"^\[\^([^\]]+)\]:", re.MULTILINE)
+FENCE_RE = re.compile(r"^( {0,3})(`{3,}|~{3,})(.*)$")
+HEADING_RE = re.compile(r"^ {0,3}#{1,6}\s+(.+?)\s*#*\s*$")
+YAML_KEY_RE = re.compile(r"^([A-Za-z][A-Za-z0-9_-]*):", re.MULTILINE)
+LINE_ANCHOR_RE = re.compile(r"\bL(\d+)(?:-(\d+))?")
+
+
+def is_frozen_evidence(path: Path, root: Path) -> bool:
+    relative = "/" + path.relative_to(root).as_posix() + "/"
+    return "/docs/evidence/" in relative
+
+
+def authored_markdown(root: Path) -> list[Path]:
+    return [
+        path
+        for path in sorted(root.rglob("*.md"))
+        if ".git" not in path.parts and not is_frozen_evidence(path, root)
+    ]
+
+
+def prose_lines(path: Path, errors: list[str]) -> list[tuple[int, str]]:
+    lines = path.read_text(errors="replace").splitlines()
+    prose: list[tuple[int, str]] = []
+    opened: tuple[str, int, int] | None = None
+    for line_number, line in enumerate(lines, 1):
+        match = FENCE_RE.match(line)
+        if match:
+            marker = match.group(2)
+            remainder = match.group(3)
+            if opened is None:
+                if marker[0] == "`" and "`" in remainder:
+                    errors.append(f"{path}:{line_number}: backtick fence info contains a backtick")
+                opened = (marker[0], len(marker), line_number)
+            elif marker[0] == opened[0] and len(marker) >= opened[1]:
+                if remainder.strip():
+                    errors.append(f"{path}:{line_number}: closing fence has trailing text")
+                opened = None
+            continue
+        if opened is None:
+            prose.append((line_number, line))
+    if opened is not None:
+        errors.append(f"{path}: unclosed {opened[0] * opened[1]} fence from line {opened[2]}")
+    return prose
+
+
+def local_target(raw_target: str, source: Path) -> tuple[Path, str] | None:
+    target = raw_target.strip()
+    if target.startswith("<") and target.endswith(">"):
+        target = target[1:-1]
+    path_text, _, fragment = target.partition("#")
+    path_text = unquote(path_text)
+    if re.match(r"^[A-Za-z][A-Za-z0-9+.-]*:", path_text):
+        return None
+    path = source if not path_text else (source.parent / path_text).resolve()
+    return path, unquote(fragment)
+
+
+def markdown_anchors(path: Path) -> set[str]:
+    anchors: set[str] = set()
+    counts: dict[str, int] = {}
+    for line in path.read_text(errors="replace").splitlines():
+        match = HEADING_RE.match(line)
+        if not match:
+            continue
+        heading = re.sub(r"<[^>]+>", "", match.group(1))
+        explicit = re.search(r"\{#([A-Za-z0-9_.:-]+)\}\s*$", heading)
+        if explicit:
+            anchors.add(explicit.group(1))
+            heading = heading[: explicit.start()].rstrip()
+        slug = re.sub(r"[^\w\s-]", "", heading.lower())
+        slug = re.sub(r"[\s-]+", "-", slug).strip("-")
+        duplicate = counts.get(slug, 0)
+        counts[slug] = duplicate + 1
+        anchors.add(slug if duplicate == 0 else f"{slug}-{duplicate}")
+    return anchors
+
+
+def check_frontmatter(root: Path, errors: list[str]) -> None:
+    for path in sorted(root.rglob("SKILL.md")):
+        if ".git" in path.parts:
+            continue
+        lines = path.read_text(errors="replace").splitlines()
+        if not lines or lines[0] != "---":
+            errors.append(f"{path}: missing YAML frontmatter")
+            continue
+        try:
+            closing = lines.index("---", 1)
+        except ValueError:
+            errors.append(f"{path}: unclosed YAML frontmatter")
+            continue
+        keys = YAML_KEY_RE.findall("\n".join(lines[1:closing]))
+        if sorted(keys) != ["description", "name"]:
+            errors.append(f"{path}: frontmatter keys must be name and description, got {keys}")
+        values = {}
+        for line in lines[1:closing]:
+            if ":" not in line:
+                errors.append(f"{path}: invalid frontmatter line {line!r}")
+                continue
+            key, raw_value = line.split(":", 1)
+            value = raw_value.strip()
+            if value[:1] in {"\"", "'"} and (len(value) < 2 or value[-1] != value[0]):
+                errors.append(f"{path}: unterminated quoted frontmatter value for {key}")
+            values[key] = value.strip("'\"")
+        if not values.get("name") or not values.get("description"):
+            errors.append(f"{path}: name and description must be nonempty")
+        elif not re.fullmatch(r"[a-z0-9-]+", values["name"]):
+            errors.append(f"{path}: invalid skill name {values['name']!r}")
+
+
+def audit(root: Path) -> list[str]:
+    root = root.resolve()
+    errors: list[str] = []
+    check_frontmatter(root, errors)
+    for path in authored_markdown(root):
+        prose = prose_lines(path, errors)
+        prose_text = "\n".join(line for _, line in prose)
+        definitions = set(FOOTNOTE_DEF_RE.findall(prose_text))
+        uses = set(FOOTNOTE_RE.findall(prose_text))
+        for missing in sorted(uses - definitions):
+            errors.append(f"{path}: missing footnote definition [^{missing}]")
+        for line_number, line in prose:
+            if line.count("[") != line.count("]"):
+                errors.append(f"{path}:{line_number}: unbalanced square brackets")
+            for match in LINK_RE.finditer(line):
+                resolved = local_target(match.group(1), path)
+                if resolved is None:
+                    continue
+                target, fragment = resolved
+                if not target.exists():
+                    errors.append(f"{path}:{line_number}: missing local link {match.group(1)!r}")
+                    continue
+                if fragment and target.is_file() and target.suffix.lower() == ".md":
+                    if fragment not in markdown_anchors(target):
+                        errors.append(f"{path}:{line_number}: missing Markdown fragment #{fragment}")
+                relative = "/" + target.as_posix() + "/"
+                if "/docs/evidence/" not in relative or not target.is_file():
+                    continue
+                line_count = len(target.read_text(errors="replace").splitlines())
+                for anchor in LINE_ANCHOR_RE.finditer(line[match.end():]):
+                    start = int(anchor.group(1))
+                    end = int(anchor.group(2) or anchor.group(1))
+                    if start < 1 or end < start or end > line_count:
+                        errors.append(
+                            f"{path}:{line_number}: invalid evidence anchor L{start}-{end} for "
+                            f"{target} ({line_count} lines)"
+                        )
+    return errors
+
+
+def write_fixture(root: Path) -> None:
+    (root / "docs/evidence").mkdir(parents=True)
+    (root / "docs/evidence/source.md").write_text("one\ntwo\nthree\n")
+    (root / "SKILL.md").write_text(
+        "---\nname: fixture\ndescription: fixture skill\n---\n\n# Fixture\n"
+    )
+    (root / "guide.md").write_text(
+        "# Fixture section\n"
+        "[skill](SKILL.md)\n"
+        "[section](guide.md#fixture-section)\n"
+        "![image](docs/evidence/source.md)\n"
+        "[cache](docs/evidence/source.md): L2-3\n"
+        "Claim.[^source]\n\n"
+        "[^source]: Evidence.\n"
+    )
+
+
+def self_test() -> None:
+    mutations = [
+        (
+            "missing local link",
+            lambda root: (root / "guide.md").write_text(
+                (root / "guide.md").read_text() + "\n[broken](missing.md)\n"
+            ),
+        ),
+        (
+            "missing local link",
+            lambda root: (root / "guide.md").write_text(
+                (root / "guide.md").read_text().replace(
+                    "docs/evidence/source.md)", "missing-image.png)", 1
+                )
+            ),
+        ),
+        (
+            "unclosed",
+            lambda root: (root / "guide.md").write_text(
+                (root / "guide.md").read_text() + "\n```python\n"
+            ),
+        ),
+        (
+            "backtick fence info",
+            lambda root: (root / "guide.md").write_text(
+                (root / "guide.md").read_text() + "\n```py`thon\n```\n"
+            ),
+        ),
+        (
+            "missing footnote",
+            lambda root: (root / "guide.md").write_text(
+                (root / "guide.md").read_text() + "\nMissing.[^absent]\n"
+            ),
+        ),
+        (
+            "frontmatter keys",
+            lambda root: (root / "SKILL.md").write_text(
+                (root / "SKILL.md").read_text().replace("name:", "title:")
+            ),
+        ),
+        (
+            "unterminated quoted frontmatter",
+            lambda root: (root / "SKILL.md").write_text(
+                (root / "SKILL.md").read_text().replace(
+                    "description: fixture skill", 'description: "fixture skill'
+                )
+            ),
+        ),
+        (
+            "invalid evidence anchor",
+            lambda root: (root / "guide.md").write_text(
+                (root / "guide.md").read_text().replace("L2-3", "L0")
+            ),
+        ),
+        (
+            "invalid evidence anchor",
+            lambda root: (root / "guide.md").write_text(
+                (root / "guide.md").read_text().replace("L2-3", "L3-2")
+            ),
+        ),
+        (
+            "missing Markdown fragment",
+            lambda root: (root / "guide.md").write_text(
+                (root / "guide.md").read_text().replace(
+                    "#fixture-section", "#missing-section"
+                )
+            ),
+        ),
+        (
+            "unbalanced square brackets",
+            lambda root: (root / "guide.md").write_text(
+                (root / "guide.md").read_text() + "\n**Broken** [label\n"
+            ),
+        ),
+    ]
+    with tempfile.TemporaryDirectory() as directory:
+        clean = Path(directory) / "clean"
+        write_fixture(clean)
+        assert not audit(clean), audit(clean)
+    for expected, mutate in mutations:
+        with tempfile.TemporaryDirectory() as directory:
+            root = Path(directory)
+            write_fixture(root)
+            mutate(root)
+            errors = audit(root)
+            assert any(expected in error for error in errors), (expected, errors)
+    print(f"audit self-test: PASS ({len(mutations)} injected defects rejected)")
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("root", nargs="?", type=Path, default=Path.cwd())
+    parser.add_argument("--self-test", action="store_true")
+    args = parser.parse_args()
+    errors = audit(args.root)
+    if errors:
+        print("\n".join(errors))
+        return 1
+    if args.self_test:
+        self_test()
+    print(
+        f"audit: PASS ({len(authored_markdown(args.root.resolve()))} authored Markdown files, "
+        f"{len(list(args.root.resolve().rglob('SKILL.md')))} skills)"
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())