Add fail-fast skill audit

docs/spec/20260712_epistemic_audit_benchmark.md

@@ -0,0 +1,72 @@
+# Tighten debugging claims, audit the skill, and benchmark agent behavior
+
+## Goal
+Remove overconfident debugging claims, add a cheap repository audit, then measure whether loading `ml-debug` improves agent diagnosis on seeded failures.
+
+## Scope
+In: `PLAYBOOK.md`, `refs/diagnostics.md`, `pinn/SKILL.md`, authored-markdown integrity checks, a `just audit` recipe, and an isolated A/B benchmark in a git worktree.
+Out: end-to-end case-study prose, new debugging domains, compatibility shims, and unrelated evidence collection.
+
+## Requirements
+- R1: Advice must separate observations from diagnoses. Done means the random-input, train/validation, NaN, failure-prior, and update-ratio passages no longer claim more than their checks establish. VERIFY: targeted searches plus human review of the changed paragraphs.
+- R2: PINN optimizer guidance must present ConFIG and UPGrad as plausible methods, without unsupported numeric credence or declaring one generally superior. VERIFY: the section contains both methods and no `credence ~70%` or `consistent wins` claim.
+- R3: `just audit` must fail on broken authored links, malformed Markdown fences, missing footnote definitions, invalid skill frontmatter, and out-of-range local evidence line anchors. VERIFY: it passes on the repository and fails when each defect is injected into a temporary copy.
+- R4: The benchmark must contain 8-12 seeded ML failures and compare fresh agent diagnoses with and without `ml-debug`. Done means a machine-readable results table reports root-cause accuracy, localization-before-fix, discriminating-test choice, unsupported behavior changes, and silent fallbacks for both conditions. VERIFY: rerun the scorer from raw outputs and reproduce the summary.
+- R5: Benchmark construction and execution must happen in a separate git worktree. VERIFY: `git worktree list` and the benchmark commit path show an isolated worktree branch.
+
+## Tasks
+- [x] T1 (R1-R2): Correct the overconfident documentation.
+  - steps: remove numeric pseudo-priors; qualify random-input evidence; rewrite train/validation triage; replace blanket clamp/epsilon advice with finite assertions and localization; distinguish the Adam update from `lr * grad`; soften PINN optimizer comparison and include UPGrad
+  - verify: `rg -n 'Data pipeline.*~40%|Loss function.*~20%|Conflict-free gradient methods.*credence|ConFIG.*recommended|consistent wins|Overfitting, not a bug|add clamp/eps|update-to-data' PLAYBOOK.md refs/diagnostics.md pinn/SKILL.md`
+  - success: none of the rejected claims remain; replacement text names what each check establishes
+  - likely_fail: old overconfident wording survives in one duplicate location; repository-wide search catches it
+  - sneaky_fail: prose changes but still localizes a cause from weak evidence; manual observation-versus-inference review catches it
+  - UAT: "when I follow a diagnostic, it tells me what I observed, what remains plausible, and where to inspect next"
+- [x] T2 (R3): Add the authored-markdown and skill audit.
+  - steps: add one small audit script and a `just audit` recipe; exclude frozen scraped evidence from ordinary link checking while validating explicit evidence anchors used by authored files
+  - verify: `just --dry-run audit && just audit`
+  - success: the clean repository passes with a short summary
+  - likely_fail: the known deleted RL process-log link or malformed heading fails the first run; fix the source
+  - sneaky_fail: the audit always exits zero; mutation tests inject one defect per check and require nonzero exit
+  - UAT: "when an authored link, fence, footnote, frontmatter, or evidence anchor breaks, `just audit` names the file and exits nonzero"
+- [/] T3 (R1-R3): Review, humanize, verify, commit, and push the documentation/audit chunk.
+  - verify: `python3 /home/wassname/.agents/skills/humanizer/lint.py --help`, the selected lint command, `just audit`, `git diff --check`, and external review
+  - success: checks pass and review findings are resolved or recorded
+  - likely_fail: humanizer catches repeated AI patterns or external review finds an overclaim; revise and rerun
+  - sneaky_fail: checks pass but user-facing meaning regresses; fresh-eyes review compares the changed passages to R1-R2
+  - UAT: "the committed diff is small, readable, and its audit output is linked in this spec"
+- [ ] T4 (R4-R5): Build the seeded-failure benchmark in a separate worktree.
+  - steps: create 8-12 compact cases with hidden answer keys; run fresh agent sessions in control and skill conditions; retain raw outputs; score only explicit evidence in outputs
+  - verify: benchmark validation command checks case count, unique IDs, hidden keys, raw output completeness, and score reproducibility
+  - success: both conditions have the same cases and model settings, with no answer-key leakage
+  - likely_fail: agent runner or model access is unavailable; record the exact failure and keep a runnable harness
+  - sneaky_fail: treatment prompt leaks intended diagnoses or cases are easier in one condition; prompt diff and case-ID pairing checks catch it
+  - UAT: "I can inspect each raw diagnosis and reproduce the aggregate A/B table from it"
+- [ ] T5 (R4-R5): Fresh-eyes review the benchmark evidence, then merge the completed benchmark chunk.
+  - verify: reviewer reproduces scoring for a sample without seeing aggregate conclusions, then `git diff --check` and benchmark audit pass
+  - success: reviewer agrees with the sampled scores or corrections are applied before merge
+  - likely_fail: rubric requires subjective reconstruction; tighten evidence fields and rescore
+  - sneaky_fail: scorer rewards verbosity or keyword matching rather than diagnosis; reviewer checks decisions against raw outputs and answer keys
+  - UAT: "the final results table links to raw outputs and survives independent rescoring"
+
+## Context
+- The repo is fail-fast research code: checks should raise on invalid state rather than clamp, fill, skip, or fall back.
+- Frozen `docs/evidence/` files contain scraped links that are not expected to resolve locally. Authored files should resolve all local links.
+- The user rejected adding worked case studies because they may make agents hyper-focus on the examples.
+- The benchmark is last and must use a worktree.
+
+## Log
+- Precise failure percentages in `PLAYBOOK.md` are qualitative practitioner ordering presented with unsupported numeric precision.
+- For Adam/AdamW, `lr * grad` is not the applied parameter update because moments, normalization, and decoupled weight decay alter the update.
+- `just audit` passes 19 authored Markdown files and three skills; its self-test rejects 11 injected defect types.
+
+## TODO
+
+## Errors
+| Task | Error | Resolution |
+|------|-------|------------|
+| T1-T2 | Both `apply_patch` entry points failed because the sandbox could not configure loopback. | Used exact count-asserted replacements, then reviewed the complete `git diff`; no partial patch landed. |
+| T2 | The first audit treated an evidence directory as a file and escaped the self-test fence. | Restricted anchor reads to files and wrote a real fence. |
+| T3 | DeepSeek returned only a promise to inspect files; GLM produced no output in about 15 minutes. | Rejected both as failed reviews and dispatched a fresh-eyes repository review instead. |
+| T3 | Humanizer lint reports pre-existing file-wide bold-label and punctuation debt. | Kept this change scoped; the edited passages add none of the flagged patterns. |
+| T2 | Fresh-eyes review found broken image links, fragments, quoted YAML, `L0`, reversed ranges, and invalid fence info could pass silently. | Added each case to the parser and mutation suite. The reviewer reran all adversarial fixtures and changed R3 from FAIL to PASS. |

justfile

@@ -0,0 +1,5 @@
+default:
+  @just --list
+
+audit:
+  python3 scripts/audit.py --self-test .

rl/SKILL.md

@@ -139,7 +139,7 @@ Sometimes (rarely) you don't. Schulman:
 
 ## Sources
 
-**Evidence map**: [docs/ml_debug_folklore.argdown](../docs/ml_debug_folklore.argdown) traces each claim to verbatim quotes across 21 evidence files in [docs/evidence/](../docs/evidence/). Process log at [docs/ml_debug_folklore_log.md](../docs/ml_debug_folklore_log.md).
+**Evidence map**: [docs/ml_debug_folklore.argdown](../docs/ml_debug_folklore.argdown) traces claims to verbatim quotes in [docs/evidence/](../docs/evidence/).
 
 ### Talks
 - Schulman, "Nuts and Bolts of Deep RL Experimentation," Deep RL Bootcamp 2017

scripts/audit.py

@@ -0,0 +1,286 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import re
+import sys
+import tempfile
+from pathlib import Path
+from urllib.parse import unquote
+
+LINK_RE = re.compile(r"!?\[[^\]]*\]\(([^)]+)\)")
+FOOTNOTE_RE = re.compile(r"\[\^([^\]]+)\]")
+FOOTNOTE_DEF_RE = re.compile(r"^\[\^([^\]]+)\]:", re.MULTILINE)
+FENCE_RE = re.compile(r"^( {0,3})(`{3,}|~{3,})(.*)$")
+HEADING_RE = re.compile(r"^ {0,3}#{1,6}\s+(.+?)\s*#*\s*$")
+YAML_KEY_RE = re.compile(r"^([A-Za-z][A-Za-z0-9_-]*):", re.MULTILINE)
+LINE_ANCHOR_RE = re.compile(r"\bL(\d+)(?:-(\d+))?")
+
+
+def is_frozen_evidence(path: Path, root: Path) -> bool:
+    relative = "/" + path.relative_to(root).as_posix() + "/"
+    return "/docs/evidence/" in relative
+
+
+def authored_markdown(root: Path) -> list[Path]:
+    return [
+        path
+        for path in sorted(root.rglob("*.md"))
+        if ".git" not in path.parts and not is_frozen_evidence(path, root)
+    ]
+
+
+def prose_lines(path: Path, errors: list[str]) -> list[tuple[int, str]]:
+    lines = path.read_text(errors="replace").splitlines()
+    prose: list[tuple[int, str]] = []
+    opened: tuple[str, int, int] | None = None
+    for line_number, line in enumerate(lines, 1):
+        match = FENCE_RE.match(line)
+        if match:
+            marker = match.group(2)
+            remainder = match.group(3)
+            if opened is None:
+                if marker[0] == "`" and "`" in remainder:
+                    errors.append(f"{path}:{line_number}: backtick fence info contains a backtick")
+                opened = (marker[0], len(marker), line_number)
+            elif marker[0] == opened[0] and len(marker) >= opened[1]:
+                if remainder.strip():
+                    errors.append(f"{path}:{line_number}: closing fence has trailing text")
+                opened = None
+            continue
+        if opened is None:
+            prose.append((line_number, line))
+    if opened is not None:
+        errors.append(f"{path}: unclosed {opened[0] * opened[1]} fence from line {opened[2]}")
+    return prose
+
+
+def local_target(raw_target: str, source: Path) -> tuple[Path, str] | None:
+    target = raw_target.strip()
+    if target.startswith("<") and target.endswith(">"):
+        target = target[1:-1]
+    path_text, _, fragment = target.partition("#")
+    path_text = unquote(path_text)
+    if re.match(r"^[A-Za-z][A-Za-z0-9+.-]*:", path_text):
+        return None
+    path = source if not path_text else (source.parent / path_text).resolve()
+    return path, unquote(fragment)
+
+
+def markdown_anchors(path: Path) -> set[str]:
+    anchors: set[str] = set()
+    counts: dict[str, int] = {}
+    for line in path.read_text(errors="replace").splitlines():
+        match = HEADING_RE.match(line)
+        if not match:
+            continue
+        heading = re.sub(r"<[^>]+>", "", match.group(1))
+        explicit = re.search(r"\{#([A-Za-z0-9_.:-]+)\}\s*$", heading)
+        if explicit:
+            anchors.add(explicit.group(1))
+            heading = heading[: explicit.start()].rstrip()
+        slug = re.sub(r"[^\w\s-]", "", heading.lower())
+        slug = re.sub(r"[\s-]+", "-", slug).strip("-")
+        duplicate = counts.get(slug, 0)
+        counts[slug] = duplicate + 1
+        anchors.add(slug if duplicate == 0 else f"{slug}-{duplicate}")
+    return anchors
+
+
+def check_frontmatter(root: Path, errors: list[str]) -> None:
+    for path in sorted(root.rglob("SKILL.md")):
+        if ".git" in path.parts:
+            continue
+        lines = path.read_text(errors="replace").splitlines()
+        if not lines or lines[0] != "---":
+            errors.append(f"{path}: missing YAML frontmatter")
+            continue
+        try:
+            closing = lines.index("---", 1)
+        except ValueError:
+            errors.append(f"{path}: unclosed YAML frontmatter")
+            continue
+        keys = YAML_KEY_RE.findall("\n".join(lines[1:closing]))
+        if sorted(keys) != ["description", "name"]:
+            errors.append(f"{path}: frontmatter keys must be name and description, got {keys}")
+        values = {}
+        for line in lines[1:closing]:
+            if ":" not in line:
+                errors.append(f"{path}: invalid frontmatter line {line!r}")
+                continue
+            key, raw_value = line.split(":", 1)
+            value = raw_value.strip()
+            if value[:1] in {"\"", "'"} and (len(value) < 2 or value[-1] != value[0]):
+                errors.append(f"{path}: unterminated quoted frontmatter value for {key}")
+            values[key] = value.strip("'\"")
+        if not values.get("name") or not values.get("description"):
+            errors.append(f"{path}: name and description must be nonempty")
+        elif not re.fullmatch(r"[a-z0-9-]+", values["name"]):
+            errors.append(f"{path}: invalid skill name {values['name']!r}")
+
+
+def audit(root: Path) -> list[str]:
+    root = root.resolve()
+    errors: list[str] = []
+    check_frontmatter(root, errors)
+    for path in authored_markdown(root):
+        prose = prose_lines(path, errors)
+        prose_text = "\n".join(line for _, line in prose)
+        definitions = set(FOOTNOTE_DEF_RE.findall(prose_text))
+        uses = set(FOOTNOTE_RE.findall(prose_text))
+        for missing in sorted(uses - definitions):
+            errors.append(f"{path}: missing footnote definition [^{missing}]")
+        for line_number, line in prose:
+            if line.count("[") != line.count("]"):
+                errors.append(f"{path}:{line_number}: unbalanced square brackets")
+            for match in LINK_RE.finditer(line):
+                resolved = local_target(match.group(1), path)
+                if resolved is None:
+                    continue
+                target, fragment = resolved
+                if not target.exists():
+                    errors.append(f"{path}:{line_number}: missing local link {match.group(1)!r}")
+                    continue
+                if fragment and target.is_file() and target.suffix.lower() == ".md":
+                    if fragment not in markdown_anchors(target):
+                        errors.append(f"{path}:{line_number}: missing Markdown fragment #{fragment}")
+                relative = "/" + target.as_posix() + "/"
+                if "/docs/evidence/" not in relative or not target.is_file():
+                    continue
+                line_count = len(target.read_text(errors="replace").splitlines())
+                for anchor in LINE_ANCHOR_RE.finditer(line[match.end():]):
+                    start = int(anchor.group(1))
+                    end = int(anchor.group(2) or anchor.group(1))
+                    if start < 1 or end < start or end > line_count:
+                        errors.append(
+                            f"{path}:{line_number}: invalid evidence anchor L{start}-{end} for "
+                            f"{target} ({line_count} lines)"
+                        )
+    return errors
+
+
+def write_fixture(root: Path) -> None:
+    (root / "docs/evidence").mkdir(parents=True)
+    (root / "docs/evidence/source.md").write_text("one\ntwo\nthree\n")
+    (root / "SKILL.md").write_text(
+        "---\nname: fixture\ndescription: fixture skill\n---\n\n# Fixture\n"
+    )
+    (root / "guide.md").write_text(
+        "# Fixture section\n"
+        "[skill](SKILL.md)\n"
+        "[section](guide.md#fixture-section)\n"
+        "![image](docs/evidence/source.md)\n"
+        "[cache](docs/evidence/source.md): L2-3\n"
+        "Claim.[^source]\n\n"
+        "[^source]: Evidence.\n"
+    )
+
+
+def self_test() -> None:
+    mutations = [
+        (
+            "missing local link",
+            lambda root: (root / "guide.md").write_text(
+                (root / "guide.md").read_text() + "\n[broken](missing.md)\n"
+            ),
+        ),
+        (
+            "missing local link",
+            lambda root: (root / "guide.md").write_text(
+                (root / "guide.md").read_text().replace(
+                    "docs/evidence/source.md)", "missing-image.png)", 1
+                )
+            ),
+        ),
+        (
+            "unclosed",
+            lambda root: (root / "guide.md").write_text(
+                (root / "guide.md").read_text() + "\n```python\n"
+            ),
+        ),
+        (
+            "backtick fence info",
+            lambda root: (root / "guide.md").write_text(
+                (root / "guide.md").read_text() + "\n```py`thon\n```\n"
+            ),
+        ),
+        (
+            "missing footnote",
+            lambda root: (root / "guide.md").write_text(
+                (root / "guide.md").read_text() + "\nMissing.[^absent]\n"
+            ),
+        ),
+        (
+            "frontmatter keys",
+            lambda root: (root / "SKILL.md").write_text(
+                (root / "SKILL.md").read_text().replace("name:", "title:")
+            ),
+        ),
+        (
+            "unterminated quoted frontmatter",
+            lambda root: (root / "SKILL.md").write_text(
+                (root / "SKILL.md").read_text().replace(
+                    "description: fixture skill", 'description: "fixture skill'
+                )
+            ),
+        ),
+        (
+            "invalid evidence anchor",
+            lambda root: (root / "guide.md").write_text(
+                (root / "guide.md").read_text().replace("L2-3", "L0")
+            ),
+        ),
+        (
+            "invalid evidence anchor",
+            lambda root: (root / "guide.md").write_text(
+                (root / "guide.md").read_text().replace("L2-3", "L3-2")
+            ),
+        ),
+        (
+            "missing Markdown fragment",
+            lambda root: (root / "guide.md").write_text(
+                (root / "guide.md").read_text().replace(
+                    "#fixture-section", "#missing-section"
+                )
+            ),
+        ),
+        (
+            "unbalanced square brackets",
+            lambda root: (root / "guide.md").write_text(
+                (root / "guide.md").read_text() + "\n**Broken** [label\n"
+            ),
+        ),
+    ]
+    with tempfile.TemporaryDirectory() as directory:
+        clean = Path(directory) / "clean"
+        write_fixture(clean)
+        assert not audit(clean), audit(clean)
+    for expected, mutate in mutations:
+        with tempfile.TemporaryDirectory() as directory:
+            root = Path(directory)
+            write_fixture(root)
+            mutate(root)
+            errors = audit(root)
+            assert any(expected in error for error in errors), (expected, errors)
+    print(f"audit self-test: PASS ({len(mutations)} injected defects rejected)")
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("root", nargs="?", type=Path, default=Path.cwd())
+    parser.add_argument("--self-test", action="store_true")
+    args = parser.parse_args()
+    errors = audit(args.root)
+    if errors:
+        print("\n".join(errors))
+        return 1
+    if args.self_test:
+        self_test()
+    print(
+        f"audit: PASS ({len(authored_markdown(args.root.resolve()))} authored Markdown files, "
+        f"{len(list(args.root.resolve().rglob('SKILL.md')))} skills)"
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())