diff --git a/docs/spec/20260712_epistemic_audit_benchmark.md b/docs/spec/20260712_epistemic_audit_benchmark.md new file mode 100644 index 0000000..9dfae44 --- /dev/null +++ b/docs/spec/20260712_epistemic_audit_benchmark.md @@ -0,0 +1,72 @@ +# Tighten debugging claims, audit the skill, and benchmark agent behavior + +## Goal +Remove overconfident debugging claims, add a cheap repository audit, then measure whether loading `ml-debug` improves agent diagnosis on seeded failures. + +## Scope +In: `PLAYBOOK.md`, `refs/diagnostics.md`, `pinn/SKILL.md`, authored-markdown integrity checks, a `just audit` recipe, and an isolated A/B benchmark in a git worktree. +Out: end-to-end case-study prose, new debugging domains, compatibility shims, and unrelated evidence collection. + +## Requirements +- R1: Advice must separate observations from diagnoses. Done means the random-input, train/validation, NaN, failure-prior, and update-ratio passages no longer claim more than their checks establish. VERIFY: targeted searches plus human review of the changed paragraphs. +- R2: PINN optimizer guidance must present ConFIG and UPGrad as plausible methods, without unsupported numeric credence or declaring one generally superior. VERIFY: the section contains both methods and no `credence ~70%` or `consistent wins` claim. +- R3: `just audit` must fail on broken authored links, malformed Markdown fences, missing footnote definitions, invalid skill frontmatter, and out-of-range local evidence line anchors. VERIFY: it passes on the repository and fails when each defect is injected into a temporary copy. +- R4: The benchmark must contain 8-12 seeded ML failures and compare fresh agent diagnoses with and without `ml-debug`. Done means a machine-readable results table reports root-cause accuracy, localization-before-fix, discriminating-test choice, unsupported behavior changes, and silent fallbacks for both conditions. VERIFY: rerun the scorer from raw outputs and reproduce the summary. +- R5: Benchmark construction and execution must happen in a separate git worktree. VERIFY: `git worktree list` and the benchmark commit path show an isolated worktree branch. + +## Tasks +- [x] T1 (R1-R2): Correct the overconfident documentation. + - steps: remove numeric pseudo-priors; qualify random-input evidence; rewrite train/validation triage; replace blanket clamp/epsilon advice with finite assertions and localization; distinguish the Adam update from `lr * grad`; soften PINN optimizer comparison and include UPGrad + - verify: `rg -n 'Data pipeline.*~40%|Loss function.*~20%|Conflict-free gradient methods.*credence|ConFIG.*recommended|consistent wins|Overfitting, not a bug|add clamp/eps|update-to-data' PLAYBOOK.md refs/diagnostics.md pinn/SKILL.md` + - success: none of the rejected claims remain; replacement text names what each check establishes + - likely_fail: old overconfident wording survives in one duplicate location; repository-wide search catches it + - sneaky_fail: prose changes but still localizes a cause from weak evidence; manual observation-versus-inference review catches it + - UAT: "when I follow a diagnostic, it tells me what I observed, what remains plausible, and where to inspect next" +- [x] T2 (R3): Add the authored-markdown and skill audit. + - steps: add one small audit script and a `just audit` recipe; exclude frozen scraped evidence from ordinary link checking while validating explicit evidence anchors used by authored files + - verify: `just --dry-run audit && just audit` + - success: the clean repository passes with a short summary + - likely_fail: the known deleted RL process-log link or malformed heading fails the first run; fix the source + - sneaky_fail: the audit always exits zero; mutation tests inject one defect per check and require nonzero exit + - UAT: "when an authored link, fence, footnote, frontmatter, or evidence anchor breaks, `just audit` names the file and exits nonzero" +- [/] T3 (R1-R3): Review, humanize, verify, commit, and push the documentation/audit chunk. + - verify: `python3 /home/wassname/.agents/skills/humanizer/lint.py --help`, the selected lint command, `just audit`, `git diff --check`, and external review + - success: checks pass and review findings are resolved or recorded + - likely_fail: humanizer catches repeated AI patterns or external review finds an overclaim; revise and rerun + - sneaky_fail: checks pass but user-facing meaning regresses; fresh-eyes review compares the changed passages to R1-R2 + - UAT: "the committed diff is small, readable, and its audit output is linked in this spec" +- [ ] T4 (R4-R5): Build the seeded-failure benchmark in a separate worktree. + - steps: create 8-12 compact cases with hidden answer keys; run fresh agent sessions in control and skill conditions; retain raw outputs; score only explicit evidence in outputs + - verify: benchmark validation command checks case count, unique IDs, hidden keys, raw output completeness, and score reproducibility + - success: both conditions have the same cases and model settings, with no answer-key leakage + - likely_fail: agent runner or model access is unavailable; record the exact failure and keep a runnable harness + - sneaky_fail: treatment prompt leaks intended diagnoses or cases are easier in one condition; prompt diff and case-ID pairing checks catch it + - UAT: "I can inspect each raw diagnosis and reproduce the aggregate A/B table from it" +- [ ] T5 (R4-R5): Fresh-eyes review the benchmark evidence, then merge the completed benchmark chunk. + - verify: reviewer reproduces scoring for a sample without seeing aggregate conclusions, then `git diff --check` and benchmark audit pass + - success: reviewer agrees with the sampled scores or corrections are applied before merge + - likely_fail: rubric requires subjective reconstruction; tighten evidence fields and rescore + - sneaky_fail: scorer rewards verbosity or keyword matching rather than diagnosis; reviewer checks decisions against raw outputs and answer keys + - UAT: "the final results table links to raw outputs and survives independent rescoring" + +## Context +- The repo is fail-fast research code: checks should raise on invalid state rather than clamp, fill, skip, or fall back. +- Frozen `docs/evidence/` files contain scraped links that are not expected to resolve locally. Authored files should resolve all local links. +- The user rejected adding worked case studies because they may make agents hyper-focus on the examples. +- The benchmark is last and must use a worktree. + +## Log +- Precise failure percentages in `PLAYBOOK.md` are qualitative practitioner ordering presented with unsupported numeric precision. +- For Adam/AdamW, `lr * grad` is not the applied parameter update because moments, normalization, and decoupled weight decay alter the update. +- `just audit` passes 19 authored Markdown files and three skills; its self-test rejects 11 injected defect types. + +## TODO + +## Errors +| Task | Error | Resolution | +|------|-------|------------| +| T1-T2 | Both `apply_patch` entry points failed because the sandbox could not configure loopback. | Used exact count-asserted replacements, then reviewed the complete `git diff`; no partial patch landed. | +| T2 | The first audit treated an evidence directory as a file and escaped the self-test fence. | Restricted anchor reads to files and wrote a real fence. | +| T3 | DeepSeek returned only a promise to inspect files; GLM produced no output in about 15 minutes. | Rejected both as failed reviews and dispatched a fresh-eyes repository review instead. | +| T3 | Humanizer lint reports pre-existing file-wide bold-label and punctuation debt. | Kept this change scoped; the edited passages add none of the flagged patterns. | +| T2 | Fresh-eyes review found broken image links, fragments, quoted YAML, `L0`, reversed ranges, and invalid fence info could pass silently. | Added each case to the parser and mutation suite. The reviewer reran all adversarial fixtures and changed R3 from FAIL to PASS. | diff --git a/justfile b/justfile new file mode 100644 index 0000000..cc09816 --- /dev/null +++ b/justfile @@ -0,0 +1,5 @@ +default: + @just --list + +audit: + python3 scripts/audit.py --self-test . diff --git a/rl/SKILL.md b/rl/SKILL.md index a9e7af0..dbe3255 100644 --- a/rl/SKILL.md +++ b/rl/SKILL.md @@ -139,7 +139,7 @@ Sometimes (rarely) you don't. Schulman: ## Sources -**Evidence map**: [docs/ml_debug_folklore.argdown](../docs/ml_debug_folklore.argdown) traces each claim to verbatim quotes across 21 evidence files in [docs/evidence/](../docs/evidence/). Process log at [docs/ml_debug_folklore_log.md](../docs/ml_debug_folklore_log.md). +**Evidence map**: [docs/ml_debug_folklore.argdown](../docs/ml_debug_folklore.argdown) traces claims to verbatim quotes in [docs/evidence/](../docs/evidence/). ### Talks - Schulman, "Nuts and Bolts of Deep RL Experimentation," Deep RL Bootcamp 2017 diff --git a/scripts/audit.py b/scripts/audit.py new file mode 100644 index 0000000..5d566da --- /dev/null +++ b/scripts/audit.py @@ -0,0 +1,286 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import re +import sys +import tempfile +from pathlib import Path +from urllib.parse import unquote + +LINK_RE = re.compile(r"!?\[[^\]]*\]\(([^)]+)\)") +FOOTNOTE_RE = re.compile(r"\[\^([^\]]+)\]") +FOOTNOTE_DEF_RE = re.compile(r"^\[\^([^\]]+)\]:", re.MULTILINE) +FENCE_RE = re.compile(r"^( {0,3})(`{3,}|~{3,})(.*)$") +HEADING_RE = re.compile(r"^ {0,3}#{1,6}\s+(.+?)\s*#*\s*$") +YAML_KEY_RE = re.compile(r"^([A-Za-z][A-Za-z0-9_-]*):", re.MULTILINE) +LINE_ANCHOR_RE = re.compile(r"\bL(\d+)(?:-(\d+))?") + + +def is_frozen_evidence(path: Path, root: Path) -> bool: + relative = "/" + path.relative_to(root).as_posix() + "/" + return "/docs/evidence/" in relative + + +def authored_markdown(root: Path) -> list[Path]: + return [ + path + for path in sorted(root.rglob("*.md")) + if ".git" not in path.parts and not is_frozen_evidence(path, root) + ] + + +def prose_lines(path: Path, errors: list[str]) -> list[tuple[int, str]]: + lines = path.read_text(errors="replace").splitlines() + prose: list[tuple[int, str]] = [] + opened: tuple[str, int, int] | None = None + for line_number, line in enumerate(lines, 1): + match = FENCE_RE.match(line) + if match: + marker = match.group(2) + remainder = match.group(3) + if opened is None: + if marker[0] == "`" and "`" in remainder: + errors.append(f"{path}:{line_number}: backtick fence info contains a backtick") + opened = (marker[0], len(marker), line_number) + elif marker[0] == opened[0] and len(marker) >= opened[1]: + if remainder.strip(): + errors.append(f"{path}:{line_number}: closing fence has trailing text") + opened = None + continue + if opened is None: + prose.append((line_number, line)) + if opened is not None: + errors.append(f"{path}: unclosed {opened[0] * opened[1]} fence from line {opened[2]}") + return prose + + +def local_target(raw_target: str, source: Path) -> tuple[Path, str] | None: + target = raw_target.strip() + if target.startswith("<") and target.endswith(">"): + target = target[1:-1] + path_text, _, fragment = target.partition("#") + path_text = unquote(path_text) + if re.match(r"^[A-Za-z][A-Za-z0-9+.-]*:", path_text): + return None + path = source if not path_text else (source.parent / path_text).resolve() + return path, unquote(fragment) + + +def markdown_anchors(path: Path) -> set[str]: + anchors: set[str] = set() + counts: dict[str, int] = {} + for line in path.read_text(errors="replace").splitlines(): + match = HEADING_RE.match(line) + if not match: + continue + heading = re.sub(r"<[^>]+>", "", match.group(1)) + explicit = re.search(r"\{#([A-Za-z0-9_.:-]+)\}\s*$", heading) + if explicit: + anchors.add(explicit.group(1)) + heading = heading[: explicit.start()].rstrip() + slug = re.sub(r"[^\w\s-]", "", heading.lower()) + slug = re.sub(r"[\s-]+", "-", slug).strip("-") + duplicate = counts.get(slug, 0) + counts[slug] = duplicate + 1 + anchors.add(slug if duplicate == 0 else f"{slug}-{duplicate}") + return anchors + + +def check_frontmatter(root: Path, errors: list[str]) -> None: + for path in sorted(root.rglob("SKILL.md")): + if ".git" in path.parts: + continue + lines = path.read_text(errors="replace").splitlines() + if not lines or lines[0] != "---": + errors.append(f"{path}: missing YAML frontmatter") + continue + try: + closing = lines.index("---", 1) + except ValueError: + errors.append(f"{path}: unclosed YAML frontmatter") + continue + keys = YAML_KEY_RE.findall("\n".join(lines[1:closing])) + if sorted(keys) != ["description", "name"]: + errors.append(f"{path}: frontmatter keys must be name and description, got {keys}") + values = {} + for line in lines[1:closing]: + if ":" not in line: + errors.append(f"{path}: invalid frontmatter line {line!r}") + continue + key, raw_value = line.split(":", 1) + value = raw_value.strip() + if value[:1] in {"\"", "'"} and (len(value) < 2 or value[-1] != value[0]): + errors.append(f"{path}: unterminated quoted frontmatter value for {key}") + values[key] = value.strip("'\"") + if not values.get("name") or not values.get("description"): + errors.append(f"{path}: name and description must be nonempty") + elif not re.fullmatch(r"[a-z0-9-]+", values["name"]): + errors.append(f"{path}: invalid skill name {values['name']!r}") + + +def audit(root: Path) -> list[str]: + root = root.resolve() + errors: list[str] = [] + check_frontmatter(root, errors) + for path in authored_markdown(root): + prose = prose_lines(path, errors) + prose_text = "\n".join(line for _, line in prose) + definitions = set(FOOTNOTE_DEF_RE.findall(prose_text)) + uses = set(FOOTNOTE_RE.findall(prose_text)) + for missing in sorted(uses - definitions): + errors.append(f"{path}: missing footnote definition [^{missing}]") + for line_number, line in prose: + if line.count("[") != line.count("]"): + errors.append(f"{path}:{line_number}: unbalanced square brackets") + for match in LINK_RE.finditer(line): + resolved = local_target(match.group(1), path) + if resolved is None: + continue + target, fragment = resolved + if not target.exists(): + errors.append(f"{path}:{line_number}: missing local link {match.group(1)!r}") + continue + if fragment and target.is_file() and target.suffix.lower() == ".md": + if fragment not in markdown_anchors(target): + errors.append(f"{path}:{line_number}: missing Markdown fragment #{fragment}") + relative = "/" + target.as_posix() + "/" + if "/docs/evidence/" not in relative or not target.is_file(): + continue + line_count = len(target.read_text(errors="replace").splitlines()) + for anchor in LINE_ANCHOR_RE.finditer(line[match.end():]): + start = int(anchor.group(1)) + end = int(anchor.group(2) or anchor.group(1)) + if start < 1 or end < start or end > line_count: + errors.append( + f"{path}:{line_number}: invalid evidence anchor L{start}-{end} for " + f"{target} ({line_count} lines)" + ) + return errors + + +def write_fixture(root: Path) -> None: + (root / "docs/evidence").mkdir(parents=True) + (root / "docs/evidence/source.md").write_text("one\ntwo\nthree\n") + (root / "SKILL.md").write_text( + "---\nname: fixture\ndescription: fixture skill\n---\n\n# Fixture\n" + ) + (root / "guide.md").write_text( + "# Fixture section\n" + "[skill](SKILL.md)\n" + "[section](guide.md#fixture-section)\n" + "![image](docs/evidence/source.md)\n" + "[cache](docs/evidence/source.md): L2-3\n" + "Claim.[^source]\n\n" + "[^source]: Evidence.\n" + ) + + +def self_test() -> None: + mutations = [ + ( + "missing local link", + lambda root: (root / "guide.md").write_text( + (root / "guide.md").read_text() + "\n[broken](missing.md)\n" + ), + ), + ( + "missing local link", + lambda root: (root / "guide.md").write_text( + (root / "guide.md").read_text().replace( + "docs/evidence/source.md)", "missing-image.png)", 1 + ) + ), + ), + ( + "unclosed", + lambda root: (root / "guide.md").write_text( + (root / "guide.md").read_text() + "\n```python\n" + ), + ), + ( + "backtick fence info", + lambda root: (root / "guide.md").write_text( + (root / "guide.md").read_text() + "\n```py`thon\n```\n" + ), + ), + ( + "missing footnote", + lambda root: (root / "guide.md").write_text( + (root / "guide.md").read_text() + "\nMissing.[^absent]\n" + ), + ), + ( + "frontmatter keys", + lambda root: (root / "SKILL.md").write_text( + (root / "SKILL.md").read_text().replace("name:", "title:") + ), + ), + ( + "unterminated quoted frontmatter", + lambda root: (root / "SKILL.md").write_text( + (root / "SKILL.md").read_text().replace( + "description: fixture skill", 'description: "fixture skill' + ) + ), + ), + ( + "invalid evidence anchor", + lambda root: (root / "guide.md").write_text( + (root / "guide.md").read_text().replace("L2-3", "L0") + ), + ), + ( + "invalid evidence anchor", + lambda root: (root / "guide.md").write_text( + (root / "guide.md").read_text().replace("L2-3", "L3-2") + ), + ), + ( + "missing Markdown fragment", + lambda root: (root / "guide.md").write_text( + (root / "guide.md").read_text().replace( + "#fixture-section", "#missing-section" + ) + ), + ), + ( + "unbalanced square brackets", + lambda root: (root / "guide.md").write_text( + (root / "guide.md").read_text() + "\n**Broken** [label\n" + ), + ), + ] + with tempfile.TemporaryDirectory() as directory: + clean = Path(directory) / "clean" + write_fixture(clean) + assert not audit(clean), audit(clean) + for expected, mutate in mutations: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + write_fixture(root) + mutate(root) + errors = audit(root) + assert any(expected in error for error in errors), (expected, errors) + print(f"audit self-test: PASS ({len(mutations)} injected defects rejected)") + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("root", nargs="?", type=Path, default=Path.cwd()) + parser.add_argument("--self-test", action="store_true") + args = parser.parse_args() + errors = audit(args.root) + if errors: + print("\n".join(errors)) + return 1 + if args.self_test: + self_test() + print( + f"audit: PASS ({len(authored_markdown(args.root.resolve()))} authored Markdown files, " + f"{len(list(args.root.resolve().rglob('SKILL.md')))} skills)" + ) + return 0 + + +if __name__ == "__main__": + sys.exit(main())