diff --git a/README.md b/README.md index 5485002..385fb07 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,21 @@ # pi-goals -Plan mode for agreeing on goals before any code gets written. Each goal names the subtle failure mode -that could fake a "done" and the discriminator that tells real success from it, plus subtasks and the -evidence checked at sign-off. It lives in one markdown file. A widget keeps the goals in front of you -through compaction, a reminder nudges the agent to keep the file current, and a goal is signed off -only after a read-only subagent checks its evidence. +Plan mode for agreeing on goals before any code gets written. Each goal names the subtle failure +mode that could fake a "done" and the discriminator that tells real success from it. Everything +lives in one markdown file, `.pi/plan.md`, which the agent edits with its normal edit tool and which +is injected verbatim every turn so it survives compaction. A goal is signed off only after a fresh +read-only judge checks its evidence against the repo. + +Design bet (v2): the plan file is for LLMs and the human, not for TypeScript. There is no parser and +no schema. The format is a convention taught by a prompt; the judge, being a model, reads the file +natively, finds the claimed goal itself (wording drift is fine), runs the goal's `verify` command +itself, and validates the evidence in words. This deleted ~800 lines of v1 (parser, exact-string +goal matching, verify runner, JSON-stream judge transport, review menus) and with them the footguns +they caused. Like [pi-milestones](https://github.com/Neuron-Mr-White/UniPi/tree/main/packages/milestone) and -[burneikis/pi-plan](https://github.com/burneikis/pi-plan), it guides rather than guards: a form and a -process the agent follows. [pi-lgtm](https://github.com/wassname/pi-lgtm) was my earlier, more complex -attempt. +[burneikis/pi-plan](https://github.com/burneikis/pi-plan), it guides rather than guards. +[pi-lgtm](https://github.com/wassname/pi-lgtm) was my earlier, more complex attempt. ## Install @@ -29,67 +35,20 @@ pi -e npm:@wassname2/pi-goals /goals CSV export for the report view ``` -`/goals` enters plan mode and starts a conversation; the description is an optional seed, so plain -`/goals` works too. From there: +`/goals` enters plan mode and starts a conversation; the objective is an optional seed. From there: -1. Plan. The agent explores read-only, asks about anything unclear, and writes the goals into - `.pi/goals.md`. -2. Review. You get a menu: Ready, Edit (ask the agent to revise), Open in `$EDITOR`, or Cancel. On - Ready you choose whether to keep the current context or start fresh and compacted. -3. Work. Each turn the active goal is injected so it survives compaction, and a reminder nudges the - agent to keep `goals.md` current and keep going. When a goal's discriminator is satisfied the agent - calls `CompleteGoal`, which runs `verify` and a read-only judge, then marks the goal done and logs it. +1. Plan. The agent explores read-only (edit/write are blocked except on the plan file itself), asks + about anything unclear, and drafts the goals into `.pi/plan.md`. +2. Review. Read the file; a two-option prompt asks Ready or keep planning. To revise, just reply. +3. Work. Each turn the whole plan file is injected (byte-identical when unchanged, so the KV cache + holds), plus a reminder when the file went untouched for a turn. The agent ticks subtasks, + appends to `## Log`, fills `evidence:`, and calls `CompleteGoal` when a discriminator is + satisfied. -Other commands: `/goals clear` empties `.pi/goals.md`; `/goals judge ` picks a specific -model for the sign-off judge (the default is your current model). +Other commands: `/goals clear` empties the plan file; `/goals judge ` picks a specific +model for the sign-off judge (default: your current session model, else pi's default). -## Example - -``` -/goals audit the papers dir metadata and clean up empty dirs -``` - -The agent explores read-only, drafts the goal with a subtle failure mode and the discriminator that -beats it, and stops for review: - -```markdown -## Goals - -1. [ ] goal: Audit steering/ metadata and remove empty dirs - - subtle failure mode: report written but counts are zero (resolver errored silently) - - discriminator: report shows the XXXX count before/after AND a non-zero rename count - - tasks: - 1. [ ] dry-run the metadata resolve - 2. [ ] remove the empty _artifacts dirs - 3. [ ] write the report - - evidence: - - -``` - -You choose Ready. The agent works the subtasks, fills `evidence` (each item an artifact plus a short -read of it), and calls `CompleteGoal`: - -```markdown - - evidence: - - > scripts/metadata_report.txt: XXXX 52 -> 4, 146 empty _artifacts removed - - > 48 files renamed; almost certain done, the silent-resolver failure mode is ruled out -``` - -A fresh read-only subagent re-checks the evidence against the repo and the discriminator, then -returns its verdict and reasoning: - -``` -Signed off "Audit steering/ metadata and remove empty dirs". Marked done in goals.md. - ---- sign-off judge --- -metadata_report.txt present; counts 52 -> 4 confirmed; rename log shows 48 renamed (not zero). -VERDICT: accept -``` - -## The goals.md format - -One project-local file, `/.pi/goals.md` (gitignored), holds the title, a context block, the -goals, and a short append-only log. A fresh `/goals` draft replaces it. +## The plan.md format (a convention, not a schema) ```markdown # ship the cache layer @@ -110,65 +69,46 @@ Latency target came from the SLO review; keep the existing client API. # Future work / out of scope -- distributed cache - ## Log - 2026-06-15 14:02 cache client wired; eviction next ``` -- A goal is a numbered checkbox line beginning `goal:`; the checkbox carries its state (`[ ]` open, - `[/]` active, `[x]` done, `[-]` cancelled). Goals are matched by their text, so the number is just - for you to reference. +- A goal is a checkbox line beginning `goal:` (`[ ]` open, `[/]` active, `[x]` done, `[-]` + cancelled). That one line pattern is the only thing the extension itself reads, for the widget. - The `discriminator` is the success test, written while planning: the positive observation that the - goal succeeded and that none of the `subtle failure mode`s could fake (a count moved, a test - exercised the path, a metric beat noise), not just that a failure was avoided. `evidence` is the - proof, filled at sign-off: each item pairs a durable artifact (a quoted and linked log, a table, a - metric) with a short read of it. `verify`, when present, is the deterministic first stage. -- Subtasks are any checkbox without a `goal:` prefix, under `- tasks:`. The agent ticks them, appends - to `## Log`, and sets a goal `[/]` when it starts it; only `CompleteGoal` writes `[x]`. Several - goals can be active at once. + goal succeeded and that none of the `subtle failure mode`s could fake. `evidence` is the proof, + filled at sign-off: each item pairs a durable artifact with a short read of it. +- Small format deviations are fine; the file is read by the human and the judge, not a parser. +- The agent prunes finished goals itself when the file gets long (evidence survives in git history + and `## Log`). ## Signing off a goal (`CompleteGoal`) -`CompleteGoal(goal)` (matched by the goal's text) is the only tool that marks a goal done; everything -else is the agent editing the file. It reads the goal's `evidence:` block from `.pi/goals.md`, then: +`CompleteGoal(goal)` is the one blessed tool. It spawns a read-only `pi` subprocess (`-p +--no-session --no-extensions`, tools `read,bash,grep,find,ls`, edit/write excluded) with the whole +plan file and the claimed goal. The judge finds the goal, re-derives from the cited artifacts rather +than trusting claims, runs `verify` if the goal names one, and returns `VERDICT: accept | reject` +plus what's missing. -1. If the goal has a `verify:` command, it runs. A non-zero exit rejects right away, no model call. -2. Then a read-only `pi` subprocess (a fresh `--no-session` context, so it never sees the working - agent's transcript) inspects the `evidence:` against the repo, the `discriminator`, and the - `subtle failure mode`. It re-derives from the cited artifacts rather than trusting the claim, so - list real artifacts, not assertions. -3. On accept, the goal flips to `[x]` and a `## Log` line is written. On judge reject, it stays open - and the agent is told what is missing. If the judge subprocess times out or its transport/model - fails after any `verify:` command has passed, the goal still flips to `[x]` with a `judge - inconclusive` log line and any partial judge output in the result. Either way the judge's - reasoning comes back in the result. - -The judge defaults to the current session model and streams partial output while it runs. If the -session model is not visible to the extension, the `--model` flag is omitted and pi uses its own -configured default, so the judge always runs. `/goals judge ` is an optional override -for an independent cross-family check; never required. +- accept: a sign-off line is appended to `## Log` and the agent ticks the goal `[x]` itself. The + tool-written log line is the audit trail; a hand-tick without one shows in the diff. +- reject: the goal stays open and the agent gets the missing list. +- judge unavailable (timeout, transport, no VERDICT line): accepted inconclusive, logged as such. + The working agent is never blocked on judge infra. ## Prompts -All model-facing text lives in [`src/prompts.ts`](src/prompts.ts), in flow order, so you can read the -whole process top to bottom. +All model-facing text lives in [`src/prompts.ts`](src/prompts.ts), in flow order. ## Develop ```bash pi -e ./src/index.ts # load locally -npm test # vitest: parser + sign-off record logic +npm test # vitest: judge argv invariants + appendLog npm run typecheck npm run lint ``` -## Not (yet) included - -- No autonomous re-prompt loop. The reminder nudges the agent within a turn, but the turn still ends - and hands back to you; nothing auto-re-prompts until the goals are done. -- The plan and execution phases can't yet run on different, sticky models. - ## License MIT diff --git a/src/index.ts b/src/index.ts index 14784f7..b48add5 100644 --- a/src/index.ts +++ b/src/index.ts @@ -1,107 +1,78 @@ /** - * pi-goals — plan mode that sets up goals with evidence, tracked in one .pi/goals.md, signed off by a - * read-only subagent check. A successor to pi-lgtm, kept deliberately small (≈ burneikis/pi-plan - * plus the additions: goals + a discriminator + a subtle failure mode + subtasks, a sign-off check, - * a widget, a reminder). A goal's success test is its discriminator: the observation that tells real - * success from the named failure mode. + * pi-goals v2 — plan mode drafts goals into one .pi/plan.md, the agent works them with its normal + * Edit tool, and a fresh read-only judge signs each goal off through the one blessed tool, + * CompleteGoal. * - * Philosophy (spec D3): the form guides, it does not gate. The agent edits goals.md with its normal - * Edit tool. The one blessed tool is CompleteGoal, which runs the sign-off check and records it. The - * reminder + the injected plan + git/widget visibility carry the process; we trust the agent's - * judgement rather than guarding it. + * The v1 lesson: the parser existed so TypeScript could read plan.md, but almost every reader is a + * model. So v2 has NO parser and no schema. The harness does exactly three things for a + * cooperative-but-confused model: + * 1. memory — inject plan.md verbatim every turn (survives compaction; byte-identical when + * unchanged so the KV cache holds; stale copies stripped by the context hook) + * 2. format — a skeleton convention taught in planDrafting (prompts.ts), not validated + * 3. eyes — CompleteGoal spawns a read-only pi subprocess (--no-session) that gets the whole + * plan file plus the claimed goal, finds the goal itself (tolerates wording drift), + * runs the goal's verify command itself, and returns VERDICT: accept|reject * - * Flow: - * /goals [objective] -> plan mode (conversational): objective is an optional seed; agent explores - * read-only, asks, then drafts goals into .pi/goals.md (planDrafting guides) - * agent_end -> review menu (Ready / Edit / $EDITOR / Cancel); Ready offers compaction - * execution -> each turn, inject the plan summary (survives compaction) + a reminder; - * agent works goals, ticks subtasks, appends ## Log, calls CompleteGoal - * CompleteGoal -> optional deterministic verify, then a read-only oracle judge -> accept - * flips status:done + logs; reject returns what's missing + * The judge subsumes what v1 did in code: goal matching (no findGoal), evidence validation (a + * placeholder gets rejected in words), verify execution, and format reading. The extension's only + * write is appending a sign-off line to ## Log — the audit trail. The agent ticks [x] itself; a + * hand-tick without a matching tool-written log line is visible in the diff. * - * The plan file lives at /.pi/goals.md (project-local, gitignored, like pi-tasks), not in the - * repo. A fresh /goals draft just replaces it (the "overwrite" staleness rule). - * - * Plan mode is read-only: the tool_call hook blocks edit/write (except goals.md itself) and mutating - * bash while drafting, so code isn't written before the goals are agreed. Read-only bash exploration - * stays open (blocklist, not allowlist). - * - * Not built (FIXME): no plan-vs-exec model switch on accept (plan-model stickiness); noted at its - * call site below. + * Judge unavailable (timeout/transport) => accepted_inconclusive: the working agent is never + * blocked on judge infra; the log line says the judge didn't run. * * All model-facing text lives in prompts.ts, in flow order. */ -import { spawn, spawnSync } from "node:child_process"; +import { spawn } from "node:child_process"; import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; import { basename, join, resolve } from "node:path"; -import type { ExtensionAPI, ExtensionCommandContext, ExtensionContext } from "@earendil-works/pi-coding-agent"; -import { getMarkdownTheme } from "@earendil-works/pi-coding-agent"; -import { Container, Markdown, Spacer, Text } from "@earendil-works/pi-tui"; +import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent"; import { Type } from "@sinclair/typebox"; -import { counts, findGoal, type Goal, type PlanDoc, parse, pruneCompleted, recordSignOff, type SignOff } from "./plan-file.js"; -import { - completeGoalDescription, - completeGoalParamDescription, - evidenceJudgeSystem, - evidenceJudgeUser, - planDrafting, - planInjection, - reminder, -} from "./prompts.js"; +import { completeGoalDescription, completeGoalParamDescription, judgeSystem, judgeUser, planDrafting, reminder } from "./prompts.js"; const STATE = "pi-goals-state"; -const PLAN_CONTEXT = "pi-goals-context"; // injected plan-mode guidance, stripped from history later +const PLAN_CONTEXT = "pi-goals-context"; // injected plan/guidance, stale copies stripped by the context hook const STATUS_KEY = "pi-goals"; const WIDGET_KEY = "pi-goals-widget"; -// Tools the sign-off judge gets: read-only inspection + bash (for git log, cat, running scripts to -// inspect). File mutators (edit, write) are blocked so the judge cannot modify anything. -// Names match pi's internal tool registry (grep→ffgrep, find→fffind, etc.). +const PLAN_REL = ".pi/plan.md"; +// Judge toolset: read-only inspection plus bash (git log, cat, running the goal's verify command). +// edit/write are excluded so the judge cannot modify anything. Names match pi's tool registry. const JUDGE_TOOLS = ["read", "bash", "grep", "find", "ls"]; const JUDGE_BLOCKED_TOOLS = ["edit", "write"]; const JUDGE_TIMEOUT_MS = 600_000; -// File mutators blocked while drafting goals (read-only plan mode, like narumiruna/pi-plan-mode), so -// code isn't written before goals are agreed. The one allowed write is goals.md itself (the -// deliverable). A read-only task (a pure search) can still be explored in plan mode by nature. +// Plan mode is read-only by convention AND a light gate: edit/write are blocked (except plan.md, +// the deliverable). bash stays open — the prompt says don't mutate; guide, don't gate (spec D3). const PLAN_MODE_BLOCKED_TOOLS = ["edit", "write"]; -// bash is dual-use, so block it only when the command looks mutating; read-only exploration (cat, rg, -// git log, running a script to inspect) stays open. Blocklist, not allowlist: keep exploration -// frictionless and just stop the obvious mutators. List adapted from narumiruna/pi-plan-mode; the -// redirect rule catches `> file` / `>> file` / `>| file` but not fd-dups like `2>&1` or `>&2`. -const MUTATING_BASH_PATTERNS: RegExp[] = [ - /\b(rm|rmdir|mv|cp|mkdir|touch|chmod|chown|chgrp|ln|tee|truncate|dd)\b/i, - />\s*[^&\s]/, // redirect to a file (write/append/clobber), excludes 2>&1 and >&2 - /\bnpm\s+(install|uninstall|update|ci|link|publish|version)\b/i, - /\byarn\s+(add|remove|install|publish|upgrade)\b/i, - /\bpnpm\s+(add|remove|install|publish|update)\b/i, - /\bbun\s+(add|remove|install|update|publish)\b/i, - /\bpip\s+(install|uninstall)\b/i, - /\buv\s+(add|remove|sync|lock|pip\s+install)\b/i, - /\bgit\s+(add|commit|push|pull|merge|rebase|reset|checkout|switch|stash|cherry-pick|revert|tag|init|clone)\b/i, - /\b(sudo|su|kill|pkill|killall|reboot|shutdown)\b/i, - /\bsystemctl\s+(start|stop|restart|enable|disable)\b/i, - /\b(vim?|nano|emacs|code|subl)\b/i, -]; -const PLAN_REL = ".pi/goals.md"; // project-local, gitignored (pi-tasks convention); shown in the widget + +// The one regex in the whole extension: a checkbox line beginning "goal:", for the widget and the +// "any goals open?" reminder condition. Everything else reads the file as prose. +const GOAL_LINE = /^\s*(?:\d+\.|[-*])\s*\[([ xX/-])\]\s*goal:\s*(.*)$/i; +type GoalStatus = "open" | "active" | "done" | "cancelled"; +const CHAR_TO_STATUS: Record = { " ": "open", "/": "active", x: "done", "-": "cancelled" }; + +function scanGoals(plan: string): Array<{ status: GoalStatus; subject: string }> { + const goals: Array<{ status: GoalStatus; subject: string }> = []; + for (const line of plan.split("\n")) { + const m = GOAL_LINE.exec(line); + if (m) goals.push({ status: CHAR_TO_STATUS[m[1].toLowerCase()] ?? "open", subject: m[2].trim() }); + } + return goals; +} interface PlanState { isPlanMode: boolean; - objective: string | null; - /** Optional model ref for the sign-off judge; unset => use the current session model. */ + /** Optional model ref for the sign-off judge; unset => current session model, else pi's default. */ judgeModel: string | null; } export default function piGoalsExtension(pi: ExtensionAPI): void { - let state: PlanState = { isPlanMode: false, objective: null, judgeModel: null }; - // Reminder cadence: fire when an active goal exists but goals.md was not touched since last turn. + let state: PlanState = { isPlanMode: false, judgeModel: null }; + // Reminder cadence: fire when goals are open but plan.md was untouched since the last turn. let lastInjectedPlan = ""; - // newSession is only on the command-handler context; agent_end's ctx lacks it. Save it from /goals. - let savedCmdCtx: ExtensionCommandContext | null = null; - const planPath = (ctx: ExtensionContext) => join(ctx.cwd, ".pi", "goals.md"); + const planPath = (ctx: ExtensionContext) => join(ctx.cwd, ".pi", "plan.md"); const readPlan = (ctx: ExtensionContext): string => (existsSync(planPath(ctx)) ? readFileSync(planPath(ctx), "utf-8") : ""); - // Our programmatic writes (clear, CompleteGoal). The agent creates/edits the file with its own Edit - // tool; this just makes sure .pi/ exists for our writes. const writePlan = (ctx: ExtensionContext, content: string): void => { mkdirSync(join(ctx.cwd, ".pi"), { recursive: true }); writeFileSync(planPath(ctx), content); @@ -114,334 +85,103 @@ export default function piGoalsExtension(pi: ExtensionAPI): void { function updateWidget(ctx: ExtensionContext): void { if (state.isPlanMode) { ctx.ui.setStatus(STATUS_KEY, ctx.ui.theme.fg("warning", "planning")); - ctx.ui.setWidget(WIDGET_KEY, ["pi-goals: drafting goals", `Write goals to ${PLAN_REL}, then review.`]); + ctx.ui.setWidget(WIDGET_KEY, [`pi-goals: drafting goals in ${PLAN_REL}`]); return; } - const doc = parse(readPlan(ctx)); - if (doc.goals.length === 0) { + const goals = scanGoals(readPlan(ctx)); + if (goals.length === 0) { ctx.ui.setStatus(STATUS_KEY, undefined); ctx.ui.setWidget(WIDGET_KEY, undefined); return; } - const c = counts(doc); - ctx.ui.setStatus(STATUS_KEY, ctx.ui.theme.fg("accent", `◷ ${c.done}/${doc.goals.length} goals`)); - ctx.ui.setWidget(WIDGET_KEY, goalWidgetLines(doc, ctx)); + const done = goals.filter((g) => g.status === "done").length; + ctx.ui.setStatus(STATUS_KEY, ctx.ui.theme.fg("accent", `◷ ${done}/${goals.length} goals`)); + const mark: Record = { done: "✔", active: "▸", open: "◻", cancelled: "✗" }; + // Only live goals get lines so finished work never pushes current work off screen. + const live = goals.filter((g) => g.status === "active" || g.status === "open"); + ctx.ui.setWidget(WIDGET_KEY, [ctx.ui.theme.fg("muted", PLAN_REL), ...live.map((g) => `${mark[g.status]} ${g.subject}`)]); } - function goalWidgetLines(doc: PlanDoc, ctx: ExtensionContext): string[] { - const mark: Record = { done: "✔", active: "▸", open: "◻", cancelled: "✗" }; - // Header doubles as the file path (clickable) so we don't spend a second line on a "Goals:" label - // plus a footer path -- one line carries both. Title trails the path when set. - const header = ctx.ui.theme.fg("muted", doc.title ? `${PLAN_REL}: ${doc.title}` : PLAN_REL); - const lines = [header]; - // Only the live work (active + open) gets lines; finished goals (done/cancelled) are hidden so - // they never push the current work off screen. The done count lives in the status bar (◷ n/N - // goals) and the full history in goals.md / the ## Log. - const live = doc.goals.filter((g) => g.status === "active" || g.status === "open"); - for (const g of live) { - const total = g.subtasks.length; - const done = g.subtasks.filter((s) => s.status === "done").length; - lines.push(`${mark[g.status]} ${g.subject}${total ? ` (${done}/${total} tasks)` : ""}`); - } - return lines; - } - - // --- plan mode: setup ------------------------------------------------------------------------- + // --- /goals: enter plan mode (or clear / set judge) -------------------------------------------- pi.registerCommand("goals", { - description: "Plan mode: set up goals (with evidence) in goals.md, then work them. /goals | /goals clear", + description: `Plan mode: draft goals into ${PLAN_REL}, review, then work them. /goals | /goals clear | /goals judge `, handler: async (args, ctx) => { - savedCmdCtx = ctx; // ctx here is an ExtensionCommandContext (has newSession); keep it for later const arg = args.trim(); if (arg === "clear") { - await clearPlan(ctx); + writePlan(ctx, ""); + state = { ...state, isPlanMode: false }; + persist(); + updateWidget(ctx); + ctx.ui.notify(`Cleared ${PLAN_REL}.`, "info"); return; } if (arg.startsWith("judge")) { - setJudge(arg.slice("judge".length).trim(), ctx); + const ref = arg.slice("judge".length).trim(); + state = { ...state, judgeModel: ref || null }; + persist(); + ctx.ui.notify(ref ? `Sign-off judge model set to ${ref}` : "Sign-off judge reset to the session model", "info"); return; } - // Conversational entry (like narumiruna/pi-plan-mode): /goals enters plan mode and starts a - // dialogue. The objective is an optional seed, not a required arg, so there's no awkward - // "type your objective" prompt; the agent explores read-only and asks before drafting. A - // fresh draft just replaces .pi/goals.md (the "overwrite" staleness rule). - const objective = arg || null; - state = { ...state, isPlanMode: true, objective }; + state = { ...state, isPlanMode: true }; persist(); updateWidget(ctx); - const seed = objective - ? `We're in plan mode. Objective: ${objective}\n\nExplore the repo read-only and ask me anything unclear. When the objective is nailed down, draft (or replace) the goals in ${planPath(ctx)}, then stop for review.` - : `We're in plan mode. Tell me what you want to plan. Explore read-only and ask questions as needed; when the objective is clear, draft the goals in ${planPath(ctx)} and stop for review.`; + const seed = arg + ? `We're in plan mode. Objective: ${arg}\n\nExplore the repo read-only and ask me anything unclear. When the objective is nailed down, draft (or replace) the plan in ${planPath(ctx)}, then stop for review.` + : `We're in plan mode. Tell me what you want to plan. Explore read-only and ask questions as needed; when the objective is clear, draft the plan in ${planPath(ctx)} and stop for review.`; pi.sendUserMessage(seed, { deliverAs: "followUp" }); }, }); - function setJudge(ref: string, ctx: ExtensionContext): void { - state = { ...state, judgeModel: ref || null }; - persist(); - ctx.ui.notify(ref ? `Sign-off judge model set to ${ref}` : "Sign-off judge reset to the current session model", "info"); - } - - async function clearPlan(ctx: ExtensionContext): Promise { - if (!existsSync(planPath(ctx))) { - ctx.ui.notify("No goals.md to clear.", "info"); - return; - } - if (ctx.hasUI) { - const choice = await ctx.ui.select(`Clear ${PLAN_REL}?`, [ - "Cancel", - "Prune completed goals (keep active/open + log)", - "Clear everything", - ]); - if (!choice || choice.startsWith("Cancel")) return; - if (choice.startsWith("Prune")) { - writePlan(ctx, pruneCompleted(readPlan(ctx))); - updateWidget(ctx); - ctx.ui.notify(`Pruned completed goals from ${PLAN_REL}.`, "info"); - return; - } - } - writePlan(ctx, ""); - state = { ...state, isPlanMode: false, objective: null }; - persist(); - updateWidget(ctx); - ctx.ui.notify(`Cleared ${PLAN_REL}.`, "info"); - } - - // --- review loop (after the agent drafts the plan) -------------------------------------------- - - async function reviewLoop(ctx: ExtensionContext): Promise { - while (true) { - const doc = parse(readPlan(ctx)); - const choice = await ctx.ui.select(`Goals: ${doc.goals.length} goal(s). What next?`, [ - "Ready — start working the plan", - "Edit — ask the agent to revise", - "Open in $EDITOR", - "Cancel — leave plan mode", - ]); - if (!choice || choice.startsWith("Cancel")) { - exitPlanMode(ctx); - ctx.ui.notify("Left plan mode. goals.md kept.", "info"); - return; - } - if (choice.startsWith("Ready")) return startExecution(ctx); - if (choice.startsWith("Edit")) { - const changes = await ctx.ui.editor("What should change about the plan?", ""); - if (changes?.trim()) { - pi.sendUserMessage(`Revise the plan at ${planPath(ctx)} with these changes, same format:\n\n${changes.trim()}`, { deliverAs: "followUp" }); - return; // agent_end re-opens the review loop - } - continue; - } - if (choice.startsWith("Open")) { - const editor = process.env.EDITOR || process.env.VISUAL || "vi"; - spawnSync(editor, [planPath(ctx)], { stdio: "inherit" }); - } - } - } - - function exitPlanMode(ctx: ExtensionContext): void { - state = { ...state, isPlanMode: false }; - persist(); - updateWidget(ctx); - } - - async function startExecution(ctx: ExtensionContext): Promise { - // FIXME(model-switch): the plan phase should be able to run on a sticky plan model and execution - // on a different one (see README "Not yet included"). newSession can't switch the model yet; wire - // this when pi exposes a model override on newSession. - // Offer a clean execution context (D13). newSession lives only on the saved command context. - let fresh = false; - if (ctx.hasUI && savedCmdCtx) { - const choice = await ctx.ui.select("Start working the plan in...", [ - "This context (keep history)", - "A fresh, compacted context", - ]); - fresh = choice?.startsWith("A fresh") ?? false; - } - const doc = parse(readPlan(ctx)); - const planFile = planPath(ctx); - const planContent = readPlan(ctx); // captured now: ctx is stale after newSession below - const parentSession = ctx.sessionManager.getSessionFile(); - const startMsg = `Work the goals in ${planFile}. Pick an open goal, mark it active (set its checkbox to [/]), work its subtasks, and when its discriminator is satisfied fill the goal's evidence: block then call CompleteGoal with the goal's desc. Keep goals.md current as you go.`; - exitPlanMode(ctx); - - if (fresh && savedCmdCtx) { - // After newSession, `ctx`/`pi` bound to the old session are stale; do post-swap work - // through the ReplacedSessionContext passed to withSession (see runner.assertActive). - const result = await savedCmdCtx.newSession({ - parentSession, - withSession: async (sessionCtx) => { - // pi.* and the outer ctx are invalidated by newSession; use the fresh sessionCtx only. - // (No setSessionName here: it lives on pi/the outer ctx, both stale now. Cosmetic, skip it.) - sessionCtx.ui.notify(planContent, "info"); - await sessionCtx.sendUserMessage(startMsg, { deliverAs: "followUp" }); - }, - }); - if (result.cancelled) { - return; - } - return; - } - if (doc.title) pi.setSessionName(`Goals: ${doc.title}`); - ctx.ui.notify(planContent, "info"); - pi.sendUserMessage(startMsg, { deliverAs: "followUp" }); - } - - // --- the one blessed tool: CompleteGoal ------------------------------------------------------- - - pi.registerTool({ - name: "CompleteGoal", - label: "Goal signoff", - description: completeGoalDescription, - parameters: Type.Object({ - goal: Type.String({ description: completeGoalParamDescription }), - }), - async execute(_id, params, signal, onUpdate, ctx) { - const content = readPlan(ctx); - const goal = findGoal(parse(content), params.goal); - if (!goal) return text(`No goal "${params.goal}" in goals.md. Use the exact text after "goal:".`, true); - if (goal.evidence.length === 0) { - return text(`Goal "${goal.subject}" has no evidence yet. Add an evidence: list to the goal in goals.md (artifacts + a short read showing the discriminator is satisfied), then call CompleteGoal.`, true); - } - - const handleUpdate = (partial: { content: Array<{ type: "text"; text: string }>; details: SignOffDetails }) => { - onUpdate?.(partial); - }; - - const judgeModel = state.judgeModel ?? currentSessionModelRef(ctx); - const { outcome, reasoning, durationMs } = await decideSignOff(goal, goal.evidence.join("\n"), goal.evidence, judgeModel, ctx.cwd, signal, handleUpdate); - const res = recordSignOff(content, goal.subject, stamp(), outcome); - if (res.content !== content) writePlan(ctx, res.content); - updateWidget(ctx); - const detail = reasoning ? `\n\n--- sign-off judge ---\n${reasoning}` : ""; - const outcomeLabel = - outcome.kind === "accepted" ? "accepted" : - outcome.kind === "accepted_inconclusive" ? "accepted_inconclusive" : - outcome.kind === "verify_failed" ? "verify_failed" : - "rejected"; - const details: SignOffDetails = { - goal: goal.subject, - outcome: outcomeLabel, - durationMs, - verifyCommand: goal.verify ?? undefined, - verifyExitCode: outcome.kind === "verify_failed" ? outcome.exitCode : undefined, - judgeModel: judgeModel ?? "pi default", - reasoning, - isError: res.isError, - }; - return textWithDetails(res.message + detail, details, res.isError); - }, - - renderCall(args, theme) { - const goalText = args.goal.length > 80 ? `${args.goal.slice(0, 80)}...` : args.goal; - return new Text( - `${theme.fg("toolTitle", theme.bold("goal signoff "))}${theme.fg("dim", goalText)}`, - 0, 0, - ); - }, - - renderResult(result, { expanded }, theme) { - const details = result.details as SignOffDetails | undefined; - const body = result.content[0]?.type === "text" ? result.content[0].text : "(no output)"; - if (!details || details.outcome === "running") return new Text(body, 0, 0); - - const accepted = details.outcome === "accepted" || details.outcome === "accepted_inconclusive"; - const icon = accepted ? theme.fg("success", "✔") : theme.fg("error", "✗"); - const outcomeText = - details.outcome === "accepted" ? "accepted" : - details.outcome === "accepted_inconclusive" ? "accepted (judge inconclusive)" : - details.outcome === "verify_failed" ? `verify failed (exit ${details.verifyExitCode})` : - "rejected"; - const header = `${icon} ${theme.fg("toolTitle", theme.bold("goal signoff "))}${theme.fg("accent", outcomeText)}`; - const duration = details.durationMs < 1000 ? `${details.durationMs}ms` : `${(details.durationMs / 1000).toFixed(1)}s`; - const sub = [details.judgeModel, duration].filter(Boolean).join(" · "); - - if (!expanded) { - let text = header; - if (sub) text += `\n${theme.fg("dim", sub)}`; - text += `\n\n${theme.fg("toolOutput", body.slice(0, 500))}`; - if (body.length > 500) text += theme.fg("dim", "..."); - text += `\n${theme.fg("muted", "(Ctrl+O to expand)")}`; - return new Text(text, 0, 0); - } - - const container = new Container(); - container.addChild(new Text(header, 0, 0)); - if (sub) container.addChild(new Text(theme.fg("dim", sub), 0, 0)); - if (details.verifyCommand) { - container.addChild(new Spacer(1)); - container.addChild(new Text(theme.fg("muted", `verify: ${details.verifyCommand}`), 0, 0)); - } - container.addChild(new Spacer(1)); - container.addChild(new Text(theme.fg("muted", "Judge"), 0, 0)); - container.addChild(new Markdown(body.trim(), 0, 0, getMarkdownTheme())); - return container; - }, - }); - - // --- hooks ------------------------------------------------------------------------------------ + // --- hooks -------------------------------------------------------------------------------------- pi.on("before_agent_start", async (_event, ctx) => { if (state.isPlanMode) { - // Read-only is enforced in the tool_call hook below (blocks edit/write while planning). return { message: { customType: PLAN_CONTEXT, content: `${planDrafting}\n\nWrite the plan to ${planPath(ctx)}.`, display: false } }; } - const doc = parse(readPlan(ctx)); - if (doc.goals.length === 0) return; - - const active = doc.goals.find((g) => g.status === "active") ?? doc.goals.find((g) => g.status === "open") ?? null; - const c = counts(doc); - let body = planInjection({ - title: doc.title, - activeGoal: active - ? { - subject: active.subject, - discriminator: active.discriminator, - openSubtasks: active.subtasks.filter((s) => s.status !== "done" && s.status !== "cancelled").map((s) => s.text), - } - : null, - lastLogLine: doc.log.at(-1) ?? null, - counts: { done: c.done, open: c.open + c.active }, - }); - // Reminder fires when there is an active goal but goals.md was untouched since the last turn. - const planNow = readPlan(ctx); - if (active && planNow === lastInjectedPlan) body += `\n\n${reminder}`; - lastInjectedPlan = planNow; + const plan = readPlan(ctx); + const goals = scanGoals(plan); + if (goals.length === 0) return; + // The plan file itself IS the injection: no parsing, no summarizing, the model sees the + // literal file it edits. Byte-identical when unchanged, so the prefix cache holds. + let body = `Current plan (${PLAN_REL}; keep it updated with your edit tool):\n\n${plan}`; + const live = goals.some((g) => g.status === "active" || g.status === "open"); + if (live && plan === lastInjectedPlan) body += `\n\n${reminder}`; + lastInjectedPlan = plan; return { message: { customType: PLAN_CONTEXT, content: body, display: false } }; }); - // Enforce read-only planning: block file mutators while in plan mode so code isn't written before - // the goals are agreed. The agent draws back to read/grep/find/ls and read-only bash to explore. + // Plan mode gate: block edit/write except on plan.md itself. bash stays open (guide, not gate). pi.on("tool_call", async (event, ctx) => { if (!state.isPlanMode) return; - // edit/write: blocked, except writing goals.md itself (the deliverable of plan mode). if (PLAN_MODE_BLOCKED_TOOLS.includes(event.toolName)) { const target = (event.input as { path?: string }).path; if (target && resolve(ctx.cwd, target) === resolve(planPath(ctx))) return; - return { block: true, reason: `Plan mode is read-only: agree the goals in ${PLAN_REL} and choose Ready before writing code (${event.toolName} is blocked while planning; only ${PLAN_REL} may be written).` }; - } - // bash: blocked only when the command looks mutating; read-only exploration stays open. - if (event.toolName === "bash") { - const command = (event.input as { command?: string }).command ?? ""; - if (MUTATING_BASH_PATTERNS.some((re) => re.test(command))) { - return { block: true, reason: `Plan mode is read-only: this bash command looks like it mutates state, so it's blocked while planning. Explore read-only, agree the goals in ${PLAN_REL}, then choose Ready.\nCommand: ${command}` }; - } + return { block: true, reason: `Plan mode is read-only: only ${PLAN_REL} may be written while drafting. Agree the goals first, then choose Ready.` }; } }); + // After a plan-mode turn: if goals were drafted, offer Ready. No edit menus, no fresh-session + // dance — the human reads the file and says go (or keeps talking to revise it). pi.on("agent_end", async (_event, ctx) => { if (!state.isPlanMode || !ctx.hasUI) return; - const doc = parse(readPlan(ctx)); - if (doc.goals.length === 0) { - ctx.ui.notify("No goals found in goals.md yet — ask the agent to draft them.", "warning"); - return; - } - await reviewLoop(ctx); + if (scanGoals(readPlan(ctx)).length === 0) return; // still exploring/asking; nothing to review yet + const choice = await ctx.ui.select(`Plan drafted in ${PLAN_REL}. Ready?`, [ + "Ready — start working the plan", + "Keep planning (reply to revise)", + ]); + if (!choice?.startsWith("Ready")) return; + state = { ...state, isPlanMode: false }; + persist(); + updateWidget(ctx); + pi.sendUserMessage( + `Work the goals in ${planPath(ctx)}. Pick an open goal, mark it active ([/]), work its subtasks, and when its discriminator is satisfied fill its evidence: list, then call CompleteGoal with the goal's text. Keep the plan file current as you go.`, + { deliverAs: "followUp" }, + ); }); - // Keep only the freshest injected plan summary; strip stale ones so history does not bloat and - // the model never sees an out-of-date plan. (The current turn's injection is the one kept.) + // Keep only the freshest injected plan; strip stale ones so history doesn't bloat and the model + // never sees an out-of-date plan. pi.on("context", async (event) => { const isCtx = (m: unknown) => (m as { customType?: string }).customType === PLAN_CONTEXT; let lastIdx = -1; @@ -459,84 +199,97 @@ export default function piGoalsExtension(pi: ExtensionAPI): void { if (last?.data) state = { ...state, ...last.data }; updateWidget(ctx); }); + + // --- the one blessed tool: CompleteGoal --------------------------------------------------------- + + pi.registerTool({ + name: "CompleteGoal", + label: "Goal signoff", + description: completeGoalDescription, + parameters: Type.Object({ + goal: Type.String({ description: completeGoalParamDescription }), + }), + async execute(_id, params, signal, onUpdate, ctx) { + const plan = readPlan(ctx); + if (!plan.trim()) return result(`No plan file at ${PLAN_REL}.`, true); + + const judgeModel = state.judgeModel ?? (ctx.model ? `${ctx.model.provider}/${ctx.model.id}` : null); + onUpdate?.({ content: [{ type: "text", text: `Read-only judge (${judgeModel ?? "pi default"}) inspecting: ${params.goal}` }], details: {} }); + const judge = await runJudge(judgeUser({ goal: params.goal, plan, planPath: PLAN_REL }), judgeModel, ctx.cwd, signal); + + if (signal?.aborted) return result("Sign-off aborted.", true); + + const log = (entry: string) => writePlan(ctx, appendLog(readPlan(ctx), `${stamp()} ${entry}`)); + // Judge infra failure must not block the working agent: accept inconclusive, say so in the log. + if (judge.error) { + log(`signed off "${params.goal}" (judge unavailable: ${oneLine(judge.error)})`); + updateWidget(ctx); + return result( + `Judge unavailable (${judge.error}). Accepted inconclusive — logged. Tick the goal [x] in ${PLAN_REL} yourself.${judge.output ? `\n\npartial judge output:\n${judge.output}` : ""}`, + ); + } + + const verdictLine = judge.output.split("\n").find((l) => /^\s*VERDICT\s*:/i.test(l)) ?? ""; + const verdict = /^\s*VERDICT\s*:\s*(accept|reject)\s*$/i.exec(verdictLine)?.[1]?.toLowerCase(); + const reasoning = judge.output.length > 2000 ? `...\n${judge.output.slice(-2000)}` : judge.output; + + if (verdict === "accept") { + log(`signed off "${params.goal}" (judge accept)`); + updateWidget(ctx); + return result(`Sign-off ACCEPTED. Tick the goal [x] in ${PLAN_REL} (the log line is already appended).\n\n--- judge ---\n${reasoning}`); + } + if (verdict === "reject") { + const missing = judge.output.match(/missing\s*:\s*([\s\S]*)$/i)?.[1].trim() || judge.output.slice(-500); + log(`reject "${params.goal}": ${oneLine(missing)}`); + updateWidget(ctx); + return result(`Sign-off REJECTED. Missing:\n${missing}\n\n--- judge ---\n${reasoning}`, true); + } + // No VERDICT line: same fail-forward as judge-unavailable. + log(`signed off "${params.goal}" (judge inconclusive: no VERDICT line)`); + updateWidget(ctx); + return result(`Judge returned no VERDICT line. Accepted inconclusive — logged. Tick the goal [x] in ${PLAN_REL} yourself.\n\n--- judge ---\n${reasoning || "(no output)"}`); + }, + }); } -// --- helpers (module scope; pure enough to keep out of the closure) ------------------------------- +// --- helpers (module scope) -------------------------------------------------------------------- -/** Structured details returned by CompleteGoal so renderCall/renderResult can show metadata. */ -interface SignOffDetails { - goal: string; - outcome: "accepted" | "accepted_inconclusive" | "rejected" | "verify_failed" | "running"; - phase?: string; // "verifying" | "spawning" | "judging" — while running - durationMs: number; - verifyCommand?: string; - verifyExitCode?: number; - judgeModel?: string; - reasoning: string; - isError?: boolean; -} - -function text(s: string, isError = false) { - return { content: [{ type: "text" as const, text: s }], details: { isError }, isError }; -} - -function textWithDetails(s: string, details: SignOffDetails, isError = false) { - return { content: [{ type: "text" as const, text: s }], details, isError }; +function result(text: string, isError = false) { + return { content: [{ type: "text" as const, text }], details: {}, isError }; } function stamp(): string { return new Date().toISOString().slice(0, 16).replace("T", " "); } -function currentSessionModelRef(ctx: ExtensionContext): string | null { - return ctx.model ? `${ctx.model.provider}/${ctx.model.id}` : null; +function oneLine(s: string): string { + return s.replace(/\s+/g, " ").trim().slice(0, 200); } -/** Decide a sign-off: deterministic verify first (cheap; skip the model call if it fails), then the judge. - * Returns the outcome plus the judge's (or verify's) reasoning so CompleteGoal can show WHY. */ -async function decideSignOff( - goal: Goal, - evidence: string, - paths: string[], - judgeModel: string | null, - cwd: string, - signal: AbortSignal | undefined, - onUpdate?: (partial: { content: Array<{ type: "text"; text: string }>; details: SignOffDetails }) => void, -): Promise<{ outcome: SignOff; reasoning: string; durationMs: number }> { - const startedAt = Date.now(); - const emit = (phase: string, text: string) => { - onUpdate?.({ - content: [{ type: "text" as const, text }], - details: { goal: goal.subject, outcome: "running", phase, durationMs: Date.now() - startedAt, verifyCommand: goal.verify ?? undefined, judgeModel: judgeModel ?? undefined, reasoning: "" }, - }); - }; - let verifyResult: { command: string; exitCode: number; outputTail: string } | null = null; - if (goal.verify) { - emit("verifying", `Running verify: ${goal.verify}`); - verifyResult = runVerify(goal.verify, cwd, signal); - if (verifyResult.exitCode !== 0) { - return { - outcome: { kind: "verify_failed", exitCode: verifyResult.exitCode, outputTail: verifyResult.outputTail }, - reasoning: `verify \`${goal.verify}\` exited ${verifyResult.exitCode}:\n${verifyResult.outputTail}`, - durationMs: Date.now() - startedAt, - }; - } +/** Append one line under ## Log (creating the section at EOF if absent). The extension's only write. */ +export function appendLog(text: string, entry: string): string { + const lines = text.split("\n"); + const line = `- ${entry}`; + const header = lines.findIndex((l) => /^##\s+Log\s*$/i.test(l)); + if (header === -1) return `${text.replace(/\n+$/, "")}\n\n## Log\n${line}\n`; + let insertAt = header + 1; + for (let i = header + 1; i < lines.length; i++) { + if (/^#{1,6}\s/.test(lines[i])) break; + if (/^\s*-\s+/.test(lines[i])) insertAt = i + 1; } - const verdict = await runJudge(goal, evidence, paths, verifyResult, judgeModel, cwd, signal, onUpdate); - const outcome: SignOff = - verdict.kind === "accepted" - ? { kind: "accepted" } - : verdict.kind === "inconclusive" - ? { kind: "accepted_inconclusive", reason: verdict.reason } - : { kind: "rejected", missing: verdict.missing }; - return { outcome, reasoning: verdict.reasoning, durationMs: verdict.durationMs }; + lines.splice(insertAt, 0, line); + return lines.join("\n"); } -/** Run the goal's verify command. It is agent-authored and trusted (single-user machine, guide-not-guard). */ -function runVerify(command: string, cwd: string, signal: AbortSignal | undefined): { command: string; exitCode: number; outputTail: string } { - const res = spawnSync("sh", ["-c", command], { cwd, encoding: "utf-8", signal, timeout: 600_000 }); - const out = `${res.stdout ?? ""}${res.stderr ?? ""}`; - return { command, exitCode: res.status ?? 1, outputTail: out.split("\n").slice(-30).join("\n") }; +/** Build the pi argv for the read-only judge. `--model` is omitted when no explicit/session model is + * set, so pi falls back to its configured default — the judge always runs. `--no-extensions` keeps + * the judge minimal and immune to a broken third-party extension taking down every sign-off. + * Exported for the unit test that locks these invariants. */ +export function buildJudgeArgs(judgeModel: string | null): string[] { + const args = ["-p", "--no-session", "--no-extensions"]; + if (judgeModel) args.push("--model", judgeModel); + args.push("--tools", JUDGE_TOOLS.join(","), "--exclude-tools", JUDGE_BLOCKED_TOOLS.join(","), "--append-system-prompt", judgeSystem); + return args; } /** Locate the pi binary the same way the oracle extension does, so spawning works under bun or node. */ @@ -548,198 +301,44 @@ function getPiInvocation(args: string[]): { command: string; args: string[] } { return { command: "pi", args }; } -function extractTextFromContent(content: unknown): string { - if (!Array.isArray(content)) return ""; - const parts: string[] = []; - for (const part of content) { - if (part && typeof part === "object" && (part as { type?: string }).type === "text") { - const text = (part as { text?: string }).text; - if (typeof text === "string" && text.trim()) parts.push(text); - } - } - return parts.join("\n\n").trim(); -} - -type JudgeResult = - | { kind: "accepted"; reasoning: string; durationMs: number } - | { kind: "rejected"; missing: string; reasoning: string; durationMs: number } - | { kind: "inconclusive"; reason: string; reasoning: string; durationMs: number }; - -/** Stage 2: a read-only pi subprocess inspects the evidence against the repo and returns a verdict. */ -/** Build the pi argv for the read-only judge. `--model` is omitted when no explicit/session model is - * set, so pi falls back to its configured default -- the judge always runs, never pre-emptively - * fails as "no model". Exported for a unit test that locks this invariant (an empty `--model ""` - * would make every sign-off silently inconclusive). */ -export function buildJudgeArgs(judgeModel: string | null): string[] { - const args = ["--mode", "json", "-p", "--no-session"]; - if (judgeModel) args.push("--model", judgeModel); - args.push("--tools", JUDGE_TOOLS.join(","), "--exclude-tools", JUDGE_BLOCKED_TOOLS.join(","), "--append-system-prompt", evidenceJudgeSystem); - return args; -} - -/** Stage 2: a read-only pi subprocess inspects the evidence against the repo and returns a verdict. */ +/** Spawn the read-only judge subprocess (plain `pi -p`: stdout is the final response text). */ async function runJudge( - goal: Goal, - evidence: string, - paths: string[], - verifyResult: { command: string; exitCode: number; outputTail: string } | null, + task: string, judgeModel: string | null, cwd: string, signal: AbortSignal | undefined, - onUpdate?: (partial: { content: Array<{ type: "text"; text: string }>; details: SignOffDetails }) => void, -): Promise { - const startedAt = Date.now(); - const emit = (phase: string, text: string) => { - onUpdate?.({ - content: [{ type: "text" as const, text }], - details: { goal: goal.subject, outcome: "running", phase, durationMs: Date.now() - startedAt, verifyCommand: goal.verify ?? undefined, judgeModel: judgeModel ?? undefined, reasoning: "" }, - }); - }; - const task = evidenceJudgeUser({ - subject: goal.subject, - discriminator: goal.discriminator, - failure_modes: goal.failure_modes, - verify: goal.verify ?? null, - verifyResult, - evidence, - paths, - }); +): Promise<{ output: string; error?: string }> { const args = buildJudgeArgs(judgeModel); args.push(task); - - emit("spawning", `Spawning read-only judge for: ${goal.subject}`); const inv = getPiInvocation(args); - // The judge runs in-place against this checkout (cwd is passed to spawn and the read-only tools - // read from it); pi --no-session does not clone into the parent. Proven and re-checked by - // scripts/check-judge-footprint.sh, which reproduces this invocation and asserts no parent clone. - const judge = await new Promise<{ output: string; error?: string; aborted?: boolean }>((resolve) => { + // Runs in-place against this checkout; pi --no-session does not clone into the parent + // (proven by scripts/check-judge-footprint.sh). + return new Promise((resolvePromise) => { let settled = false; - let stdoutBuffer = ""; - let finalOutput = ""; - let currentText = ""; + let stdout = ""; let stderr = ""; - let lastStopReason: string | undefined; - let lastErrorMessage: string | undefined; - let lastProgressAt = 0; - const partialOutput = (): string => [finalOutput, currentText, stderr.trim()].filter(Boolean).join("\n\n").trim(); - const emitProgress = (text: string, force = false) => { - const now = Date.now(); - if (!force && now - lastProgressAt < 750) return; - lastProgressAt = now; - emit("judging", text); + const done = (r: { output: string; error?: string }) => { + if (!settled) { + settled = true; + clearTimeout(timer); + resolvePromise(r); + } }; const proc = spawn(inv.command, inv.args, { cwd, shell: false, stdio: ["ignore", "pipe", "pipe"], signal }); - if (!proc.stdout || !proc.stderr) throw new Error("judge subprocess stdio must be piped"); const timer = setTimeout(() => { - if (!settled) { - settled = true; - proc.kill(); - resolve({ output: partialOutput(), error: `judge timed out after ${JUDGE_TIMEOUT_MS / 1000}s` }); - } + proc.kill(); + done({ output: stdout.trim(), error: `judge timed out after ${JUDGE_TIMEOUT_MS / 1000}s` }); }, JUDGE_TIMEOUT_MS); - - const processLine = (line: string) => { - if (!line.trim()) return; - let event: any; - try { - event = JSON.parse(line); - } catch { - return; - } - - if (event.type === "message_start" && event.message?.role === "assistant") { - currentText = ""; - emitProgress("Judge is reading evidence...", true); - return; - } - if (event.type === "message_update" && event.assistantMessageEvent?.type === "text_delta") { - currentText += event.assistantMessageEvent.delta ?? ""; - emitProgress(currentText || "Judge is reading evidence..."); - return; - } - if (event.type === "message_end" && event.message?.role === "assistant") { - const text = extractTextFromContent(event.message.content) || currentText; - if (text) finalOutput = text; - currentText = ""; - lastStopReason = typeof event.message.stopReason === "string" ? event.message.stopReason : undefined; - lastErrorMessage = typeof event.message.errorMessage === "string" ? event.message.errorMessage : undefined; - emitProgress(finalOutput || "Judge finished without text.", true); - } - }; - - proc.stdout.on("data", (data) => { - stdoutBuffer += data.toString(); - const lines = stdoutBuffer.split("\n"); - stdoutBuffer = lines.pop() || ""; - for (const line of lines) processLine(line); + proc.stdout?.on("data", (d) => { + stdout += d.toString(); }); - proc.stderr.on("data", (data) => { - stderr += data.toString(); - emitProgress(partialOutput() || "Judge subprocess wrote stderr."); + proc.stderr?.on("data", (d) => { + stderr += d.toString(); }); proc.on("close", (code) => { - if (!settled) { - settled = true; - clearTimeout(timer); - if (stdoutBuffer.trim()) processLine(stdoutBuffer); - if (lastStopReason === "error" || lastStopReason === "aborted") { - resolve({ output: finalOutput, error: lastErrorMessage || `judge model ${lastStopReason}`, aborted: signal?.aborted }); - return; - } - if ((code ?? 0) !== 0) { - resolve({ output: finalOutput, error: stderr.trim() || finalOutput || `judge subprocess exited ${code ?? 1}` }); - return; - } - resolve({ output: finalOutput }); - } - }); - proc.on("error", (e) => { - if (!settled) { - settled = true; - clearTimeout(timer); - resolve({ output: "", error: `judge subprocess failed: ${e.message}`, aborted: signal?.aborted }); - } + if ((code ?? 0) !== 0) done({ output: stdout.trim(), error: stderr.trim() || `judge subprocess exited ${code ?? 1}` }); + else done({ output: stdout.trim() }); }); + proc.on("error", (e) => done({ output: stdout.trim(), error: `judge subprocess failed: ${e.message}` })); }); - - if (judge.error) { - const partial = judge.output.trim(); - if (judge.aborted) { - return { - kind: "rejected", - missing: judge.error, - reasoning: `VERDICT: reject\nmissing: ${judge.error}`, - durationMs: Date.now() - startedAt, - }; - } - return { - kind: "inconclusive", - reason: judge.error, - reasoning: partial - ? `VERDICT: inconclusive\nreason: ${judge.error}\n\npartial judge output:\n${partial}` - : `VERDICT: inconclusive\nreason: ${judge.error}`, - durationMs: Date.now() - startedAt, - }; - } - - const clean = judge.output.trim(); - const verdictLine = clean.split("\n").find((l) => /^\s*VERDICT\s*:/i.test(l)) ?? ""; - const verdictMatch = /^\s*VERDICT\s*:\s*(accept|reject)\s*$/i.exec(verdictLine); - if (!verdictMatch) { - return { - kind: "inconclusive", - reason: clean ? "judge returned no exact VERDICT line" : "judge finished without returning any text", - reasoning: clean || "VERDICT: inconclusive\nreason: judge finished without returning any text", - durationMs: Date.now() - startedAt, - }; - } - const missingMatch = clean.match(/missing\s*:\s*([\s\S]*)$/i); - // The judge's own words (inspection + verdict), so CompleteGoal can show them. The verdict is at the - // end, so keep the tail when it's long. - const trimmed = clean.trim(); - const reasoning = trimmed.length > 1800 ? `...\n${trimmed.slice(-1800)}` : trimmed; - if (verdictMatch[1].toLowerCase() === "accept") return { kind: "accepted", reasoning, durationMs: Date.now() - startedAt }; - - const missing = missingMatch?.[1].trim() || clean.slice(-500) || "judge gave no reason"; - return { kind: "rejected", missing, reasoning, durationMs: Date.now() - startedAt }; } diff --git a/src/plan-file.ts b/src/plan-file.ts deleted file mode 100644 index 301de5f..0000000 --- a/src/plan-file.ts +++ /dev/null @@ -1,325 +0,0 @@ -/** - * plan-file.ts — read goals.md, and the two writes CompleteGoal needs. That is all. - * - * Pure module, no pi deps, so it unit-tests without a runtime. The file is the canonical store and - * the agent edits it with its normal Edit tool (create goals, tick subtasks, fill evidence), guided - * by the format in prompts.ts and the reminder -- the form guides, it does not gate. The only - * programmatic writers are setGoalStatus + appendLog, used by CompleteGoal to record an accepted - * sign-off; both touch one line so the diff stays readable. - * - * Format (markdown, checkbox-first, made to be skim-reviewed by a human): - * - * # - * - * - * - * ## Goals - * - * 1. [ ] goal: <- state in the checkbox: [ ] open [/] active [x] done [-] cancelled - * - discriminator: - * - subtle failure mode: - * - verify: - * - tasks: - * 1. [x] <- a subtask is any checkbox WITHOUT a "goal:" prefix - * 2. [/] - * 3. [-] <- [-] or ~~[ ]~~ both read as cancelled - * - evidence: <- empty at planning; filled at sign-off, read by CompleteGoal - * - > - * 2. [ ] goal: - * - * # Future work / out of scope - * - * ## Log - * - - * - * A goal/subtask's state lives in its checkbox (single source of truth, renders natively). Goals are - * matched by their (the text after "goal:"); the list number is human-facing only. Only - * CompleteGoal writes a goal's [x]; the agent sets [/] when it starts one. - */ - -export type GoalStatus = "open" | "active" | "done" | "cancelled"; - -export interface Subtask { - text: string; - status: GoalStatus; -} - -export interface Goal { - /** The text after "goal:" in the header line; the handle CompleteGoal matches on. */ - subject: string; - status: GoalStatus; - /** Positive observation(s) that the goal succeeded AND that no failure mode could fake. The success test. Written at planning. */ - discriminator: string[]; - /** Subtle ways a "done" could be wrong (look-like-success failures). Written at planning. */ - failure_modes: string[]; - /** Optional command that exits 0 only when the discriminator passes (the cheap deterministic gate). */ - verify?: string; - /** Proof the discriminator passed, pointing at durable artifacts. Written at completion; read by CompleteGoal. */ - evidence: string[]; - subtasks: Subtask[]; -} - -export interface PlanDoc { - title: string; - goals: Goal[]; - /** Verbatim ## Log lines, including the leading "- ". */ - log: string[]; -} - -const TITLE = /^#\s+(.+?)\s*$/; // the first single-# H1 -const GOALS_HEADER = /^##\s+Goals\s*$/i; -const LOG_HEADER = /^##\s+Log\s*$/i; -const ANY_HEADER = /^#{1,6}\s/; -// A goal: a numbered or bulleted checkbox item whose text begins "goal:". -const GOAL_ITEM = /^\s*(?:\d+\.|[-*])\s*\[([ xX/-])\]\s*goal:\s*(.*)$/i; -// A section marker bullet under a goal (the trailing colon is optional, e.g. "- tasks"). -const KEY_LINE = /^\s*[-*]\s*(discriminator|subtle failure modes?|failure_modes?|verify|tasks?|evidence)\s*:?\s*(.*)$/i; -// Any list item (numbered or bulleted); used for subtasks and for list items inside the sections. -const LIST_ITEM = /^\s*(?:\d+\.|[-*])\s+(.*)$/; -// A checkbox inside a list-item body (subtask). A leading/trailing ~~ marks it cancelled. -const CHECKBOX_BODY = /^(~~)?\s*\[([ xX/-])\]\s*(.*)$/; - -const CHAR_TO_STATUS: Record = { " ": "open", "/": "active", x: "done", "-": "cancelled" }; -const STATUS_TO_CHAR: Record = { open: " ", active: "/", done: "x", cancelled: "-" }; - -function normalizeKey(raw: string): "discriminator" | "failure_modes" | "verify" | "tasks" | "evidence" { - const k = raw.toLowerCase(); - if (k.startsWith("discriminator")) return "discriminator"; - if (k.startsWith("verify")) return "verify"; - if (k.startsWith("task")) return "tasks"; - if (k.startsWith("evidence")) return "evidence"; - return "failure_modes"; // "subtle failure mode(s)" / "failure_mode(s)" -} - -export function parse(text: string): PlanDoc { - const lines = text.split("\n"); - let title = ""; - const goals: Goal[] = []; - const log: string[] = []; - - let cur: Goal | null = null; - let curList: string[] | null = null; // the discriminator/failure_modes/evidence list "- " items append to - let inGoals = false; - let inLog = false; - - const flush = () => { - if (cur) goals.push(cur); - cur = null; - curList = null; - }; - - for (const line of lines) { - const tM = TITLE.exec(line); - if (tM && !title && !GOALS_HEADER.test(line) && !LOG_HEADER.test(line)) { - title = tM[1].trim(); - continue; - } - if (GOALS_HEADER.test(line)) { - flush(); - inGoals = true; - inLog = false; - continue; - } - if (LOG_HEADER.test(line)) { - flush(); - inGoals = false; - inLog = true; - continue; - } - // Any other header (e.g. "# Future work") ends the goals / log section. - if (ANY_HEADER.test(line)) { - flush(); - inGoals = false; - inLog = false; - continue; - } - - if (inLog) { - if (/^\s*-\s+/.test(line)) log.push(line); - continue; - } - if (!inGoals) continue; // title + context prose between the title and ## Goals - - const goalM = GOAL_ITEM.exec(line); - if (goalM) { - flush(); - cur = { - subject: goalM[2].trim(), - status: CHAR_TO_STATUS[goalM[1].toLowerCase()] ?? "open", - discriminator: [], - failure_modes: [], - evidence: [], - subtasks: [], - }; - continue; - } - if (!cur) continue; - - const keyM = KEY_LINE.exec(line); - if (keyM) { - const key = normalizeKey(keyM[1]); - const inlineVal = keyM[2].trim(); - if (key === "verify") { - cur.verify = inlineVal || undefined; - curList = null; - } else if (key === "tasks") { - curList = null; // subtasks are identified by being a checkbox; this marker is cosmetic - } else { - curList = cur[key]; // discriminator | failure_modes | evidence - if (inlineVal) curList.push(inlineVal); - } - continue; - } - - const listM = LIST_ITEM.exec(line); - if (listM) { - const body = listM[1]; - const cb = CHECKBOX_BODY.exec(body); - if (cb) { - // A checkbox without a "goal:" prefix is a subtask of the current goal. - const cancelled = cb[1] === "~~" || body.includes("~~"); - const status = cancelled ? "cancelled" : (CHAR_TO_STATUS[cb[2].toLowerCase()] ?? "open"); - cur.subtasks.push({ text: cb[3].replace(/~~/g, "").trim(), status }); - curList = null; - continue; - } - // A plain "- " / "> " item belongs to the current section (discriminator/failure/evidence). - if (curList) curList.push(body.trim()); - continue; - } - - // A non-empty, non-"- " line continues the current item, so multi-line evidence (a block quote - // of a log, a table, an interpretation line) stays attached to its item. Blank lines are skipped. - if (curList && line.trim() !== "" && curList.length > 0) { - curList[curList.length - 1] += `\n${line.trim()}`; - } - } - flush(); - - return { title, goals, log }; -} - -export function findGoal(doc: PlanDoc, subject: string): Goal | undefined { - const want = subject.trim(); - return doc.goals.find((g) => g.subject === want); -} - -export function counts(doc: PlanDoc): { done: number; open: number; active: number } { - const c = { done: 0, open: 0, active: 0 }; - for (const g of doc.goals) { - if (g.status === "done") c.done++; - else if (g.status === "active") c.active++; - else if (g.status === "open") c.open++; - } - return c; -} - -/** Flip a goal's checkbox in place, matched by its subject (the one write CompleteGoal needs). */ -export function setGoalStatus(text: string, subject: string, status: GoalStatus): string { - const lines = text.split("\n"); - const want = subject.trim(); - for (let i = 0; i < lines.length; i++) { - const m = GOAL_ITEM.exec(lines[i]); - if (m && m[2].trim() === want) { - lines[i] = lines[i].replace(/\[[ xX/-]\]/, `[${STATUS_TO_CHAR[status]}]`); - return lines.join("\n"); - } - } - throw new Error(`Goal "${subject}" not found`); -} - -/** - * Drop done+cancelled goal blocks (their header line through the last of their subtask/section lines) - * from goals.md, keeping the title, context, active/open goals, and the ## Log. Pure. Lets the human - * prune old finished goals so the widget/file stay short across sessions, without losing the log trail. - */ -export function pruneCompleted(text: string): string { - const lines = text.split("\n"); - const out: string[] = []; - let inGoals = false; - let dropping = false; - for (const line of lines) { - if (GOALS_HEADER.test(line)) { - inGoals = true; - dropping = false; - out.push(line); - continue; - } - if (ANY_HEADER.test(line)) { - // Any other header (## Log, "# Future work") ends the goals section. - inGoals = false; - dropping = false; - out.push(line); - continue; - } - if (inGoals) { - const m = GOAL_ITEM.exec(line); - if (m) { - const status = CHAR_TO_STATUS[m[1].toLowerCase()] ?? "open"; - dropping = status === "done" || status === "cancelled"; - } - if (dropping) continue; // skip the goal line and everything under it until the next goal/header - } - out.push(line); - } - return out.join("\n"); -} - -/** - * The outcome of a sign-off attempt, decided by CompleteGoal (which runs verify + the judge). Kept - * separate from the I/O so the record logic below is pure and testable. - */ -export type SignOff = - | { kind: "verify_failed"; exitCode: number; outputTail: string } - | { kind: "rejected"; missing: string } - | { kind: "accepted" } - | { kind: "accepted_inconclusive"; reason: string }; - -/** Apply a sign-off outcome to goals.md text: accept flips the goal checkbox to [x] + logs; reject only logs. Pure. */ -export function recordSignOff( - text: string, - subject: string, - when: string, - outcome: SignOff, -): { content: string; message: string; isError: boolean } { - const goal = findGoal(parse(text), subject); - if (!goal) return { content: text, message: `No goal "${subject}" in goals.md.`, isError: true }; - - if (outcome.kind === "verify_failed") { - const content = appendLog(text, `${when} reject "${subject}": verify exit ${outcome.exitCode}`); - return { content, message: `Sign-off rejected: verify failed (exit ${outcome.exitCode}).\n${outcome.outputTail}`, isError: true }; - } - if (outcome.kind === "rejected") { - const oneLine = outcome.missing.replace(/\s+/g, " ").trim().slice(0, 200); - const content = appendLog(text, `${when} reject "${subject}": ${oneLine}`); - return { content, message: `Sign-off rejected. Missing:\n${outcome.missing}`, isError: true }; - } - const flipped = setGoalStatus(text, subject, "done"); - if (outcome.kind === "accepted_inconclusive") { - const oneLine = outcome.reason.replace(/\s+/g, " ").trim().slice(0, 200); - const content = appendLog(flipped, `${when} signed off "${subject}" (judge inconclusive: ${oneLine})`); - return { - content, - message: `Signed off "${subject}". Marked done in goals.md.\nJudge inconclusive: ${outcome.reason}`, - isError: false, - }; - } - const content = appendLog(flipped, `${when} signed off "${subject}" (judge accept)`); - return { content, message: `Signed off "${subject}". Marked done in goals.md.`, isError: false }; -} - -/** Append one verbatim line to ## Log (creating the section if absent). The other CompleteGoal write. */ -export function appendLog(text: string, entry: string): string { - const lines = text.split("\n"); - const line = `- ${entry}`; - const header = lines.findIndex((l) => LOG_HEADER.test(l)); - if (header === -1) return `${text.replace(/\n+$/, "")}\n\n## Log\n${line}\n`; - - let insertAt = header + 1; - for (let i = header + 1; i < lines.length; i++) { - if (ANY_HEADER.test(lines[i])) break; - if (/^\s*-\s+/.test(lines[i])) insertAt = i + 1; - } - lines.splice(insertAt, 0, line); - return lines.join("\n"); -} diff --git a/src/prompts.ts b/src/prompts.ts index 3b8d270..22d0cde 100644 --- a/src/prompts.ts +++ b/src/prompts.ts @@ -1,60 +1,41 @@ /** - * pi-goals — all model-facing text, in flow order. + * pi-goals v2 — all model-facing text, in flow order. * - * Philosophy: the form guides a process; it does not police one. The agent can - * edit goals.md freely. These prompts + the goals.md structure make the right path - * the easy path. The only step that is genuinely rigorous is the evidence judge - * (7), and even that is reached by guiding the agent to call CompleteGoal, not by - * trapping it. Bypasses stay visible in the git diff and the widget. + * Design: plan.md is for LLMs and the human, not for TypeScript. There is no parser and no schema; + * the skeleton below is a convention the drafting prompt teaches, the working agent maintains with + * its normal Edit tool, and the judge reads natively. The harness does three things for a + * cooperative-but-confused model: memory (inject the file verbatim every turn), format guidance + * (the skeleton), and fresh eyes (the read-only judge in CompleteGoal). * - * Flow (this file is ordered the way the agent meets each text, so it reads as one pass): - * SETUP (plan mode) 1. planDrafting — drafts goals (read-only phase) - * EXEC, each turn start 2. planInjection — "here is your plan, where you are" - * EXEC, periodic 3. reminder — the typed nudge that drives upkeep + autonomy - * EXEC, loop continue 4. continuation — keep going toward the active goal - * EXEC, after each turn 5. loopJudge — continue / pause (cheap, foolable, ok) - * SIGN-OFF, agent-side 6. completeGoalTool — the CompleteGoal tool desc + param the agent reads - * SIGN-OFF, judge-side 7. evidenceJudge — read-only verify (rigorous; the one real check) - * - * Read top to bottom to see the whole process. 5 and 7 embody the design contrast: - * the cheap-foolable loop gate vs the must-not-be-fooled sign-off. - * - * WIRED in index.ts: 1 planDrafting, 2 planInjection, 3 reminder, 6 completeGoalTool, 7 evidenceJudge. - * NOT YET WIRED: 4 continuation and 5 loopJudge define the autonomous re-prompt loop, which is - * intentionally not built in v1 (an until-done-style loop was judged too complex). They stay here so - * the full intended flow is reviewable; wire them if/when the loop is added. + * Flow: + * SETUP (plan mode) 1. planDrafting — draft goals into plan.md (read-only phase) + * EXEC, each turn start 2. (the plan.md file itself, injected verbatim by index.ts) + * EXEC, periodic 3. reminder — upkeep + autonomy nudge when plan.md went untouched + * SIGN-OFF, agent-side 4. completeGoal* — the one blessed tool's description + * SIGN-OFF, judge-side 5. judgeSystem/judgeUser — the one rigorous check * * The goal's test is the DISCRIMINATOR: the concrete observation that tells real success from the - * named subtle failure mode. It replaces a vague "done_when". Evidence is empty at planning and - * filled at sign-off (you don't always know the exact artifacts up front; the judge checks them then). + * named subtle failure mode. Evidence is empty at planning and filled at sign-off. */ /* ───────────────────────────────────────────────────────────────────────── - * 1. planDrafting — SETUP, plan mode - * - * System guidance for the plan-phase agent. This phase is read-only (edit/write - * and mutating bash are blocked by a tool hook): explore, then draft goals into - * goals.md. The fields here are the whole "elicitation"; the human reviews this - * output before any execution. + * 1. planDrafting — SETUP, plan mode (read-only: edit/write blocked except plan.md) * ──────────────────────────────────────────────────────────────────────── */ export const planDrafting = `\ You are in plan mode. The objective may arrive through conversation, not as one up-front command. -Explore the repository read-only first, then ask: resolve discoverable facts by looking them up, and -only ask the human when the answer is a genuine intent or preference choice that exploration can't -settle. Don't write goals that branch on something you could just check. Do not write or run code in -this phase (edit and write are blocked, and so is mutating bash). If the ask is itself read-only -(e.g. research, a search, a report), explore enough to scope it, but leave the actual deliverable for -after the human approves the plan. When the objective is clear, draft goals into goals.md and stop -for review. Produce a plan the human will review and approve. +Explore the repository read-only first: resolve discoverable facts by looking them up, and only ask +the human when the answer is a genuine intent or preference choice. Do not write or run code in this +phase (edit/write are blocked except for the plan file; don't mutate state via bash either). When +the objective is clear, draft the plan file and stop for review. -Right-size it, don't force structure that isn't there: -- Default to ONE goal. Add another only when it's a genuinely separate checkpoint you'd want signed - off on its own (it can pass or fail independently). Most objectives are 1-2 goals. -- Subtasks are the steps inside a goal. Add them when a goal has 3+ distinct steps; skip them for a - single-action goal. Don't pad with trivial steps. +Right-size it: +- Default to ONE goal. Add another only when it's a genuinely separate checkpoint that can pass or + fail on its own. Most objectives are 1-2 goals. +- Subtasks are the steps inside a goal; add them when a goal has 3+ distinct steps, skip otherwise. - Don't invent goals to look thorough. When in doubt, merge. -Write the whole file in this shape (markdown checkboxes, made to be skim-reviewed): +Write the plan file in roughly this shape (it's a convention, not a schema -- the file is read +directly by the human and a judge model, so clarity beats conformance; small deviations are fine): # @@ -68,208 +49,90 @@ Write the whole file in this shape (markdown checkboxes, made to be skim-reviewe - verify: - tasks: 1. [ ] - 2. [ ] - - evidence: - - -2. [ ] goal: <...> + - evidence: (empty until sign-off) # Future work / out of scope -- - ## Log -Keep it lean and legible: -- A goal is a checkbox line beginning "goal:"; its state is the checkbox ([ ] open, [/] active, [x] - done, [-] cancelled). Leave goals [ ] at planning. The number is just for the human to reference. -- subtle failure mode + discriminator are the heart of this. List the ways a "done" could look - achieved but not be (empty/zero-count output, a silently-errored step, a gamed test, a flat/no-op - result that dodged every trap and still showed nothing; these are examples, find the ones that fit). -- The discriminator is the POSITIVE observation that the goal actually succeeded AND that none of - those failure modes could have produced. It must show success happened -- the count moved the right - way, the test really exercised the path, the metric beat noise -- not merely that a failure was - ruled out: avoiding every failure mode is necessary, not sufficient. Name the success signal first, - then check it isn't something a failure mode could fake. Keep it terse. -- The discriminator is the success test, written now, in place of a vague "done": make it a concrete, - checkable observation about a real artifact (a file, a test result, a committed diff, a metric), not - about goals.md's own checkbox. -- subtasks: any checkbox WITHOUT a "goal:" prefix, under "- tasks:". Use [/] for in progress and [-] - for cancelled/impossible. -- verify: prefer one when the discriminator is a test, build, threshold, or metric: a green check or - a printed number beats prose. Omit it otherwise. -- evidence stays empty at planning. You don't always know the exact artifacts up front, and that's - fine: you fill evidence at sign-off, and a fresh read-only judge checks it then. +Conventions: +- A goal is a checkbox line beginning "goal:". Checkbox state: [ ] open, [/] active, [x] done, + [-] cancelled. Leave goals [ ] at planning. +- subtle failure mode + discriminator are the heart of this. Name the ways a "done" could look + achieved but not be (empty output, a silently-errored step, a gamed test, a no-op that dodged + every trap and showed nothing). The discriminator is the POSITIVE observation that success + happened -- the count moved, the test exercised the real path, the metric beat noise -- and that + none of the failure modes could fake. Ruling out failures is necessary, not sufficient. +- Make the discriminator a concrete, checkable observation about a real artifact (a file, a test + result, a committed diff, a metric), never about the plan file's own checkbox. +- evidence stays empty at planning; you fill it at sign-off and a fresh read-only judge checks it. When the goals are drafted, present them and stop for review. Do not begin execution.`; /* ───────────────────────────────────────────────────────────────────────── - * 2. planInjection — EXEC, injected at each agent start (and after compaction) - * - * A late user-role message, NOT a system-prompt mutation (keeps the prefix cache - * valid). Built from the parsed plan. MUST be byte-identical when nothing changed: - * fixed field order, no volatile timestamps. Pass only the active goal + its open - * subtasks + the last log line, not the whole file. - * ──────────────────────────────────────────────────────────────────────── */ -export function planInjection(p: { - title: string; - activeGoal: { subject: string; discriminator: string[]; openSubtasks: string[] } | null; - lastLogLine: string | null; - counts: { done: number; open: number }; -}): string { - if (!p.activeGoal) { - return `.pi/goals.md: ${p.title}\nNo active goal. ${p.counts.open} open, ${p.counts.done} done. Pick the next goal (set its checkbox to [/]) or run /goals.`; - } - const subtasks = p.activeGoal.openSubtasks.length - ? p.activeGoal.openSubtasks.map((s) => ` - [ ] ${s}`).join("\n") - : " (no open subtasks)"; - const disc = p.activeGoal.discriminator.length ? p.activeGoal.discriminator.join("; ") : "(none set)"; - return `\ -.pi/goals.md: ${p.title} -Active goal: ${p.activeGoal.subject} -discriminator (the success test): ${disc} -Open subtasks: -${subtasks} -Last log: ${p.lastLogLine ?? "(none yet)"} -Progress: ${p.counts.done} done, ${p.counts.open} open.`; -} - -/* ───────────────────────────────────────────────────────────────────────── - * 3. reminder — EXEC, periodic system-reminder - * - * The typed nudge. This is both the housekeeping and the autonomy engine — it is - * what makes the process get followed without a hard gate. Fires after a turn that - * left goals.md untouched while a goal is active. Keep the wording stable so it - * doesn't thrash the cache. + * 3. reminder — EXEC, appended to the injected plan when plan.md went + * untouched for a whole turn while goals are open. Wording stable (cache). * ──────────────────────────────────────────────────────────────────────── */ export const reminder = `\ -Keep goals.md current as you work: -- tasks: tick the subtasks you've finished ([/] for in progress); add any you've discovered. -- log: append ONE short line to ## Log (append, don't rewrite earlier lines). -- goal: when the active goal's discriminator is satisfied, fill its evidence: block in goals.md (a - list pointing at durable artifacts), then call CompleteGoal with the goal's desc. Don't tick the - goal [x] by hand; CompleteGoal reads the evidence, runs the check, and writes [x]. -- otherwise: keep working toward the active goal. Don't stop to ask unless you're genuinely blocked; - if blocked, say what's blocking it. +Keep the plan file current as you work (edit it directly): +- tick finished subtasks ([/] in progress), add discovered ones +- append ONE short line to ## Log (append, don't rewrite earlier lines) +- when the active goal's discriminator is satisfied, fill its evidence: list (each item = a durable + artifact + a short read of it), then call CompleteGoal. Don't tick a goal [x] before CompleteGoal + accepts; the sign-off log line is the audit trail. +- if the file has grown long, prune finished goals (their evidence lives in git history and ## Log) +- otherwise keep working toward the active goal; don't stop to ask unless genuinely blocked `; /* ───────────────────────────────────────────────────────────────────────── - * 4. continuation — EXEC, the loop's "keep going" turn - * - * Hermes-style. A plain user-role message appended when the loop judge (5) says - * continue. Does not mutate the system prompt, so the cache holds. - * ──────────────────────────────────────────────────────────────────────── */ -export const continuation = `\ -Continue toward the active goal in goals.md. If its discriminator is now satisfied, fill the goal's -evidence: block (durable artifacts, e.g. saved logs, committed diffs, files, not just claims) and -then call CompleteGoal with the goal's desc. If you're blocked, state what's blocking it.`; - -/* ───────────────────────────────────────────────────────────────────────── - * 5. loopJudge — EXEC, runs after each turn to decide continue / pause - * - * Cheap, conservative, fail-open. Reads only the agent's last response, so it CAN - * be fooled by an asserted "done" — that's acceptable: its worst case is a - * premature pause, caught by you or the iteration budget. It does NOT sign goals - * off; that's the evidence judge's job. Return strict JSON, no prose. - * ──────────────────────────────────────────────────────────────────────── */ -export const loopJudgeSystem = `\ -You decide whether an autonomous coding agent should keep working or pause for the human. -Be conservative: only pause when the work is plainly finished or plainly blocked. When in -doubt, continue. You are not verifying correctness; a later read-only judge does that. -Reply with ONLY a JSON object, no other text: {"done": boolean, "reason": ""}. -Set done=true only if the agent's last message shows the active goal's discriminator is satisfied, -or the agent says it is blocked and needs the human.`; - -export function loopJudgeUser(p: { discriminator: string; lastResponse: string }): string { - return `\ -Active goal discriminator (the success test): ${p.discriminator} - -Agent's last message: -""" -${p.lastResponse} -""" - -{"done": ?, "reason": ?}`; -} - -/* ───────────────────────────────────────────────────────────────────────── - * 6. completeGoalTool — SIGN-OFF, agent-side - * - * The description + param the agent reads on the one blessed tool, CompleteGoal. - * This is where the agent meets the sign-off: it fills evidence and calls the - * tool, which then runs verify + the judge (7). Kept here with the rest of the - * model-facing text so the whole process reads top to bottom. + * 4. completeGoal — SIGN-OFF, agent-side: the one blessed tool * ──────────────────────────────────────────────────────────────────────── */ export const completeGoalDescription = - "Sign off a goal once its discriminator is satisfied. First fill the goal's evidence: block in " + - "goals.md: a list where each item pairs a durable artifact with a short read of it (a quoted+linked " + - "log, a table plus how to read it, or a metric plus what it shows; quote the key lines and link the " + - "rest, not a pasted blob or a bare claim). The read must show the success POSITIVELY happened (the " + - "result is present, the count moved the right way, the metric beat noise), not just that a failure " + - "was avoided; ruling out the failure modes is necessary but not sufficient. Then call this with the " + - "goal's desc (the text after 'goal:'). Runs the goal's verify command (if any) then a read-only " + - "subagent that inspects that evidence against the repo and the discriminator. On accept, the goal is " + - "marked done and logged; on judge reject, it stays open and you get what is missing. If the " + - "subagent times out or its transport/model fails after any verify command has passed, the goal is " + - "marked done with a judge-inconclusive log line. The subagent's reasoning or partial output is " + - "returned either way."; + "Sign off a goal once its discriminator is satisfied. First fill the goal's evidence: list in the " + + "plan file: each item pairs a durable artifact with a short read of it (a quoted+linked log, a " + + "table plus how to read it, a metric plus what it shows -- not a bare claim). The read must show " + + "success POSITIVELY happened, not just that failures were avoided. Then call this with the goal's " + + "text (the line after 'goal:'; small wording drift is fine). A fresh read-only judge inspects the " + + "repo, runs the goal's verify command if it names one, and returns accept or reject with what's " + + "missing. On accept (or if the judge itself is unavailable), a sign-off line is appended to ## Log " + + "and YOU tick the goal [x]. On reject the goal stays open."; -export const completeGoalParamDescription = "The goal's desc: the exact text after 'goal:' in its line."; +export const completeGoalParamDescription = "The goal's text: the line after 'goal:' in the plan file."; /* ───────────────────────────────────────────────────────────────────────── - * 7. evidenceJudge — SIGN-OFF, judge-side; the one rigorous check - * - * Runs inside CompleteGoal, on a read-only pi subprocess (fresh context via - * --no-session, so it never sees the working agent's transcript; override to a - * different vendor for an independent cross-family check). It re-derives from the - * repo rather than trusting the agent's transcription, and judges whether the - * evidence satisfies the discriminator and rules out the named failure mode. - * - * The transport gives it read/grep/find/ls. The prompt below imposes the verdict - * contract — the subprocess returns prose by default, so parse the VERDICT line. + * 5. judge — SIGN-OFF, judge-side: the one rigorous check. Runs on a fresh + * read-only pi subprocess (--no-session) so it never sees the working + * agent's transcript. It gets the WHOLE plan file: it finds the goal, + * reads discriminator/failure modes/evidence itself (no parser between). * ──────────────────────────────────────────────────────────────────────── */ -export const evidenceJudgeSystem = `\ +export const judgeSystem = `\ You are a read-only reviewer signing off a coding goal. Do not trust claims; verify. -Use read/grep/find/ls to inspect the repository and the cited artifacts yourself. Re-read the -files, logs, and diffs the evidence points to; if something it asserts isn't on disk, you can't -confirm it. Judge whether the evidence shows the goal POSITIVELY succeeded -- the discriminator's -success signal is actually present, not just that the failure modes were dodged. Avoiding every -failure mode is necessary but not sufficient: a run can rule out each trap and still have produced -nothing, so reject "no problems found" that lacks the positive result. Then check the named subtle -failure modes are genuinely ruled out, not just unmentioned. If a verify command was run, -judge whether it really tests the discriminator or could pass while the failure mode still holds; a -tautological or skipped test is a reject. +Use read/grep/find/ls/bash to inspect the repository and the cited artifacts yourself. Re-read the +files, logs, and diffs the evidence points to; if something asserted isn't on disk, you can't +confirm it. If the goal names a verify: command, run it and check it really tests the discriminator +rather than passing tautologically. Judge whether the evidence shows the goal POSITIVELY succeeded +-- the discriminator's success signal is actually present, not just that the named failure modes +were dodged; a run can rule out every trap and still have produced nothing. Then check each subtle +failure mode is genuinely ruled out, not just unmentioned. Finish with exactly these two lines and nothing after: VERDICT: accept | reject missing: `; -export function evidenceJudgeUser(p: { - subject: string; - discriminator: string[]; - failure_modes: string[]; - verify: string | null; - verifyResult: { command: string; exitCode: number; outputTail: string } | null; - evidence: string; - paths: string[]; -}): string { - const verifyBlock = p.verify - ? `verify command: ${p.verify}\nverify result: exit ${p.verifyResult?.exitCode ?? "n/a"}\n${p.verifyResult?.outputTail ?? ""}` - : "verify command: none (no deterministic check for this goal)"; - return `\ -Goal: ${p.subject} -discriminator (must be satisfied): -${p.discriminator.map((d) => ` - ${d}`).join("\n") || " (none stated, note this)"} -subtle failure modes (must be ruled out): -${p.failure_modes.map((f) => ` - ${f}`).join("\n") || " (none stated)"} +export function judgeUser(p: { goal: string; plan: string; planPath: string }): string { + return `\ +The working agent claims this goal is complete: -${verifyBlock} + goal: ${p.goal} -Agent's evidence: -${p.evidence} +Below is the full plan file (${p.planPath}). Find that goal in it (tolerate small wording drift; if +you cannot find a matching goal at all, reject and say so). Read its discriminator, subtle failure +modes, verify command, and evidence list from the file itself. -Artifacts it points to (inspect these): -${p.paths.map((x) => ` - ${x}`).join("\n") || " (none listed, note this)"} +--- plan file --- +${p.plan} +--- end plan file --- -Verify the evidence satisfies the discriminator and rules out the failure modes. Then give your VERDICT.`; +Inspect the repo and artifacts, run verify if present, then give your VERDICT.`; } diff --git a/test/append-log.test.ts b/test/append-log.test.ts new file mode 100644 index 0000000..2925d6f --- /dev/null +++ b/test/append-log.test.ts @@ -0,0 +1,17 @@ +import { describe, expect, it } from "vitest"; +import { appendLog } from "../src/index.js"; + +describe("appendLog (the extension's only plan-file write)", () => { + it("creates ## Log at EOF when absent", () => { + const out = appendLog("# Plan\n\n## Goals\n\n1. [ ] goal: x\n", "2026-07-03 10:00 signed off \"x\" (judge accept)"); + expect(out).toContain("## Log\n- 2026-07-03 10:00 signed off"); + }); + + it("appends after the last existing log line, before any following header", () => { + const plan = "# Plan\n\n## Log\n- first\n- second\n\n# Future work\n- later\n"; + const out = appendLog(plan, "third"); + const lines = out.split("\n"); + expect(lines[lines.indexOf("- second") + 1]).toBe("- third"); + expect(out.indexOf("- third")).toBeLessThan(out.indexOf("# Future work")); + }); +}); diff --git a/test/judge-args.test.ts b/test/judge-args.test.ts new file mode 100644 index 0000000..9704cac --- /dev/null +++ b/test/judge-args.test.ts @@ -0,0 +1,31 @@ +import { describe, expect, it } from "vitest"; +import { buildJudgeArgs } from "../src/index.js"; + +describe("buildJudgeArgs", () => { + it("omits --model when judgeModel is null (pi uses its configured default; never a pre-emptive 'no model' failure)", () => { + const args = buildJudgeArgs(null); + expect(args).not.toContain("--model"); + // an empty --model "" would make every sign-off silently inconclusive -- guard against it + const i = args.indexOf("--model"); + expect(i).toBe(-1); + }); + + it("includes --model when an explicit/session model is set", () => { + const args = buildJudgeArgs("openrouter/~anthropic/claude-haiku-latest"); + const i = args.indexOf("--model"); + expect(i).not.toBe(-1); + expect(args[i + 1]).toBe("openrouter/~anthropic/claude-haiku-latest"); + }); + + it("always sets --no-session, --no-extensions, the read-only tool allowlist, and edit/write exclusion", () => { + for (const m of [null, "some/model"]) { + const args = buildJudgeArgs(m); + expect(args).toContain("--no-session"); + expect(args).toContain("--no-extensions"); // a broken global extension must not take down sign-offs + expect(args).toContain("--tools"); + expect(args.some((a) => a.startsWith("read,bash,grep,find,ls"))).toBe(true); + expect(args).toContain("--exclude-tools"); + expect(args.some((a) => a.includes("edit") && a.includes("write"))).toBe(true); + } + }); +}); diff --git a/test/plan-file.test.ts b/test/plan-file.test.ts deleted file mode 100644 index 8374ff4..0000000 --- a/test/plan-file.test.ts +++ /dev/null @@ -1,220 +0,0 @@ -import { describe, expect, it } from "vitest"; -import { appendLog, counts, findGoal, parse, pruneCompleted, recordSignOff, setGoalStatus } from "../src/plan-file.js"; - -const SAMPLE = `# papers audit - -Clean up steering/ metadata and kill empty dirs. Keep it read-only until I approve. - -## Goals - -1. [/] goal: Implement cache layer - - discriminator: hit-rate > 0.8 in load-test.log (a bypass reads ~0) - - subtle failure mode: cache silently bypassed, latency ok by luck - - verify: pytest tests/cache -q - - tasks: - 1. [x] wire cache client - 2. [/] eviction policy - 3. ~~[ ]~~ distributed cache, out of scope - - evidence: - - > load-test.log: p95=41ms - - > hit-rate 0.93 (not bypassed) -2. [ ] goal: Document the API - - discriminator: every public fn has a docstring; sphinx warns on none - - subtle failure mode: docstrings exist but are stale - -# Future work / out of scope - -- distributed cache - -## Log -- 2026-06-15 14:02 cache client wired; eviction next -`; - -/** Multiset line diff: lines b adds vs removes vs a (order-insensitive, so insertions score added:1). */ -function lineDelta(a: string, b: string): { added: number; removed: number } { - const count = (s: string) => { - const m = new Map(); - for (const l of s.split("\n")) m.set(l, (m.get(l) ?? 0) + 1); - return m; - }; - const ma = count(a); - const mb = count(b); - let added = 0; - let removed = 0; - for (const k of new Set([...ma.keys(), ...mb.keys()])) { - const d = (mb.get(k) ?? 0) - (ma.get(k) ?? 0); - if (d > 0) added += d; - else if (d < 0) removed += -d; - } - return { added, removed }; -} - -describe("parse", () => { - const doc = parse(SAMPLE); - - it("reads the title and both goals (matched by subject)", () => { - expect(doc.title).toBe("papers audit"); - expect(doc.goals.map((g) => g.subject)).toEqual(["Implement cache layer", "Document the API"]); - }); - - it("reads goal status from the checkbox", () => { - expect(findGoal(doc, "Implement cache layer")?.status).toBe("active"); // [/] - expect(findGoal(doc, "Document the API")?.status).toBe("open"); // [ ] - }); - - it("reads discriminator, subtle failure mode, and verify as separate fields", () => { - const g = findGoal(doc, "Implement cache layer"); - expect(g?.discriminator).toEqual(["hit-rate > 0.8 in load-test.log (a bypass reads ~0)"]); - expect(g?.failure_modes).toEqual(["cache silently bypassed, latency ok by luck"]); - expect(g?.verify).toBe("pytest tests/cache -q"); - }); - - it("reads subtasks with their checkbox state, strikethrough as cancelled", () => { - const g = findGoal(doc, "Implement cache layer"); - expect(g?.subtasks).toEqual([ - { text: "wire cache client", status: "done" }, - { text: "eviction policy", status: "active" }, - { text: "distributed cache, out of scope", status: "cancelled" }, - ]); - }); - - it("reads the evidence block separate from the other lists", () => { - const g = findGoal(doc, "Implement cache layer"); - expect(g?.evidence).toEqual(["> load-test.log: p95=41ms", "> hit-rate 0.93 (not bypassed)"]); - expect(findGoal(doc, "Document the API")?.evidence).toEqual([]); // a goal with no evidence parses to [] - }); - - it("keeps a multi-line evidence item together (quote + interpretation)", () => { - const doc2 = parse( - `# x\n\n## Goals\n\n1. [ ] goal: G\n - discriminator: report has non-zero counts\n - evidence:\n - > report.txt: counts 52 -> 4\n remaining 4 = index + 3 notes\n almost certain the discriminator passes\n - > second item, single line\n`, - ); - expect(findGoal(doc2, "G")?.evidence).toEqual([ - "> report.txt: counts 52 -> 4\nremaining 4 = index + 3 notes\nalmost certain the discriminator passes", - "> second item, single line", - ]); - }); - - it("reads the log verbatim and counts by status", () => { - expect(doc.log).toEqual(["- 2026-06-15 14:02 cache client wired; eviction next"]); - expect(counts(doc)).toEqual({ done: 0, open: 1, active: 1 }); - }); - - it("ignores the Future work section, does not read it as goals or log", () => { - expect(doc.goals).toHaveLength(2); - expect(doc.log).toHaveLength(1); - }); -}); - -describe("the two CompleteGoal writes (minimal diff)", () => { - it("setGoalStatus replaces exactly one line, scoped to the right goal", () => { - const next = setGoalStatus(SAMPLE, "Implement cache layer", "done"); - expect(lineDelta(SAMPLE, next)).toEqual({ added: 1, removed: 1 }); - expect(findGoal(parse(next), "Implement cache layer")?.status).toBe("done"); - expect(findGoal(parse(next), "Document the API")?.status).toBe("open"); // untouched - }); - - it("setGoalStatus keeps the number and goal: prefix, flips only the checkbox", () => { - expect(setGoalStatus(SAMPLE, "Implement cache layer", "done")).toContain("1. [x] goal: Implement cache layer"); - expect(setGoalStatus(SAMPLE, "Document the API", "cancelled")).toContain("2. [-] goal: Document the API"); - }); - - it("setGoalStatus throws on an unknown subject", () => { - expect(() => setGoalStatus(SAMPLE, "no such goal", "done")).toThrow(); - }); - - it("appendLog adds exactly one line under ## Log", () => { - const next = appendLog(SAMPLE, "2026-06-15 15:00 eviction done"); - expect(lineDelta(SAMPLE, next)).toEqual({ added: 1, removed: 0 }); - expect(parse(next).log).toEqual([ - "- 2026-06-15 14:02 cache client wired; eviction next", - "- 2026-06-15 15:00 eviction done", - ]); - }); - - it("appendLog creates the section when absent", () => { - const noLog = "# x\n\n## Goals\n\n1. [ ] goal: y\n - discriminator: z\n"; - expect(parse(appendLog(noLog, "first entry")).log).toEqual(["- first entry"]); - }); -}); - -describe("recordSignOff (CompleteGoal's pure record logic)", () => { - const WHEN = "2026-06-15 16:00"; - - it("accept flips status:done and logs a sign-off line", () => { - const r = recordSignOff(SAMPLE, "Implement cache layer", WHEN, { kind: "accepted" }); - expect(r.isError).toBe(false); - const doc = parse(r.content); - expect(findGoal(doc, "Implement cache layer")?.status).toBe("done"); - expect(doc.log.at(-1)).toBe(`- ${WHEN} signed off "Implement cache layer" (judge accept)`); - }); - - it("accepted_inconclusive still marks done and logs the judge failure", () => { - const r = recordSignOff(SAMPLE, "Implement cache layer", WHEN, { - kind: "accepted_inconclusive", - reason: "judge timed out after 120s", - }); - expect(r.isError).toBe(false); - const doc = parse(r.content); - expect(findGoal(doc, "Implement cache layer")?.status).toBe("done"); - expect(doc.log.at(-1)).toBe(`- ${WHEN} signed off "Implement cache layer" (judge inconclusive: judge timed out after 120s)`); - expect(r.message).toContain("Judge inconclusive: judge timed out after 120s"); - }); - - it("verify_failed only logs a reject line, status stays active", () => { - const r = recordSignOff(SAMPLE, "Implement cache layer", WHEN, { kind: "verify_failed", exitCode: 1, outputTail: "boom" }); - expect(r.isError).toBe(true); - const doc = parse(r.content); - expect(findGoal(doc, "Implement cache layer")?.status).toBe("active"); // NOT marked done - expect(doc.log.at(-1)).toBe(`- ${WHEN} reject "Implement cache layer": verify exit 1`); - }); - - it("rejected logs the (one-lined) missing reason, status stays", () => { - const r = recordSignOff(SAMPLE, "Implement cache layer", WHEN, { kind: "rejected", missing: "no\nsaved\nbench log" }); - expect(r.isError).toBe(true); - expect(findGoal(parse(r.content), "Implement cache layer")?.status).toBe("active"); - expect(parse(r.content).log.at(-1)).toBe(`- ${WHEN} reject "Implement cache layer": no saved bench log`); - }); - - it("unknown goal returns an error and does not touch the file", () => { - const r = recordSignOff(SAMPLE, "nope", WHEN, { kind: "accepted" }); - expect(r.isError).toBe(true); - expect(r.content).toBe(SAMPLE); - }); -}); - -describe("pruneCompleted (drop finished goals, keep the rest)", () => { - // SAMPLE has one active + one open goal; add a done and a cancelled one to prune. - const WITH_FINISHED = `# papers audit - -Context line kept. - -## Goals - -1. [x] goal: Old finished thing - - discriminator: shipped - - tasks: - 1. [x] did it - - evidence: - - > done.log -2. [/] goal: Implement cache layer - - discriminator: hit-rate > 0.8 -3. [-] goal: Abandoned idea - - discriminator: n/a -4. [ ] goal: Document the API - - discriminator: docstrings - -## Log -- 2026-06-15 14:02 note -`; - - it("removes done and cancelled goals, keeps active/open + title + log", () => { - const out = parse(pruneCompleted(WITH_FINISHED)); - expect(out.goals.map((g) => g.subject)).toEqual(["Implement cache layer", "Document the API"]); - expect(out.title).toBe("papers audit"); - expect(out.log.at(-1)).toBe("- 2026-06-15 14:02 note"); - }); - - it("is a no-op when nothing is finished", () => { - expect(pruneCompleted(SAMPLE)).toBe(SAMPLE); - }); -});