mirror of
https://github.com/wassname/pi-goals.git
synced 2026-07-24 13:10:41 +08:00
Compare commits
19
Commits
e0470a0c6d
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
 wassnameandClaudypoo
|
16827de45d | ||
|
wassnameandClaudypoo
|
7d5342f332 | ||
|
wassnameandClaudypoo
|
72ac6cf357 | ||
|
wassnameandClaudypoo
|
924910e942 | ||
|
wassnameandClaudypoo
|
92d3f6d725 | ||
|
wassnameandClaudypoo
|
03e88d34ad | ||
|
wassnameandClaudypoo
|
4b622a22fa | ||
|
wassnameandClaudypoo
|
67daed312f | ||
|
wassnameandClaudypoo
|
8586b26ba8 | ||
|
wassnameandClaudypoo
|
485be236ce | ||
|
wassname
|
632012cc6e | ||
|
wassnameandClaudypoo
|
c0f80b869e | ||
|
wassnameandClaudypoo
|
5d88502e4d | ||
|
wassnameandClaudypoo
|
0dfd6612ce | ||
|
wassname
|
acb04dbfa2 | ||
|
wassnameandClaudypoo
|
de01894348 | ||
|
wassnameandClaudypoo
|
87c6c4d440 | ||
|
wassname
|
14e757cfe1 | ||
|
wassname
|
9a4da96d56 |
@@ -1,26 +1,29 @@
|
||||
# pi-goals
|
||||
|
||||
Plan mode for agreeing on goals before any code gets written. Each goal names the subtle failure mode
|
||||
that could fake a "done" and the discriminator that tells real success from it, plus subtasks and the
|
||||
evidence checked at sign-off. It lives in one markdown file. A widget keeps the goals in front of you
|
||||
through compaction, a reminder nudges the agent to keep the file current, and a goal is signed off
|
||||
only after a read-only subagent checks its evidence.
|
||||
Plan mode for agreeing on goals before any code gets written. Each goal names the subtle failure
|
||||
mode that could fake a "done" and the discriminator that tells real success from it. Everything
|
||||
lives in one markdown file, `.pi/plan.md`, which the agent edits with its normal edit tool and which
|
||||
is injected verbatim every turn so it survives compaction. A goal is signed off only after a fresh
|
||||
read-only judge checks its evidence against the repo.
|
||||
|
||||
Design bet (v2): the plan file is for LLMs and the human, not for TypeScript. There is no parser and
|
||||
no schema. The format is a convention taught by a prompt; the judge, being a model, reads the file
|
||||
natively, finds the claimed goal itself (wording drift is fine), runs the goal's `verify` command
|
||||
itself, and validates the evidence in words. This deleted ~800 lines of v1 (parser, exact-string
|
||||
goal matching, verify runner, JSON-stream judge transport, review menus) and with them the footguns
|
||||
they caused.
|
||||
|
||||
Like [pi-milestones](https://github.com/Neuron-Mr-White/UniPi/tree/main/packages/milestone) and
|
||||
[burneikis/pi-plan](https://github.com/burneikis/pi-plan), it guides rather than guards: a form and a
|
||||
process the agent follows. [pi-lgtm](https://github.com/wassname/pi-lgtm) was my earlier, more complex
|
||||
attempt.
|
||||
[burneikis/pi-plan](https://github.com/burneikis/pi-plan), it guides rather than guards.
|
||||
[pi-lgtm](https://github.com/wassname/pi-lgtm) was my earlier, more complex attempt.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
pi install npm:@wassname2/pi-goals
|
||||
```
|
||||
|
||||
Or run it without installing:
|
||||
Not yet published to npm; run from a checkout:
|
||||
|
||||
```bash
|
||||
pi -e npm:@wassname2/pi-goals
|
||||
git clone https://github.com/wassname/pi-plan && cd pi-plan && npm install
|
||||
pi -e ./src/index.ts
|
||||
```
|
||||
|
||||
## Use
|
||||
@@ -29,67 +32,22 @@ pi -e npm:@wassname2/pi-goals
|
||||
/goals CSV export for the report view
|
||||
```
|
||||
|
||||
`/goals` enters plan mode and starts a conversation; the description is an optional seed, so plain
|
||||
`/goals` works too. From there:
|
||||
`/goals` enters plan mode and starts a conversation; the objective is an optional seed. From there:
|
||||
|
||||
1. Plan. The agent explores read-only, asks about anything unclear, and writes the goals into
|
||||
`.pi/goals.md`.
|
||||
2. Review. You get a menu: Ready, Edit (ask the agent to revise), Open in `$EDITOR`, or Cancel. On
|
||||
Ready you choose whether to keep the current context or start fresh and compacted.
|
||||
3. Work. Each turn the active goal is injected so it survives compaction, and a reminder nudges the
|
||||
agent to keep `goals.md` current and keep going. When a goal's discriminator is satisfied the agent
|
||||
calls `CompleteGoal`, which runs `verify` and a read-only judge, then marks the goal done and logs it.
|
||||
1. Plan. The agent explores read-only (edit/write are blocked except on the plan file itself), asks
|
||||
about anything unclear, and drafts the goals into `.pi/plan.md`.
|
||||
2. Review. Read the file; a two-option prompt asks Ready or keep planning. To revise, just reply.
|
||||
3. Work. Each turn the whole plan file is injected (byte-identical when unchanged, so the KV cache
|
||||
holds), plus a reminder when the file went untouched for a turn. The agent ticks subtasks,
|
||||
appends to `## Log`, fills `evidence:`, and calls `CompleteGoal` when a discriminator is
|
||||
satisfied.
|
||||
|
||||
Other commands: `/goals clear` empties `.pi/goals.md`; `/goals judge <model-ref>` picks a specific
|
||||
model for the sign-off judge (the default is your current model).
|
||||
Other commands: `/goals clear` empties the plan file; `/goals judge <model-ref>` picks a specific
|
||||
model for the sign-off judge (default: your current session model, else pi's default).
|
||||
|
||||
## Example
|
||||
Coming from v1: a leftover `.pi/goals.md` is renamed to `.pi/plan.md` on session start.
|
||||
|
||||
```
|
||||
/goals audit the papers dir metadata and clean up empty dirs
|
||||
```
|
||||
|
||||
The agent explores read-only, drafts the goal with a subtle failure mode and the discriminator that
|
||||
beats it, and stops for review:
|
||||
|
||||
```markdown
|
||||
## Goals
|
||||
|
||||
1. [ ] goal: Audit steering/ metadata and remove empty dirs
|
||||
- subtle failure mode: report written but counts are zero (resolver errored silently)
|
||||
- discriminator: report shows the XXXX count before/after AND a non-zero rename count
|
||||
- tasks:
|
||||
1. [ ] dry-run the metadata resolve
|
||||
2. [ ] remove the empty _artifacts dirs
|
||||
3. [ ] write the report
|
||||
- evidence:
|
||||
- <empty until sign-off>
|
||||
```
|
||||
|
||||
You choose Ready. The agent works the subtasks, fills `evidence` (each item an artifact plus a short
|
||||
read of it), and calls `CompleteGoal`:
|
||||
|
||||
```markdown
|
||||
- evidence:
|
||||
- > scripts/metadata_report.txt: XXXX 52 -> 4, 146 empty _artifacts removed
|
||||
- > 48 files renamed; almost certain done, the silent-resolver failure mode is ruled out
|
||||
```
|
||||
|
||||
A fresh read-only subagent re-checks the evidence against the repo and the discriminator, then
|
||||
returns its verdict and reasoning:
|
||||
|
||||
```
|
||||
Signed off "Audit steering/ metadata and remove empty dirs". Marked done in goals.md.
|
||||
|
||||
--- sign-off judge ---
|
||||
metadata_report.txt present; counts 52 -> 4 confirmed; rename log shows 48 renamed (not zero).
|
||||
VERDICT: accept
|
||||
```
|
||||
|
||||
## The goals.md format
|
||||
|
||||
One project-local file, `<cwd>/.pi/goals.md` (gitignored), holds the title, a context block, the
|
||||
goals, and a short append-only log. A fresh `/goals` draft replaces it.
|
||||
## The plan.md format (a convention, not a schema)
|
||||
|
||||
```markdown
|
||||
# ship the cache layer
|
||||
@@ -110,60 +68,58 @@ Latency target came from the SLO review; keep the existing client API.
|
||||
|
||||
# Future work / out of scope
|
||||
|
||||
- distributed cache
|
||||
|
||||
## Log
|
||||
- 2026-06-15 14:02 cache client wired; eviction next
|
||||
```
|
||||
|
||||
- A goal is a numbered checkbox line beginning `goal:`; the checkbox carries its state (`[ ]` open,
|
||||
`[/]` active, `[x]` done, `[-]` cancelled). Goals are matched by their text, so the number is just
|
||||
for you to reference.
|
||||
- A goal is a checkbox line beginning `goal:` (`[ ]` open, `[/]` active, `[x]` done, `[-]`
|
||||
cancelled). That one line pattern is the only thing the extension itself reads (widget counts,
|
||||
reminder cadence, the Ready prompt); everything else in the file is prose to the extension.
|
||||
- The `discriminator` is the success test, written while planning: the positive observation that the
|
||||
goal succeeded and that none of the `subtle failure mode`s could fake (a count moved, a test
|
||||
exercised the path, a metric beat noise), not just that a failure was avoided. `evidence` is the
|
||||
proof, filled at sign-off: each item pairs a durable artifact (a quoted and linked log, a table, a
|
||||
metric) with a short read of it. `verify`, when present, is the deterministic first stage.
|
||||
- Subtasks are any checkbox without a `goal:` prefix, under `- tasks:`. The agent ticks them, appends
|
||||
to `## Log`, and sets a goal `[/]` when it starts it; only `CompleteGoal` writes `[x]`. Several
|
||||
goals can be active at once.
|
||||
goal succeeded and that none of the `subtle failure mode`s could fake. `evidence` is the proof,
|
||||
filled at sign-off: each item pairs a durable artifact with a short read of it. Prefer committed
|
||||
artifacts (files, tests, diffs); `.pi/` is usually gitignored, so evidence there is judge-time
|
||||
proof only and won't survive in history.
|
||||
- Small format deviations are fine; the file is read by the human and the judge, not a parser.
|
||||
- The agent prunes finished goals itself when the file gets long (evidence survives in git history
|
||||
and `## Log`).
|
||||
|
||||
## Signing off a goal (`CompleteGoal`)
|
||||
|
||||
`CompleteGoal(goal)` (matched by the goal's text) is the only tool that marks a goal done; everything
|
||||
else is the agent editing the file. It reads the goal's `evidence:` block from `.pi/goals.md`, then:
|
||||
`CompleteGoal(goal)` is the one blessed tool. It spawns a strictly read-only `pi` subprocess (`-p
|
||||
--no-session --no-extensions`, tools `read,grep,find,ls` -- no bash, no edit/write) with the whole
|
||||
plan file and the claimed goal. The judge cannot execute anything, so it never re-runs your verify
|
||||
command (which may be a 10-hour training job); the agent runs `verify` itself and saves the output
|
||||
as evidence. The judge reviews evidence discipline in order -- anything here at all? each item
|
||||
quoted and attributed? provenance visible (how was this produced)? do the quotes match the cited
|
||||
files on disk? -- and only then the substance, returning `VERDICT: accept | reject` plus what's
|
||||
missing. It reads the live working tree, not HEAD: uncommitted work counts, and committing before
|
||||
sign-off is for durable evidence, not for the judge's visibility.
|
||||
|
||||
1. If the goal has a `verify:` command, it runs. A non-zero exit rejects right away, no model call.
|
||||
2. Otherwise a read-only `pi` subprocess (a fresh `--no-session` context, so it never sees the working
|
||||
agent's transcript) inspects the `evidence:` against the repo, the `discriminator`, and the
|
||||
`subtle failure mode`. It re-derives from the cited artifacts rather than trusting the claim, so
|
||||
list real artifacts, not assertions.
|
||||
3. On accept, the goal flips to `[x]` and a `## Log` line is written. On reject, it stays open and the
|
||||
agent is told what is missing. Either way the judge's reasoning comes back in the result.
|
||||
|
||||
The judge defaults to your current model (a fresh context, same weights). Point it at another with
|
||||
`/goals judge <provider/model>` for an independent cross-family check.
|
||||
- accept: a sign-off line is appended to `## Log` and the goal is ticked `[x]` in the same write
|
||||
(exact goal-line match only; on wording drift the result asks the agent to tick it). The
|
||||
tool-written log line is the audit trail; a hand-tick without one shows in the diff.
|
||||
- every run saves the judge's full transcript to `.pi/judge/<stamp>.md`, referenced from the log
|
||||
line, so "what did the judge actually check?" stays answerable after the fact.
|
||||
- reject: the goal stays open and the agent gets the missing list.
|
||||
- judge ran but failed/errored/timed out, or returned no VERDICT line: accepted inconclusive,
|
||||
logged as such. There is no pre-emptive "no model" path -- a null judgeModel just omits
|
||||
`--model` so pi's configured default runs the judge, so inconclusive always means "ran but
|
||||
failed", never "couldn't start". The working agent is never blocked on judge infra.
|
||||
|
||||
## Prompts
|
||||
|
||||
All model-facing text lives in [`src/prompts.ts`](src/prompts.ts), in flow order, so you can read the
|
||||
whole process top to bottom.
|
||||
All model-facing text lives in [`src/prompts.ts`](src/prompts.ts), in flow order.
|
||||
|
||||
## Develop
|
||||
|
||||
```bash
|
||||
pi -e ./src/index.ts # load locally
|
||||
npm test # vitest: parser + sign-off record logic
|
||||
npm test # vitest: judge argv invariants, appendLog, decideSignOff fail-forward
|
||||
npm run typecheck
|
||||
npm run lint
|
||||
```
|
||||
|
||||
## Not (yet) included
|
||||
|
||||
- No autonomous re-prompt loop. The reminder nudges the agent within a turn, but the turn still ends
|
||||
and hands back to you; nothing auto-re-prompts until the goals are done.
|
||||
- The plan and execution phases can't yet run on different, sticky models.
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
# CompleteGoal fail-forward on judge failure
|
||||
|
||||
## Goal
|
||||
Make `CompleteGoal` stop rejecting verified goals just because the read-only judge subprocess times out. Keep the judge useful when it works, and make failures explicit in the log/result.
|
||||
|
||||
## Scope
|
||||
In: `CompleteGoal` sign-off behavior, judge transport, tests, docs.
|
||||
Out: broader autonomous loop work, plan-mode UX, model auto-selection.
|
||||
|
||||
## Requirements
|
||||
- R1: If `verify:` fails, the goal is rejected immediately. Done means: existing `verify_failed` behavior remains. VERIFY: unit test for pure sign-off record still passes.
|
||||
- R2: If `verify:` passes and the judge accepts, mark the goal done as before. Done means: log records normal judge accept. VERIFY: unit test for accepted sign-off still passes.
|
||||
- R3: If any `verify:` command passes but the judge times out or subprocess/model transport fails, mark the goal done with an explicit inconclusive-judge log. Goals without `verify:` use the same fail-forward rule once evidence exists. VERIFY: a unit test records accepted status and a log line containing `judge inconclusive`.
|
||||
- R4: Judge transport should parse `pi --mode json` message events instead of raw `-p` terminal output. Done means: code captures final assistant text and provider stop errors distinctly. VERIFY: `npm run typecheck` and tests pass.
|
||||
- R5: The judge should behave like oracle where it matters: explicit model, live streamed progress, and a timeout large enough for a cold reasoning turn. Done means: unset `/goals judge` resolves to the current session model when visible; if no model is visible, no implicit Pi default is used and sign-off is `judge inconclusive`. `message_update` emits throttled progress, and timeout is 600s. VERIFY: `npm run typecheck` and fresh-eyes diff review.
|
||||
|
||||
## Tasks
|
||||
- [x] T1 (R3): Add an accepted-with-warning sign-off outcome.
|
||||
- verify: `npm test`
|
||||
- success: test shows status `[x]` plus `judge inconclusive` in `## Log`
|
||||
- likely_fail: timeout still records `reject`
|
||||
- sneaky_fail: accepted status lands but log hides judge failure
|
||||
- UAT: [test/plan-file.test.ts](/home/wassname/.pi/agent/git/github.com/wassname/pi-plan/test/plan-file.test.ts)
|
||||
- [x] T2 (R4): Switch judge subprocess to JSON-mode parsing.
|
||||
- verify: `npm run typecheck`
|
||||
- success: no TypeScript errors, judge code has no ANSI-terminal parsing dependency
|
||||
- likely_fail: compile errors around streamed event shape
|
||||
- sneaky_fail: model error produces empty output and gets parsed as reject instead of transport failure
|
||||
- UAT: [src/index.ts](/home/wassname/.pi/agent/git/github.com/wassname/pi-plan/src/index.ts)
|
||||
- [x] T3 (docs): Update README sign-off semantics.
|
||||
- verify: `rg "inconclusive|timeout|judge accept" README.md src test`
|
||||
- success: docs name fail-forward behavior
|
||||
- likely_fail: README still says all rejects keep goal open
|
||||
- sneaky_fail: docs imply subagent evidence was accepted when it timed out
|
||||
- UAT: [README.md](/home/wassname/.pi/agent/git/github.com/wassname/pi-plan/README.md)
|
||||
- [x] T4 (R5): Copy oracle's reliability shape for model/progress.
|
||||
- verify: `npm run typecheck`
|
||||
- success: `CompleteGoal` passes the current session model to the judge when no override is set, never spawns without `--model`, and streamed judge deltas are surfaced through `onUpdate`
|
||||
- likely_fail: judge still runs without `--model`
|
||||
- sneaky_fail: user sees no progress for several minutes and kills a working judge
|
||||
- UAT: [src/index.ts](/home/wassname/.pi/agent/git/github.com/wassname/pi-plan/src/index.ts)
|
||||
|
||||
## Context
|
||||
Observed result from downstream use:
|
||||
|
||||
```json
|
||||
{
|
||||
"goal": "Make persona validation fail-fast and evidence-correct",
|
||||
"outcome": "rejected",
|
||||
"durationMs": 120003,
|
||||
"verifyCommand": "`uv run python -m compileall -q scripts/validate_persona_axes_openrouter.py`",
|
||||
"reasoning": "VERDICT: reject\nmissing: judge timed out after 120s",
|
||||
"isError": true
|
||||
}
|
||||
```
|
||||
|
||||
Interpretation: latest surfaced output proves the internal judge timed out. It does not prove the verify command passed, though earlier logs indicated that pattern.
|
||||
|
||||
## Log
|
||||
- 2026-06-29 current `runJudge` uses raw `pi -p --no-session` output plus ANSI stripping; oracle uses `--mode json` and parses message events, which is likely more reliable.
|
||||
- 2026-06-29 unset `/goals judge` spawns the judge without `--model`, so Pi resolves its configured default model; do not describe this as the current session model.
|
||||
- 2026-06-29 timeout/transport failure now maps to `accepted_inconclusive`, preserving partial output in reasoning when available.
|
||||
- 2026-06-29 fresh-eyes review found loose `/accept/i` verdict parsing and caller-abort fail-forward risk; fixed exact verdict parsing and made caller abort reject.
|
||||
- 2026-06-29 oracle comparison suggests the important reliability pieces are explicit model selection, JSON streaming, live partial output, and no short wrapper timeout; updated CompleteGoal to use the current session model when visible, never spawn without `--model`, stream throttled progress, and wait 600s.
|
||||
|
||||
## TODO
|
||||
- Consider making `CompleteGoal` expose `verifyExitCode: 0` and `judgeOutcome` separately in details.
|
||||
|
||||
## Errors
|
||||
| Task | Error | Resolution |
|
||||
|------|-------|------------|
|
||||
Generated
+3385
File diff suppressed because it is too large
Load Diff
Executable
+43
@@ -0,0 +1,43 @@
|
||||
#!/usr/bin/env bash
|
||||
# Replaces the stale FIXME(side-effect) claim in src/index.ts with a checked fact.
|
||||
#
|
||||
# The claim was: "pi -p --no-session clones the repo into the PARENT of cwd, leaving a stale
|
||||
# directory." Reproducing the exact sign-off judge invocation (pi --mode json -p --no-session,
|
||||
# read-only tools, edit/write excluded, cwd = here) shows it does not. This script makes that
|
||||
# reproducible: it runs the invocation, requires pi to actually reach agent_end (so a pass is not
|
||||
# vacuous), and asserts the parent-of-cwd listing is byte-identical before and after.
|
||||
#
|
||||
# Exit 0 = judge leaves no clone in the parent. Exit 1 = either pi did not run, or it polluted.
|
||||
# Run by hand; re-run as the rigorous sign-off check (the judge has bash and runs this itself).
|
||||
set -u
|
||||
|
||||
PARENT="$(cd "$PWD/.." && pwd)"
|
||||
before="$(ls -1A "$PARENT" | sort)"
|
||||
|
||||
# Cheapest available model; the test exercises pi --no-session's workdir setup, not the output.
|
||||
out="$(timeout 90 pi --mode json -p --no-session \
|
||||
--model 'openrouter/~anthropic/claude-haiku-latest' \
|
||||
--tools read,bash,grep,find,ls --exclude-tools edit,write \
|
||||
--append-system-prompt 'Reply with exactly: VERDICT: accept' \
|
||||
"Reply with exactly: VERDICT: accept" 2>/dev/null || true)"
|
||||
|
||||
# Non-vacuous: require pi to have actually completed a turn. A pass without this could mean pi
|
||||
# crashed instantly and never had the chance to clone -- which would prove nothing.
|
||||
if ! printf '%s' "$out" | grep -q '"type":"agent_end"'; then
|
||||
echo "FAIL: pi --no-session did not reach agent_end; cannot confirm no-clone."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
after="$(ls -1A "$PARENT" | sort)"
|
||||
|
||||
echo "parent: $PARENT"
|
||||
echo "--- before ---"; echo "$before"
|
||||
echo "--- after ---"; echo "$after"
|
||||
|
||||
if [ "$before" == "$after" ]; then
|
||||
echo "PASS: parent-of-cwd listing identical before/after; no clone created."
|
||||
exit 0
|
||||
fi
|
||||
echo "FAIL: parent-of-cwd listing changed. Diff (< before, > after):"
|
||||
diff <(printf '%s\n' "$before") <(printf '%s\n' "$after") | head -20
|
||||
exit 1
|
||||
Executable
+14
@@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
# Structural gate for the goal's `verify:` field. Cheap and deterministic: no API calls.
|
||||
# Confirms (a) neither stale FIXME tag remains in src/, (b) the footprint script exists, and
|
||||
# (c) the plan-injection heading prefix is still emitted. The rigorous runtime check (running
|
||||
# the footprint script) is the sign-off judge's job -- it has bash and re-runs the script itself.
|
||||
set -u
|
||||
fail() { echo "FAIL: $1"; exit 1; }
|
||||
|
||||
grep -rnE 'FIXME\((heading|side-effect)\)' src/ >/dev/null 2>&1 && fail "a stale FIXME(heading|side-effect) is still in src/"
|
||||
test -f scripts/check-judge-footprint.sh || fail "scripts/check-judge-footprint.sh is missing"
|
||||
grep -q '\.pi/goals\.md:' src/prompts.ts || fail "the .pi/goals.md: heading prefix was dropped from src/prompts.ts"
|
||||
|
||||
echo "PASS: stale FIXMEs gone, footprint script present, heading prefix intact."
|
||||
exit 0
|
||||
@@ -0,0 +1,104 @@
|
||||
diff --git a/README.md b/README.md
|
||||
index ce4056a..5485002 100644
|
||||
--- a/README.md
|
||||
+++ b/README.md
|
||||
@@ -145,9 +145,9 @@ else is the agent editing the file. It reads the goal's `evidence:` block from `
|
||||
reasoning comes back in the result.
|
||||
|
||||
The judge defaults to the current session model and streams partial output while it runs. If the
|
||||
-current model is not visible to the extension, `CompleteGoal` does not fall back to Pi's implicit
|
||||
-default; it signs off as `judge inconclusive` and tells you to set `/goals judge <provider/model>`.
|
||||
-Point it at another model for an independent cross-family check.
|
||||
+session model is not visible to the extension, the `--model` flag is omitted and pi uses its own
|
||||
+configured default, so the judge always runs. `/goals judge <provider/model>` is an optional override
|
||||
+for an independent cross-family check; never required.
|
||||
|
||||
## Prompts
|
||||
|
||||
diff --git a/src/index.ts b/src/index.ts
|
||||
index 9eb2a16..14784f7 100644
|
||||
--- a/src/index.ts
|
||||
+++ b/src/index.ts
|
||||
@@ -326,7 +326,7 @@ export default function piGoalsExtension(pi: ExtensionAPI): void {
|
||||
durationMs,
|
||||
verifyCommand: goal.verify ?? undefined,
|
||||
verifyExitCode: outcome.kind === "verify_failed" ? outcome.exitCode : undefined,
|
||||
- judgeModel: judgeModel ?? "no explicit judge model",
|
||||
+ judgeModel: judgeModel ?? "pi default",
|
||||
reasoning,
|
||||
isError: res.isError,
|
||||
};
|
||||
@@ -522,14 +522,6 @@ async function decideSignOff(
|
||||
};
|
||||
}
|
||||
}
|
||||
- if (!judgeModel) {
|
||||
- const reason = "no explicit judge model available; set /goals judge <provider/model>";
|
||||
- return {
|
||||
- outcome: { kind: "accepted_inconclusive", reason },
|
||||
- reasoning: `VERDICT: inconclusive\nreason: ${reason}`,
|
||||
- durationMs: Date.now() - startedAt,
|
||||
- };
|
||||
- }
|
||||
const verdict = await runJudge(goal, evidence, paths, verifyResult, judgeModel, cwd, signal, onUpdate);
|
||||
const outcome: SignOff =
|
||||
verdict.kind === "accepted"
|
||||
@@ -573,13 +565,25 @@ type JudgeResult =
|
||||
| { kind: "rejected"; missing: string; reasoning: string; durationMs: number }
|
||||
| { kind: "inconclusive"; reason: string; reasoning: string; durationMs: number };
|
||||
|
||||
+/** Stage 2: a read-only pi subprocess inspects the evidence against the repo and returns a verdict. */
|
||||
+/** Build the pi argv for the read-only judge. `--model` is omitted when no explicit/session model is
|
||||
+ * set, so pi falls back to its configured default -- the judge always runs, never pre-emptively
|
||||
+ * fails as "no model". Exported for a unit test that locks this invariant (an empty `--model ""`
|
||||
+ * would make every sign-off silently inconclusive). */
|
||||
+export function buildJudgeArgs(judgeModel: string | null): string[] {
|
||||
+ const args = ["--mode", "json", "-p", "--no-session"];
|
||||
+ if (judgeModel) args.push("--model", judgeModel);
|
||||
+ args.push("--tools", JUDGE_TOOLS.join(","), "--exclude-tools", JUDGE_BLOCKED_TOOLS.join(","), "--append-system-prompt", evidenceJudgeSystem);
|
||||
+ return args;
|
||||
+}
|
||||
+
|
||||
/** Stage 2: a read-only pi subprocess inspects the evidence against the repo and returns a verdict. */
|
||||
async function runJudge(
|
||||
goal: Goal,
|
||||
evidence: string,
|
||||
paths: string[],
|
||||
verifyResult: { command: string; exitCode: number; outputTail: string } | null,
|
||||
- judgeModel: string,
|
||||
+ judgeModel: string | null,
|
||||
cwd: string,
|
||||
signal: AbortSignal | undefined,
|
||||
onUpdate?: (partial: { content: Array<{ type: "text"; text: string }>; details: SignOffDetails }) => void,
|
||||
@@ -600,14 +604,14 @@ async function runJudge(
|
||||
evidence,
|
||||
paths,
|
||||
});
|
||||
- const args = ["--mode", "json", "-p", "--no-session", "--model", judgeModel, "--tools", JUDGE_TOOLS.join(","), "--exclude-tools", JUDGE_BLOCKED_TOOLS.join(","), "--append-system-prompt", evidenceJudgeSystem];
|
||||
+ const args = buildJudgeArgs(judgeModel);
|
||||
args.push(task);
|
||||
|
||||
emit("spawning", `Spawning read-only judge for: ${goal.subject}`);
|
||||
const inv = getPiInvocation(args);
|
||||
- // FIXME(side-effect): pi -p --no-session clones the repo into the PARENT of cwd (so alongside
|
||||
- // the working dir), leaving a stale directory. The judge should run in a temp dir or inside the
|
||||
- // existing repo checkout so it doesn't pollute the user's workspace.
|
||||
+ // The judge runs in-place against this checkout (cwd is passed to spawn and the read-only tools
|
||||
+ // read from it); pi --no-session does not clone into the parent. Proven and re-checked by
|
||||
+ // scripts/check-judge-footprint.sh, which reproduces this invocation and asserts no parent clone.
|
||||
const judge = await new Promise<{ output: string; error?: string; aborted?: boolean }>((resolve) => {
|
||||
let settled = false;
|
||||
let stdoutBuffer = "";
|
||||
diff --git a/src/prompts.ts b/src/prompts.ts
|
||||
index 03faea8..3b8d270 100644
|
||||
--- a/src/prompts.ts
|
||||
+++ b/src/prompts.ts
|
||||
@@ -117,8 +117,6 @@ export function planInjection(p: {
|
||||
counts: { done: number; open: number };
|
||||
}): string {
|
||||
if (!p.activeGoal) {
|
||||
- // FIXME(heading): user wants the heading to show ".pi/goals.md: <title>" so the filename is explicit
|
||||
- // even in the injection. Currently says "Goals (goals.md):" which is close but not the same.
|
||||
return `.pi/goals.md: ${p.title}\nNo active goal. ${p.counts.open} open, ${p.counts.done} done. Pick the next goal (set its checkbox to [/]) or run /goals.`;
|
||||
}
|
||||
const subtasks = p.activeGoal.openSubtasks.length
|
||||
@@ -0,0 +1,30 @@
|
||||
diff --git a/src/index.ts b/src/index.ts
|
||||
index 9eb2a16..cdd7b45 100644
|
||||
--- a/src/index.ts
|
||||
+++ b/src/index.ts
|
||||
@@ -605,9 +605,9 @@ async function runJudge(
|
||||
|
||||
emit("spawning", `Spawning read-only judge for: ${goal.subject}`);
|
||||
const inv = getPiInvocation(args);
|
||||
- // FIXME(side-effect): pi -p --no-session clones the repo into the PARENT of cwd (so alongside
|
||||
- // the working dir), leaving a stale directory. The judge should run in a temp dir or inside the
|
||||
- // existing repo checkout so it doesn't pollute the user's workspace.
|
||||
+ // The judge runs in-place against this checkout (cwd is passed to spawn and the read-only tools
|
||||
+ // read from it); pi --no-session does not clone into the parent. Proven and re-checked by
|
||||
+ // scripts/check-judge-footprint.sh, which reproduces this invocation and asserts no parent clone.
|
||||
const judge = await new Promise<{ output: string; error?: string; aborted?: boolean }>((resolve) => {
|
||||
let settled = false;
|
||||
let stdoutBuffer = "";
|
||||
diff --git a/src/prompts.ts b/src/prompts.ts
|
||||
index 03faea8..3b8d270 100644
|
||||
--- a/src/prompts.ts
|
||||
+++ b/src/prompts.ts
|
||||
@@ -117,8 +117,6 @@ export function planInjection(p: {
|
||||
counts: { done: number; open: number };
|
||||
}): string {
|
||||
if (!p.activeGoal) {
|
||||
- // FIXME(heading): user wants the heading to show ".pi/goals.md: <title>" so the filename is explicit
|
||||
- // even in the injection. Currently says "Goals (goals.md):" which is close but not the same.
|
||||
return `.pi/goals.md: ${p.title}\nNo active goal. ${p.counts.open} open, ${p.counts.done} done. Pick the next goal (set its checkbox to [/]) or run /goals.`;
|
||||
}
|
||||
const subtasks = p.activeGoal.openSubtasks.length
|
||||
+323
-457
@@ -1,106 +1,83 @@
|
||||
/**
|
||||
* pi-goals — plan mode that sets up goals with evidence, tracked in one .pi/goals.md, signed off by a
|
||||
* read-only subagent check. A successor to pi-lgtm, kept deliberately small (≈ burneikis/pi-plan
|
||||
* plus the additions: goals + a discriminator + a subtle failure mode + subtasks, a sign-off check,
|
||||
* a widget, a reminder). A goal's success test is its discriminator: the observation that tells real
|
||||
* success from the named failure mode.
|
||||
* pi-goals v2 — plan mode drafts goals into one .pi/plan.md, the agent works them with its normal
|
||||
* Edit tool, and a fresh read-only judge signs each goal off through the one blessed tool,
|
||||
* CompleteGoal.
|
||||
*
|
||||
* Philosophy (spec D3): the form guides, it does not gate. The agent edits goals.md with its normal
|
||||
* Edit tool. The one blessed tool is CompleteGoal, which runs the sign-off check and records it. The
|
||||
* reminder + the injected plan + git/widget visibility carry the process; we trust the agent's
|
||||
* judgement rather than guarding it.
|
||||
* The v1 lesson: the parser existed so TypeScript could read plan.md, but almost every reader is a
|
||||
* model. So v2 has NO parser and no schema. The harness does exactly three things for a
|
||||
* cooperative-but-confused model:
|
||||
* 1. memory — inject plan.md verbatim every turn (survives compaction; byte-identical when
|
||||
* unchanged so the KV cache holds; stale copies stripped by the context hook)
|
||||
* 2. format — a skeleton convention taught in planDrafting (prompts.ts), not validated
|
||||
* 3. eyes — CompleteGoal spawns a strictly read-only pi subprocess (--no-session, no bash)
|
||||
* that gets the whole plan file plus the claimed goal, finds the goal itself
|
||||
* (tolerates wording drift), checks the evidence (including the agent's saved
|
||||
* verify output) against the repo, and returns VERDICT: accept|reject
|
||||
*
|
||||
* Flow:
|
||||
* /goals [objective] -> plan mode (conversational): objective is an optional seed; agent explores
|
||||
* read-only, asks, then drafts goals into .pi/goals.md (planDrafting guides)
|
||||
* agent_end -> review menu (Ready / Edit / $EDITOR / Cancel); Ready offers compaction
|
||||
* execution -> each turn, inject the plan summary (survives compaction) + a reminder;
|
||||
* agent works goals, ticks subtasks, appends ## Log, calls CompleteGoal
|
||||
* CompleteGoal -> optional deterministic verify, then a read-only oracle judge -> accept
|
||||
* flips status:done + logs; reject returns what's missing
|
||||
* The judge subsumes what v1 did in code: goal matching (no findGoal), evidence validation (a
|
||||
* placeholder gets rejected in words), and format reading. The extension's only
|
||||
* writes are the sign-off: append a log line to ## Log (the audit trail) and tick the goal [x] when
|
||||
* an exact goal line matches (on drift the agent ticks, and the result says so). A hand-tick
|
||||
* without a matching tool-written log line is visible in the diff either way.
|
||||
*
|
||||
* The plan file lives at <cwd>/.pi/goals.md (project-local, gitignored, like pi-tasks), not in the
|
||||
* repo. A fresh /goals draft just replaces it (the "overwrite" staleness rule).
|
||||
*
|
||||
* Plan mode is read-only: the tool_call hook blocks edit/write (except goals.md itself) and mutating
|
||||
* bash while drafting, so code isn't written before the goals are agreed. Read-only bash exploration
|
||||
* stays open (blocklist, not allowlist).
|
||||
*
|
||||
* Not built (FIXME): no plan-vs-exec model switch on accept (plan-model stickiness); noted at its
|
||||
* call site below.
|
||||
* Judge ran but failed/errored/timed out, or returned no VERDICT line => accepted_inconclusive: the
|
||||
* working agent is never blocked on judge infra; the log line says the judge ran but failed. There
|
||||
* is no pre-emptive "no model" path -- a null judgeModel just omits --model so pi's configured
|
||||
* default runs the judge, so inconclusive always means "ran but failed", never "couldn't start".
|
||||
*
|
||||
* All model-facing text lives in prompts.ts, in flow order.
|
||||
*/
|
||||
|
||||
import { spawn, spawnSync } from "node:child_process";
|
||||
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { spawn } from "node:child_process";
|
||||
import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs";
|
||||
import { basename, join, resolve } from "node:path";
|
||||
import type { ExtensionAPI, ExtensionCommandContext, ExtensionContext } from "@earendil-works/pi-coding-agent";
|
||||
import { getMarkdownTheme } from "@earendil-works/pi-coding-agent";
|
||||
import { Container, Markdown, Spacer, Text } from "@earendil-works/pi-tui";
|
||||
import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent";
|
||||
import { Type } from "@sinclair/typebox";
|
||||
import { counts, findGoal, type Goal, type PlanDoc, parse, recordSignOff, type SignOff } from "./plan-file.js";
|
||||
import {
|
||||
completeGoalDescription,
|
||||
completeGoalParamDescription,
|
||||
evidenceJudgeSystem,
|
||||
evidenceJudgeUser,
|
||||
planDrafting,
|
||||
planInjection,
|
||||
reminder,
|
||||
} from "./prompts.js";
|
||||
import { completeGoalDescription, completeGoalParamDescription, judgeSystem, judgeUser, planDrafting, reminder } from "./prompts.js";
|
||||
|
||||
const STATE = "pi-goals-state";
|
||||
const PLAN_CONTEXT = "pi-goals-context"; // injected plan-mode guidance, stripped from history later
|
||||
const PLAN_CONTEXT = "pi-goals-context"; // injected plan/guidance, stale copies stripped by the context hook
|
||||
const STATUS_KEY = "pi-goals";
|
||||
const WIDGET_KEY = "pi-goals-widget";
|
||||
// Tools the sign-off judge gets: read-only inspection + bash (for git log, cat, running scripts to
|
||||
// inspect). File mutators (edit, write) are blocked so the judge cannot modify anything.
|
||||
// Names match pi's internal tool registry (grep→ffgrep, find→fffind, etc.).
|
||||
const JUDGE_TOOLS = ["read", "bash", "grep", "find", "ls"];
|
||||
const PLAN_REL = ".pi/plan.md";
|
||||
// Judge toolset: strictly read-only, NO bash -- the judge can never execute or mutate anything, and
|
||||
// in particular never re-runs a verify command (which may be a 10-hour training job). The agent runs
|
||||
// verify itself and saves the output as evidence; the judge reads it. Names match pi's tool registry.
|
||||
const JUDGE_TOOLS = ["read", "grep", "find", "ls"];
|
||||
const JUDGE_BLOCKED_TOOLS = ["edit", "write"];
|
||||
// File mutators blocked while drafting goals (read-only plan mode, like narumiruna/pi-plan-mode), so
|
||||
// code isn't written before goals are agreed. The one allowed write is goals.md itself (the
|
||||
// deliverable). A read-only task (a pure search) can still be explored in plan mode by nature.
|
||||
const JUDGE_TIMEOUT_MS = 600_000;
|
||||
// Plan mode is read-only by convention AND a light gate: edit/write are blocked (except plan.md,
|
||||
// the deliverable). bash stays open — the prompt says don't mutate; guide, don't gate (spec D3).
|
||||
const PLAN_MODE_BLOCKED_TOOLS = ["edit", "write"];
|
||||
// bash is dual-use, so block it only when the command looks mutating; read-only exploration (cat, rg,
|
||||
// git log, running a script to inspect) stays open. Blocklist, not allowlist: keep exploration
|
||||
// frictionless and just stop the obvious mutators. List adapted from narumiruna/pi-plan-mode; the
|
||||
// redirect rule catches `> file` / `>> file` / `>| file` but not fd-dups like `2>&1` or `>&2`.
|
||||
const MUTATING_BASH_PATTERNS: RegExp[] = [
|
||||
/\b(rm|rmdir|mv|cp|mkdir|touch|chmod|chown|chgrp|ln|tee|truncate|dd)\b/i,
|
||||
/>\s*[^&\s]/, // redirect to a file (write/append/clobber), excludes 2>&1 and >&2
|
||||
/\bnpm\s+(install|uninstall|update|ci|link|publish|version)\b/i,
|
||||
/\byarn\s+(add|remove|install|publish|upgrade)\b/i,
|
||||
/\bpnpm\s+(add|remove|install|publish|update)\b/i,
|
||||
/\bbun\s+(add|remove|install|update|publish)\b/i,
|
||||
/\bpip\s+(install|uninstall)\b/i,
|
||||
/\buv\s+(add|remove|sync|lock|pip\s+install)\b/i,
|
||||
/\bgit\s+(add|commit|push|pull|merge|rebase|reset|checkout|switch|stash|cherry-pick|revert|tag|init|clone)\b/i,
|
||||
/\b(sudo|su|kill|pkill|killall|reboot|shutdown)\b/i,
|
||||
/\bsystemctl\s+(start|stop|restart|enable|disable)\b/i,
|
||||
/\b(vim?|nano|emacs|code|subl)\b/i,
|
||||
];
|
||||
const PLAN_REL = ".pi/goals.md"; // project-local, gitignored (pi-tasks convention); shown in the widget
|
||||
|
||||
// The one regex in the whole extension: a checkbox line beginning "goal:", for the widget and the
|
||||
// "any goals open?" reminder condition. Everything else reads the file as prose.
|
||||
const GOAL_LINE = /^\s*(?:\d+\.|[-*])\s*\[([ xX/-])\]\s*goal:\s*(.*)$/i;
|
||||
type GoalStatus = "open" | "active" | "done" | "cancelled";
|
||||
const CHAR_TO_STATUS: Record<string, GoalStatus> = { " ": "open", "/": "active", x: "done", "-": "cancelled" };
|
||||
|
||||
function scanGoals(plan: string): Array<{ status: GoalStatus; subject: string }> {
|
||||
const goals: Array<{ status: GoalStatus; subject: string }> = [];
|
||||
for (const line of plan.split("\n")) {
|
||||
const m = GOAL_LINE.exec(line);
|
||||
if (m) goals.push({ status: CHAR_TO_STATUS[m[1].toLowerCase()] ?? "open", subject: m[2].trim() });
|
||||
}
|
||||
return goals;
|
||||
}
|
||||
|
||||
interface PlanState {
|
||||
isPlanMode: boolean;
|
||||
objective: string | null;
|
||||
/** Optional model ref for the sign-off judge; unset => the subprocess uses pi's default model. */
|
||||
/** Optional model ref for the sign-off judge; unset => current session model, else pi's default. */
|
||||
judgeModel: string | null;
|
||||
}
|
||||
|
||||
export default function piGoalsExtension(pi: ExtensionAPI): void {
|
||||
let state: PlanState = { isPlanMode: false, objective: null, judgeModel: null };
|
||||
// Reminder cadence: fire when an active goal exists but goals.md was not touched since last turn.
|
||||
let state: PlanState = { isPlanMode: false, judgeModel: null };
|
||||
// Reminder cadence: fire when goals are open but plan.md was untouched since the last turn.
|
||||
let lastInjectedPlan = "";
|
||||
// newSession is only on the command-handler context; agent_end's ctx lacks it. Save it from /goals.
|
||||
let savedCmdCtx: ExtensionCommandContext | null = null;
|
||||
|
||||
const planPath = (ctx: ExtensionContext) => join(ctx.cwd, ".pi", "goals.md");
|
||||
const planPath = (ctx: ExtensionContext) => join(ctx.cwd, ".pi", "plan.md");
|
||||
const readPlan = (ctx: ExtensionContext): string => (existsSync(planPath(ctx)) ? readFileSync(planPath(ctx), "utf-8") : "");
|
||||
// Our programmatic writes (clear, CompleteGoal). The agent creates/edits the file with its own Edit
|
||||
// tool; this just makes sure .pi/ exists for our writes.
|
||||
const writePlan = (ctx: ExtensionContext, content: string): void => {
|
||||
mkdirSync(join(ctx.cwd, ".pi"), { recursive: true });
|
||||
writeFileSync(planPath(ctx), content);
|
||||
@@ -113,309 +90,114 @@ export default function piGoalsExtension(pi: ExtensionAPI): void {
|
||||
function updateWidget(ctx: ExtensionContext): void {
|
||||
if (state.isPlanMode) {
|
||||
ctx.ui.setStatus(STATUS_KEY, ctx.ui.theme.fg("warning", "planning"));
|
||||
ctx.ui.setWidget(WIDGET_KEY, ["pi-goals: drafting goals", `Write goals to ${PLAN_REL}, then review.`]);
|
||||
ctx.ui.setWidget(WIDGET_KEY, [`pi-goals: drafting goals in ${PLAN_REL}`]);
|
||||
return;
|
||||
}
|
||||
const doc = parse(readPlan(ctx));
|
||||
if (doc.goals.length === 0) {
|
||||
const goals = scanGoals(readPlan(ctx));
|
||||
if (goals.length === 0) {
|
||||
ctx.ui.setStatus(STATUS_KEY, undefined);
|
||||
ctx.ui.setWidget(WIDGET_KEY, undefined);
|
||||
return;
|
||||
}
|
||||
const c = counts(doc);
|
||||
ctx.ui.setStatus(STATUS_KEY, ctx.ui.theme.fg("accent", `◷ ${c.done}/${doc.goals.length} goals`));
|
||||
ctx.ui.setWidget(WIDGET_KEY, [...goalWidgetLines(doc), ctx.ui.theme.fg("muted", PLAN_REL)]);
|
||||
const done = goals.filter((g) => g.status === "done").length;
|
||||
ctx.ui.setStatus(STATUS_KEY, ctx.ui.theme.fg("accent", `◷ ${done}/${goals.length} goals`));
|
||||
const mark: Record<GoalStatus, string> = { done: "✔", active: "▸", open: "◻", cancelled: "✗" };
|
||||
// Only live goals get lines so finished work never pushes current work off screen.
|
||||
const live = goals.filter((g) => g.status === "active" || g.status === "open");
|
||||
ctx.ui.setWidget(WIDGET_KEY, [ctx.ui.theme.fg("muted", PLAN_REL), ...live.map((g) => `${mark[g.status]} ${g.subject}`)]);
|
||||
}
|
||||
|
||||
function goalWidgetLines(doc: PlanDoc): string[] {
|
||||
const mark: Record<Goal["status"], string> = { done: "✔", active: "▸", open: "◻", cancelled: "✗" };
|
||||
const lines = [`Goals: ${doc.title || "(untitled)"}`];
|
||||
for (const g of doc.goals) {
|
||||
// Show every goal with its status glyph (✔ done, ▸ active, ◻ open, ✗ cancelled) so finished
|
||||
// goals read as checked off rather than vanishing. Plans are small, so this stays readable.
|
||||
const total = g.subtasks.length;
|
||||
const done = g.subtasks.filter((s) => s.status === "done").length;
|
||||
lines.push(`${mark[g.status]} ${g.subject}${total ? ` (${done}/${total} tasks)` : ""}`);
|
||||
}
|
||||
return lines;
|
||||
}
|
||||
|
||||
// --- plan mode: setup -------------------------------------------------------------------------
|
||||
// --- /goals: enter plan mode (or clear / set judge) --------------------------------------------
|
||||
|
||||
pi.registerCommand("goals", {
|
||||
description: "Plan mode: set up goals (with evidence) in goals.md, then work them. /goals <objective>",
|
||||
description: `Plan mode: draft goals into ${PLAN_REL}, review, then work them. /goals <objective> | /goals clear | /goals judge <model>`,
|
||||
handler: async (args, ctx) => {
|
||||
savedCmdCtx = ctx; // ctx here is an ExtensionCommandContext (has newSession); keep it for later
|
||||
const arg = args.trim();
|
||||
if (arg === "clear") {
|
||||
await clearPlan(ctx);
|
||||
writePlan(ctx, "");
|
||||
state = { ...state, isPlanMode: false };
|
||||
persist();
|
||||
updateWidget(ctx);
|
||||
ctx.ui.notify(`Cleared ${PLAN_REL}.`, "info");
|
||||
return;
|
||||
}
|
||||
if (arg.startsWith("judge")) {
|
||||
setJudge(arg.slice("judge".length).trim(), ctx);
|
||||
const ref = arg.slice("judge".length).trim();
|
||||
state = { ...state, judgeModel: ref || null };
|
||||
persist();
|
||||
ctx.ui.notify(ref ? `Sign-off judge model set to ${ref}` : "Sign-off judge reset to the session model", "info");
|
||||
return;
|
||||
}
|
||||
// Conversational entry (like narumiruna/pi-plan-mode): /goals enters plan mode and starts a
|
||||
// dialogue. The objective is an optional seed, not a required arg, so there's no awkward
|
||||
// "type your objective" prompt; the agent explores read-only and asks before drafting. A
|
||||
// fresh draft just replaces .pi/goals.md (the "overwrite" staleness rule).
|
||||
const objective = arg || null;
|
||||
state = { ...state, isPlanMode: true, objective };
|
||||
state = { ...state, isPlanMode: true };
|
||||
persist();
|
||||
updateWidget(ctx);
|
||||
const seed = objective
|
||||
? `We're in plan mode. Objective: ${objective}\n\nExplore the repo read-only and ask me anything unclear. When the objective is nailed down, draft (or replace) the goals in ${planPath(ctx)}, then stop for review.`
|
||||
: `We're in plan mode. Tell me what you want to plan. Explore read-only and ask questions as needed; when the objective is clear, draft the goals in ${planPath(ctx)} and stop for review.`;
|
||||
const seed = arg
|
||||
? `We're in plan mode. Objective: ${arg}\n\nExplore the repo read-only and ask me anything unclear. When the objective is nailed down, draft (or replace) the plan in ${planPath(ctx)}, then stop for review.`
|
||||
: `We're in plan mode. Tell me what you want to plan. Explore read-only and ask questions as needed; when the objective is clear, draft the plan in ${planPath(ctx)} and stop for review.`;
|
||||
pi.sendUserMessage(seed, { deliverAs: "followUp" });
|
||||
},
|
||||
});
|
||||
|
||||
function setJudge(ref: string, ctx: ExtensionContext): void {
|
||||
state = { ...state, judgeModel: ref || null };
|
||||
persist();
|
||||
ctx.ui.notify(ref ? `Sign-off judge model set to ${ref}` : "Sign-off judge reset to the default model", "info");
|
||||
}
|
||||
|
||||
async function clearPlan(ctx: ExtensionContext): Promise<void> {
|
||||
if (!existsSync(planPath(ctx))) {
|
||||
ctx.ui.notify("No goals.md to clear.", "info");
|
||||
return;
|
||||
}
|
||||
if (ctx.hasUI) {
|
||||
const ok = await ctx.ui.select(`Clear ${PLAN_REL}?`, ["Cancel", "Clear goals.md"]);
|
||||
if (ok !== "Clear goals.md") return;
|
||||
}
|
||||
writePlan(ctx, "");
|
||||
state = { ...state, isPlanMode: false, objective: null };
|
||||
persist();
|
||||
updateWidget(ctx);
|
||||
ctx.ui.notify(`Cleared ${PLAN_REL}.`, "info");
|
||||
}
|
||||
|
||||
// --- review loop (after the agent drafts the plan) --------------------------------------------
|
||||
|
||||
async function reviewLoop(ctx: ExtensionContext): Promise<void> {
|
||||
while (true) {
|
||||
const doc = parse(readPlan(ctx));
|
||||
const choice = await ctx.ui.select(`Goals: ${doc.goals.length} goal(s). What next?`, [
|
||||
"Ready — start working the plan",
|
||||
"Edit — ask the agent to revise",
|
||||
"Open in $EDITOR",
|
||||
"Cancel — leave plan mode",
|
||||
]);
|
||||
if (!choice || choice.startsWith("Cancel")) {
|
||||
exitPlanMode(ctx);
|
||||
ctx.ui.notify("Left plan mode. goals.md kept.", "info");
|
||||
return;
|
||||
}
|
||||
if (choice.startsWith("Ready")) return startExecution(ctx);
|
||||
if (choice.startsWith("Edit")) {
|
||||
const changes = await ctx.ui.editor("What should change about the plan?", "");
|
||||
if (changes?.trim()) {
|
||||
pi.sendUserMessage(`Revise the plan at ${planPath(ctx)} with these changes, same format:\n\n${changes.trim()}`, { deliverAs: "followUp" });
|
||||
return; // agent_end re-opens the review loop
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (choice.startsWith("Open")) {
|
||||
const editor = process.env.EDITOR || process.env.VISUAL || "vi";
|
||||
spawnSync(editor, [planPath(ctx)], { stdio: "inherit" });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function exitPlanMode(ctx: ExtensionContext): void {
|
||||
state = { ...state, isPlanMode: false };
|
||||
persist();
|
||||
updateWidget(ctx);
|
||||
}
|
||||
|
||||
async function startExecution(ctx: ExtensionContext): Promise<void> {
|
||||
// FIXME(model-switch): the plan phase should be able to run on a sticky plan model and execution
|
||||
// on a different one (see README "Not yet included"). newSession can't switch the model yet; wire
|
||||
// this when pi exposes a model override on newSession.
|
||||
// Offer a clean execution context (D13). newSession lives only on the saved command context.
|
||||
let fresh = false;
|
||||
if (ctx.hasUI && savedCmdCtx) {
|
||||
const choice = await ctx.ui.select("Start working the plan in...", [
|
||||
"This context (keep history)",
|
||||
"A fresh, compacted context",
|
||||
]);
|
||||
fresh = choice?.startsWith("A fresh") ?? false;
|
||||
}
|
||||
const doc = parse(readPlan(ctx));
|
||||
const planFile = planPath(ctx);
|
||||
const planContent = readPlan(ctx); // captured now: ctx is stale after newSession below
|
||||
const parentSession = ctx.sessionManager.getSessionFile();
|
||||
const startMsg = `Work the goals in ${planFile}. Pick an open goal, mark it active (set its checkbox to [/]), work its subtasks, and when its discriminator is satisfied fill the goal's evidence: block then call CompleteGoal with the goal's desc. Keep goals.md current as you go.`;
|
||||
exitPlanMode(ctx);
|
||||
|
||||
if (fresh && savedCmdCtx) {
|
||||
// After newSession, `ctx`/`pi` bound to the old session are stale; do post-swap work
|
||||
// through the ReplacedSessionContext passed to withSession (see runner.assertActive).
|
||||
const result = await savedCmdCtx.newSession({
|
||||
parentSession,
|
||||
withSession: async (sessionCtx) => {
|
||||
// pi.* and the outer ctx are invalidated by newSession; use the fresh sessionCtx only.
|
||||
// (No setSessionName here: it lives on pi/the outer ctx, both stale now. Cosmetic, skip it.)
|
||||
sessionCtx.ui.notify(planContent, "info");
|
||||
await sessionCtx.sendUserMessage(startMsg, { deliverAs: "followUp" });
|
||||
},
|
||||
});
|
||||
if (result.cancelled) {
|
||||
return;
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (doc.title) pi.setSessionName(`Goals: ${doc.title}`);
|
||||
ctx.ui.notify(planContent, "info");
|
||||
pi.sendUserMessage(startMsg, { deliverAs: "followUp" });
|
||||
}
|
||||
|
||||
// --- the one blessed tool: CompleteGoal -------------------------------------------------------
|
||||
|
||||
pi.registerTool({
|
||||
name: "CompleteGoal",
|
||||
label: "Goal signoff",
|
||||
description: completeGoalDescription,
|
||||
parameters: Type.Object({
|
||||
goal: Type.String({ description: completeGoalParamDescription }),
|
||||
}),
|
||||
async execute(_id, params, signal, onUpdate, ctx) {
|
||||
const content = readPlan(ctx);
|
||||
const goal = findGoal(parse(content), params.goal);
|
||||
if (!goal) return text(`No goal "${params.goal}" in goals.md. Use the exact text after "goal:".`, true);
|
||||
if (goal.evidence.length === 0) {
|
||||
return text(`Goal "${goal.subject}" has no evidence yet. Add an evidence: list to the goal in goals.md (artifacts + a short read showing the discriminator is satisfied), then call CompleteGoal.`, true);
|
||||
}
|
||||
|
||||
const handleUpdate = (partial: { content: Array<{ type: "text"; text: string }>; details: SignOffDetails }) => {
|
||||
onUpdate?.(partial);
|
||||
};
|
||||
|
||||
const { outcome, reasoning, durationMs } = await decideSignOff(goal, goal.evidence.join("\n"), goal.evidence, state.judgeModel, ctx.cwd, signal, handleUpdate);
|
||||
const res = recordSignOff(content, goal.subject, stamp(), outcome);
|
||||
if (res.content !== content) writePlan(ctx, res.content);
|
||||
updateWidget(ctx);
|
||||
const detail = reasoning ? `\n\n--- sign-off judge ---\n${reasoning}` : "";
|
||||
const outcomeLabel = outcome.kind === "accepted" ? "accepted" : outcome.kind === "verify_failed" ? "verify_failed" : "rejected";
|
||||
const details: SignOffDetails = {
|
||||
goal: goal.subject,
|
||||
outcome: outcomeLabel,
|
||||
durationMs,
|
||||
verifyCommand: goal.verify ?? undefined,
|
||||
verifyExitCode: outcome.kind === "verify_failed" ? outcome.exitCode : undefined,
|
||||
judgeModel: state.judgeModel ?? undefined,
|
||||
reasoning,
|
||||
isError: res.isError,
|
||||
};
|
||||
return textWithDetails(res.message + detail, details, res.isError);
|
||||
},
|
||||
|
||||
renderCall(args, theme) {
|
||||
const goalText = args.goal.length > 80 ? `${args.goal.slice(0, 80)}...` : args.goal;
|
||||
return new Text(
|
||||
`${theme.fg("toolTitle", theme.bold("goal signoff "))}${theme.fg("dim", goalText)}`,
|
||||
0, 0,
|
||||
);
|
||||
},
|
||||
|
||||
renderResult(result, { expanded }, theme) {
|
||||
const details = result.details as SignOffDetails | undefined;
|
||||
const body = result.content[0]?.type === "text" ? result.content[0].text : "(no output)";
|
||||
if (!details || details.outcome === "running") return new Text(body, 0, 0);
|
||||
|
||||
const icon = details.outcome === "accepted" ? theme.fg("success", "✔") : theme.fg("error", "✗");
|
||||
const outcomeText = details.outcome === "accepted" ? "accepted" : details.outcome === "verify_failed" ? `verify failed (exit ${details.verifyExitCode})` : "rejected";
|
||||
const header = `${icon} ${theme.fg("toolTitle", theme.bold("goal signoff "))}${theme.fg("accent", outcomeText)}`;
|
||||
const duration = details.durationMs < 1000 ? `${details.durationMs}ms` : `${(details.durationMs / 1000).toFixed(1)}s`;
|
||||
const sub = [details.judgeModel, duration].filter(Boolean).join(" · ");
|
||||
|
||||
if (!expanded) {
|
||||
let text = header;
|
||||
if (sub) text += `\n${theme.fg("dim", sub)}`;
|
||||
text += `\n\n${theme.fg("toolOutput", body.slice(0, 500))}`;
|
||||
if (body.length > 500) text += theme.fg("dim", "...");
|
||||
text += `\n${theme.fg("muted", "(Ctrl+O to expand)")}`;
|
||||
return new Text(text, 0, 0);
|
||||
}
|
||||
|
||||
const container = new Container();
|
||||
container.addChild(new Text(header, 0, 0));
|
||||
if (sub) container.addChild(new Text(theme.fg("dim", sub), 0, 0));
|
||||
if (details.verifyCommand) {
|
||||
container.addChild(new Spacer(1));
|
||||
container.addChild(new Text(theme.fg("muted", `verify: ${details.verifyCommand}`), 0, 0));
|
||||
}
|
||||
container.addChild(new Spacer(1));
|
||||
container.addChild(new Text(theme.fg("muted", "Judge"), 0, 0));
|
||||
container.addChild(new Markdown(body.trim(), 0, 0, getMarkdownTheme()));
|
||||
return container;
|
||||
},
|
||||
});
|
||||
|
||||
// --- hooks ------------------------------------------------------------------------------------
|
||||
// --- hooks --------------------------------------------------------------------------------------
|
||||
|
||||
pi.on("before_agent_start", async (_event, ctx) => {
|
||||
if (state.isPlanMode) {
|
||||
// Read-only is enforced in the tool_call hook below (blocks edit/write while planning).
|
||||
return { message: { customType: PLAN_CONTEXT, content: `${planDrafting}\n\nWrite the plan to ${planPath(ctx)}.`, display: false } };
|
||||
}
|
||||
const doc = parse(readPlan(ctx));
|
||||
if (doc.goals.length === 0) return;
|
||||
|
||||
const active = doc.goals.find((g) => g.status === "active") ?? doc.goals.find((g) => g.status === "open") ?? null;
|
||||
const c = counts(doc);
|
||||
let body = planInjection({
|
||||
title: doc.title,
|
||||
activeGoal: active
|
||||
? {
|
||||
subject: active.subject,
|
||||
discriminator: active.discriminator,
|
||||
openSubtasks: active.subtasks.filter((s) => s.status !== "done" && s.status !== "cancelled").map((s) => s.text),
|
||||
}
|
||||
: null,
|
||||
lastLogLine: doc.log.at(-1) ?? null,
|
||||
counts: { done: c.done, open: c.open + c.active },
|
||||
});
|
||||
// Reminder fires when there is an active goal but goals.md was untouched since the last turn.
|
||||
const planNow = readPlan(ctx);
|
||||
if (active && planNow === lastInjectedPlan) body += `\n\n${reminder}`;
|
||||
lastInjectedPlan = planNow;
|
||||
const plan = readPlan(ctx);
|
||||
const goals = scanGoals(plan);
|
||||
if (goals.length === 0) {
|
||||
// Non-empty plan but no recognizable goal line: the harness would go silently inert (no
|
||||
// widget, no injection, no reminders). Say so once instead -- cooperative but confused.
|
||||
if (!plan.trim()) return;
|
||||
return {
|
||||
message: {
|
||||
customType: PLAN_CONTEXT,
|
||||
content: `${PLAN_REL} exists but has no goal line pi-goals recognizes. A goal is a checkbox list line starting "goal:", e.g. "1. [ ] goal: <imperative>" ([ ] open, [/] active, [x] done, [-] cancelled). Reformat if it's meant to be the plan.`,
|
||||
display: false,
|
||||
},
|
||||
};
|
||||
}
|
||||
// The plan file itself IS the injection: no parsing, no summarizing, the model sees the
|
||||
// literal file it edits. Byte-identical when unchanged, so the prefix cache holds.
|
||||
let body = `Current plan (${PLAN_REL}; keep it updated with your edit tool):\n\n${plan}`;
|
||||
const live = goals.some((g) => g.status === "active" || g.status === "open");
|
||||
if (live && plan === lastInjectedPlan) body += `\n\n${reminder}`;
|
||||
lastInjectedPlan = plan;
|
||||
return { message: { customType: PLAN_CONTEXT, content: body, display: false } };
|
||||
});
|
||||
|
||||
// Enforce read-only planning: block file mutators while in plan mode so code isn't written before
|
||||
// the goals are agreed. The agent draws back to read/grep/find/ls and read-only bash to explore.
|
||||
// Plan mode gate: block edit/write except on plan.md itself. bash stays open (guide, not gate).
|
||||
pi.on("tool_call", async (event, ctx) => {
|
||||
if (!state.isPlanMode) return;
|
||||
// edit/write: blocked, except writing goals.md itself (the deliverable of plan mode).
|
||||
if (PLAN_MODE_BLOCKED_TOOLS.includes(event.toolName)) {
|
||||
const target = (event.input as { path?: string }).path;
|
||||
if (target && resolve(ctx.cwd, target) === resolve(planPath(ctx))) return;
|
||||
return { block: true, reason: `Plan mode is read-only: agree the goals in ${PLAN_REL} and choose Ready before writing code (${event.toolName} is blocked while planning; only ${PLAN_REL} may be written).` };
|
||||
}
|
||||
// bash: blocked only when the command looks mutating; read-only exploration stays open.
|
||||
if (event.toolName === "bash") {
|
||||
const command = (event.input as { command?: string }).command ?? "";
|
||||
if (MUTATING_BASH_PATTERNS.some((re) => re.test(command))) {
|
||||
return { block: true, reason: `Plan mode is read-only: this bash command looks like it mutates state, so it's blocked while planning. Explore read-only, agree the goals in ${PLAN_REL}, then choose Ready.\nCommand: ${command}` };
|
||||
}
|
||||
return { block: true, reason: `Plan mode is read-only: only ${PLAN_REL} may be written while drafting. Agree the goals first, then choose Ready.` };
|
||||
}
|
||||
});
|
||||
|
||||
// After a plan-mode turn: if goals were drafted, offer Ready. No edit menus, no fresh-session
|
||||
// dance — the human reads the file and says go (or keeps talking to revise it).
|
||||
pi.on("agent_end", async (_event, ctx) => {
|
||||
if (!state.isPlanMode || !ctx.hasUI) return;
|
||||
const doc = parse(readPlan(ctx));
|
||||
if (doc.goals.length === 0) {
|
||||
ctx.ui.notify("No goals found in goals.md yet — ask the agent to draft them.", "warning");
|
||||
return;
|
||||
}
|
||||
await reviewLoop(ctx);
|
||||
if (scanGoals(readPlan(ctx)).length === 0) return; // still exploring/asking; nothing to review yet
|
||||
const choice = await ctx.ui.select(`Plan drafted in ${PLAN_REL}. Ready?`, [
|
||||
"Ready — start working the plan",
|
||||
"Keep planning (reply to revise)",
|
||||
]);
|
||||
if (!choice?.startsWith("Ready")) return;
|
||||
state = { ...state, isPlanMode: false };
|
||||
persist();
|
||||
updateWidget(ctx);
|
||||
pi.sendUserMessage(
|
||||
`Work the goals in ${planPath(ctx)}. Pick an open goal, mark it active ([/]), work its subtasks, and when its discriminator is satisfied fill its evidence: list, then call CompleteGoal with the goal's text. Keep the plan file current as you go.`,
|
||||
{ deliverAs: "followUp" },
|
||||
);
|
||||
});
|
||||
|
||||
// Keep only the freshest injected plan summary; strip stale ones so history does not bloat and
|
||||
// the model never sees an out-of-date plan. (The current turn's injection is the one kept.)
|
||||
// Keep only the freshest injected plan; strip stale ones so history doesn't bloat and the model
|
||||
// never sees an out-of-date plan.
|
||||
pi.on("context", async (event) => {
|
||||
const isCtx = (m: unknown) => (m as { customType?: string }).customType === PLAN_CONTEXT;
|
||||
let lastIdx = -1;
|
||||
@@ -426,6 +208,13 @@ export default function piGoalsExtension(pi: ExtensionAPI): void {
|
||||
});
|
||||
|
||||
pi.on("session_start", async (_event, ctx) => {
|
||||
// v1 wrote .pi/goals.md; v2 reads .pi/plan.md. Rename so old goals aren't silently invisible
|
||||
// (dogfood finding). Claude: one-time migration, delete once v1 files are gone from the wild.
|
||||
const v1Path = join(ctx.cwd, ".pi", "goals.md");
|
||||
if (existsSync(v1Path) && !existsSync(planPath(ctx))) {
|
||||
renameSync(v1Path, planPath(ctx));
|
||||
ctx.ui.notify(`Renamed .pi/goals.md -> ${PLAN_REL} (v2 filename).`, "info");
|
||||
}
|
||||
const last = ctx.sessionManager
|
||||
.getEntries()
|
||||
.filter((e: { type?: string; customType?: string }) => e.type === "custom" && e.customType === STATE)
|
||||
@@ -433,75 +222,190 @@ export default function piGoalsExtension(pi: ExtensionAPI): void {
|
||||
if (last?.data) state = { ...state, ...last.data };
|
||||
updateWidget(ctx);
|
||||
});
|
||||
|
||||
// --- the one blessed tool: CompleteGoal ---------------------------------------------------------
|
||||
|
||||
pi.registerTool({
|
||||
name: "CompleteGoal",
|
||||
label: "Goal signoff",
|
||||
description: completeGoalDescription,
|
||||
parameters: Type.Object({
|
||||
goal: Type.String({ description: completeGoalParamDescription }),
|
||||
}),
|
||||
async execute(_id, params, signal, onUpdate, ctx) {
|
||||
const plan = readPlan(ctx);
|
||||
if (!plan.trim()) return result(`No plan file at ${PLAN_REL}.`, true);
|
||||
|
||||
const judgeModel = state.judgeModel ?? (ctx.model ? `${ctx.model.provider}/${ctx.model.id}` : null);
|
||||
onUpdate?.({ content: [{ type: "text", text: `Read-only judge (${judgeModel ?? "pi default"}) inspecting: ${params.goal}` }], details: {} });
|
||||
// decideSignOff runs the judge and derives the outcome + the one log line. judgeModel is never
|
||||
// checked pre-emptively: null just means pi's configured default runs (buildJudgeArgs omits
|
||||
// --model), so accepted_inconclusive always means "the judge ran but failed", never "no model".
|
||||
let judgeRaw: JudgeResult | null = null;
|
||||
const outcome = await decideSignOff({ goal: params.goal, plan, judgeModel }, signal, async (task) => {
|
||||
judgeRaw = await runJudge(task, judgeModel, ctx.cwd, signal);
|
||||
return judgeRaw;
|
||||
});
|
||||
// Persist the judge's full transcript so "did the judge really re-run verify?" is answerable
|
||||
// after the fact (dogfood finding: with only the one log line, an accept is unauditable).
|
||||
let transcriptNote = "";
|
||||
if (judgeRaw !== null) {
|
||||
const raw: JudgeResult = judgeRaw;
|
||||
mkdirSync(join(ctx.cwd, ".pi", "judge"), { recursive: true });
|
||||
const rel = `.pi/judge/${stamp().replace(/[: ]/g, "-")}.md`;
|
||||
writeFileSync(join(ctx.cwd, rel), `goal: ${params.goal}\nmodel: ${judgeModel ?? "pi default"}\nerror: ${raw.error ?? "none"}\n\n${raw.output}\n`);
|
||||
transcriptNote = ` (${rel})`;
|
||||
}
|
||||
if (outcome.logEntry) {
|
||||
// Sign-off write: tick the goal [x] (exact-subject match; dogfood showed agent bookkeeping
|
||||
// is the drift point) and append the audit log line, one write. On wording drift the tick
|
||||
// falls to the agent and the result says so -- both paths are explicit, never silent.
|
||||
let updated = readPlan(ctx);
|
||||
let tickNote = "";
|
||||
if (outcome.logEntry.startsWith("signed off")) {
|
||||
const ticked = tickGoal(updated, params.goal);
|
||||
updated = ticked ?? updated;
|
||||
tickNote = ticked
|
||||
? `\n\nGoal ticked [x] in ${PLAN_REL}.`
|
||||
: `\n\nNo exact goal line matched your wording -- tick it [x] in ${PLAN_REL} yourself.`;
|
||||
}
|
||||
writePlan(ctx, appendLog(updated, `${stamp()} ${outcome.logEntry}${transcriptNote}`));
|
||||
updateWidget(ctx);
|
||||
return result(outcome.resultText + tickNote, outcome.isError);
|
||||
}
|
||||
return result(outcome.resultText, outcome.isError);
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
// --- helpers (module scope; pure enough to keep out of the closure) -------------------------------
|
||||
// --- helpers (module scope) --------------------------------------------------------------------
|
||||
|
||||
/** Structured details returned by CompleteGoal so renderCall/renderResult can show metadata. */
|
||||
interface SignOffDetails {
|
||||
goal: string;
|
||||
outcome: "accepted" | "rejected" | "verify_failed" | "running";
|
||||
phase?: string; // "verifying" | "spawning" | "judging" — while running
|
||||
durationMs: number;
|
||||
verifyCommand?: string;
|
||||
verifyExitCode?: number;
|
||||
judgeModel?: string;
|
||||
reasoning: string;
|
||||
isError?: boolean;
|
||||
}
|
||||
|
||||
function text(s: string, isError = false) {
|
||||
return { content: [{ type: "text" as const, text: s }], details: { isError }, isError };
|
||||
}
|
||||
|
||||
function textWithDetails(s: string, details: SignOffDetails, isError = false) {
|
||||
return { content: [{ type: "text" as const, text: s }], details, isError };
|
||||
function result(text: string, isError = false) {
|
||||
return { content: [{ type: "text" as const, text }], details: {}, isError };
|
||||
}
|
||||
|
||||
/** Local time, not UTC: agents freehand-stamp their manual ## Log lines from the local clock they
|
||||
* see, so a UTC tool stamp made the trail read as two different afternoons (dogfood finding). */
|
||||
function stamp(): string {
|
||||
return new Date().toISOString().slice(0, 16).replace("T", " ");
|
||||
const d = new Date();
|
||||
const p = (n: number) => String(n).padStart(2, "0");
|
||||
return `${d.getFullYear()}-${p(d.getMonth() + 1)}-${p(d.getDate())} ${p(d.getHours())}:${p(d.getMinutes())}`;
|
||||
}
|
||||
|
||||
/** Decide a sign-off: deterministic verify first (cheap; skip the model call if it fails), then the judge.
|
||||
* Returns the outcome plus the judge's (or verify's) reasoning so CompleteGoal can show WHY. */
|
||||
async function decideSignOff(
|
||||
goal: Goal,
|
||||
evidence: string,
|
||||
paths: string[],
|
||||
judgeModel: string | null,
|
||||
cwd: string,
|
||||
function oneLine(s: string): string {
|
||||
return s.replace(/\s+/g, " ").trim().slice(0, 200);
|
||||
}
|
||||
|
||||
/** A judge run's result: stdout output, plus an error string when the subprocess failed/timed out. */
|
||||
export interface JudgeResult {
|
||||
output: string;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
/** Inputs to a sign-off decision. judgeModel is null when no explicit/session model is set. */
|
||||
export interface SignOffInput {
|
||||
goal: string;
|
||||
plan: string;
|
||||
judgeModel: string | null;
|
||||
}
|
||||
|
||||
/** The outcome of a sign-off: the reply text, whether it's a hard error, and the one ## Log line to
|
||||
* append (null when nothing should be written, e.g. aborted before any verdict). */
|
||||
export interface SignOffOutcome {
|
||||
resultText: string;
|
||||
isError: boolean;
|
||||
logEntry: string | null;
|
||||
}
|
||||
|
||||
/** Run the judge and decide accept / reject / accepted_inconclusive. Pure aside from the injected
|
||||
* judge runner, so the unit test can lock the fail-forward invariant: judgeModel is NEVER checked
|
||||
* here, so a null model still reaches runJudge (pi's configured default runs it), and the only
|
||||
* producers of accepted_inconclusive are the judge-error and no-VERDICT paths -- i.e. "the judge
|
||||
* ran but failed", never "no model". The execute() wrapper does the plan-file write + widget.
|
||||
* Exported for the unit test that locks this invariant. */
|
||||
export async function decideSignOff(
|
||||
input: SignOffInput,
|
||||
signal: AbortSignal | undefined,
|
||||
onUpdate?: (partial: { content: Array<{ type: "text"; text: string }>; details: SignOffDetails }) => void,
|
||||
): Promise<{ outcome: SignOff; reasoning: string; durationMs: number }> {
|
||||
const startedAt = Date.now();
|
||||
const emit = (phase: string, text: string) => {
|
||||
onUpdate?.({
|
||||
content: [{ type: "text" as const, text }],
|
||||
details: { goal: goal.subject, outcome: "running", phase, durationMs: Date.now() - startedAt, verifyCommand: goal.verify ?? undefined, judgeModel: judgeModel ?? undefined, reasoning: "" },
|
||||
});
|
||||
};
|
||||
let verifyResult: { command: string; exitCode: number; outputTail: string } | null = null;
|
||||
if (goal.verify) {
|
||||
emit("verifying", `Running verify: ${goal.verify}`);
|
||||
verifyResult = runVerify(goal.verify, cwd, signal);
|
||||
if (verifyResult.exitCode !== 0) {
|
||||
return {
|
||||
outcome: { kind: "verify_failed", exitCode: verifyResult.exitCode, outputTail: verifyResult.outputTail },
|
||||
reasoning: `verify \`${goal.verify}\` exited ${verifyResult.exitCode}:\n${verifyResult.outputTail}`,
|
||||
durationMs: Date.now() - startedAt,
|
||||
};
|
||||
}
|
||||
runJudgeFn: (task: string) => Promise<JudgeResult>,
|
||||
): Promise<SignOffOutcome> {
|
||||
const task = judgeUser({ goal: input.goal, plan: input.plan, planPath: PLAN_REL });
|
||||
const judge = await runJudgeFn(task);
|
||||
|
||||
if (signal?.aborted) return { resultText: "Sign-off aborted.", isError: true, logEntry: null };
|
||||
|
||||
// Judge ran but failed/errored/timed out: fail forward, say so in the log.
|
||||
if (judge.error) {
|
||||
const partial = judge.output ? `\n\npartial judge output:\n${judge.output}` : "";
|
||||
return {
|
||||
resultText: `Judge ran but failed (${judge.error}). Accepted inconclusive — logged.${partial}`,
|
||||
isError: false,
|
||||
logEntry: `signed off "${input.goal}" (judge inconclusive: ran but failed: ${oneLine(judge.error)})`,
|
||||
};
|
||||
}
|
||||
const verdict = await runJudge(goal, evidence, paths, verifyResult, judgeModel, cwd, signal, onUpdate);
|
||||
const outcome: SignOff = verdict.accept ? { kind: "accepted" } : { kind: "rejected", missing: verdict.missing };
|
||||
return { outcome, reasoning: verdict.reasoning, durationMs: verdict.durationMs };
|
||||
|
||||
const verdictLine = judge.output.split("\n").find((l) => /^\s*VERDICT\s*:/i.test(l)) ?? "";
|
||||
const verdict = /^\s*VERDICT\s*:\s*(accept|reject)\s*$/i.exec(verdictLine)?.[1]?.toLowerCase();
|
||||
const reasoning = judge.output.length > 2000 ? `...\n${judge.output.slice(-2000)}` : judge.output;
|
||||
|
||||
if (verdict === "accept") {
|
||||
return {
|
||||
resultText: `Sign-off ACCEPTED (log line appended).\n\n--- judge ---\n${reasoning}`,
|
||||
isError: false,
|
||||
logEntry: `signed off "${input.goal}" (judge accept)`,
|
||||
};
|
||||
}
|
||||
if (verdict === "reject") {
|
||||
const missing = judge.output.match(/missing\s*:\s*([\s\S]*)$/i)?.[1].trim() || judge.output.slice(-500);
|
||||
return {
|
||||
resultText: `Sign-off REJECTED. Missing:\n${missing}\n\n--- judge ---\n${reasoning}`,
|
||||
isError: true,
|
||||
logEntry: `reject "${input.goal}": ${oneLine(missing)}`,
|
||||
};
|
||||
}
|
||||
// No VERDICT line: same fail-forward as a judge error -- the judge ran but didn't answer.
|
||||
return {
|
||||
resultText: `Judge returned no VERDICT line. Accepted inconclusive — logged.\n\n--- judge ---\n${reasoning || "(no output)"}`,
|
||||
isError: false,
|
||||
logEntry: `signed off "${input.goal}" (judge inconclusive: no VERDICT line)`,
|
||||
};
|
||||
}
|
||||
|
||||
/** Run the goal's verify command. It is agent-authored and trusted (single-user machine, guide-not-guard). */
|
||||
function runVerify(command: string, cwd: string, signal: AbortSignal | undefined): { command: string; exitCode: number; outputTail: string } {
|
||||
const res = spawnSync("sh", ["-c", command], { cwd, encoding: "utf-8", signal, timeout: 600_000 });
|
||||
const out = `${res.stdout ?? ""}${res.stderr ?? ""}`;
|
||||
return { command, exitCode: res.status ?? 1, outputTail: out.split("\n").slice(-30).join("\n") };
|
||||
/** Tick the goal line whose subject exactly matches `goal` (trimmed, case-insensitive) to [x].
|
||||
* Null when there is no unique exact match (wording drift / duplicates) -- the caller then asks the
|
||||
* agent to tick it itself. Reuses GOAL_LINE; deliberately NOT fuzzy, that's the judge's job. */
|
||||
export function tickGoal(plan: string, goal: string): string | null {
|
||||
const lines = plan.split("\n");
|
||||
const want = goal.trim().toLowerCase();
|
||||
const hits = lines.flatMap((l, i) => (GOAL_LINE.exec(l)?.[2].trim().toLowerCase() === want ? [i] : []));
|
||||
if (hits.length !== 1) return null;
|
||||
lines[hits[0]] = lines[hits[0]].replace(/\[[ xX/-]\]/, "[x]");
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
/** Append one line under ## Log (creating the section at EOF if absent). */
|
||||
export function appendLog(text: string, entry: string): string {
|
||||
const lines = text.split("\n");
|
||||
const line = `- ${entry}`;
|
||||
const header = lines.findIndex((l) => /^##\s+Log\s*$/i.test(l));
|
||||
if (header === -1) return `${text.replace(/\n+$/, "")}\n\n## Log\n${line}\n`;
|
||||
let insertAt = header + 1;
|
||||
for (let i = header + 1; i < lines.length; i++) {
|
||||
if (/^#{1,6}\s/.test(lines[i])) break;
|
||||
if (/^\s*-\s+/.test(lines[i])) insertAt = i + 1;
|
||||
}
|
||||
lines.splice(insertAt, 0, line);
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
/** Build the pi argv for the read-only judge. `--model` is omitted when no explicit/session model is
|
||||
* set, so pi falls back to its configured default — the judge always runs. `--no-extensions` keeps
|
||||
* the judge minimal and immune to a broken third-party extension taking down every sign-off.
|
||||
* Exported for the unit test that locks these invariants. */
|
||||
export function buildJudgeArgs(judgeModel: string | null): string[] {
|
||||
const args = ["-p", "--no-session", "--no-extensions"];
|
||||
if (judgeModel) args.push("--model", judgeModel);
|
||||
args.push("--tools", JUDGE_TOOLS.join(","), "--exclude-tools", JUDGE_BLOCKED_TOOLS.join(","), "--append-system-prompt", judgeSystem);
|
||||
return args;
|
||||
}
|
||||
|
||||
/** Locate the pi binary the same way the oracle extension does, so spawning works under bun or node. */
|
||||
@@ -513,82 +417,44 @@ function getPiInvocation(args: string[]): { command: string; args: string[] } {
|
||||
return { command: "pi", args };
|
||||
}
|
||||
|
||||
/** Stage 2: a read-only pi subprocess inspects the evidence against the repo and returns a verdict. */
|
||||
/** Spawn the read-only judge subprocess (plain `pi -p`: stdout is the final response text). */
|
||||
async function runJudge(
|
||||
goal: Goal,
|
||||
evidence: string,
|
||||
paths: string[],
|
||||
verifyResult: { command: string; exitCode: number; outputTail: string } | null,
|
||||
task: string,
|
||||
judgeModel: string | null,
|
||||
cwd: string,
|
||||
signal: AbortSignal | undefined,
|
||||
onUpdate?: (partial: { content: Array<{ type: "text"; text: string }>; details: SignOffDetails }) => void,
|
||||
): Promise<{ accept: boolean; missing: string; reasoning: string; durationMs: number }> {
|
||||
const startedAt = Date.now();
|
||||
const emit = (phase: string, text: string) => {
|
||||
onUpdate?.({
|
||||
content: [{ type: "text" as const, text }],
|
||||
details: { goal: goal.subject, outcome: "running", phase, durationMs: Date.now() - startedAt, verifyCommand: goal.verify ?? undefined, judgeModel: judgeModel ?? undefined, reasoning: "" },
|
||||
});
|
||||
};
|
||||
const task = evidenceJudgeUser({
|
||||
subject: goal.subject,
|
||||
discriminator: goal.discriminator,
|
||||
failure_modes: goal.failure_modes,
|
||||
verify: goal.verify ?? null,
|
||||
verifyResult,
|
||||
evidence,
|
||||
paths,
|
||||
});
|
||||
const args = ["-p", "--no-session", "--tools", JUDGE_TOOLS.join(","), "--exclude-tools", JUDGE_BLOCKED_TOOLS.join(","), "--append-system-prompt", evidenceJudgeSystem];
|
||||
if (judgeModel) args.push("--model", judgeModel);
|
||||
): Promise<JudgeResult> {
|
||||
const args = buildJudgeArgs(judgeModel);
|
||||
args.push(task);
|
||||
|
||||
emit("spawning", `Spawning read-only judge for: ${goal.subject}`);
|
||||
const inv = getPiInvocation(args);
|
||||
// FIXME(side-effect): pi -p --no-session clones the repo into the PARENT of cwd (so alongside
|
||||
// the working dir), leaving a stale directory. The judge should run in a temp dir or inside the
|
||||
// existing repo checkout so it doesn't pollute the user's workspace.
|
||||
const JUDGE_TIMEOUT_MS = 120_000;
|
||||
const output = await new Promise<string>((resolve) => {
|
||||
// Runs in-place against this checkout; pi --no-session does not clone into the parent
|
||||
// (proven by scripts/check-judge-footprint.sh).
|
||||
return new Promise((resolvePromise) => {
|
||||
let settled = false;
|
||||
const timer = setTimeout(() => {
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
const done = (r: { output: string; error?: string }) => {
|
||||
if (!settled) {
|
||||
settled = true;
|
||||
proc.kill();
|
||||
resolve(`VERDICT: reject\nmissing: judge timed out after ${JUDGE_TIMEOUT_MS / 1000}s`);
|
||||
clearTimeout(timer);
|
||||
resolvePromise(r);
|
||||
}
|
||||
}, JUDGE_TIMEOUT_MS);
|
||||
};
|
||||
const proc = spawn(inv.command, inv.args, { cwd, shell: false, stdio: ["ignore", "pipe", "pipe"], signal });
|
||||
let out = "";
|
||||
proc.stdout.on("data", (d) => (out += d));
|
||||
proc.stderr.on("data", (d) => (out += d));
|
||||
proc.on("close", () => {
|
||||
if (!settled) {
|
||||
settled = true;
|
||||
clearTimeout(timer);
|
||||
resolve(out);
|
||||
}
|
||||
const timer = setTimeout(() => {
|
||||
proc.kill();
|
||||
done({ output: stdout.trim(), error: `judge timed out after ${JUDGE_TIMEOUT_MS / 1000}s` });
|
||||
}, JUDGE_TIMEOUT_MS);
|
||||
proc.stdout?.on("data", (d) => {
|
||||
stdout += d.toString();
|
||||
});
|
||||
proc.on("error", (e) => {
|
||||
if (!settled) {
|
||||
settled = true;
|
||||
clearTimeout(timer);
|
||||
resolve(`VERDICT: reject\nmissing: judge subprocess failed: ${e.message}`);
|
||||
}
|
||||
proc.stderr?.on("data", (d) => {
|
||||
stderr += d.toString();
|
||||
});
|
||||
proc.on("close", (code) => {
|
||||
if ((code ?? 0) !== 0) done({ output: stdout.trim(), error: stderr.trim() || `judge subprocess exited ${code ?? 1}` });
|
||||
else done({ output: stdout.trim() });
|
||||
});
|
||||
proc.on("error", (e) => done({ output: stdout.trim(), error: `judge subprocess failed: ${e.message}` }));
|
||||
});
|
||||
|
||||
// The subprocess emits ANSI/CSI control codes in -p mode; strip them so they don't leak into `missing`.
|
||||
const clean = output.replace(/\u001b\[[0-9;?]*[ -/]*[@-~]/g, "");
|
||||
|
||||
const verdictLine = clean.split("\n").find((l) => /^\s*VERDICT\s*:/i.test(l)) ?? "";
|
||||
const accept = /accept/i.test(verdictLine);
|
||||
const missingMatch = clean.match(/missing\s*:\s*([\s\S]*)$/i);
|
||||
const missing = accept ? "" : (missingMatch?.[1].trim() || clean.trim().slice(-500) || "judge gave no reason");
|
||||
// The judge's own words (inspection + verdict), so CompleteGoal can show them. The verdict is at the
|
||||
// end, so keep the tail when it's long.
|
||||
const trimmed = clean.trim();
|
||||
const reasoning = trimmed.length > 1800 ? `...\n${trimmed.slice(-1800)}` : trimmed;
|
||||
return { accept, missing, reasoning, durationMs: Date.now() - startedAt };
|
||||
}
|
||||
|
||||
@@ -1,278 +0,0 @@
|
||||
/**
|
||||
* plan-file.ts — read goals.md, and the two writes CompleteGoal needs. That is all.
|
||||
*
|
||||
* Pure module, no pi deps, so it unit-tests without a runtime. The file is the canonical store and
|
||||
* the agent edits it with its normal Edit tool (create goals, tick subtasks, fill evidence), guided
|
||||
* by the format in prompts.ts and the reminder -- the form guides, it does not gate. The only
|
||||
* programmatic writers are setGoalStatus + appendLog, used by CompleteGoal to record an accepted
|
||||
* sign-off; both touch one line so the diff stays readable.
|
||||
*
|
||||
* Format (markdown, checkbox-first, made to be skim-reviewed by a human):
|
||||
*
|
||||
* # <plan title>
|
||||
*
|
||||
* <context: the user's ask, preferences, decisions>
|
||||
*
|
||||
* ## Goals
|
||||
*
|
||||
* 1. [ ] goal: <desc> <- state in the checkbox: [ ] open [/] active [x] done [-] cancelled
|
||||
* - discriminator: <positive observation that the goal succeeded, that no failure below could fake>
|
||||
* - subtle failure mode: <a way this looks done but isn't>
|
||||
* - verify: <optional shell command that exits 0 only when the discriminator passes>
|
||||
* - tasks:
|
||||
* 1. [x] <subtask> <- a subtask is any checkbox WITHOUT a "goal:" prefix
|
||||
* 2. [/] <subtask>
|
||||
* 3. [-] <subtask> <- [-] or ~~[ ]~~ both read as cancelled
|
||||
* - evidence: <- empty at planning; filled at sign-off, read by CompleteGoal
|
||||
* - > <artifact path / link / metric, plus a short read of it>
|
||||
* 2. [ ] goal: <desc>
|
||||
*
|
||||
* # Future work / out of scope
|
||||
*
|
||||
* ## Log
|
||||
* - <verbatim append-only line>
|
||||
*
|
||||
* A goal/subtask's state lives in its checkbox (single source of truth, renders natively). Goals are
|
||||
* matched by their <desc> (the text after "goal:"); the list number is human-facing only. Only
|
||||
* CompleteGoal writes a goal's [x]; the agent sets [/] when it starts one.
|
||||
*/
|
||||
|
||||
export type GoalStatus = "open" | "active" | "done" | "cancelled";
|
||||
|
||||
export interface Subtask {
|
||||
text: string;
|
||||
status: GoalStatus;
|
||||
}
|
||||
|
||||
export interface Goal {
|
||||
/** The text after "goal:" in the header line; the handle CompleteGoal matches on. */
|
||||
subject: string;
|
||||
status: GoalStatus;
|
||||
/** Positive observation(s) that the goal succeeded AND that no failure mode could fake. The success test. Written at planning. */
|
||||
discriminator: string[];
|
||||
/** Subtle ways a "done" could be wrong (look-like-success failures). Written at planning. */
|
||||
failure_modes: string[];
|
||||
/** Optional command that exits 0 only when the discriminator passes (the cheap deterministic gate). */
|
||||
verify?: string;
|
||||
/** Proof the discriminator passed, pointing at durable artifacts. Written at completion; read by CompleteGoal. */
|
||||
evidence: string[];
|
||||
subtasks: Subtask[];
|
||||
}
|
||||
|
||||
export interface PlanDoc {
|
||||
title: string;
|
||||
goals: Goal[];
|
||||
/** Verbatim ## Log lines, including the leading "- ". */
|
||||
log: string[];
|
||||
}
|
||||
|
||||
const TITLE = /^#\s+(.+?)\s*$/; // the first single-# H1
|
||||
const GOALS_HEADER = /^##\s+Goals\s*$/i;
|
||||
const LOG_HEADER = /^##\s+Log\s*$/i;
|
||||
const ANY_HEADER = /^#{1,6}\s/;
|
||||
// A goal: a numbered or bulleted checkbox item whose text begins "goal:".
|
||||
const GOAL_ITEM = /^\s*(?:\d+\.|[-*])\s*\[([ xX/-])\]\s*goal:\s*(.*)$/i;
|
||||
// A section marker bullet under a goal (the trailing colon is optional, e.g. "- tasks").
|
||||
const KEY_LINE = /^\s*[-*]\s*(discriminator|subtle failure modes?|failure_modes?|verify|tasks?|evidence)\s*:?\s*(.*)$/i;
|
||||
// Any list item (numbered or bulleted); used for subtasks and for list items inside the sections.
|
||||
const LIST_ITEM = /^\s*(?:\d+\.|[-*])\s+(.*)$/;
|
||||
// A checkbox inside a list-item body (subtask). A leading/trailing ~~ marks it cancelled.
|
||||
const CHECKBOX_BODY = /^(~~)?\s*\[([ xX/-])\]\s*(.*)$/;
|
||||
|
||||
const CHAR_TO_STATUS: Record<string, GoalStatus> = { " ": "open", "/": "active", x: "done", "-": "cancelled" };
|
||||
const STATUS_TO_CHAR: Record<GoalStatus, string> = { open: " ", active: "/", done: "x", cancelled: "-" };
|
||||
|
||||
function normalizeKey(raw: string): "discriminator" | "failure_modes" | "verify" | "tasks" | "evidence" {
|
||||
const k = raw.toLowerCase();
|
||||
if (k.startsWith("discriminator")) return "discriminator";
|
||||
if (k.startsWith("verify")) return "verify";
|
||||
if (k.startsWith("task")) return "tasks";
|
||||
if (k.startsWith("evidence")) return "evidence";
|
||||
return "failure_modes"; // "subtle failure mode(s)" / "failure_mode(s)"
|
||||
}
|
||||
|
||||
export function parse(text: string): PlanDoc {
|
||||
const lines = text.split("\n");
|
||||
let title = "";
|
||||
const goals: Goal[] = [];
|
||||
const log: string[] = [];
|
||||
|
||||
let cur: Goal | null = null;
|
||||
let curList: string[] | null = null; // the discriminator/failure_modes/evidence list "- " items append to
|
||||
let inGoals = false;
|
||||
let inLog = false;
|
||||
|
||||
const flush = () => {
|
||||
if (cur) goals.push(cur);
|
||||
cur = null;
|
||||
curList = null;
|
||||
};
|
||||
|
||||
for (const line of lines) {
|
||||
const tM = TITLE.exec(line);
|
||||
if (tM && !title && !GOALS_HEADER.test(line) && !LOG_HEADER.test(line)) {
|
||||
title = tM[1].trim();
|
||||
continue;
|
||||
}
|
||||
if (GOALS_HEADER.test(line)) {
|
||||
flush();
|
||||
inGoals = true;
|
||||
inLog = false;
|
||||
continue;
|
||||
}
|
||||
if (LOG_HEADER.test(line)) {
|
||||
flush();
|
||||
inGoals = false;
|
||||
inLog = true;
|
||||
continue;
|
||||
}
|
||||
// Any other header (e.g. "# Future work") ends the goals / log section.
|
||||
if (ANY_HEADER.test(line)) {
|
||||
flush();
|
||||
inGoals = false;
|
||||
inLog = false;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (inLog) {
|
||||
if (/^\s*-\s+/.test(line)) log.push(line);
|
||||
continue;
|
||||
}
|
||||
if (!inGoals) continue; // title + context prose between the title and ## Goals
|
||||
|
||||
const goalM = GOAL_ITEM.exec(line);
|
||||
if (goalM) {
|
||||
flush();
|
||||
cur = {
|
||||
subject: goalM[2].trim(),
|
||||
status: CHAR_TO_STATUS[goalM[1].toLowerCase()] ?? "open",
|
||||
discriminator: [],
|
||||
failure_modes: [],
|
||||
evidence: [],
|
||||
subtasks: [],
|
||||
};
|
||||
continue;
|
||||
}
|
||||
if (!cur) continue;
|
||||
|
||||
const keyM = KEY_LINE.exec(line);
|
||||
if (keyM) {
|
||||
const key = normalizeKey(keyM[1]);
|
||||
const inlineVal = keyM[2].trim();
|
||||
if (key === "verify") {
|
||||
cur.verify = inlineVal || undefined;
|
||||
curList = null;
|
||||
} else if (key === "tasks") {
|
||||
curList = null; // subtasks are identified by being a checkbox; this marker is cosmetic
|
||||
} else {
|
||||
curList = cur[key]; // discriminator | failure_modes | evidence
|
||||
if (inlineVal) curList.push(inlineVal);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
const listM = LIST_ITEM.exec(line);
|
||||
if (listM) {
|
||||
const body = listM[1];
|
||||
const cb = CHECKBOX_BODY.exec(body);
|
||||
if (cb) {
|
||||
// A checkbox without a "goal:" prefix is a subtask of the current goal.
|
||||
const cancelled = cb[1] === "~~" || body.includes("~~");
|
||||
const status = cancelled ? "cancelled" : (CHAR_TO_STATUS[cb[2].toLowerCase()] ?? "open");
|
||||
cur.subtasks.push({ text: cb[3].replace(/~~/g, "").trim(), status });
|
||||
curList = null;
|
||||
continue;
|
||||
}
|
||||
// A plain "- " / "> " item belongs to the current section (discriminator/failure/evidence).
|
||||
if (curList) curList.push(body.trim());
|
||||
continue;
|
||||
}
|
||||
|
||||
// A non-empty, non-"- " line continues the current item, so multi-line evidence (a block quote
|
||||
// of a log, a table, an interpretation line) stays attached to its item. Blank lines are skipped.
|
||||
if (curList && line.trim() !== "" && curList.length > 0) {
|
||||
curList[curList.length - 1] += `\n${line.trim()}`;
|
||||
}
|
||||
}
|
||||
flush();
|
||||
|
||||
return { title, goals, log };
|
||||
}
|
||||
|
||||
export function findGoal(doc: PlanDoc, subject: string): Goal | undefined {
|
||||
const want = subject.trim();
|
||||
return doc.goals.find((g) => g.subject === want);
|
||||
}
|
||||
|
||||
export function counts(doc: PlanDoc): { done: number; open: number; active: number } {
|
||||
const c = { done: 0, open: 0, active: 0 };
|
||||
for (const g of doc.goals) {
|
||||
if (g.status === "done") c.done++;
|
||||
else if (g.status === "active") c.active++;
|
||||
else if (g.status === "open") c.open++;
|
||||
}
|
||||
return c;
|
||||
}
|
||||
|
||||
/** Flip a goal's checkbox in place, matched by its subject (the one write CompleteGoal needs). */
|
||||
export function setGoalStatus(text: string, subject: string, status: GoalStatus): string {
|
||||
const lines = text.split("\n");
|
||||
const want = subject.trim();
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const m = GOAL_ITEM.exec(lines[i]);
|
||||
if (m && m[2].trim() === want) {
|
||||
lines[i] = lines[i].replace(/\[[ xX/-]\]/, `[${STATUS_TO_CHAR[status]}]`);
|
||||
return lines.join("\n");
|
||||
}
|
||||
}
|
||||
throw new Error(`Goal "${subject}" not found`);
|
||||
}
|
||||
|
||||
/**
|
||||
* The outcome of a sign-off attempt, decided by CompleteGoal (which runs verify + the judge). Kept
|
||||
* separate from the I/O so the record logic below is pure and testable.
|
||||
*/
|
||||
export type SignOff =
|
||||
| { kind: "verify_failed"; exitCode: number; outputTail: string }
|
||||
| { kind: "rejected"; missing: string }
|
||||
| { kind: "accepted" };
|
||||
|
||||
/** Apply a sign-off outcome to goals.md text: accept flips the goal checkbox to [x] + logs; reject only logs. Pure. */
|
||||
export function recordSignOff(
|
||||
text: string,
|
||||
subject: string,
|
||||
when: string,
|
||||
outcome: SignOff,
|
||||
): { content: string; message: string; isError: boolean } {
|
||||
const goal = findGoal(parse(text), subject);
|
||||
if (!goal) return { content: text, message: `No goal "${subject}" in goals.md.`, isError: true };
|
||||
|
||||
if (outcome.kind === "verify_failed") {
|
||||
const content = appendLog(text, `${when} reject "${subject}": verify exit ${outcome.exitCode}`);
|
||||
return { content, message: `Sign-off rejected: verify failed (exit ${outcome.exitCode}).\n${outcome.outputTail}`, isError: true };
|
||||
}
|
||||
if (outcome.kind === "rejected") {
|
||||
const oneLine = outcome.missing.replace(/\s+/g, " ").trim().slice(0, 200);
|
||||
const content = appendLog(text, `${when} reject "${subject}": ${oneLine}`);
|
||||
return { content, message: `Sign-off rejected. Missing:\n${outcome.missing}`, isError: true };
|
||||
}
|
||||
const flipped = setGoalStatus(text, subject, "done");
|
||||
const content = appendLog(flipped, `${when} signed off "${subject}" (judge accept)`);
|
||||
return { content, message: `Signed off "${subject}". Marked done in goals.md.`, isError: false };
|
||||
}
|
||||
|
||||
/** Append one verbatim line to ## Log (creating the section if absent). The other CompleteGoal write. */
|
||||
export function appendLog(text: string, entry: string): string {
|
||||
const lines = text.split("\n");
|
||||
const line = `- ${entry}`;
|
||||
const header = lines.findIndex((l) => LOG_HEADER.test(l));
|
||||
if (header === -1) return `${text.replace(/\n+$/, "")}\n\n## Log\n${line}\n`;
|
||||
|
||||
let insertAt = header + 1;
|
||||
for (let i = header + 1; i < lines.length; i++) {
|
||||
if (ANY_HEADER.test(lines[i])) break;
|
||||
if (/^\s*-\s+/.test(lines[i])) insertAt = i + 1;
|
||||
}
|
||||
lines.splice(insertAt, 0, line);
|
||||
return lines.join("\n");
|
||||
}
|
||||
+101
-218
@@ -1,60 +1,41 @@
|
||||
/**
|
||||
* pi-goals — all model-facing text, in flow order.
|
||||
* pi-goals v2 — all model-facing text, in flow order.
|
||||
*
|
||||
* Philosophy: the form guides a process; it does not police one. The agent can
|
||||
* edit goals.md freely. These prompts + the goals.md structure make the right path
|
||||
* the easy path. The only step that is genuinely rigorous is the evidence judge
|
||||
* (7), and even that is reached by guiding the agent to call CompleteGoal, not by
|
||||
* trapping it. Bypasses stay visible in the git diff and the widget.
|
||||
* Design: plan.md is for LLMs and the human, not for TypeScript. There is no parser and no schema;
|
||||
* the skeleton below is a convention the drafting prompt teaches, the working agent maintains with
|
||||
* its normal Edit tool, and the judge reads natively. The harness does three things for a
|
||||
* cooperative-but-confused model: memory (inject the file verbatim every turn), format guidance
|
||||
* (the skeleton), and fresh eyes (the read-only judge in CompleteGoal).
|
||||
*
|
||||
* Flow (this file is ordered the way the agent meets each text, so it reads as one pass):
|
||||
* SETUP (plan mode) 1. planDrafting — drafts goals (read-only phase)
|
||||
* EXEC, each turn start 2. planInjection — "here is your plan, where you are"
|
||||
* EXEC, periodic 3. reminder — the typed nudge that drives upkeep + autonomy
|
||||
* EXEC, loop continue 4. continuation — keep going toward the active goal
|
||||
* EXEC, after each turn 5. loopJudge — continue / pause (cheap, foolable, ok)
|
||||
* SIGN-OFF, agent-side 6. completeGoalTool — the CompleteGoal tool desc + param the agent reads
|
||||
* SIGN-OFF, judge-side 7. evidenceJudge — read-only verify (rigorous; the one real check)
|
||||
*
|
||||
* Read top to bottom to see the whole process. 5 and 7 embody the design contrast:
|
||||
* the cheap-foolable loop gate vs the must-not-be-fooled sign-off.
|
||||
*
|
||||
* WIRED in index.ts: 1 planDrafting, 2 planInjection, 3 reminder, 6 completeGoalTool, 7 evidenceJudge.
|
||||
* NOT YET WIRED: 4 continuation and 5 loopJudge define the autonomous re-prompt loop, which is
|
||||
* intentionally not built in v1 (an until-done-style loop was judged too complex). They stay here so
|
||||
* the full intended flow is reviewable; wire them if/when the loop is added.
|
||||
* Flow:
|
||||
* SETUP (plan mode) 1. planDrafting — draft goals into plan.md (read-only phase)
|
||||
* EXEC, each turn start 2. (the plan.md file itself, injected verbatim by index.ts)
|
||||
* EXEC, periodic 3. reminder — upkeep + autonomy nudge when plan.md went untouched
|
||||
* SIGN-OFF, agent-side 4. completeGoal* — the one blessed tool's description
|
||||
* SIGN-OFF, judge-side 5. judgeSystem/judgeUser — the one rigorous check
|
||||
*
|
||||
* The goal's test is the DISCRIMINATOR: the concrete observation that tells real success from the
|
||||
* named subtle failure mode. It replaces a vague "done_when". Evidence is empty at planning and
|
||||
* filled at sign-off (you don't always know the exact artifacts up front; the judge checks them then).
|
||||
* named subtle failure mode. Evidence is empty at planning and filled at sign-off.
|
||||
*/
|
||||
|
||||
/* ─────────────────────────────────────────────────────────────────────────
|
||||
* 1. planDrafting — SETUP, plan mode
|
||||
*
|
||||
* System guidance for the plan-phase agent. This phase is read-only (edit/write
|
||||
* and mutating bash are blocked by a tool hook): explore, then draft goals into
|
||||
* goals.md. The fields here are the whole "elicitation"; the human reviews this
|
||||
* output before any execution.
|
||||
* 1. planDrafting — SETUP, plan mode (read-only: edit/write blocked except plan.md)
|
||||
* ──────────────────────────────────────────────────────────────────────── */
|
||||
export const planDrafting = `\
|
||||
You are in plan mode. The objective may arrive through conversation, not as one up-front command.
|
||||
Explore the repository read-only first, then ask: resolve discoverable facts by looking them up, and
|
||||
only ask the human when the answer is a genuine intent or preference choice that exploration can't
|
||||
settle. Don't write goals that branch on something you could just check. Do not write or run code in
|
||||
this phase (edit and write are blocked, and so is mutating bash). If the ask is itself read-only
|
||||
(e.g. research, a search, a report), explore enough to scope it, but leave the actual deliverable for
|
||||
after the human approves the plan. When the objective is clear, draft goals into goals.md and stop
|
||||
for review. Produce a plan the human will review and approve.
|
||||
Explore the repository read-only first: resolve discoverable facts by looking them up, and only ask
|
||||
the human when the answer is a genuine intent or preference choice. Do not write or run code in this
|
||||
phase (edit/write are blocked except for the plan file; don't mutate state via bash either). When
|
||||
the objective is clear, draft the plan file and stop for review.
|
||||
|
||||
Right-size it, don't force structure that isn't there:
|
||||
- Default to ONE goal. Add another only when it's a genuinely separate checkpoint you'd want signed
|
||||
off on its own (it can pass or fail independently). Most objectives are 1-2 goals.
|
||||
- Subtasks are the steps inside a goal. Add them when a goal has 3+ distinct steps; skip them for a
|
||||
single-action goal. Don't pad with trivial steps.
|
||||
Right-size it:
|
||||
- Default to ONE goal. Add another only when it's a genuinely separate checkpoint that can pass or
|
||||
fail on its own. Most objectives are 1-2 goals.
|
||||
- Subtasks are the steps inside a goal; add them when a goal has 3+ distinct steps, skip otherwise.
|
||||
- Don't invent goals to look thorough. When in doubt, merge.
|
||||
|
||||
Write the whole file in this shape (markdown checkboxes, made to be skim-reviewed):
|
||||
Write the plan file in roughly this shape (it's a convention, not a schema -- the file is read
|
||||
directly by the human and a judge model, so clarity beats conformance; small deviations are fine):
|
||||
|
||||
# <short plan title>
|
||||
|
||||
@@ -65,211 +46,113 @@ Write the whole file in this shape (markdown checkboxes, made to be skim-reviewe
|
||||
1. [ ] goal: <one short imperative line>
|
||||
- subtle failure mode: <a way this could look done but isn't>
|
||||
- discriminator: <the concrete observation that tells real success from that failure>
|
||||
- verify: <optional shell command that exits 0 only when the discriminator passes; omit if not testable>
|
||||
- verify: <optional shell command that exits 0 only when the discriminator passes; omit if not
|
||||
testable. YOU run it at sign-off time and save its output as evidence; the judge only reads>
|
||||
- tasks:
|
||||
1. [ ] <subtask>
|
||||
2. [ ] <subtask>
|
||||
- evidence:
|
||||
- <leave empty now; filled at sign-off>
|
||||
2. [ ] goal: <...>
|
||||
- evidence: (empty until sign-off)
|
||||
|
||||
# Future work / out of scope
|
||||
|
||||
- <anything deliberately not in these goals>
|
||||
|
||||
## Log
|
||||
|
||||
Keep it lean and legible:
|
||||
- A goal is a checkbox line beginning "goal:"; its state is the checkbox ([ ] open, [/] active, [x]
|
||||
done, [-] cancelled). Leave goals [ ] at planning. The number is just for the human to reference.
|
||||
- subtle failure mode + discriminator are the heart of this. List the ways a "done" could look
|
||||
achieved but not be (empty/zero-count output, a silently-errored step, a gamed test, a flat/no-op
|
||||
result that dodged every trap and still showed nothing; these are examples, find the ones that fit).
|
||||
- The discriminator is the POSITIVE observation that the goal actually succeeded AND that none of
|
||||
those failure modes could have produced. It must show success happened -- the count moved the right
|
||||
way, the test really exercised the path, the metric beat noise -- not merely that a failure was
|
||||
ruled out: avoiding every failure mode is necessary, not sufficient. Name the success signal first,
|
||||
then check it isn't something a failure mode could fake. Keep it terse.
|
||||
- The discriminator is the success test, written now, in place of a vague "done": make it a concrete,
|
||||
checkable observation about a real artifact (a file, a test result, a committed diff, a metric), not
|
||||
about goals.md's own checkbox.
|
||||
- subtasks: any checkbox WITHOUT a "goal:" prefix, under "- tasks:". Use [/] for in progress and [-]
|
||||
for cancelled/impossible.
|
||||
- verify: prefer one when the discriminator is a test, build, threshold, or metric: a green check or
|
||||
a printed number beats prose. Omit it otherwise.
|
||||
- evidence stays empty at planning. You don't always know the exact artifacts up front, and that's
|
||||
fine: you fill evidence at sign-off, and a fresh read-only judge checks it then.
|
||||
Conventions:
|
||||
- A goal is a checkbox line beginning "goal:". Checkbox state: [ ] open, [/] active, [x] done,
|
||||
[-] cancelled. Leave goals [ ] at planning.
|
||||
- subtle failure mode + discriminator are the heart of this. Name the ways a "done" could look
|
||||
achieved but not be (empty output, a silently-errored step, a gamed test, a no-op that dodged
|
||||
every trap and showed nothing). The discriminator is the POSITIVE observation that success
|
||||
happened -- the count moved, the test exercised the real path, the metric beat noise -- and that
|
||||
none of the failure modes could fake. Ruling out failures is necessary, not sufficient.
|
||||
- Make the discriminator a concrete, checkable observation about a real artifact (a file, a test
|
||||
result, a committed diff, a metric), never about the plan file's own checkbox.
|
||||
- evidence stays empty at planning; you fill it at sign-off and a fresh read-only judge checks it.
|
||||
Cite durable artifacts a future reader can open: committed files, test names, git diffs. .pi/ is
|
||||
usually gitignored, so files there prove things only at judge time, not in history.
|
||||
|
||||
When the goals are drafted, present them and stop for review. Do not begin execution.`;
|
||||
|
||||
/* ─────────────────────────────────────────────────────────────────────────
|
||||
* 2. planInjection — EXEC, injected at each agent start (and after compaction)
|
||||
*
|
||||
* A late user-role message, NOT a system-prompt mutation (keeps the prefix cache
|
||||
* valid). Built from the parsed plan. MUST be byte-identical when nothing changed:
|
||||
* fixed field order, no volatile timestamps. Pass only the active goal + its open
|
||||
* subtasks + the last log line, not the whole file.
|
||||
* ──────────────────────────────────────────────────────────────────────── */
|
||||
export function planInjection(p: {
|
||||
title: string;
|
||||
activeGoal: { subject: string; discriminator: string[]; openSubtasks: string[] } | null;
|
||||
lastLogLine: string | null;
|
||||
counts: { done: number; open: number };
|
||||
}): string {
|
||||
if (!p.activeGoal) {
|
||||
// FIXME(heading): user wants the heading to show ".pi/goals.md: <title>" so the filename is explicit
|
||||
// even in the injection. Currently says "Goals (goals.md):" which is close but not the same.
|
||||
return `.pi/goals.md: ${p.title}\nNo active goal. ${p.counts.open} open, ${p.counts.done} done. Pick the next goal (set its checkbox to [/]) or run /goals.`;
|
||||
}
|
||||
const subtasks = p.activeGoal.openSubtasks.length
|
||||
? p.activeGoal.openSubtasks.map((s) => ` - [ ] ${s}`).join("\n")
|
||||
: " (no open subtasks)";
|
||||
const disc = p.activeGoal.discriminator.length ? p.activeGoal.discriminator.join("; ") : "(none set)";
|
||||
return `\
|
||||
.pi/goals.md: ${p.title}
|
||||
Active goal: ${p.activeGoal.subject}
|
||||
discriminator (the success test): ${disc}
|
||||
Open subtasks:
|
||||
${subtasks}
|
||||
Last log: ${p.lastLogLine ?? "(none yet)"}
|
||||
Progress: ${p.counts.done} done, ${p.counts.open} open.`;
|
||||
}
|
||||
|
||||
/* ─────────────────────────────────────────────────────────────────────────
|
||||
* 3. reminder — EXEC, periodic system-reminder
|
||||
*
|
||||
* The typed nudge. This is both the housekeeping and the autonomy engine — it is
|
||||
* what makes the process get followed without a hard gate. Fires after a turn that
|
||||
* left goals.md untouched while a goal is active. Keep the wording stable so it
|
||||
* doesn't thrash the cache.
|
||||
* 3. reminder — EXEC, appended to the injected plan when plan.md went
|
||||
* untouched for a whole turn while goals are open. Wording stable (cache).
|
||||
* ──────────────────────────────────────────────────────────────────────── */
|
||||
export const reminder = `\
|
||||
<system-reminder>
|
||||
Keep goals.md current as you work:
|
||||
- tasks: tick the subtasks you've finished ([/] for in progress); add any you've discovered.
|
||||
- log: append ONE short line to ## Log (append, don't rewrite earlier lines).
|
||||
- goal: when the active goal's discriminator is satisfied, fill its evidence: block in goals.md (a
|
||||
list pointing at durable artifacts), then call CompleteGoal with the goal's desc. Don't tick the
|
||||
goal [x] by hand; CompleteGoal reads the evidence, runs the check, and writes [x].
|
||||
- otherwise: keep working toward the active goal. Don't stop to ask unless you're genuinely blocked;
|
||||
if blocked, say what's blocking it.
|
||||
Keep the plan file current as you work (edit it directly):
|
||||
- tick finished subtasks ([/] in progress), add discovered ones
|
||||
- append ONE short line to ## Log (append, don't rewrite earlier lines)
|
||||
- when the active goal's discriminator is satisfied, fill its evidence: list (each item = a durable
|
||||
artifact + a verbatim quote you actually observed + a short read of it), then call CompleteGoal. Don't tick a goal [x] before CompleteGoal
|
||||
accepts; the sign-off log line is the audit trail.
|
||||
- if the file has grown long, prune finished goals (their evidence lives in git history and ## Log)
|
||||
- otherwise keep working toward the active goal; don't stop to ask unless genuinely blocked
|
||||
</system-reminder>`;
|
||||
|
||||
/* ─────────────────────────────────────────────────────────────────────────
|
||||
* 4. continuation — EXEC, the loop's "keep going" turn
|
||||
*
|
||||
* Hermes-style. A plain user-role message appended when the loop judge (5) says
|
||||
* continue. Does not mutate the system prompt, so the cache holds.
|
||||
* ──────────────────────────────────────────────────────────────────────── */
|
||||
export const continuation = `\
|
||||
Continue toward the active goal in goals.md. If its discriminator is now satisfied, fill the goal's
|
||||
evidence: block (durable artifacts, e.g. saved logs, committed diffs, files, not just claims) and
|
||||
then call CompleteGoal with the goal's desc. If you're blocked, state what's blocking it.`;
|
||||
|
||||
/* ─────────────────────────────────────────────────────────────────────────
|
||||
* 5. loopJudge — EXEC, runs after each turn to decide continue / pause
|
||||
*
|
||||
* Cheap, conservative, fail-open. Reads only the agent's last response, so it CAN
|
||||
* be fooled by an asserted "done" — that's acceptable: its worst case is a
|
||||
* premature pause, caught by you or the iteration budget. It does NOT sign goals
|
||||
* off; that's the evidence judge's job. Return strict JSON, no prose.
|
||||
* ──────────────────────────────────────────────────────────────────────── */
|
||||
export const loopJudgeSystem = `\
|
||||
You decide whether an autonomous coding agent should keep working or pause for the human.
|
||||
Be conservative: only pause when the work is plainly finished or plainly blocked. When in
|
||||
doubt, continue. You are not verifying correctness; a later read-only judge does that.
|
||||
Reply with ONLY a JSON object, no other text: {"done": boolean, "reason": "<one sentence>"}.
|
||||
Set done=true only if the agent's last message shows the active goal's discriminator is satisfied,
|
||||
or the agent says it is blocked and needs the human.`;
|
||||
|
||||
export function loopJudgeUser(p: { discriminator: string; lastResponse: string }): string {
|
||||
return `\
|
||||
Active goal discriminator (the success test): ${p.discriminator}
|
||||
|
||||
Agent's last message:
|
||||
"""
|
||||
${p.lastResponse}
|
||||
"""
|
||||
|
||||
{"done": ?, "reason": ?}`;
|
||||
}
|
||||
|
||||
/* ─────────────────────────────────────────────────────────────────────────
|
||||
* 6. completeGoalTool — SIGN-OFF, agent-side
|
||||
*
|
||||
* The description + param the agent reads on the one blessed tool, CompleteGoal.
|
||||
* This is where the agent meets the sign-off: it fills evidence and calls the
|
||||
* tool, which then runs verify + the judge (7). Kept here with the rest of the
|
||||
* model-facing text so the whole process reads top to bottom.
|
||||
* 4. completeGoal — SIGN-OFF, agent-side: the one blessed tool
|
||||
* ──────────────────────────────────────────────────────────────────────── */
|
||||
export const completeGoalDescription =
|
||||
"Sign off a goal once its discriminator is satisfied. First fill the goal's evidence: block in " +
|
||||
"goals.md: a list where each item pairs a durable artifact with a short read of it (a quoted+linked " +
|
||||
"log, a table plus how to read it, or a metric plus what it shows; quote the key lines and link the " +
|
||||
"rest, not a pasted blob or a bare claim). The read must show the success POSITIVELY happened (the " +
|
||||
"result is present, the count moved the right way, the metric beat noise), not just that a failure " +
|
||||
"was avoided; ruling out the failure modes is necessary but not sufficient. Then call this with the " +
|
||||
"goal's desc (the text after 'goal:'). Runs the goal's verify command (if any) then a read-only " +
|
||||
"subagent that inspects that evidence against the repo and the discriminator. On accept, the goal is " +
|
||||
"marked done and logged; on reject, it stays open and you get what is missing. The subagent's " +
|
||||
"reasoning is returned either way.";
|
||||
"Sign off a goal once its discriminator is satisfied. First fill the goal's evidence: list in the " +
|
||||
"plan file: each item pairs a durable artifact with a short read of it (a quoted+linked log, a " +
|
||||
"table plus how to read it, a metric plus what it shows -- not a bare claim). Quote verbatim from " +
|
||||
"output you actually observed; never reconstruct numbers from memory. If you couldn't see an " +
|
||||
"output, rerun it or write that you couldn't -- an honest gap beats a plausible fabrication. If " +
|
||||
"the goal names a verify: command, run it yourself first and save its output to a file cited in " +
|
||||
"the evidence: the judge cannot execute anything and will reject a claimed pass with no saved " +
|
||||
"output. The read must show success POSITIVELY happened, not just that failures were avoided. " +
|
||||
"Then call this with the goal's text (the line after 'goal:'; small wording drift is fine). A " +
|
||||
"fresh strictly-read-only judge inspects the LIVE WORKING TREE (uncommitted changes included; " +
|
||||
"committing first is for durability, not visibility) and returns accept or reject with what's " +
|
||||
"missing. On accept (or if the judge itself failed), a sign-off line is appended to ## Log " +
|
||||
"and the goal is ticked [x] for you; the result says if you must tick it yourself. On reject the " +
|
||||
"goal stays open.";
|
||||
|
||||
export const completeGoalParamDescription = "The goal's desc: the exact text after 'goal:' in its line.";
|
||||
export const completeGoalParamDescription = "The goal's text: the line after 'goal:' in the plan file.";
|
||||
|
||||
/* ─────────────────────────────────────────────────────────────────────────
|
||||
* 7. evidenceJudge — SIGN-OFF, judge-side; the one rigorous check
|
||||
*
|
||||
* Runs inside CompleteGoal, on a read-only pi subprocess (fresh context via
|
||||
* --no-session, so it never sees the working agent's transcript; override to a
|
||||
* different vendor for an independent cross-family check). It re-derives from the
|
||||
* repo rather than trusting the agent's transcription, and judges whether the
|
||||
* evidence satisfies the discriminator and rules out the named failure mode.
|
||||
*
|
||||
* The transport gives it read/grep/find/ls. The prompt below imposes the verdict
|
||||
* contract — the subprocess returns prose by default, so parse the VERDICT line.
|
||||
* 5. judge — SIGN-OFF, judge-side: the one rigorous check. Runs on a fresh
|
||||
* read-only pi subprocess (--no-session) so it never sees the working
|
||||
* agent's transcript. It gets the WHOLE plan file: it finds the goal,
|
||||
* reads discriminator/failure modes/evidence itself (no parser between).
|
||||
* ──────────────────────────────────────────────────────────────────────── */
|
||||
export const evidenceJudgeSystem = `\
|
||||
You are a read-only reviewer signing off a coding goal. Do not trust claims; verify.
|
||||
Use read/grep/find/ls to inspect the repository and the cited artifacts yourself. Re-read the
|
||||
files, logs, and diffs the evidence points to; if something it asserts isn't on disk, you can't
|
||||
confirm it. Judge whether the evidence shows the goal POSITIVELY succeeded -- the discriminator's
|
||||
success signal is actually present, not just that the failure modes were dodged. Avoiding every
|
||||
failure mode is necessary but not sufficient: a run can rule out each trap and still have produced
|
||||
nothing, so reject "no problems found" that lacks the positive result. Then check the named subtle
|
||||
failure modes are genuinely ruled out, not just unmentioned. If a verify command was run,
|
||||
judge whether it really tests the discriminator or could pass while the failure mode still holds; a
|
||||
tautological or skipped test is a reject.
|
||||
export const judgeSystem = `\
|
||||
You are a strictly read-only reviewer signing off a coding goal. You cannot execute anything: judge
|
||||
by reading (read/grep/find/ls). Never re-run the work or its verify command -- it may be a 10-hour
|
||||
job; the agent must bring you its saved output. Your job is evidence discipline, checked in order:
|
||||
|
||||
1. Anything here? An empty or placeholder evidence: list -> reject: "there's nothing here -- fill
|
||||
the evidence and try again."
|
||||
2. Quoted and attributed? Each item needs a source (file path / command) plus a verbatim quote of
|
||||
what was observed, plus a one-line read. A bare claim -> reject: "you didn't quote and
|
||||
attribute it."
|
||||
3. Provenance? It must be visible HOW each result was produced (the command run, where its output
|
||||
was saved). Results with no origin -> reject: "I see the results, but how did you get them?"
|
||||
4. Spot-check: open the cited files. A quote or number that doesn't match what's on disk means the
|
||||
evidence was reconstructed from memory, not observed -> reject and ask for re-observed
|
||||
evidence, even if the goal otherwise looks met.
|
||||
5. Substance, only once 1-4 hold: does the evidence show the discriminator's success signal
|
||||
POSITIVELY happened -- not just that the named failure modes were dodged; a run can rule out
|
||||
every trap and still have produced nothing. Is each subtle failure mode genuinely ruled out,
|
||||
not just unmentioned? If the goal names a verify: command, its saved output must be among the
|
||||
evidence, and the command must actually test the discriminator rather than pass tautologically.
|
||||
|
||||
Finish with exactly these two lines and nothing after:
|
||||
VERDICT: accept | reject
|
||||
missing: <empty if accept; otherwise a short list of what's needed before this can be accepted>`;
|
||||
|
||||
export function evidenceJudgeUser(p: {
|
||||
subject: string;
|
||||
discriminator: string[];
|
||||
failure_modes: string[];
|
||||
verify: string | null;
|
||||
verifyResult: { command: string; exitCode: number; outputTail: string } | null;
|
||||
evidence: string;
|
||||
paths: string[];
|
||||
}): string {
|
||||
const verifyBlock = p.verify
|
||||
? `verify command: ${p.verify}\nverify result: exit ${p.verifyResult?.exitCode ?? "n/a"}\n${p.verifyResult?.outputTail ?? ""}`
|
||||
: "verify command: none (no deterministic check for this goal)";
|
||||
return `\
|
||||
Goal: ${p.subject}
|
||||
discriminator (must be satisfied):
|
||||
${p.discriminator.map((d) => ` - ${d}`).join("\n") || " (none stated, note this)"}
|
||||
subtle failure modes (must be ruled out):
|
||||
${p.failure_modes.map((f) => ` - ${f}`).join("\n") || " (none stated)"}
|
||||
export function judgeUser(p: { goal: string; plan: string; planPath: string }): string {
|
||||
return `\
|
||||
The working agent claims this goal is complete:
|
||||
|
||||
${verifyBlock}
|
||||
goal: ${p.goal}
|
||||
|
||||
Agent's evidence:
|
||||
${p.evidence}
|
||||
Below is the full plan file (${p.planPath}). Find that goal in it (tolerate small wording drift; if
|
||||
you cannot find a matching goal at all, reject and say so). Read its discriminator, subtle failure
|
||||
modes, verify command, and evidence list from the file itself.
|
||||
|
||||
Artifacts it points to (inspect these):
|
||||
${p.paths.map((x) => ` - ${x}`).join("\n") || " (none listed, note this)"}
|
||||
--- plan file ---
|
||||
${p.plan}
|
||||
--- end plan file ---
|
||||
|
||||
Verify the evidence satisfies the discriminator and rules out the failure modes. Then give your VERDICT.`;
|
||||
Read the cited artifacts (you cannot execute anything), then give your VERDICT.`;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { appendLog } from "../src/index.js";
|
||||
|
||||
describe("appendLog (the extension's only plan-file write)", () => {
|
||||
it("creates ## Log at EOF when absent", () => {
|
||||
const out = appendLog("# Plan\n\n## Goals\n\n1. [ ] goal: x\n", "2026-07-03 10:00 signed off \"x\" (judge accept)");
|
||||
expect(out).toContain("## Log\n- 2026-07-03 10:00 signed off");
|
||||
});
|
||||
|
||||
it("appends after the last existing log line, before any following header", () => {
|
||||
const plan = "# Plan\n\n## Log\n- first\n- second\n\n# Future work\n- later\n";
|
||||
const out = appendLog(plan, "third");
|
||||
const lines = out.split("\n");
|
||||
expect(lines[lines.indexOf("- second") + 1]).toBe("- third");
|
||||
expect(out.indexOf("- third")).toBeLessThan(out.indexOf("# Future work"));
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,63 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { decideSignOff, type JudgeResult } from "../src/index.js";
|
||||
|
||||
// decideSignOff is the fail-forward invariant: judgeModel is NEVER checked pre-emptively, so a null
|
||||
// model still reaches runJudge (pi's configured default runs it), and the only producers of
|
||||
// accepted_inconclusive are the judge-error and no-VERDICT paths -- i.e. "the judge ran but failed",
|
||||
// never "no model". The judge runner is injected so these tests never spawn a real subprocess.
|
||||
describe("decideSignOff (fail-forward invariant)", () => {
|
||||
it("proceeds to runJudge even when judgeModel is null (no pre-emptive 'no model' inconclusive)", async () => {
|
||||
const runJudge = vi.fn().mockResolvedValue({ output: "VERDICT: accept\nall good" });
|
||||
const out = await decideSignOff({ goal: "x", plan: "# plan\n1. [ ] goal: x\n", judgeModel: null }, undefined, runJudge);
|
||||
expect(runJudge).toHaveBeenCalledOnce(); // reached the judge -- no pre-emptive return on null model
|
||||
expect(out.isError).toBe(false);
|
||||
expect(out.logEntry).toContain("judge accept");
|
||||
});
|
||||
|
||||
it("a judge-subprocess error yields accepted_inconclusive with a 'ran but failed' reason", async () => {
|
||||
const runJudge = vi.fn().mockResolvedValue({ output: "", error: "judge subprocess exited 1" } satisfies JudgeResult);
|
||||
const out = await decideSignOff({ goal: "x", plan: "# plan\n", judgeModel: null }, undefined, runJudge);
|
||||
expect(runJudge).toHaveBeenCalledOnce();
|
||||
expect(out.isError).toBe(false); // accepted inconclusive, not a hard error that blocks the agent
|
||||
expect(out.resultText.toLowerCase()).toContain("accepted inconclusive");
|
||||
expect(out.resultText).toContain("ran but failed"); // inconclusive means ran but failed, not "no model"
|
||||
expect(out.logEntry).toContain("ran but failed");
|
||||
expect(out.logEntry).toContain("subprocess exited 1");
|
||||
});
|
||||
|
||||
it("a judge timeout is also accepted_inconclusive (ran but failed)", async () => {
|
||||
const runJudge = vi.fn().mockResolvedValue({ output: "partial", error: "judge timed out after 600s" });
|
||||
const out = await decideSignOff({ goal: "x", plan: "# plan\n", judgeModel: null }, undefined, runJudge);
|
||||
expect(out.isError).toBe(false);
|
||||
expect(out.resultText.toLowerCase()).toContain("accepted inconclusive");
|
||||
expect(out.logEntry).toContain("ran but failed");
|
||||
expect(out.logEntry).toContain("timed out");
|
||||
expect(out.resultText).toContain("partial judge output:\npartial");
|
||||
});
|
||||
|
||||
it("no VERDICT line is accepted_inconclusive too (judge ran but didn't answer)", async () => {
|
||||
const runJudge = vi.fn().mockResolvedValue({ output: "I looked but forgot the verdict line" });
|
||||
const out = await decideSignOff({ goal: "x", plan: "# plan\n", judgeModel: null }, undefined, runJudge);
|
||||
expect(out.isError).toBe(false);
|
||||
expect(out.resultText).toContain("no VERDICT line");
|
||||
expect(out.logEntry).toContain("no VERDICT line");
|
||||
});
|
||||
|
||||
it("rejects when the judge returns VERDICT: reject", async () => {
|
||||
const runJudge = vi.fn().mockResolvedValue({ output: "VERDICT: reject\nmissing: evidence, tests" });
|
||||
const out = await decideSignOff({ goal: "x", plan: "# plan\n", judgeModel: "openrouter/claude" }, undefined, runJudge);
|
||||
expect(out.isError).toBe(true);
|
||||
expect(out.resultText).toContain("REJECTED");
|
||||
expect(out.resultText).toContain("evidence, tests");
|
||||
expect(out.logEntry).toContain("reject");
|
||||
});
|
||||
|
||||
it("writes nothing when aborted after the judge ran", async () => {
|
||||
const runJudge = vi.fn().mockResolvedValue({ output: "VERDICT: accept" });
|
||||
const ctrl = new AbortController();
|
||||
ctrl.abort();
|
||||
const out = await decideSignOff({ goal: "x", plan: "# plan\n", judgeModel: null }, ctrl.signal, runJudge);
|
||||
expect(out.logEntry).toBeNull();
|
||||
expect(out.isError).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,33 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { buildJudgeArgs } from "../src/index.js";
|
||||
|
||||
describe("buildJudgeArgs", () => {
|
||||
it("omits --model when judgeModel is null (pi uses its configured default; never a pre-emptive 'no model' failure)", () => {
|
||||
const args = buildJudgeArgs(null);
|
||||
expect(args).not.toContain("--model");
|
||||
// an empty --model "" would make every sign-off silently inconclusive -- guard against it
|
||||
const i = args.indexOf("--model");
|
||||
expect(i).toBe(-1);
|
||||
});
|
||||
|
||||
it("includes --model <ref> when an explicit/session model is set", () => {
|
||||
const args = buildJudgeArgs("openrouter/~anthropic/claude-haiku-latest");
|
||||
const i = args.indexOf("--model");
|
||||
expect(i).not.toBe(-1);
|
||||
expect(args[i + 1]).toBe("openrouter/~anthropic/claude-haiku-latest");
|
||||
});
|
||||
|
||||
it("always sets --no-session, --no-extensions, the read-only tool allowlist, and edit/write exclusion", () => {
|
||||
for (const m of [null, "some/model"]) {
|
||||
const args = buildJudgeArgs(m);
|
||||
expect(args).toContain("--no-session");
|
||||
expect(args).toContain("--no-extensions"); // a broken global extension must not take down sign-offs
|
||||
expect(args).toContain("--tools");
|
||||
expect(args.some((a) => a.startsWith("read,grep,find,ls"))).toBe(true);
|
||||
// no bash: the judge must never be able to execute (or re-run a 10-hour verify) or mutate
|
||||
expect(args.some((a) => a.includes("bash"))).toBe(false);
|
||||
expect(args).toContain("--exclude-tools");
|
||||
expect(args.some((a) => a.includes("edit") && a.includes("write"))).toBe(true);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -1,171 +0,0 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { appendLog, counts, findGoal, parse, recordSignOff, setGoalStatus } from "../src/plan-file.js";
|
||||
|
||||
const SAMPLE = `# papers audit
|
||||
|
||||
Clean up steering/ metadata and kill empty dirs. Keep it read-only until I approve.
|
||||
|
||||
## Goals
|
||||
|
||||
1. [/] goal: Implement cache layer
|
||||
- discriminator: hit-rate > 0.8 in load-test.log (a bypass reads ~0)
|
||||
- subtle failure mode: cache silently bypassed, latency ok by luck
|
||||
- verify: pytest tests/cache -q
|
||||
- tasks:
|
||||
1. [x] wire cache client
|
||||
2. [/] eviction policy
|
||||
3. ~~[ ]~~ distributed cache, out of scope
|
||||
- evidence:
|
||||
- > load-test.log: p95=41ms
|
||||
- > hit-rate 0.93 (not bypassed)
|
||||
2. [ ] goal: Document the API
|
||||
- discriminator: every public fn has a docstring; sphinx warns on none
|
||||
- subtle failure mode: docstrings exist but are stale
|
||||
|
||||
# Future work / out of scope
|
||||
|
||||
- distributed cache
|
||||
|
||||
## Log
|
||||
- 2026-06-15 14:02 cache client wired; eviction next
|
||||
`;
|
||||
|
||||
/** Multiset line diff: lines b adds vs removes vs a (order-insensitive, so insertions score added:1). */
|
||||
function lineDelta(a: string, b: string): { added: number; removed: number } {
|
||||
const count = (s: string) => {
|
||||
const m = new Map<string, number>();
|
||||
for (const l of s.split("\n")) m.set(l, (m.get(l) ?? 0) + 1);
|
||||
return m;
|
||||
};
|
||||
const ma = count(a);
|
||||
const mb = count(b);
|
||||
let added = 0;
|
||||
let removed = 0;
|
||||
for (const k of new Set([...ma.keys(), ...mb.keys()])) {
|
||||
const d = (mb.get(k) ?? 0) - (ma.get(k) ?? 0);
|
||||
if (d > 0) added += d;
|
||||
else if (d < 0) removed += -d;
|
||||
}
|
||||
return { added, removed };
|
||||
}
|
||||
|
||||
describe("parse", () => {
|
||||
const doc = parse(SAMPLE);
|
||||
|
||||
it("reads the title and both goals (matched by subject)", () => {
|
||||
expect(doc.title).toBe("papers audit");
|
||||
expect(doc.goals.map((g) => g.subject)).toEqual(["Implement cache layer", "Document the API"]);
|
||||
});
|
||||
|
||||
it("reads goal status from the checkbox", () => {
|
||||
expect(findGoal(doc, "Implement cache layer")?.status).toBe("active"); // [/]
|
||||
expect(findGoal(doc, "Document the API")?.status).toBe("open"); // [ ]
|
||||
});
|
||||
|
||||
it("reads discriminator, subtle failure mode, and verify as separate fields", () => {
|
||||
const g = findGoal(doc, "Implement cache layer");
|
||||
expect(g?.discriminator).toEqual(["hit-rate > 0.8 in load-test.log (a bypass reads ~0)"]);
|
||||
expect(g?.failure_modes).toEqual(["cache silently bypassed, latency ok by luck"]);
|
||||
expect(g?.verify).toBe("pytest tests/cache -q");
|
||||
});
|
||||
|
||||
it("reads subtasks with their checkbox state, strikethrough as cancelled", () => {
|
||||
const g = findGoal(doc, "Implement cache layer");
|
||||
expect(g?.subtasks).toEqual([
|
||||
{ text: "wire cache client", status: "done" },
|
||||
{ text: "eviction policy", status: "active" },
|
||||
{ text: "distributed cache, out of scope", status: "cancelled" },
|
||||
]);
|
||||
});
|
||||
|
||||
it("reads the evidence block separate from the other lists", () => {
|
||||
const g = findGoal(doc, "Implement cache layer");
|
||||
expect(g?.evidence).toEqual(["> load-test.log: p95=41ms", "> hit-rate 0.93 (not bypassed)"]);
|
||||
expect(findGoal(doc, "Document the API")?.evidence).toEqual([]); // a goal with no evidence parses to []
|
||||
});
|
||||
|
||||
it("keeps a multi-line evidence item together (quote + interpretation)", () => {
|
||||
const doc2 = parse(
|
||||
`# x\n\n## Goals\n\n1. [ ] goal: G\n - discriminator: report has non-zero counts\n - evidence:\n - > report.txt: counts 52 -> 4\n remaining 4 = index + 3 notes\n almost certain the discriminator passes\n - > second item, single line\n`,
|
||||
);
|
||||
expect(findGoal(doc2, "G")?.evidence).toEqual([
|
||||
"> report.txt: counts 52 -> 4\nremaining 4 = index + 3 notes\nalmost certain the discriminator passes",
|
||||
"> second item, single line",
|
||||
]);
|
||||
});
|
||||
|
||||
it("reads the log verbatim and counts by status", () => {
|
||||
expect(doc.log).toEqual(["- 2026-06-15 14:02 cache client wired; eviction next"]);
|
||||
expect(counts(doc)).toEqual({ done: 0, open: 1, active: 1 });
|
||||
});
|
||||
|
||||
it("ignores the Future work section, does not read it as goals or log", () => {
|
||||
expect(doc.goals).toHaveLength(2);
|
||||
expect(doc.log).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("the two CompleteGoal writes (minimal diff)", () => {
|
||||
it("setGoalStatus replaces exactly one line, scoped to the right goal", () => {
|
||||
const next = setGoalStatus(SAMPLE, "Implement cache layer", "done");
|
||||
expect(lineDelta(SAMPLE, next)).toEqual({ added: 1, removed: 1 });
|
||||
expect(findGoal(parse(next), "Implement cache layer")?.status).toBe("done");
|
||||
expect(findGoal(parse(next), "Document the API")?.status).toBe("open"); // untouched
|
||||
});
|
||||
|
||||
it("setGoalStatus keeps the number and goal: prefix, flips only the checkbox", () => {
|
||||
expect(setGoalStatus(SAMPLE, "Implement cache layer", "done")).toContain("1. [x] goal: Implement cache layer");
|
||||
expect(setGoalStatus(SAMPLE, "Document the API", "cancelled")).toContain("2. [-] goal: Document the API");
|
||||
});
|
||||
|
||||
it("setGoalStatus throws on an unknown subject", () => {
|
||||
expect(() => setGoalStatus(SAMPLE, "no such goal", "done")).toThrow();
|
||||
});
|
||||
|
||||
it("appendLog adds exactly one line under ## Log", () => {
|
||||
const next = appendLog(SAMPLE, "2026-06-15 15:00 eviction done");
|
||||
expect(lineDelta(SAMPLE, next)).toEqual({ added: 1, removed: 0 });
|
||||
expect(parse(next).log).toEqual([
|
||||
"- 2026-06-15 14:02 cache client wired; eviction next",
|
||||
"- 2026-06-15 15:00 eviction done",
|
||||
]);
|
||||
});
|
||||
|
||||
it("appendLog creates the section when absent", () => {
|
||||
const noLog = "# x\n\n## Goals\n\n1. [ ] goal: y\n - discriminator: z\n";
|
||||
expect(parse(appendLog(noLog, "first entry")).log).toEqual(["- first entry"]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("recordSignOff (CompleteGoal's pure record logic)", () => {
|
||||
const WHEN = "2026-06-15 16:00";
|
||||
|
||||
it("accept flips status:done and logs a sign-off line", () => {
|
||||
const r = recordSignOff(SAMPLE, "Implement cache layer", WHEN, { kind: "accepted" });
|
||||
expect(r.isError).toBe(false);
|
||||
const doc = parse(r.content);
|
||||
expect(findGoal(doc, "Implement cache layer")?.status).toBe("done");
|
||||
expect(doc.log.at(-1)).toBe(`- ${WHEN} signed off "Implement cache layer" (judge accept)`);
|
||||
});
|
||||
|
||||
it("verify_failed only logs a reject line, status stays active", () => {
|
||||
const r = recordSignOff(SAMPLE, "Implement cache layer", WHEN, { kind: "verify_failed", exitCode: 1, outputTail: "boom" });
|
||||
expect(r.isError).toBe(true);
|
||||
const doc = parse(r.content);
|
||||
expect(findGoal(doc, "Implement cache layer")?.status).toBe("active"); // NOT marked done
|
||||
expect(doc.log.at(-1)).toBe(`- ${WHEN} reject "Implement cache layer": verify exit 1`);
|
||||
});
|
||||
|
||||
it("rejected logs the (one-lined) missing reason, status stays", () => {
|
||||
const r = recordSignOff(SAMPLE, "Implement cache layer", WHEN, { kind: "rejected", missing: "no\nsaved\nbench log" });
|
||||
expect(r.isError).toBe(true);
|
||||
expect(findGoal(parse(r.content), "Implement cache layer")?.status).toBe("active");
|
||||
expect(parse(r.content).log.at(-1)).toBe(`- ${WHEN} reject "Implement cache layer": no saved bench log`);
|
||||
});
|
||||
|
||||
it("unknown goal returns an error and does not touch the file", () => {
|
||||
const r = recordSignOff(SAMPLE, "nope", WHEN, { kind: "accepted" });
|
||||
expect(r.isError).toBe(true);
|
||||
expect(r.content).toBe(SAMPLE);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,32 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { tickGoal } from "../src/index.js";
|
||||
|
||||
const plan = `# Plan
|
||||
|
||||
## Goals
|
||||
|
||||
1. [/] goal: Implement the cache layer
|
||||
- tasks:
|
||||
1. [x] wire client
|
||||
2. [ ] goal: Ship the docs
|
||||
|
||||
## Log
|
||||
`;
|
||||
|
||||
describe("tickGoal (sign-off ticks the goal; agent only ticks on wording drift)", () => {
|
||||
it("ticks the exact-matching goal line, case-insensitive, leaving subtasks alone", () => {
|
||||
const out = tickGoal(plan, "implement the CACHE layer");
|
||||
expect(out).toContain("1. [x] goal: Implement the cache layer");
|
||||
expect(out).toContain("1. [x] wire client"); // subtask untouched (was already x)
|
||||
expect(out).toContain("2. [ ] goal: Ship the docs"); // other goal untouched
|
||||
});
|
||||
|
||||
it("returns null on wording drift (fuzzy matching is the judge's job, not TypeScript's)", () => {
|
||||
expect(tickGoal(plan, "Implement caching")).toBeNull();
|
||||
});
|
||||
|
||||
it("returns null when the subject matches more than one goal line", () => {
|
||||
const dup = `${plan}3. [ ] goal: Ship the docs\n`;
|
||||
expect(tickGoal(dup, "Ship the docs")).toBeNull();
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user