You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@magpie.apache.org by po...@apache.org on 2026/07/03 16:18:35 UTC
(magpie) branch main updated: Document spec-loop runner contract and harness support (#700)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/magpie.git
The following commit(s) were added to refs/heads/main by this push:
new 62c3965d Document spec-loop runner contract and harness support (#700)
62c3965d is described below
commit 62c3965d2c30e617b7237bf5a69d47970e3036ef
Author: Justin Mclean <ju...@classsoftware.com>
AuthorDate: Sat Jul 4 02:18:29 2026 +1000
Document spec-loop runner contract and harness support (#700)
---
docs/spec-driven-development.md | 23 +++--
tools/spec-loop/AGENTS.md | 5 +-
tools/spec-loop/PROMPT_update.md | 6 +-
tools/spec-loop/README.md | 2 +-
tools/spec-loop/loop.sh | 10 +-
tools/spec-loop/specs/README.md | 1 +
tools/spec-loop/specs/overview.md | 1 +
tools/spec-loop/specs/spec-loop-runner.md | 154 ++++++++++++++++++++++++++++++
8 files changed, 181 insertions(+), 21 deletions(-)
diff --git a/docs/spec-driven-development.md b/docs/spec-driven-development.md
index 2b6b172f..91e91a06 100644
--- a/docs/spec-driven-development.md
+++ b/docs/spec-driven-development.md
@@ -57,7 +57,7 @@ Phases two and three are the loop, swapping prompts:
|---|---|---|---|
| **plan** | `loop.sh plan` | Compares specs against the code; rewrites the plan as prioritised work items. | No |
| **build** | `loop.sh` | Implements the single highest-priority work item on its own branch; validates; commits there. | Yes, on a work-item branch |
-| **update** | `loop.sh update` | Inverse of plan: finds functionality the code has but the specs don't, and brings the specs back in sync. | Yes, on `spec/sync-specs` |
+| **update** | `loop.sh update` | Inverse of plan: finds functionality the code has but the specs don't, and brings the specs back in sync. | Yes, on `sync-specs-<timestamp>` |
| **consolidate** | `loop.sh consolidate` | Shrinks the plan when it grows too long, without dropping planned work. | Yes, the plan only |
Every beat loads the same operational context —
@@ -90,7 +90,7 @@ PR**. Before plan/build iterations, the runner snapshots open PRs so the
plan beat does not add work already in flight and the build beat skips
planned items that an open PR already covers. The build beat returns to
the integration base, then carves out a
-`spec/<slug>` branch for the single work item it is about to implement. It
+bare `<slug>` branch for the single work item it is about to implement. It
never commits feature work to the base branch, and `loop.sh` stops if it
detects that happening. The result of a run is a fan of independent
branches, each carrying exactly one change — each independently
@@ -109,8 +109,8 @@ confirmation. The loop honours that: it ends every iteration at a **local
commit** and prints the exact commands for the human to run:
```bash
-git push -u origin spec/<slug>
-gh pr create --web --base main --head spec/<slug> \
+git push -u origin <slug>
+gh pr create --web --base main --head <slug> \
--title "<subject>" --body-file <prepared-body>
```
@@ -165,7 +165,7 @@ importance:
`--disallowedTools "Bash(git push *)" "Bash(gh *)"` when using the
Claude harness.
3. **Structural containment.** Every iteration works on its own
- `spec/<slug>` branch, the loop guards against commits landing on the
+ bare `<slug>` branch, the loop guards against commits landing on the
base branch, and the prompts forbid push/PR. The human-in-the-loop
gate is not removed — it is *relocated* from per-tool-call to the
push / PR / merge boundary, where the human reviews a finished branch.
@@ -190,8 +190,10 @@ or corrects the specs (a `proposed` area that now has a shipped skill
becomes `experimental`; a drifted *Where it lives* is corrected; genuinely
new functionality gets a new topic-named spec). It edits **only** the spec
directory — it documents reality, it doesn't change it — and lands as one
-reviewable `spec/sync-specs` PR. Run it after a batch of normal PRs merges,
-or on a schedule.
+reviewable `sync-specs-<timestamp>` PR. The runner owns
+`tools/spec-loop/.last-sync`: it passes incremental-scope guidance into
+the prompt, then amends or creates the marker commit after the sync
+finishes. Run it after a batch of normal PRs merges, or on a schedule.
## Layout
@@ -211,7 +213,8 @@ tools/spec-loop/
├── security-issue-lifecycle.md privacy-llm-gate.md
├── agent-isolation-sandbox.md cve-tooling.md
├── adoption-and-setup.md adapters.md
- └── meta-and-quality-tooling.md
+ ├── meta-and-quality-tooling.md spec-loop-runner.md
+ └── security-reporting.md reviewer-routing.md
```
## Quick start
@@ -226,8 +229,8 @@ $EDITOR tools/spec-loop/IMPLEMENTATION_PLAN.md
# 3. Review the branch it produced, then push + open the PR yourself.
git log --oneline -1
-git push -u origin spec/<slug>
-gh pr create --web --base main --head spec/<slug> --title "…" --body-file …
+git push -u origin <slug>
+gh pr create --web --base main --head <slug> --title "…" --body-file …
# Later: someone merged skills outside the loop — resync the specs.
./tools/spec-loop/loop.sh update 1
diff --git a/tools/spec-loop/AGENTS.md b/tools/spec-loop/AGENTS.md
index 03d96c0a..b34ba318 100644
--- a/tools/spec-loop/AGENTS.md
+++ b/tools/spec-loop/AGENTS.md
@@ -58,8 +58,9 @@ commit.
not sibling specs and not `IMPLEMENTATION_PLAN.md` (avoids cross-branch
conflicts; the plan is reconciled by a later `plan` pass).
- The **`update`** beat (specs fell behind code others contributed)
- branches `sync-specs` and edits `specs/` **only** — it documents
- reality, it never changes a skill, tool, or doc outside the spec dir.
+ branches `sync-specs-<timestamp>` and edits `specs/` **only** — it
+ documents reality, it never changes a skill, tool, or doc outside the
+ spec dir. The runner, not the prompt, owns `.last-sync`.
- The runner feeds each iteration **both** the open PRs and the local
work-item branches as in-flight work. Because the loop never pushes, a
built-but-un-pushed item exists only as a local branch with no PR, so the
diff --git a/tools/spec-loop/PROMPT_update.md b/tools/spec-loop/PROMPT_update.md
index 5ea7d317..c0f5b21e 100644
--- a/tools/spec-loop/PROMPT_update.md
+++ b/tools/spec-loop/PROMPT_update.md
@@ -71,9 +71,9 @@ gh pr create --web --base <integration-base> --head <sync-branch> \
Rules:
- **Edit specs only.** This beat changes `tools/spec-loop/specs/` and
- the indexes. It must NOT change any skill, tool, or doc outside the
- spec directory — it documents reality, it does not alter it. The
- marker file `.last-sync` is owned by `loop.sh`; do not touch it.
+ the spec indexes. It must NOT change any skill, tool, or doc outside the
+ spec directory — it documents reality, it does not alter it. The marker
+ file `.last-sync` is owned by `loop.sh`; do not touch it.
- Confirm with a code search before recording something as present or
absent. Do not invent behaviour the code does not have.
- Keep the RFCs untouched — they are a separate governance layer.
diff --git a/tools/spec-loop/README.md b/tools/spec-loop/README.md
index a95eaf3a..ea7c566c 100644
--- a/tools/spec-loop/README.md
+++ b/tools/spec-loop/README.md
@@ -100,7 +100,7 @@ See the SECURITY notes in [`loop.sh`](loop.sh).
item.
- **update** — the inverse of plan: scans the code for functionality not
yet described by a spec (someone contributed it the normal way) and
- brings the specs back in sync, on a `sync-specs` branch.
+ brings the specs back in sync, on a `sync-specs-<timestamp>` branch.
- **consolidate** — shrinks the plan without losing planned work (build
auto-switches to this when the plan grows past ~500 lines).
diff --git a/tools/spec-loop/loop.sh b/tools/spec-loop/loop.sh
index ba40d998..5a36809e 100755
--- a/tools/spec-loop/loop.sh
+++ b/tools/spec-loop/loop.sh
@@ -256,7 +256,7 @@ open_pr_context() {
# BASE SHA the specs were last synced against. Read from the working tree
# (so a half-finished sync counts); fall back to the control branch via
# `git show` for the case where BASE is already checked out. The update
-# prompt overwrites this file at the end of every successful sync.
+# runner amends or creates this file at the end of every successful sync.
update_scope_context() {
echo ""
echo "## Incremental scope — only re-inspect what changed since the last sync"
@@ -269,8 +269,8 @@ update_scope_context() {
prev="$(git show "$TOOLING_REF:$marker" 2>/dev/null | tr -d '[:space:]')"
fi
if [ -z "$prev" ]; then
- echo "No \`$marker\` recorded — do a full inventory, then write the new"
- echo "BASE SHA to that file as part of the sync commit."
+ echo "No \`$marker\` recorded — do a full inventory. The runner will"
+ echo "write the new BASE SHA after the sync commit finishes."
return 0
fi
echo "The last sync marker (\`$marker\`) is \`$prev\`. The current BASE HEAD"
@@ -287,8 +287,8 @@ update_scope_context() {
echo "subjects appear in the diff. If the diff is empty there is nothing to"
echo "sync; exit without creating a branch or commit."
echo ""
- echo "When you finish the sync, overwrite \`$marker\` with \`$BASE_HEAD\` and"
- echo "include it in the sync commit so the next run picks up from here."
+ echo "Do not edit \`$marker\`; the runner owns it and will amend or create"
+ echo "the marker commit after the sync finishes."
}
local_branch_context() {
diff --git a/tools/spec-loop/specs/README.md b/tools/spec-loop/specs/README.md
index 4792e142..b6ce3929 100644
--- a/tools/spec-loop/specs/README.md
+++ b/tools/spec-loop/specs/README.md
@@ -39,6 +39,7 @@ Start with [`overview.md`](overview.md), then:
[`adapters.md`](adapters.md),
[`project-agnosticism.md`](project-agnosticism.md),
[`meta-and-quality-tooling.md`](meta-and-quality-tooling.md),
+ [`spec-loop-runner.md`](spec-loop-runner.md),
[`security-reporting.md`](security-reporting.md),
[`reviewer-routing.md`](reviewer-routing.md),
[`skill-reconciler.md`](skill-reconciler.md).
diff --git a/tools/spec-loop/specs/overview.md b/tools/spec-loop/specs/overview.md
index a3e7fb62..4a00a6b0 100644
--- a/tools/spec-loop/specs/overview.md
+++ b/tools/spec-loop/specs/overview.md
@@ -54,6 +54,7 @@ Each mode is an independently toggleable set of skills. Maturity mirrors
| Adapters (Gmail / PonyMail / Jira / GitHub / mail-source / forwarder-relay / mail-archive / github-body-field / github-rollup) | [adapters.md](adapters.md) |
| Project-agnosticism (de-ASF coupling) | [project-agnosticism.md](project-agnosticism.md) |
| Meta & quality tooling | [meta-and-quality-tooling.md](meta-and-quality-tooling.md) |
+| Spec-loop runner (plan/build/update/consolidate + headless harnesses) | [spec-loop-runner.md](spec-loop-runner.md) |
| Reviewer routing (experimental, Agentic Triage) | [reviewer-routing.md](reviewer-routing.md) |
| Cross-project skill reconciler (experimental, infra) | [skill-reconciler.md](skill-reconciler.md) |
diff --git a/tools/spec-loop/specs/spec-loop-runner.md b/tools/spec-loop/specs/spec-loop-runner.md
new file mode 100644
index 00000000..6662a6f3
--- /dev/null
+++ b/tools/spec-loop/specs/spec-loop-runner.md
@@ -0,0 +1,154 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+
+---
+title: Spec-loop runner
+status: experimental
+kind: feature
+mode: infra
+source: >
+ tools/spec-loop/README.md, tools/spec-loop/loop.sh,
+ tools/spec-loop/PROMPT_plan.md, tools/spec-loop/PROMPT_build.md,
+ tools/spec-loop/PROMPT_update.md, tools/spec-loop/PROMPT_consolidate.md,
+ and docs/spec-driven-development.md.
+acceptance:
+ - The loop supports plan, build, update, and consolidate beats with one
+ fresh agent context per iteration.
+ - Build and update beats never commit to the integration base; they create
+ reviewable local branches and stop before push or PR creation.
+ - Open PRs and local work-item branches are passed into plan/build prompts
+ as in-flight work so the loop does not duplicate already-built items.
+ - The headless-agent harness contract is explicit for Claude Code, Codex,
+ Cursor, Gemini CLI, and OpenCode.
+ - The update marker `tools/spec-loop/.last-sync` is owned by the runner,
+ not by the prompt.
+---
+
+# Spec-loop runner
+
+## What it does
+
+The spec-loop runner is the framework's local, review-first development
+loop. It runs a headless agent CLI against fixed prompts, one iteration at
+a time, to compare specs with the working tree, produce planned changes,
+back-fill specs after normal contributions, or consolidate a long
+implementation plan.
+
+The loop is not an autonomous merge system. Its boundary is local branch
+creation: it may edit, validate, and commit locally, but it never pushes
+and never opens a pull request.
+
+## Where it lives
+
+- `tools/spec-loop/loop.sh` — Bash runner for the four beats.
+- `tools/spec-loop/PROMPT_plan.md` — gap-analysis prompt that rewrites
+ `IMPLEMENTATION_PLAN.md`.
+- `tools/spec-loop/PROMPT_build.md` — implementation prompt for exactly
+ one work item.
+- `tools/spec-loop/PROMPT_update.md` — spec back-fill prompt for
+ functionality that landed outside the loop.
+- `tools/spec-loop/PROMPT_consolidate.md` — plan-size reduction prompt.
+- `tools/spec-loop/AGENTS.md` — loop-specific operational context:
+ repository map, validation commands, branch rules, hard limits, and
+ commit rules.
+- `tools/spec-loop/README.md` — operator quickstart.
+- `docs/spec-driven-development.md` — full explanation of the loop's
+ posture and lifecycle.
+
+## Behaviour & contract
+
+- **Four beats, one mechanism.** `plan` compares specs to code and updates
+ the implementation plan without committing. `build` implements one
+ uncovered work item. `update` compares code to specs and back-fills the
+ spec directory. `consolidate` shrinks the implementation plan without
+ dropping planned work.
+- **One work item, one branch, one PR.** Build work creates a bare
+ `<slug>` branch off `SPEC_LOOP_BASE` and commits exactly one work item
+ there. Update work creates a uniquely named `sync-specs-<timestamp>`
+ branch. Consolidate commits only the plan file on the control branch.
+- **No remote state changes.** The runner and prompts forbid `git push`
+ and `gh pr create`; each successful build/update beat prints the
+ human-run push and `gh pr create --web` commands instead.
+- **In-flight duplicate guard.** Before each relevant iteration, the
+ runner appends open PR context and local work-item branch context to the
+ prompt. Plan/build treat both as in-flight work. This matters because a
+ loop-built branch may be local-only and invisible to GitHub.
+- **Control branch vs integration base.** `TOOLING_REF` is captured before
+ checkout. When the integration base does not carry the spec-loop files,
+ the runner tells the agent to read prompts/specs/plan from the control
+ branch with `git show`, while implementing product changes on the work
+ branch.
+- **Update marker ownership.** `tools/spec-loop/.last-sync` records the
+ base SHA last synced by the update beat. The runner reads it to append
+ incremental-scope guidance, then amends or creates a marker commit after
+ the agent finishes. Prompts must not instruct the agent to edit the
+ marker.
+- **Plan-size hysteresis.** Build mode switches to one consolidate pass
+ when `IMPLEMENTATION_PLAN.md` exceeds `SPEC_LOOP_PLAN_MAX`, then builds
+ even if planned work alone keeps the file over the threshold. The latch
+ resets once the plan drops below the threshold.
+
+## Headless harness contract
+
+Every supported harness must provide the same loop-level behaviour: run
+one non-interactive agent iteration in the repository root, receive the
+assembled prompt, allow local edits/validation/commits under the external
+sandbox, and stop without pushing or opening a PR.
+
+| Harness | Prompt transport | Cwd contract | Unattended flag | Model override | Output mode | Extra denial |
+|---|---|---|---|---|---|---|
+| Claude Code | stdin to `claude -p` | launched from repo root | `--dangerously-skip-permissions` | `--model` | `--output-format` | `--disallowedTools` denies push and `gh` |
+| Codex | stdin to `codex exec -` | `--cd "$ROOT"` | `--dangerously-bypass-approvals-and-sandbox` | `--model` | `--json` for stream JSON | external sandbox and exec policy |
+| Cursor | positional prompt to `cursor agent --print` or `cursor-agent --print` | `--workspace "$ROOT"` | `--force --trust` | `--model` | `--output-format` | external sandbox and Cursor policy |
+| Gemini CLI | `--prompt "<prompt>"` | launched from repo root | `--yolo` | `--model` | default CLI output | external sandbox and Gemini policy |
+| OpenCode | positional prompt to `opencode run` | launched from repo root | `--auto` | `--model` | `--format json` for stream JSON | external sandbox and OpenCode policy |
+
+`SPEC_LOOP_AGENT` chooses the CLI. `SPEC_LOOP_HARNESS` chooses the
+invocation convention and defaults from the agent basename. Adding a new
+harness means extending this matrix, documenting the safety boundary, and
+updating `loop.sh` in the same change.
+
+## Out of scope
+
+- Auto-pushing, auto-opening PRs, auto-merging, or changing remote state.
+- Replacing the project sandbox or adding new filesystem/network
+ permissions.
+- Teaching individual skills how to do their domain work; those contracts
+ live in the corresponding functional specs.
+- Editing `docs/rfcs/`; RFCs remain the separate governance layer.
+
+## Acceptance criteria
+
+1. `loop.sh` accepts `plan`, `build`, `update`, and `consolidate`, plus a
+ bare numeric build count, and rejects unknown modes or non-numeric
+ iteration counts.
+2. Build/update iterations check out `SPEC_LOOP_BASE`, create a non-base
+ branch, and stop if a commit lands on the base.
+3. Plan/build prompts receive both open PR context and local work-item
+ branch context.
+4. Update prompts receive incremental-scope guidance from `.last-sync`
+ when present, but the runner remains the only writer of `.last-sync`.
+5. Claude Code, Codex, Cursor, Gemini CLI, and OpenCode are documented in
+ the headless harness matrix and implemented in `loop.sh`.
+6. Operator docs and prompts agree on branch naming: build uses bare
+ `<slug>` branches; update uses `sync-specs-<timestamp>` branches.
+7. Security docs describe unattended agent flags as harness-specific
+ agent-level bypasses that must run under the external sandbox.
+
+## Validation
+
+```bash
+bash -n tools/spec-loop/loop.sh
+shellcheck tools/spec-loop/loop.sh
+uv run --project tools/spec-validator --group dev spec-validate
+```
+
+## Known gaps
+
+- There is no deterministic fixture test for prompt assembly, harness
+ command construction, or `.last-sync` marker amendment.
+- The non-Claude harnesses rely on external policy/config plus the OS
+ sandbox for push/PR denial; only Claude has a per-invocation hard-deny
+ flag in the current runner.
+- The update beat's incremental-scope mapping is path-based; it does not
+ yet map changed files to likely spec topics with a deterministic helper.