[PR] test(spec-loop): add runner fixture coverage [magpie]

["justinmclean (via GitHub)" <gi...@apache.org>]

justinmclean opened a new pull request, #735:
URL: https://github.com/apache/magpie/pull/735

   ## Summary
   
   Add deterministic fixture coverage for the spec-loop runner
   
   Extracts the spec-loop runner's deterministic logic into a testable library and adds a fixture suite that exercises it, closing the runner's long-standing "no fixture test" gap.
   
   - **New `tools/spec-loop/lib.sh`**: pulls prompt assembly, harness command rendering, agent launch, and `.last-sync` marker helpers out of `loop.sh` into standalone, sourceable functions. `loop.sh` now sources them (net ~85 fewer lines there), so the same code paths the runner uses are the ones under test.
   - **New `tools/spec-loop/tests/test_runner_fixtures.sh`**: deterministic Bash fixtures covering prompt assembly per beat (build vs update scoping), harness argv construction across agents, and the `.last-sync` marker write/branch-name helpers. No network, no agent, no `uv`.
   - **`spec-validator` wiring**: adds check #10, `validate_spec_loop_runner_fixtures`, which runs the fixture suite via `bash` when the file is present and surfaces any non-zero exit as a validation violation. `spec-validate` now executes the runner tests as part of normal validation, with a pytest added for the new validator branch.
   - **Spec update** (`specs/spec-loop-runner.md`): records `lib.sh` and the test file under *Where it lives* and *source*, adds them to the Validation block (`shellcheck`), and removes the now-resolved *Known gap* about missing fixture tests.
   
   The runner previously had no deterministic test for prompt assembly, harness command construction, or marker handling, so refactors risked silently breaking how prompts and agent invocations are built. Splitting the logic into `lib.sh` makes it unit-testable, and routing the suite through `spec-validate` keeps it running automatically rather than by hand.
   
   The bash suite passes (`spec-loop runner fixtures: OK`); the pytest covers the new validator hook.
   
   Generated-by: Claude (Opus 4.7)
   
   ## Type of change
   
   - [ ] Skill change (`.claude/skills/<name>/`) — eval fixtures updated below
   - [ ] Tool / bridge contract (`tools/<system>/*.md`)
   - [X] Python package (`tools/*/` with `pyproject.toml`)
   - [ ] Groovy reference impl
   - [ ] Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
   - [ ] Documentation (`docs/`, `README.md`, `CONTRIBUTING.md`)
   - [ ] Project template (`projects/_template/`)
   - [ ] CI / dev loop (`prek`, workflows, validators)
   - [ ] Other:
   
   ## Test plan
   
   - [ ] `prek run --all-files` passes
   - [X] For Python packages touched: `uv run pytest` / `ruff check` / `mypy` passes
   - [ ] For Groovy bridges touched: command-line invocation tested end-to-end
   - [ ] For skill changes: eval suite passes for the affected skill
         (`PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/<skill>/`)
   - [ ] For skill *behaviour* changes: a new or updated eval fixture is included in this PR
         (a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)
   - [X] Other:  bash suite passes
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] test(spec-loop): add runner fixture coverage [magpie]

["potiuk (via GitHub)" <gi...@apache.org>]

potiuk merged PR #735:
URL: https://github.com/apache/magpie/pull/735


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org