[PR] fix(skill-evals): give template-less eval steps a neutral user prompt [magpie]

["dpol1 (via GitHub)" <gi...@apache.org>]

dpol1 opened a new pull request, #1524:
URL: https://github.com/apache/magpie/pull/1524

   <!-- SPDX-License-Identifier: Apache-2.0
        https://www.apache.org/licenses/LICENSE-2.0 -->
   
   ## Summary
   
   - The eval runner's default user prompt, used by every step without a `user-prompt-template.md`, was the `security-issue-import` Step 2a template:
     a tracker corpus, a reporter roster, the incoming report, and "Apply the semantic sweep and reporter-identity check. Return JSON only."
     33 other steps (the `release-*` steps, `reviewer-routing`, `non-asf-profile-smoke`) sent that framing to the model, with an empty corpus and a `(none)` roster, next to a system prompt for an unrelated task.
   - The default is now the case report followed by `Return JSON only.`
     The system prompts of those steps are unchanged.
   - `security-issue-import/step-2a-semantic-sweep`, the step the old default was written for, gets its own `user-prompt-template.md` with the old text.
     Its rendered prompt is unchanged apart from the SPDX comment that the `add-license-headers` hook puts on every Markdown file, which the runner sends verbatim, as it already does for the 309 existing templates that carry it.
   
   ## Type of change
   
   - [ ] Skill change (`.claude/skills/<name>/`) — eval fixtures updated below
   - [ ] Tool / bridge contract (`tools/<system>/*.md`)
   - [x] Python package (`tools/*/` with `pyproject.toml`)
   - [ ] Groovy reference impl
   - [ ] Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
   - [ ] Documentation (`docs/`, `README.md`, `CONTRIBUTING.md`)
   - [ ] Project template (`projects/_template/`)
   - [ ] CI / dev loop (`prek`, workflows, validators)
   - [x] Other: eval fixture (`tools/skill-evals/evals/security-issue-import/step-2a-semantic-sweep/fixtures/user-prompt-template.md`)
   
   ## Test plan
   
   - [x] `prek run --all-files` passes
   - [x] For Python packages touched: `uv run pytest` / `ruff check` / `mypy` passes
         (`tools/skill-evals`: 175 passed; `ruff check` and `ruff format --check` clean;
         the workspace `mypy` hook does not cover `skill-evals`, which has no `[tool.mypy]` section)
   - [ ] For Groovy bridges touched: command-line invocation tested end-to-end
   - [ ] For skill changes: eval suite passes for the affected skill
   - [ ] For skill *behaviour* changes: a new or updated eval fixture is included in this PR
   - [x] Other:
     - New test: a step without `user-prompt-template.md` renders a user prompt that holds the case report and none of `Existing open trackers`, `Reporter roster`, `semantic sweep`.
       It failed on `main` before the fix.
     - New test: the rendered user prompt of `security-issue-import/step-2a-semantic-sweep/case-1-clear-duplicate` equals the SPDX comment followed by the prompt captured from `main` as a literal.
     - Replaced `test_load_step_config_uses_default_user_prompt_template_when_absent`, which asserted that the default carries `{corpus}` and `{roster}`.
     - Print mode (`python -m skill_evals.runner <dir>`, no model call):
       all 6 cases of the import step render the SPDX comment, a blank line, and then exactly the prompt `main` renders;
       `release-vote-tally/step-2-classify` now renders the report and `Return JSON only.` instead of the import framing.
     - No other template-less step has a `corpus.json` or `reporter-roster.json`, so the neutral default drops no fixture data.
     - Model evals were not run.
   
   ## RFC-AI-0004 compliance
   
   No principle is affected: the change is limited to the eval runner's prompt assembly and one eval fixture.
   
   ## Linked issues
   
   Closes #1492
   
   ## Notes for reviewers (optional)
   
   The verdicts of these steps were produced with the wrong user prompt and should be re-run with a model.
   They are the 33 step directories under `tools/skill-evals/evals/` that hold a `step-config.json` and no `user-prompt-template.md`:
   
   - `non-asf-profile-smoke/step-release-backend-preflight`
   - `release-announce-draft/step-0-preflight`, `step-2-announce-draft`, `step-3-site-bump`
   - `release-archive-sweep/step-0-preflight`, `step-1-load-listing`, `step-2-emit-commands`
   - `release-audit-report/step-0-preflight`, `step-1-gather-record`, `step-2-assemble-record`
   - `release-keys-sync/step-0-preflight`, `step-1-key-fetch`, `step-2-svn-commands`
   - `release-prepare/step-0-preflight`, `step-1-plan`, `step-2-prep`, `step-14-post`
   - `release-rc-cut/step-0-preflight`, `step-2-tag-build-sign`, `step-2b-reproducibility`, `step-3-staging`
   - `release-verify-rc/step-0-preflight`, `step-2-verify-signatures`, `step-3-verify-checksums`, `step-5-notice-license`, `step-6-binary-exclusion`, `step-6b-jvm-artefacts`, `step-8-version-consistency`, `step-9-reproducibility`
   - `release-vote-tally/step-0-preflight`, `step-2-classify`, `step-3-tally`
   - `reviewer-routing/step-0-preflight`
   
   `security-issue-import/step-2a-semantic-sweep` only gains the SPDX comment at the top of its user prompt; one re-run confirms it is unaffected.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(skill-evals): give template-less eval steps a neutral user prompt [magpie]

["potiuk (via GitHub)" <gi...@apache.org>]

potiuk merged PR #1524:
URL: https://github.com/apache/magpie/pull/1524


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org