[I] fix(skill-evals): issue-reassess is the only shipped skill without an eval suite [magpie]

["dpol1 (via GitHub)" <gi...@apache.org>]

dpol1 opened a new issue, #1138:
URL: https://github.com/apache/magpie/issues/1138

   ## What
   
   `skills/issue-reassess/` is the only one of the 71 shipped skills with no behavioural eval suite — there is no `tools/skill-evals/evals/issue-reassess/` directory. (The 71st entry under `evals/`, `non-asf-profile-smoke`, is a profile smoke, not a skill suite.)
   
   ```bash
   comm -23 <(ls skills | grep -v pyproject | sort) \
            <(ls tools/skill-evals/evals | sort)
   # issue-reassess
   ```
   
   The validator already knows:
   
   ```
   skills/issue-reassess/SKILL.md  (1 warning)
     eval-coverage: no eval suite at tools/skill-evals/evals/issue-reassess/ — add one before shipping
   ```
   
   ## Why it matters
   
   `AGENTS.md` is explicit: *"ship a behavioural eval suite under `tools/skill-evals/evals/<skill-name>/` […] A skill PR without a matching eval suite is incomplete."* The skill shipped anyway and is in active use — it has a row in the `docs/mode-economics.md` Triage table — so regressions in its classification and aggregation behaviour are currently invisible. The eval-coverage check (#9) is deliberately SOFT so in-flight eval PRs don't fail the gate, which means this gap stays silent until the skill misbehaves in front of a user.
   
   ## Suggested fix
   
   Add the suite. The skill's surfaces look directly fixturable: the per-issue verdict flow (`skills/issue-reassess/per-issue-flow.md`), pool selection (`pool-selection.md`), and verdict aggregation (`verdict-aggregation.md`). The sibling `issue-reassess-stats` suite (step-1-fetch-verdicts / step-2-classify / step-3-aggregate) is the natural model, and it already contains reassess-shaped fixture reports to crib from.
   
   ## Related
   
   - `tools/skill-and-tool-validator` check #9 (`validate_eval_coverage`) — the SOFT advisory that flags this today.
   - `tools/skill-evals/evals/issue-reassess-stats/` — the sibling suite to mirror.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] fix(skill-evals): issue-reassess is the only shipped skill without an eval suite [magpie]

["potiuk (via GitHub)" <gi...@apache.org>]

potiuk closed issue #1138: fix(skill-evals): issue-reassess is the only shipped skill without an eval suite
URL: https://github.com/apache/magpie/issues/1138


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org