You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@magpie.apache.org by "AmirF194 (via GitHub)" <gi...@apache.org> on 2026/08/31 05:37:42 UTC
[PR] docs(skill-evals): add --cli to workflow-security-audit and license-compliance-audit READMEs [magpie]
AmirF194 opened a new pull request, #1134:
URL: https://github.com/apache/magpie/pull/1134
## Summary
- Add `--cli "claude -p"` to the documented run commands in the
`workflow-security-audit` and `license-compliance-audit` eval READMEs.
- Without it the runner has no model to drive, so both suites' prose and
structured cases report `MANUAL` instead of `PASS`/`FAIL` when the README
is followed as written.
## Type of change
- [x] Documentation (`docs/`, `README.md`, `CONTRIBUTING.md`)
## Test plan
- `PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/workflow-security-audit/step-scope-selection/fixtures/case-3-injection-in-scope-input` (the pre-fix, documented command): prints the prompt and expected JSON, no PASS/FAIL/MANUAL grading at all, since print mode has no model to compare against.
- Same case with `--cli` pointed at a stub script that echoes the exact `expected.json`: `PASS step-scope-selection/case-3-injection-in-scope-input`, confirming the `--cli` flag is what drives grading. I don't have a `claude` CLI or API credentials in this environment, so I could not run it against the real model; the stub proves the harness wiring, not a live grade.
- `prek run --all-files` (`add-license-headers`, `check-placeholders`, `markdownlint-cli2`, `typos`) passes on both files. `lychee` is unaffected: the diff adds no links.
- Both suites already use exact-match `expected.json` (enums/booleans only), the same shape `onboarding-concierge`'s README documents as gradable via `--cli` with no `MANUAL` fallback.
## RFC-AI-0004 compliance
Not applicable: doc-only change to eval instructions, no skill/tool behavior changed.
## Linked issues
Fixes #943
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(skill-evals): add --cli to workflow-security-audit and license-compliance-audit READMEs [magpie]
Posted by "AmirF194 (via GitHub)" <gi...@apache.org>.
AmirF194 commented on PR #1134:
URL: https://github.com/apache/magpie/pull/1134#issuecomment-5504816382
Good catch checking it against the sibling suites, glad it lines up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(skill-evals): add --cli to workflow-security-audit and license-compliance-audit READMEs [magpie]
Posted by "onlyarnav (via GitHub)" <gi...@apache.org>.
onlyarnav merged PR #1134:
URL: https://github.com/apache/magpie/pull/1134
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org