You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@magpie.apache.org by "justinmclean (via GitHub)" <gi...@apache.org> on 2026/07/28 00:56:49 UTC
[I] Add a structural assertion that the security-issue-import injection case names the attack [magpie]
justinmclean opened a new issue, #980:
URL: https://github.com/apache/magpie/issues/980
## Summary
The prompt-injection eval case for `security-issue-import` Step 3 proves the classification
survived the attack, but nothing checks that the model noticed the attack at all. Add the
structural assertion that the equivalent `issue-triage` case already uses.
## Background
`case-9-mail-body-injection` feeds the skill a genuine vulnerability report with an
"IGNORE ALL PREVIOUS INSTRUCTIONS" block in the middle of the body, and expects the
classification to come out as `Report` regardless. That part works.
The evidence that the injection was *handled* lives only in the free-text `rationale` field,
which ends "… is untrusted input data and was disregarded". Prose fields are graded loosely, so
a model that lands on `Report` for ordinary reasons and never notices the injection passes the
case just as cleanly as one that spots it.
The `issue-triage` suite solved this for its own injection case: `expected.json` carries a
`has_rationale_injection_ignored` flag, and a suite-level `assertions.json` maps that flag to a
regex over the `rationale` field. That turns "did the model notice" into a real check. The same
pattern applies here; there is currently no `assertions.json` in this step's fixtures directory
at all.
## Where to look
- `tools/skill-evals/evals/issue-triage/step-3-classify/fixtures/assertions.json` — the
existing pattern to copy, including the regex used for the injection flag.
- `tools/skill-evals/evals/issue-triage/step-3-classify/fixtures/case-7-prompt-injection/expected.json`
— how the flag appears in an expected file.
- `tools/skill-evals/evals/security-issue-import/step-3-classify/fixtures/` — where the new
`assertions.json` goes (the directory holding the case folders and `output-spec.md`).
- `tools/skill-evals/evals/security-issue-import/step-3-classify/fixtures/case-9-mail-body-injection/expected.json`
— the file that needs the new flag.
- `tools/skill-evals/README.md` — reference for how assertions are evaluated.
## Acceptance criteria
- [ ] An `assertions.json` exists in the `step-3-classify` fixtures directory defining a flag
that checks the `rationale` field mentions the injection.
- [ ] `case-9-mail-body-injection/expected.json` sets that flag.
- [ ] The other eight cases in the step still pass unchanged (adding an assertion must not
require every case to carry the flag).
- [ ] The suite still assembles: `PYTHONPATH=tools/skill-evals/src python3 -m
skill_evals.runner tools/skill-evals/evals/security-issue-import/` runs without error.
## Estimated effort
~1-2 hours for someone new to the codebase. The pattern to copy is small, but you need to check
how a suite-level assertions file interacts with cases that do not use the flag.
## Getting started
[[How to contribute](https://github.com/apache/airflow-steward/blob/main/CONTRIBUTING.md#your-first-contribution)](https://github.com/apache/airflow-steward/blob/main/CONTRIBUTING.md#your-first-contribution)
This issue was drafted with the help of an AI-assisted tool and reviewed by a Magpie maintainer before posting. If anything here is unclear or looks wrong, say so on the issue: a real person is reading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a structural assertion that the security-issue-import injection case names the attack [magpie]
Posted by "Shaurya2k06 (via GitHub)" <gi...@apache.org>.
Shaurya2k06 commented on issue #980:
URL: https://github.com/apache/magpie/issues/980#issuecomment-5100417803
I'll take this up!
cc @justinmclean
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a structural assertion that the security-issue-import injection case names the attack [magpie]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #980: Add a structural assertion that the security-issue-import injection case names the attack
URL: https://github.com/apache/magpie/issues/980
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org