You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@magpie.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/09/29 21:48:29 UTC

[PR] fix(evals): resolve the four remaining eval failures [magpie]

potiuk opened a new pull request, #1476:
URL: https://github.com/apache/magpie/pull/1476

   Fixes the four eval failures left over from the security-family optimization (#1344).
   
   ## Summary
   
   - **issue-deduplicate step-0 case-3.** With a single argument there is no duplicate, so the expected value is `null`, and the output spec allows `null`.
   - **release-audit-report step-0 case-3.** The blocker assertion also accepts *"Planning issue not found"*, which is what a correct run prints.
   - **security-issue-triage step-2a case-1.** The two DAG-author rows in the cheat sheet overlapped. The skill now says which one applies. When the DAG author controls the value, it is the *code execution* row. The *routes user input* row applies only when someone else's input reaches the operator. The eval's system prompt carries the same note.
   - **security-issue-triage step-2.6 case-2.** The prompt's match rules required an identical code pointer for STRONG. The skill says a positive precedent of the *same shape* supports VALID. The rules now follow the skill, and the budget counts as exhausted when the input says all the calls were used.
   
   ## Test plan
   
   - [x] prek hooks on commit
   - [x] The four cases pass, run outside the sandbox
   - [x] Full triage step-2.6 (3/3) and step-2a (3/3) suites pass
   
   Part of #1344.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(evals): resolve the four remaining eval failures [magpie]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #1476:
URL: https://github.com/apache/magpie/pull/1476#issuecomment-5899652383

   I just noticed that was something my agents have not pushed yet. Completing it as the **last** thing before signing of entirely - till Sunday :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(evals): resolve the four remaining eval failures [magpie]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #1476:
URL: https://github.com/apache/magpie/pull/1476


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org