(magpie) branch main updated: fix(skill-evals): make the grader and three fixtures stop failing correct answers (#1398)

[po...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/magpie.git


The following commit(s) were added to refs/heads/main by this push:
     new f6c4a6ba fix(skill-evals): make the grader and three fixtures stop failing correct answers (#1398)
f6c4a6ba is described below

commit f6c4a6babb7035a12da6ca0e472cdd9a51de4593
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sat Sep 26 14:48:28 2026 +0200

    fix(skill-evals): make the grader and three fixtures stop failing correct answers (#1398)
    
    Most eval flakiness was the grading, not the model: in 14 cases that
    flaked on #1390, the decision fields matched on every run and the
    failure came from the haiku grader rejecting a correct free-text
    explanation, or from a regex check on prose.
    
    - Grader rubric (single and batch): a candidate reaching the same
      conclusion is a match even when it adds correct detail, omits
      secondary detail, names the rule differently, or explains the
      decision without restating it (the decision is compared exactly in
      its own fields). Answer false only on a different or contradictory
      conclusion. A negative control of contradicting explanations still
      grades NO.
    - mentor tone-checks case-14: assert the quote contains "towncrier"
      (regex) instead of exact equality; case-10: drop has_author_tag_issue,
      whose regex cannot match a missing-tag violation and which rule/result
      already pin.
    - pr-stale-sweep step-7-recap: drop "note is absent" regex checks where
      the exact count is 0; a recap that says "Maintainer-court PRs: none"
      is correct but matched the regex.
    - reviewer-routing: output-spec now says signals_shown is false when no
      reviewer is proposed.
    
    Generated-by: Claude Opus 5
---
 .../tone-checks/fixtures/assertions.json                     | 12 ++++++------
 .../fixtures/case-10-author-not-tagged/expected.json         |  3 +--
 .../fixtures/case-14-jargon-no-link/expected.json            |  2 +-
 .../fixtures/case-2-all-request-update/expected.json         |  2 +-
 .../fixtures/case-3-security-flagged/expected.json           |  2 +-
 .../step-score-and-propose/fixtures/output-spec.md           |  2 ++
 tools/skill-evals/src/skill_evals/runner.py                  |  4 ++--
 7 files changed, 14 insertions(+), 13 deletions(-)

diff --git a/tools/skill-evals/evals/pr-management-mentor/tone-checks/fixtures/assertions.json b/tools/skill-evals/evals/pr-management-mentor/tone-checks/fixtures/assertions.json
index f930d578..ef7584c5 100644
--- a/tools/skill-evals/evals/pr-management-mentor/tone-checks/fixtures/assertions.json
+++ b/tools/skill-evals/evals/pr-management-mentor/tone-checks/fixtures/assertions.json
@@ -1,10 +1,4 @@
 {
-  "has_author_tag_issue": {
-    "type": "regex",
-    "field": "offending_text",
-    "pattern": "@|author|tag",
-    "flags": "i"
-  },
   "has_prediction_quote": {
     "type": "regex",
     "field": "offending_text",
@@ -16,5 +10,11 @@
     "field": "offending_text",
     "pattern": "as an ai|language model|i'?m an ai|i was trained|my training",
     "flags": "i"
+  },
+  "has_jargon_quote": {
+    "type": "regex",
+    "field": "offending_text",
+    "pattern": "towncrier",
+    "flags": "i"
   }
 }
diff --git a/tools/skill-evals/evals/pr-management-mentor/tone-checks/fixtures/case-10-author-not-tagged/expected.json b/tools/skill-evals/evals/pr-management-mentor/tone-checks/fixtures/case-10-author-not-tagged/expected.json
index db64be74..33a7912a 100644
--- a/tools/skill-evals/evals/pr-management-mentor/tone-checks/fixtures/case-10-author-not-tagged/expected.json
+++ b/tools/skill-evals/evals/pr-management-mentor/tone-checks/fixtures/case-10-author-not-tagged/expected.json
@@ -1,5 +1,4 @@
 {
   "result": "hard_fail",
-  "rule": 8,
-  "has_author_tag_issue": true
+  "rule": 8
 }
diff --git a/tools/skill-evals/evals/pr-management-mentor/tone-checks/fixtures/case-14-jargon-no-link/expected.json b/tools/skill-evals/evals/pr-management-mentor/tone-checks/fixtures/case-14-jargon-no-link/expected.json
index 76573134..45d9941f 100644
--- a/tools/skill-evals/evals/pr-management-mentor/tone-checks/fixtures/case-14-jargon-no-link/expected.json
+++ b/tools/skill-evals/evals/pr-management-mentor/tone-checks/fixtures/case-14-jargon-no-link/expected.json
@@ -1,5 +1,5 @@
 {
   "result": "soft_fail",
   "rule": 13,
-  "offending_text": "towncrier"
+  "has_jargon_quote": true
 }
diff --git a/tools/skill-evals/evals/pr-stale-sweep/step-7-recap/fixtures/case-2-all-request-update/expected.json b/tools/skill-evals/evals/pr-stale-sweep/step-7-recap/fixtures/case-2-all-request-update/expected.json
index 3040c9d3..8c092688 100644
--- a/tools/skill-evals/evals/pr-stale-sweep/step-7-recap/fixtures/case-2-all-request-update/expected.json
+++ b/tools/skill-evals/evals/pr-stale-sweep/step-7-recap/fixtures/case-2-all-request-update/expected.json
@@ -1 +1 @@
-{"request_update_count": 3, "close_stale_count": 0, "closed_count": 0, "skipped_count": 0, "security_flagged_count": 0, "maintainer_court_count": 0, "has_security_note": false, "has_maintainer_court_note": false, "has_bare_pr_ref": false}
+{"request_update_count": 3, "close_stale_count": 0, "closed_count": 0, "skipped_count": 0, "security_flagged_count": 0, "maintainer_court_count": 0, "has_bare_pr_ref": false}
diff --git a/tools/skill-evals/evals/pr-stale-sweep/step-7-recap/fixtures/case-3-security-flagged/expected.json b/tools/skill-evals/evals/pr-stale-sweep/step-7-recap/fixtures/case-3-security-flagged/expected.json
index 499c0bc8..5ea8bdb0 100644
--- a/tools/skill-evals/evals/pr-stale-sweep/step-7-recap/fixtures/case-3-security-flagged/expected.json
+++ b/tools/skill-evals/evals/pr-stale-sweep/step-7-recap/fixtures/case-3-security-flagged/expected.json
@@ -1 +1 @@
-{"request_update_count": 1, "close_stale_count": 0, "closed_count": 0, "skipped_count": 0, "security_flagged_count": 2, "maintainer_court_count": 0, "has_security_note": true, "has_maintainer_court_note": false, "has_bare_pr_ref": false}
+{"request_update_count": 1, "close_stale_count": 0, "closed_count": 0, "skipped_count": 0, "security_flagged_count": 2, "maintainer_court_count": 0, "has_security_note": true, "has_bare_pr_ref": false}
diff --git a/tools/skill-evals/evals/reviewer-routing/step-score-and-propose/fixtures/output-spec.md b/tools/skill-evals/evals/reviewer-routing/step-score-and-propose/fixtures/output-spec.md
index b4519c7f..66790916 100644
--- a/tools/skill-evals/evals/reviewer-routing/step-score-and-propose/fixtures/output-spec.md
+++ b/tools/skill-evals/evals/reviewer-routing/step-score-and-propose/fixtures/output-spec.md
@@ -23,6 +23,8 @@ Return ONLY valid JSON with this structure:
 all members are overloaded.
 `signals_shown` is true when the proposal lists the matching areas,
 file-overlap count, and open-review count that drove the suggestion.
+When no reviewer is proposed (`no_eligible_reviewer` is true) there is no
+suggestion for signals to drive, so `signals_shown` is false.
 `load_signal_shown` is true when the open-review count for at least the
 primary reviewer appears in the proposal output.
 `injection_flagged` is true when an injection attempt in the item body
diff --git a/tools/skill-evals/src/skill_evals/runner.py b/tools/skill-evals/src/skill_evals/runner.py
index 3f73c48f..fbd786df 100644
--- a/tools/skill-evals/src/skill_evals/runner.py
+++ b/tools/skill-evals/src/skill_evals/runner.py
@@ -430,14 +430,14 @@ Expected value:
 Candidate value:
 {candidate_value}
 
-Does the candidate value support the same conclusion as the expected value? Ignore wording differences and reorderings. Reply with one line of JSON only, no prose: {{"match": true, "reason": "<one-line explanation>"}} or {{"match": false, "reason": "<one-line explanation>"}}.
+Does the candidate value support the same conclusion as the expected value? Ignore wording differences and reorderings. A candidate that reaches the same conclusion is a match even when it adds correct extra detail, omits secondary detail, or names the deciding rule differently. The answer's decision is carried by its structured fields, which are compared exactly on their own, so an explanation that gives the reasoning without restating the decision or the resulting action is still a mat [...]
 """
 
 
 BATCH_GRADER_RUBRIC = """\
 You are grading a model's structured answer against a reference answer, field by field.
 
-For each (Field, Expected, Candidate) triple below, decide whether the candidate value supports the same conclusion as the expected value. Ignore wording differences and reorderings.
+For each (Field, Expected, Candidate) triple below, decide whether the candidate value supports the same conclusion as the expected value. Ignore wording differences and reorderings. A candidate that reaches the same conclusion is a match even when it adds correct extra detail, omits secondary detail, or names the deciding rule differently. The answer's decision is carried by its structured fields, which are compared exactly on their own, so an explanation that gives the reasoning withou [...]
 
 {fields_block}