(magpie) branch main updated: fix(issue-reassess-stats): pin nature counting in step-2 classify eval spec (#1615)

[po...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/magpie.git


The following commit(s) were added to refs/heads/main by this push:
     new 5ef20f03 fix(issue-reassess-stats): pin nature counting in step-2 classify eval spec (#1615)
5ef20f03 is described below

commit 5ef20f032f609665e35b3fc6b51775280f9c7cc0
Author: Nic Crane <th...@gmail.com>
AuthorDate: Fri Oct 16 02:50:00 2026 +0100

    fix(issue-reassess-stats): pin nature counting in step-2 classify eval spec (#1615)
    
    The step-2-classify output spec lists `bug-as-advertised-partial-fix`
    as a nature bucket but never says whether `nature` is counted as given
    on each verdict or re-derived from the `cases` array. On the
    case-1-mixed-verdicts fixture the model sometimes re-buckets the one
    verdict with a mixed `cases` array into the partial-fix nature, which
    drops `bug-as-advertised` from 4 to 3 and adds a fifth cross-table row.
    Observed on two of four full-suite `claude -p` runs on main; Opus and
    Sonnet passed the case.
    
    State the rule the fixture and classify.md already assume: count
    `classification` and `nature` exactly as given, and let a mixed `cases`
    array only add the key to `partial_fixes`.
    
    Generated-by: Claude Code (Fable 5.1)
---
 .../evals/issue-reassess-stats/step-2-classify/fixtures/output-spec.md  | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tools/skill-evals/evals/issue-reassess-stats/step-2-classify/fixtures/output-spec.md b/tools/skill-evals/evals/issue-reassess-stats/step-2-classify/fixtures/output-spec.md
index a4990155..982d22ab 100644
--- a/tools/skill-evals/evals/issue-reassess-stats/step-2-classify/fixtures/output-spec.md
+++ b/tools/skill-evals/evals/issue-reassess-stats/step-2-classify/fixtures/output-spec.md
@@ -27,5 +27,7 @@ Return ONLY valid JSON with this structure:
 ```
 
 `cross_table` contains only non-zero combinations.
+Count `classification` and `nature` exactly as given on each verdict; do not re-derive or re-bucket either.
+A mixed `cases` array does not change a verdict's `nature`; it only adds the key to `partial_fixes`.
 `partial_fixes` contains keys where the `cases` array has mixed outcomes (some still-failing, some fixed).
 Do not include any text outside the JSON object.