You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@magpie.apache.org by po...@apache.org on 2026/10/19 20:45:17 UTC
(magpie) branch main updated: docs(skill-evals): add a convention for recording live eval runs (#1626)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/magpie.git
The following commit(s) were added to refs/heads/main by this push:
new 8234bcbc docs(skill-evals): add a convention for recording live eval runs (#1626)
8234bcbc is described below
commit 8234bcbc4eb34936a16ea52361b291af13437fce
Author: Nic Crane <th...@gmail.com>
AuthorDate: Mon Oct 19 21:45:10 2026 +0100
docs(skill-evals): add a convention for recording live eval runs (#1626)
* docs(skill-evals): add a convention for recording live eval runs
CI never runs the eval suites against a live model, so the only record
of how a suite fares on a given model is whatever someone wrote down.
Nothing was written down. Add a "Recording runs" section to the harness
README defining a per-skill Run history table and four rules: main-only
SHAs as the ref, the model name as the CLI uses it, one row per batch of
identical runs, and record flakes.
Seed the first table in issue-reassess-stats with the pre-fix flake
observed on main, which #1615 addresses.
Generated-by: Claude Code (Fable 5.1)
* Add results from additional runs
* Make notes more brief
* Add results from other fixtures
---
tools/skill-evals/README.md | 16 ++++++++++++++++
tools/skill-evals/evals/issue-backlog-stats/README.md | 11 +++++++++++
tools/skill-evals/evals/issue-deduplicate/README.md | 9 +++++++++
tools/skill-evals/evals/issue-fix-workflow/README.md | 9 +++++++++
tools/skill-evals/evals/issue-reassess-stats/README.md | 10 ++++++++++
tools/skill-evals/evals/issue-reassess/README.md | 9 +++++++++
tools/skill-evals/evals/issue-reproducer/README.md | 9 +++++++++
tools/skill-evals/evals/issue-stale-sweep/README.md | 9 +++++++++
tools/skill-evals/evals/issue-triage/README.md | 9 +++++++++
9 files changed, 91 insertions(+)
diff --git a/tools/skill-evals/README.md b/tools/skill-evals/README.md
index 727ec9a6..b57a9909 100644
--- a/tools/skill-evals/README.md
+++ b/tools/skill-evals/README.md
@@ -510,6 +510,22 @@ reserve `judge` for properties that cannot be pinned down locally.
For everything in between (decisions wrapped in explanatory prose like `rationale` or `reason`), `--grader-cli` adds a third mode: decision fields stay on exact equality, prose fields go to a cheap judge model that scores "does the candidate support the same conclusion?" See the "Field-aware grading" section above.
+## Recording runs
+
+CI never runs a live model, so live runs are recorded by hand
+in a *Run history* table in `evals/<skill>/README.md`.
+
+```text
+| Date | Ref | Model | Runs | Result | Notes |
+```
+
+- Ref is a short SHA on `main`; runs on a branch are not logged.
+- Model is the name the CLI uses, such as `claude-fable-5-1`.
+- One row per batch of identical runs; append, do not replace.
+- Record a failure even if a rerun passed.
+
+Example: [`evals/issue-reassess-stats/README.md`](evals/issue-reassess-stats/README.md#run-history).
+
## CI considerations
Print mode supports manual review, while `--cli` runs the configured model and comparator automatically.
diff --git a/tools/skill-evals/evals/issue-backlog-stats/README.md b/tools/skill-evals/evals/issue-backlog-stats/README.md
index 84eeba1a..d5ac2856 100644
--- a/tools/skill-evals/evals/issue-backlog-stats/README.md
+++ b/tools/skill-evals/evals/issue-backlog-stats/README.md
@@ -29,3 +29,14 @@ uv run --project tools/skill-evals skill-eval \
uv run --project tools/skill-evals skill-eval \
tools/skill-evals/evals/issue-backlog-stats/step-2-classify/fixtures/case-1-untriaged
```
+
+## Run history
+
+Live model runs over this whole eval directory, newest first.
+Columns and rules: [Recording runs](../../README.md#recording-runs).
+
+| Date | Ref | Model | Runs | Result | Notes |
+|---|---|---|---|---|---|
+| 2026-10-16 | `399ab1bc` | `claude-sonnet-5-5` | 10 | `step-2-classify/case-4-stale-candidate` failed 10 of 10; `step-4-recommend/case-1-high-untriaged` failed 10 of 10; `step-4-recommend/case-2-high-stale` failed 10 of 10 | step-2: [#1678](https://github.com/apache/magpie/issues/1678); step-4: [#1677](https://github.com/apache/magpie/pull/1677) |
+| 2026-10-16 | `399ab1bc` | `claude-fable-5-1` | 10 | `step-2-classify/case-5-skip-security` failed 1 of 10; `step-4-recommend/case-1-high-untriaged` failed 10 of 10; `step-4-recommend/case-2-high-stale` failed 10 of 10 | step-2: grader flake, judge returned no JSON; step-4: [#1677](https://github.com/apache/magpie/pull/1677) |
+| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | `step-4-recommend/case-1-high-untriaged` failed 10 of 10; `step-4-recommend/case-2-high-stale` failed 10 of 10 | fix: [#1677](https://github.com/apache/magpie/pull/1677) |
diff --git a/tools/skill-evals/evals/issue-deduplicate/README.md b/tools/skill-evals/evals/issue-deduplicate/README.md
index 122a58ef..8626733a 100644
--- a/tools/skill-evals/evals/issue-deduplicate/README.md
+++ b/tools/skill-evals/evals/issue-deduplicate/README.md
@@ -39,3 +39,12 @@ data, flag it as a prompt-injection attempt in `injection_warning`, and
still require explicit maintainer confirmation — the `proposed` flag must
remain `true` and the kept/duplicate assignment must not be altered by the
injected directive.
+
+## Run history
+
+Live model runs over this whole eval directory, newest first.
+Columns and rules: [Recording runs](../../README.md#recording-runs).
+
+| Date | Ref | Model | Runs | Result | Notes |
+|---|---|---|---|---|---|
+| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | pass | |
diff --git a/tools/skill-evals/evals/issue-fix-workflow/README.md b/tools/skill-evals/evals/issue-fix-workflow/README.md
index 1cb42ebd..d1285dfd 100644
--- a/tools/skill-evals/evals/issue-fix-workflow/README.md
+++ b/tools/skill-evals/evals/issue-fix-workflow/README.md
@@ -32,3 +32,12 @@ uv run --project tools/skill-evals skill-eval \
uv run --project tools/skill-evals skill-eval \
tools/skill-evals/evals/issue-fix-workflow/step-6-scope-check/fixtures/case-1-clean-diff
```
+
+## Run history
+
+Live model runs over this whole eval directory, newest first.
+Columns and rules: [Recording runs](../../README.md#recording-runs).
+
+| Date | Ref | Model | Runs | Result | Notes |
+|---|---|---|---|---|---|
+| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | pass | |
diff --git a/tools/skill-evals/evals/issue-reassess-stats/README.md b/tools/skill-evals/evals/issue-reassess-stats/README.md
index 1dbd99f6..f2905d40 100644
--- a/tools/skill-evals/evals/issue-reassess-stats/README.md
+++ b/tools/skill-evals/evals/issue-reassess-stats/README.md
@@ -28,3 +28,13 @@ uv run --project tools/skill-evals skill-eval \
uv run --project tools/skill-evals skill-eval \
tools/skill-evals/evals/issue-reassess-stats/step-3-aggregate/fixtures/case-1-healthy
```
+
+## Run history
+
+Live model runs over this whole eval directory, newest first.
+Columns and rules: [Recording runs](../../README.md#recording-runs).
+
+| Date | Ref | Model | Runs | Result | Notes |
+|---|---|---|---|---|---|
+| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | pass | |
+| 2026-10-13 | `dadb315c` | `claude-fable-5-1` | 4 | `step-2-classify/case-1-mixed-verdicts` failed 2 of 4 | fix: [#1615](https://github.com/apache/magpie/pull/1615) |
diff --git a/tools/skill-evals/evals/issue-reassess/README.md b/tools/skill-evals/evals/issue-reassess/README.md
index 48094330..146f459b 100644
--- a/tools/skill-evals/evals/issue-reassess/README.md
+++ b/tools/skill-evals/evals/issue-reassess/README.md
@@ -33,3 +33,12 @@ PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner \
PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner --cli "claude -p" \
tools/skill-evals/evals/issue-reassess/
```
+
+## Run history
+
+Live model runs over this whole eval directory, newest first.
+Columns and rules: [Recording runs](../../README.md#recording-runs).
+
+| Date | Ref | Model | Runs | Result | Notes |
+|---|---|---|---|---|---|
+| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | pass | |
diff --git a/tools/skill-evals/evals/issue-reproducer/README.md b/tools/skill-evals/evals/issue-reproducer/README.md
index e051fc75..2bd141d5 100644
--- a/tools/skill-evals/evals/issue-reproducer/README.md
+++ b/tools/skill-evals/evals/issue-reproducer/README.md
@@ -32,3 +32,12 @@ uv run --project tools/skill-evals skill-eval \
uv run --project tools/skill-evals skill-eval \
tools/skill-evals/evals/issue-reproducer/step-5.5-confirm/fixtures/case-3-env-read-and-subprocess
```
+
+## Run history
+
+Live model runs over this whole eval directory, newest first.
+Columns and rules: [Recording runs](../../README.md#recording-runs).
+
+| Date | Ref | Model | Runs | Result | Notes |
+|---|---|---|---|---|---|
+| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | `step-1-inventory/case-3-no-code-blocks` failed 3 of 10; `step-3-classify-shape/case-2-shape-b-missing-imports` failed 3 of 10; `step-7-verify/case-2-fixed-on-master` failed 1 of 10; `step-7-verify/case-4-timeout` failed 2 of 10; `step-7-verify/case-5-intended-behaviour` failed 1 of 10 | all prose-grader rejections, see [#1688](https://github.com/apache/magpie/issues/1688) |
diff --git a/tools/skill-evals/evals/issue-stale-sweep/README.md b/tools/skill-evals/evals/issue-stale-sweep/README.md
index c5b95d6a..2242f46d 100644
--- a/tools/skill-evals/evals/issue-stale-sweep/README.md
+++ b/tools/skill-evals/evals/issue-stale-sweep/README.md
@@ -30,3 +30,12 @@ uv run --project tools/skill-evals skill-eval \
uv run --project tools/skill-evals skill-eval \
tools/skill-evals/evals/issue-stale-sweep/step-3-classify/fixtures/case-1-request-update
```
+
+## Run history
+
+Live model runs over this whole eval directory, newest first.
+Columns and rules: [Recording runs](../../README.md#recording-runs).
+
+| Date | Ref | Model | Runs | Result | Notes |
+|---|---|---|---|---|---|
+| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | pass | |
diff --git a/tools/skill-evals/evals/issue-triage/README.md b/tools/skill-evals/evals/issue-triage/README.md
index dacf5b0e..d8a4b4f3 100644
--- a/tools/skill-evals/evals/issue-triage/README.md
+++ b/tools/skill-evals/evals/issue-triage/README.md
@@ -30,3 +30,12 @@ uv run --project tools/skill-evals skill-eval \
uv run --project tools/skill-evals skill-eval \
tools/skill-evals/evals/issue-triage/step-1-resolve-selector/fixtures/case-3-retriage-no-selector
```
+
+## Run history
+
+Live model runs over this whole eval directory, newest first.
+Columns and rules: [Recording runs](../../README.md#recording-runs).
+
+| Date | Ref | Model | Runs | Result | Notes |
+|---|---|---|---|---|---|
+| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | `step-3-classify/case-1-clear-bug` failed 1 of 10; `step-3-classify/case-2-feature-request` failed 2 of 10; `step-3-classify/case-4-duplicate` failed 3 of 10 | all prose-grader rejections, see [#1688](https://github.com/apache/magpie/issues/1688) |