You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@magpie.apache.org by po...@apache.org on 2026/10/03 23:46:32 UTC
(magpie) branch main updated: fix(security-issue-sync): settle three ambiguous bulk and push rules (#1501)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/magpie.git
The following commit(s) were added to refs/heads/main by this push:
new a5885939 fix(security-issue-sync): settle three ambiguous bulk and push rules (#1501)
a5885939 is described below
commit a5885939f0978bc8af497c1c2d8e64718a28cbc2
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Oct 4 01:46:27 2026 +0200
fix(security-issue-sync): settle three ambiguous bulk and push rules (#1501)
- Step 5b: a not-configured session probe skips the push silently on
ordinary runs, but on a fix released transition the record is still
allocated, so the recap carries the deferral blocker, as it does
for an expired session.
- Bulk pre-flight: the 24h dispatch-urgent rule now comes before the
7-day override. Under first-match ordering it could never fire,
because a reply that recent also bumps updatedAt. Same reorder in
tools/preflight-audit's classifier, whose test documented the gap.
- Bulk pre-flight: a sync CVE-... selector names its trackers
explicitly, so pre-flight never skips them, like issue numbers.
Eval cases pin each rule (5b cases 7-8, bulk-selectors cases 7-8,
with a new urgent field). On main the urgent case fails; the other
three already passed, the model having resolved those the same way.
Generated-by: Claude Opus 5
---
.../skills/issue-sync/apply-and-push.md | 6 ++--
.../magpie-security/skills/issue-sync/bulk-mode.md | 6 ++--
.../src/preflight_audit/classifier.py | 22 +++++++-------
tools/preflight-audit/tests/test_classifier.py | 19 +++---------
tools/skill-evals/README.md | 2 +-
.../evals/security-issue-sync/README.md | 4 +--
.../expected.json | 8 +++++
.../case-7-not-configured-fix-released/report.md | 33 +++++++++++++++++++++
.../expected.json | 8 +++++
.../case-8-not-configured-no-transition/report.md | 34 ++++++++++++++++++++++
.../expected.json | 14 +++++++++
.../case-7-cve-selector-never-skipped/report.md | 28 ++++++++++++++++++
.../case-8-reporter-reply-urgent/expected.json | 17 +++++++++++
.../case-8-reporter-reply-urgent/report.md | 26 +++++++++++++++++
.../step-bulk-selectors/fixtures/output-spec.md | 3 ++
15 files changed, 197 insertions(+), 33 deletions(-)
diff --git a/plugins/magpie-security/skills/issue-sync/apply-and-push.md b/plugins/magpie-security/skills/issue-sync/apply-and-push.md
index 6881711c..f0ad5009 100644
--- a/plugins/magpie-security/skills/issue-sync/apply-and-push.md
+++ b/plugins/magpie-security/skills/issue-sync/apply-and-push.md
@@ -359,7 +359,7 @@ Do **not** park the push behind a second confirmation, and do **not** defer it o
the RM owns `review-ready → publish-ready → advisory`; the `allocated → review-ready` promotion is sync's job at `fix released`.
A tracker labelled `fix released` whose record is still `allocated` (Vulnogram: `DRAFT`) after the sync is a **process bug** — it strands the RM, who is gated on `review-ready`, and silently stalls the advisory.
-**The only acceptable deferral** is a genuinely-unavailable authenticated session at sync time (the session probe fails — see the decision flow below).
+**The only acceptable deferral** is a genuinely-unavailable authenticated session at sync time (the session probe returns `expired` or `not-configured` — see the decision flow below).
Then, and only then, sync posts the **manual-paste** hand-off variant *and* raises an explicit blocker in the Step 6 recap
(*"<CVE> still in `allocated`/DRAFT — push on the next authenticated sync"*).
Completing the push is the first action of the next session-capable sync.
@@ -442,8 +442,10 @@ Step 6 below describes how to verify the state advance landed (and what to do if
- **`expired`** → skip the push and surface a one-line reminder in the Step 6 recap:
*"CVE-tool authenticated session expired — re-run the adapter's setup entrypoint (for the Vulnogram adapter, `vulnogram-api-setup`) to restore automatic push; using manual-paste hand-off this run."*
Use the manual-paste hand-off variant for any 5c comment work below.
- - **`not-configured`** → skip the push silently;
+ - **`not-configured`** → skip the push;
the manual-paste hand-off (via the `cve_authority.source_tab_url_template` link) still works.
+ Silent on other runs, but on a `fix released` transition the record still needs `allocated → review-ready`,
+ so raise the deferral blocker from the rule above in the Step 6 recap.
Use the manual-paste hand-off variant for any 5c comment work below.
3. **Extract the regenerated JSON.** Re-run the
diff --git a/plugins/magpie-security/skills/issue-sync/bulk-mode.md b/plugins/magpie-security/skills/issue-sync/bulk-mode.md
index d5548d70..78c10718 100644
--- a/plugins/magpie-security/skills/issue-sync/bulk-mode.md
+++ b/plugins/magpie-security/skills/issue-sync/bulk-mode.md
@@ -103,8 +103,8 @@ Subagents keep the per-issue mail threads, PR diffs and comment bodies out of th
| Signals | Decision | Reason recorded in recap |
|---|---|---|
- | `updatedAt` within last **7 days** AND last comment is **NOT** skill-or-bot | `dispatch` | recent human activity — safety override |
| Last comment author is **not** skill-or-bot AND `createdAt` within last **24h** | `dispatch-urgent` | reporter just replied |
+ | `updatedAt` within last **7 days** AND last comment is **NOT** skill-or-bot | `dispatch` | recent human activity — safety override |
| Closed > **30 days** ago AND has `announced` label | `skip-noop` | `post-announce; CVE published` |
| Closed > **90 days** ago AND no `announced` label | `skip-noop` | `stale closed (invalid/duplicate/abandoned)` |
| Open AND has `cve allocated` + `pr merged` + `announced` AND last comment is skill-or-bot | `skip-noop` | `all phases done; awaiting closure heuristic` |
@@ -119,7 +119,7 @@ Subagents keep the per-issue mail threads, PR diffs and comment bodies out of th
- **Never silent; never skips a named tracker.** See [Hard rules for bulk mode](#hard-rules-for-bulk-mode):
every skip is listed under *"Pre-flight skipped"* and can be `force-sync`ed,
- and explicit issue numbers (`sync #232, #233`) are never skipped — pre-flight skips only for set selectors (`sync all`, `sync announced`, label/title).
+ and explicit issue numbers (`sync #232, #233`) and CVE IDs (`sync CVE-2026-40913`) are never skipped — pre-flight skips only for set selectors (`sync all`, `sync announced`, label/title).
- **Opt-out.** `--no-preflight` in the selector (e.g. `sync all --no-preflight`) bypasses the classifier and dispatches every resolved tracker —
for trust-but-verify sweeps after a rule change.
- **Dispatch-urgent is just dispatch.** It flags the tracker in the recap as *"recent reporter activity"*;
@@ -261,7 +261,7 @@ bucket it into the Step 14→15 close-out **regardless** of an empty *Public adv
- **Link-form self-check still applies** to the merged output — every `#NNN` is a clickable link per Golden rule 2.
- **Pre-flight skips are never silent.** Every Step 1b `skip-noop` appears in the *"Pre-flight skipped"* group with the rule that fired;
the user can `force-sync <N>` any of them, and `--no-preflight` bypasses Step 1b entirely.
-- **Pre-flight never skips an explicitly-named tracker.** For named numbers (`sync #232, #233`) Step 1b runs the classifier only for context
+- **Pre-flight never skips an explicitly-named tracker.** For named numbers (`sync #232, #233`) and the trackers a CVE ID resolves to (`sync CVE-2026-40913`) Step 1b runs the classifier only for context
(so the recap can say *"#232 looks idle — sync anyway?"*) and never skips.
Skip-eligible selectors are state/label/title selectors like `sync all` or `sync announced`.
diff --git a/tools/preflight-audit/src/preflight_audit/classifier.py b/tools/preflight-audit/src/preflight_audit/classifier.py
index 1e0a852d..ec4425a0 100644
--- a/tools/preflight-audit/src/preflight_audit/classifier.py
+++ b/tools/preflight-audit/src/preflight_audit/classifier.py
@@ -149,7 +149,18 @@ def classify_issue(
last_comment_age = _days_between(now, iss.last_comment_created_at)
closed_age = _days_between(now, iss.closed_at)
- # Rule 1: 7-day updatedAt safety override — but only when the
+ # Rule 1: dispatch-urgent — a non-skill comment in the last 24h.
+ # Checked before the 7-day override, which would otherwise always
+ # catch it first (a comment that recent also bumps updatedAt).
+ if last_comment_age is not None and last_comment_age < 1 and not last_was_skill:
+ return Classification(
+ issue=iss,
+ decision=Decision.DISPATCH_URGENT,
+ reason=f"recent reply from {iss.last_comment_author} (<24h)",
+ last_is_skill_or_bot=last_was_skill,
+ )
+
+ # Rule 2: 7-day updatedAt safety override — but only when the
# recent activity wasn't itself a skill write. On a tracker the
# skill just touched, the recently-bumped updatedAt is the skill's
# own work; let downstream rules decide.
@@ -163,15 +174,6 @@ def classify_issue(
last_is_skill_or_bot=last_was_skill,
)
- # Rule 2: dispatch-urgent — a non-skill comment in the last 24h.
- if last_comment_age is not None and last_comment_age < 1 and not last_was_skill:
- return Classification(
- issue=iss,
- decision=Decision.DISPATCH_URGENT,
- reason=f"recent reply from {iss.last_comment_author} (<24h)",
- last_is_skill_or_bot=last_was_skill,
- )
-
# Rule 3: closed > 30d ago AND `announced` label → post-announce.
if iss.state == "CLOSED" and closed_age is not None and closed_age > 30 and "announced" in iss.labels:
return Classification(
diff --git a/tools/preflight-audit/tests/test_classifier.py b/tools/preflight-audit/tests/test_classifier.py
index f14bda02..4adc449b 100644
--- a/tools/preflight-audit/tests/test_classifier.py
+++ b/tools/preflight-audit/tests/test_classifier.py
@@ -98,7 +98,7 @@ def test_rule_1_skipped_when_skill_drove_recent_update():
# ---------------------------------------------------------------------------
-# Rule 2 — dispatch-urgent (non-skill comment in last 24h)
+# Rule 1 — dispatch-urgent (non-skill comment in last 24h)
# ---------------------------------------------------------------------------
@@ -111,20 +111,9 @@ def test_rule_2_urgent_when_non_skill_comment_under_24h():
last_comment_body="please re-check\n",
)
c = classify_issue(iss, now=NOW)
- # Rule 1 fires first (recent human activity); urgent is only
- # reachable when Rule 1 yielded — i.e., when the recent activity
- # was the skill. So the urgent path needs a non-skill comment
- # AFTER a skill update? Actually the rule is "Rule 1 yielded AND
- # Rule 2 fires" which happens when last comment is recent AND
- # not skill — but then Rule 1 ALSO doesn't yield, since it
- # requires skill-drove-recent-update. Both fail; we end up at
- # Rule 1's dispatch. That matches the prose: urgent is the
- # bot-vs-not distinction; when the last comment is a human reply
- # in the last 24h, the classifier dispatches (with the urgent
- # tag) if-and-only-if there is no other reason to dispatch.
- # In this synthetic case Rule 1 catches first → dispatch
- # without urgent. That's correct behaviour.
- assert c.decision == Decision.DISPATCH
+ # The 24h rule is checked before the 7-day override, so a fresh
+ # reporter reply is urgent even though it also bumps updatedAt.
+ assert c.decision == Decision.DISPATCH_URGENT
def test_rule_2_urgent_path_after_skill_only_recent_activity():
diff --git a/tools/skill-evals/README.md b/tools/skill-evals/README.md
index bfd5c8aa..71cf0a02 100644
--- a/tools/skill-evals/README.md
+++ b/tools/skill-evals/README.md
@@ -21,7 +21,7 @@ Suites are currently implemented for:
- **security-issue-triage** — 33 cases across 9 steps
- **security-issue-deduplicate** — 18 cases across 6 steps (steps 1, 2, 3, 4, 5, 6)
- **security-cve-allocate** — 20 cases across 6 steps (steps 1, 2, 3, 4, 5, 7)
-- **security-issue-sync**: 71 cases across 13 steps (1d, 1f, 2a, 2b, 2c, 3, 5b, 6, bulk-orchestration, bulk-selectors, ghsa-access-tier, guardrails, security-cc)
+- **security-issue-sync**: 75 cases across 13 steps (1d, 1f, 2a, 2b, 2c, 3, 5b, 6, bulk-orchestration, bulk-selectors, ghsa-access-tier, guardrails, security-cc)
- **security-issue-fix** — 39 cases across 12 steps (2, 4a, 4b, 4c, 4d, 4e, 4f, 4g, 5, 7, 10)
- **security-issue-invalidate** — 24 cases across 9 steps (2, 3, 4, 5a, 5b, 5d, 5e, 5f, 7)
- **security-issue-import-from-md** — 11 cases across 4 steps (1, 2, 4, 6)
diff --git a/tools/skill-evals/evals/security-issue-sync/README.md b/tools/skill-evals/evals/security-issue-sync/README.md
index 9e86d6de..4c95199b 100644
--- a/tools/skill-evals/evals/security-issue-sync/README.md
+++ b/tools/skill-evals/evals/security-issue-sync/README.md
@@ -19,8 +19,8 @@ Steps 0 (pre-flight), 1a–1c and 1e (data gathering), 1g (cve.org API check), 4
| 6 | Recap | 2 | Structural assertions; with and without CVE/draft |
| Bulk orchestration | Bucket-and-walk decision in bulk mode | 3 | All label-only (one bundled), mixed buckets (split), all CVE-affecting (all walked) |
| 1d | Actionable signals in comments and mail | 6 | Affected-versions lower bound (widen / keep + backtick-wrap), title strip and its under-3-words guard, private-scanner credit (anonymise / public-credit exemption), reporter CVSS surfaced only |
-| 5b | Push decision for the regenerated CVE JSON | 6 | Skip gate, expired session hand-off, title and private-finder hygiene gates, clean push with `allocated → review-ready`, close-out push on a `PUBLIC` record |
-| Bulk selectors | Selector resolution and pre-flight skips | 6 | Injection-shaped CVE token rejected, CVE list expansion, `sync all` skip rules, named trackers never skipped, `force-sync`, review-ready push exemption |
+| 5b | Push decision for the regenerated CVE JSON | 8 | Skip gate, expired and not-configured session hand-offs (blocker only on a `fix released` transition), title and private-finder hygiene gates, clean push with `allocated → review-ready`, close-out push on a `PUBLIC` record |
+| Bulk selectors | Selector resolution and pre-flight skips | 8 | Injection-shaped CVE token rejected, CVE list expansion, `sync all` skip rules, named trackers and CVE selectors never skipped, reporter reply under 24h is `dispatch-urgent`, `force-sync`, review-ready push exemption |
| GHSA access tier | Repository security advisory writes by access tier | 5 | Collaborator direct field edit, 403 admin hand-off relay draft, admin direct write, reporter reply by browser paste, non-collaborator relay fallback |
## Hard rules exercised
diff --git a/tools/skill-evals/evals/security-issue-sync/step-5b-push-decision/fixtures/case-7-not-configured-fix-released/expected.json b/tools/skill-evals/evals/security-issue-sync/step-5b-push-decision/fixtures/case-7-not-configured-fix-released/expected.json
new file mode 100644
index 00000000..c90d157d
--- /dev/null
+++ b/tools/skill-evals/evals/security-issue-sync/step-5b-push-decision/fixtures/case-7-not-configured-fix-released/expected.json
@@ -0,0 +1,8 @@
+{
+ "action": "hand-off",
+ "gate_failed": null,
+ "handoff_variant": "manual-paste",
+ "state_advance": null,
+ "recap_blocker": true,
+ "reason": "The probe returns not-configured, so the push is skipped and the manual-paste hand-off is used. Because this is a fix released transition, the record is still in allocated/DRAFT and the recap must carry the deferral blocker."
+}
diff --git a/tools/skill-evals/evals/security-issue-sync/step-5b-push-decision/fixtures/case-7-not-configured-fix-released/report.md b/tools/skill-evals/evals/security-issue-sync/step-5b-push-decision/fixtures/case-7-not-configured-fix-released/report.md
new file mode 100644
index 00000000..fda04f79
--- /dev/null
+++ b/tools/skill-evals/evals/security-issue-sync/step-5b-push-decision/fixtures/case-7-not-configured-fix-released/report.md
@@ -0,0 +1,33 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+
+Tracker #402 — CVE-2026-41107, "Stored XSS in the task instance notes view"
+Transition confirmed by the operator this run: `pr merged` → `fix released`
+(apache-airflow 2.10.5 is on PyPI).
+
+Step 5a result: regenerated and attached (`--attach`). The generator's
+`CNA_private.state` in the regenerated JSON is `REVIEW` (`review-ready`).
+
+CVE record state in the CVE tool (Vulnogram): `DRAFT` (`allocated`).
+
+Regenerated JSON excerpt:
+
+```json
+{
+ "containers": {
+ "cna": {
+ "title": "Stored XSS in the task instance notes view",
+ "descriptions": [{"lang": "en", "value": "A user with permission to edit task instance notes could store JavaScript that runs in the browser of another user who opens the task instance notes view. Users are recommended to upgrade to apache-airflow 2.10.5, which fixes the issue."}],
+ "problemTypes": [{"descriptions": [{"lang": "en", "type": "CWE", "cweId": "CWE-79", "description": "CWE-79: Improper Neutralization of Input During Web Page Generation ('Cross-site Scripting')"}]}],
+ "affected": [{"vendor": "Apache Software Foundation", "product": "Apache Airflow", "versions": [{"version": "0", "versionType": "semver", "lessThan": "2.10.5", "status": "affected"}]}],
+ "credits": [{"lang": "en", "value": "Maria Kowalska", "type": "finder"}]
+ }
+ },
+ "CNA_private": {"state": "REVIEW"}
+}
+```
+
+Provenance: the reporter wrote to `security@` directly under her real
+name and asked to be credited. Not a follow-up to any earlier CVE.
+
+Adapter session probe (`vulnogram-api-check`): **`not-configured`** — this operator has never set up the adapter.
diff --git a/tools/skill-evals/evals/security-issue-sync/step-5b-push-decision/fixtures/case-8-not-configured-no-transition/expected.json b/tools/skill-evals/evals/security-issue-sync/step-5b-push-decision/fixtures/case-8-not-configured-no-transition/expected.json
new file mode 100644
index 00000000..443969bb
--- /dev/null
+++ b/tools/skill-evals/evals/security-issue-sync/step-5b-push-decision/fixtures/case-8-not-configured-no-transition/expected.json
@@ -0,0 +1,8 @@
+{
+ "action": "hand-off",
+ "gate_failed": null,
+ "handoff_variant": "manual-paste",
+ "state_advance": null,
+ "recap_blocker": false,
+ "reason": "The probe returns not-configured, so the push is skipped silently and the manual-paste hand-off is used. The record is already review-ready and there is no fix released transition, so no blocker is raised."
+}
diff --git a/tools/skill-evals/evals/security-issue-sync/step-5b-push-decision/fixtures/case-8-not-configured-no-transition/report.md b/tools/skill-evals/evals/security-issue-sync/step-5b-push-decision/fixtures/case-8-not-configured-no-transition/report.md
new file mode 100644
index 00000000..003a8cd6
--- /dev/null
+++ b/tools/skill-evals/evals/security-issue-sync/step-5b-push-decision/fixtures/case-8-not-configured-no-transition/report.md
@@ -0,0 +1,34 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+
+Tracker #405 — CVE-2026-41120, "Stored XSS in the task instance notes view"
+No label transition this run. The tracker has been `fix released` since an
+earlier sync; this run only rewrote the *Short public summary for publish* field,
+which is CVE-affecting, so Step 5a regenerated the JSON.
+
+Step 5a result: regenerated and attached (`--attach`). The generator's
+`CNA_private.state` in the regenerated JSON is `REVIEW` (`review-ready`).
+
+CVE record state in the CVE tool (Vulnogram): `REVIEW` (`review-ready`).
+
+Regenerated JSON excerpt:
+
+```json
+{
+ "containers": {
+ "cna": {
+ "title": "Stored XSS in the task instance notes view",
+ "descriptions": [{"lang": "en", "value": "A user with permission to edit task instance notes could store JavaScript that runs in the browser of another user who opens the task instance notes view. Users are recommended to upgrade to apache-airflow 2.10.5, which fixes the issue."}],
+ "problemTypes": [{"descriptions": [{"lang": "en", "type": "CWE", "cweId": "CWE-79", "description": "CWE-79: Improper Neutralization of Input During Web Page Generation ('Cross-site Scripting')"}]}],
+ "affected": [{"vendor": "Apache Software Foundation", "product": "Apache Airflow", "versions": [{"version": "0", "versionType": "semver", "lessThan": "2.10.5", "status": "affected"}]}],
+ "credits": [{"lang": "en", "value": "Maria Kowalska", "type": "finder"}]
+ }
+ },
+ "CNA_private": {"state": "REVIEW"}
+}
+```
+
+Provenance: the reporter wrote to `security@` directly under her real
+name and asked to be credited. Not a follow-up to any earlier CVE.
+
+Adapter session probe (`vulnogram-api-check`): **`not-configured`** — this operator has never set up the adapter.
diff --git a/tools/skill-evals/evals/security-issue-sync/step-bulk-selectors/fixtures/case-7-cve-selector-never-skipped/expected.json b/tools/skill-evals/evals/security-issue-sync/step-bulk-selectors/fixtures/case-7-cve-selector-never-skipped/expected.json
new file mode 100644
index 00000000..84a662fb
--- /dev/null
+++ b/tools/skill-evals/evals/security-issue-sync/step-bulk-selectors/fixtures/case-7-cve-selector-never-skipped/expected.json
@@ -0,0 +1,14 @@
+{
+ "error": null,
+ "invalid_tokens": [],
+ "searched_cve_ids": [
+ "CVE-2026-40777"
+ ],
+ "resolved": [
+ 260
+ ],
+ "dispatched": [
+ 260
+ ],
+ "skipped": []
+}
diff --git a/tools/skill-evals/evals/security-issue-sync/step-bulk-selectors/fixtures/case-7-cve-selector-never-skipped/report.md b/tools/skill-evals/evals/security-issue-sync/step-bulk-selectors/fixtures/case-7-cve-selector-never-skipped/report.md
new file mode 100644
index 00000000..7eb12434
--- /dev/null
+++ b/tools/skill-evals/evals/security-issue-sync/step-bulk-selectors/fixtures/case-7-cve-selector-never-skipped/report.md
@@ -0,0 +1,28 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+
+Current time: 2026-10-03T12:00:00Z
+
+Operator selector:
+
+```text
+sync CVE-2026-40777
+```
+
+Mocked `gh search issues "CVE-2026-40777" --repo <tracker> --json number,title,body`
+(before the `--jq` body filter):
+
+```json
+[
+ {"number": 260, "title": "Scheduler pickle deserialization",
+ "body": "### CVE tool link\n\nhttps://cveprocess.apache.org/cve5/CVE-2026-40777"}
+]
+```
+
+Mocked pre-flight GraphQL state.
+"Marker" means the last comment's body starts with `<!-- apache-magpie: status-rollup v1 -->`
+— a skill-authored rollup proposal posted under the operator's own account (`potiuk`) that nobody has answered yet.
+
+| Issue | state | updatedAt | labels | last comment author | last comment createdAt | last comment body |
+|---|---|---|---|---|---|---|
+| 260 | OPEN | 2026-09-12T10:00:00Z | airflow, cve allocated, pr merged, security issue | potiuk | 2026-09-12T10:00:00Z | Marker |
diff --git a/tools/skill-evals/evals/security-issue-sync/step-bulk-selectors/fixtures/case-8-reporter-reply-urgent/expected.json b/tools/skill-evals/evals/security-issue-sync/step-bulk-selectors/fixtures/case-8-reporter-reply-urgent/expected.json
new file mode 100644
index 00000000..33026f7c
--- /dev/null
+++ b/tools/skill-evals/evals/security-issue-sync/step-bulk-selectors/fixtures/case-8-reporter-reply-urgent/expected.json
@@ -0,0 +1,17 @@
+{
+ "error": null,
+ "invalid_tokens": [],
+ "searched_cve_ids": [],
+ "resolved": [
+ 270,
+ 271
+ ],
+ "dispatched": [
+ 270,
+ 271
+ ],
+ "urgent": [
+ 270
+ ],
+ "skipped": []
+}
diff --git a/tools/skill-evals/evals/security-issue-sync/step-bulk-selectors/fixtures/case-8-reporter-reply-urgent/report.md b/tools/skill-evals/evals/security-issue-sync/step-bulk-selectors/fixtures/case-8-reporter-reply-urgent/report.md
new file mode 100644
index 00000000..9aa344c3
--- /dev/null
+++ b/tools/skill-evals/evals/security-issue-sync/step-bulk-selectors/fixtures/case-8-reporter-reply-urgent/report.md
@@ -0,0 +1,26 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+
+Current time: 2026-10-03T12:00:00Z
+
+Operator selector:
+
+```text
+sync all open
+```
+
+Mocked `gh issue list --repo <tracker> --state open --limit 100 --json number,title,labels`:
+
+```json
+[
+ {"number": 270, "title": "REST API pagination leaks hidden DAGs", "labels": [{"name": "airflow"}, {"name": "cve allocated"}]},
+ {"number": 271, "title": "Trigger form stored XSS", "labels": [{"name": "airflow"}, {"name": "needs triage"}]}
+]
+```
+
+Mocked pre-flight GraphQL state:
+
+| Issue | state | updatedAt | labels | last comment author | last comment createdAt | last comment body |
+|---|---|---|---|---|---|---|
+| 270 | OPEN | 2026-10-03T09:00:00Z | airflow, cve allocated, security issue | dana-reporter | 2026-10-03T09:00:00Z | `Any news on the fix? I can retest today…` |
+| 271 | OPEN | 2026-09-30T14:00:00Z | airflow, needs triage, security issue | erik-maintainer | 2026-09-30T14:00:00Z | `I can reproduce this on main…` |
diff --git a/tools/skill-evals/evals/security-issue-sync/step-bulk-selectors/fixtures/output-spec.md b/tools/skill-evals/evals/security-issue-sync/step-bulk-selectors/fixtures/output-spec.md
index c7cbac8d..ca4a4713 100644
--- a/tools/skill-evals/evals/security-issue-sync/step-bulk-selectors/fixtures/output-spec.md
+++ b/tools/skill-evals/evals/security-issue-sync/step-bulk-selectors/fixtures/output-spec.md
@@ -22,6 +22,7 @@ Return ONLY valid JSON with this shape:
"searched_cve_ids": [],
"resolved": [],
"dispatched": [],
+ "urgent": [],
"skipped": [],
"review_ready_push_bucket": null
}
@@ -44,6 +45,8 @@ Field rules:
once the user confirms the echoed list (or, for a confirmation-time
command, on the next turn), ascending. Covers both `dispatch` and
`dispatch-urgent` classifications.
+- `urgent`: the subset of `dispatched` classified `dispatch-urgent`,
+ ascending. Empty list when none is.
- `skipped`: one object per tracker the pre-flight classifier marks
`skip-noop`, ascending by issue number:
`{"issue": <N>, "rule": "<enum>"}` where `rule` is one of