You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@magpie.apache.org by po...@apache.org on 2026/10/19 15:38:33 UTC
(magpie) branch perf/security-4b2-sync created (now eafc6b01)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch perf/security-4b2-sync
in repository https://gitbox.apache.org/repos/asf/magpie.git
at eafc6b01 perf(security-issue-sync): compute the Step 1 date and marker observations as code
This branch includes the following new commits:
new eafc6b01 perf(security-issue-sync): compute the Step 1 date and marker observations as code
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(magpie) 01/01: perf(security-issue-sync): compute the Step 1 date and marker observations as code
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch perf/security-4b2-sync
in repository https://gitbox.apache.org/repos/asf/magpie.git
commit eafc6b01e64b8b8ebe25e579819aef97d95e6e11
Author: Jarek Potiuk <po...@apache.org>
AuthorDate: Mon Oct 19 17:34:06 2026 +0200
perf(security-issue-sync): compute the Step 1 date and marker observations as code
The Step 1 observations that are arithmetic move into the issue-sync
family:
- disclosure (1a): the issue's age and the overdue, approaching,
grace-expired and distributor-pending flags from the configured window;
- staleness (1c): whether the reporter thread is stale against the
response timeout, the clock stopped while the reporter has the last word;
- closed-announced (1g): the cve.org move (notify, already notified,
wait, blocker, retry, skip), the security-pages checklist and reminder
marker scan, and the advisory-records step and planner exit;
- release-vote (1h): an open [VOTE] thread whose version matches the
fix-PR milestone, a [RESULT] closing it.
gather.md shrinks by 42 lines, and the multi-scope split procedure moves
from the SKILL.md Guardrails to outcomes/scope-split.md behind a one-line
rule, bringing SKILL.md to 6,948 tokens (7,006 before layer 4b). The 38
remaining model cases of the skill pass.
Part of #1672
Generated-by: Claude Opus 5
---
plugins/magpie-security/skills/issue-sync/SKILL.md | 34 +--
.../magpie-security/skills/issue-sync/gather.md | 106 +++------
.../skills/issue-sync/outcomes/scope-split.md | 38 +++
tools/security-management/README.md | 7 +-
.../src/security_management/issue_sync/cli.py | 10 +-
.../src/security_management/issue_sync/state.py | 225 ++++++++++++++++++
.../tests/test_issue_sync_state.py | 256 +++++++++++++++++++++
tools/spec-loop/specs/security-issue-lifecycle.md | 5 +-
8 files changed, 574 insertions(+), 107 deletions(-)
diff --git a/plugins/magpie-security/skills/issue-sync/SKILL.md b/plugins/magpie-security/skills/issue-sync/SKILL.md
index 65030bed..4bf40253 100644
--- a/plugins/magpie-security/skills/issue-sync/SKILL.md
+++ b/plugins/magpie-security/skills/issue-sync/SKILL.md
@@ -25,7 +25,7 @@ argument-hint: "[issue-number]"
capability: capability:intake
surface_hash: sha256:5c4aad61d458d98f
license: Apache-2.0
-measured_tokens: 7282
+measured_tokens: 6948
---
<!-- Placeholder convention (see AGENTS.md#placeholder-convention-used-in-skill-files):
@@ -437,35 +437,9 @@ Fix every hit and check again; it is the floor, not the whole review, so still r
release-train state in
[`<project-config>/release-trains.md`](../../../../<project-config>/release-trains.md).
- **Multi-scope reports must be split into one tracking issue per
- scope.** When a report affects more than one scope (for example a root cause in a shared core module
- whose vector also exists in a plugin/extension component), never apply two scope labels to one issue;
- propose the split instead:
-
- 1. Keep the original issue on the scope whose milestone family ships *first*
- (usually core, whose patch releases cut faster), and drop the extra scope label from it.
- 2. Create one new issue per remaining scope via `gh issue create
- --repo <tracker>`, copying the report body
- verbatim but with a one-line preamble that says *"Split from
- [#NNN](https://github.com/<tracker>/issues/<N>) for the `<scope>` scope — see that issue for the
- full discussion history."*
- 3. Apply to each split issue:
- - exactly one scope label (see
- [`<project-config>/scope-labels.md`](../../../../<project-config>/scope-labels.md));
- - the same `cve allocated` label if a CVE is shared across
- scopes — CVE reuse is correct when the same upstream bug
- affects multiple products, with one `affected[]` entry per
- product in the CVE record;
- - the PR / advisory labels (`pr created` / `pr merged` /
- `fix released`) derived independently per scope from the same
- fix PR, because each scope rides a different release train;
- - the matching milestone for that scope (see
- [`<project-config>/milestones.md`](../../../../<project-config>/milestones.md));
- - the same assignee set as the anchor issue.
- 4. Post a cross-link comment on **each** issue pointing at the other(s).
- 5. Update the open reporter email draft, if any, to mention the split and link every tracker.
-
- Do **not** silently drop a scope label without splitting: each scope's release managers need the issue on their own milestone.
- A single issue with two scope labels is a process bug — flag it as a **blocker** and propose the split as a concrete numbered item.
+ scope.** Never apply two scope labels to one issue: a single issue with two is a process bug, flagged as a **blocker** with the split proposed as a concrete numbered item
+ (keep the original on the scope whose release ships first; one new issue per remaining scope with the report verbatim, its own scope label, labels and milestone; cross-links on each).
+ Do **not** silently drop a scope label without splitting. The procedure: [`outcomes/scope-split.md`](outcomes/scope-split.md).
---
diff --git a/plugins/magpie-security/skills/issue-sync/gather.md b/plugins/magpie-security/skills/issue-sync/gather.md
index 97ca5bfb..6d0877e1 100644
--- a/plugins/magpie-security/skills/issue-sync/gather.md
+++ b/plugins/magpie-security/skills/issue-sync/gather.md
@@ -48,15 +48,11 @@ Record:
- CVE ID (if allocated),
- link to the fixing PR(s);
- the discussion so far (comments), paying attention to the most recent activity and any stalled-for-30-days state.
-- **Disclosure deadline check.** Compute `issue_age_days` as calendar days from `createdAt` to today,
- compare it with `window_days` and `grace_period_days` from Step 0.6, and set these flags in the observed-state bag:
- - `overdue_for_disclosure: true` — `issue_age_days > window_days` **and** no `announced` label (the advisory has not been sent).
- - `approaching_window_end: true` — `issue_age_days > window_days * 0.8` (≥ 80 % of the window elapsed) and not yet `announced`;
- a yellow-flag caution in the observed-state dump, with no proposal item.
- - `grace_period_expired: true` — the issue carries `pr merged` or `fix released` **and** `issue_age_days > window_days + grace_period_days`,
- and no `announced` label.
- - `distributor_notify_pending: true` — `pre_announce_distributors: true` (Step 0.6) **and** the issue carries `pr merged` or `fix released`,
- and no `announced` label; Step 2b then proposes a pre-announcement draft for the distributor embargo list.
+- **Disclosure deadline check.** Write `{now, created_at, labels, window_days, grace_period_days, pre_announce_distributors}` (the Step 0.6 values) and run `<sm> disclosure`;
+ record its `issue_age_days` and flags in the observed-state bag:
+ `overdue_for_disclosure` (past the window, no `announced`), `approaching_window_end` (past 80 % of it — a yellow flag with no proposal item),
+ `grace_period_expired` (`pr merged` / `fix released` and past window plus grace) and `distributor_notify_pending` (Step 2b then proposes the distributor pre-announcement).
+ `<sm>` is the `security-management issue-sync` command of [`SKILL.md`](SKILL.md#prerequisites).
The flags are informational observations; they do not stop the sync.
A missing `announced` label on a tracker whose body has the `announced` date populated is a missing label (propose adding it), not overdue.
@@ -189,16 +185,13 @@ Process for finding the real reporter and the original thread:
- **every status update the security team has already sent the reporter** — this decides whether a new one is needed (Step 2b);
- the latest message, *who* sent it, and whether the ball is in our court.
-5. **Check for reporter staleness.** From the step 4 timestamps and senders, find the team's **most recent outbound message** on this thread
- and the reporter's **most recent reply**, if any.
- If the team's latest outbound message is older than `security_inbox.reporter_response_timeout_days`
- (resolved per the [configuration resolution order](../../../../AGENTS.md#configuration-resolution-order)
- — `<project-config>/project.md`, else the project's organization; both shipped organizations default to 14)
- **and** no reporter reply has landed since, mark the thread **stale** and surface it in Step 2b per the *Reporter unresponsiveness* row in
- [`signals-to-actions.md`](signals-to-actions.md#step-2b--proposed-changes-signal-to-action-lookup-table).
- It is a **proposal, not an automatic action** (policy: [`SKILL.md`](SKILL.md) Step 2b):
- the user confirms before the team proceeds without the reporter.
- Skip the check when the latest message is *from* the reporter — the clock runs only while we wait on a reply.
+5. **Check for reporter staleness.** Write the step 4 messages as `{at, from_team}` with `now` and `timeout_days`
+ (`security_inbox.reporter_response_timeout_days`, resolved per the [configuration resolution order](../../../../AGENTS.md#configuration-resolution-order); 14 in both shipped organizations)
+ and run `<sm> staleness`: the thread is **stale** when the team's latest outbound message is older than the timeout with no reporter reply since,
+ and the clock never runs while the latest message is the reporter's.
+ A stale thread is surfaced in Step 2b per the *Reporter unresponsiveness* row in
+ [`signals-to-actions.md`](signals-to-actions.md#step-2b--proposed-changes-signal-to-action-lookup-table) —
+ a **proposal, not an automatic action** (policy: [`SKILL.md`](SKILL.md) Step 2b): the user confirms before the team proceeds without the reporter.
6. **Sync a reporter-confirmed credit line into the issue body** whenever the thread holds a clear credit confirmation from the reporter
not yet reflected in the tracker's *"Reporter credited as"* field;
@@ -549,18 +542,11 @@ For each closed-`announced` tracker in this run:
```bash
uv run --project ~/.claude/magpie/vetted-ops vetted-op-read --caller security-issue-sync cve-check-published <CVE-ID>
```
-3. Interpret:
- - `state == "PUBLISHED"` → capture `datePublished` and propose the *CVE-published* reporter email in Step 2b.
- - `state == "RESERVED"` → record *"cve.org shows RESERVED; propagation not complete yet"* in the observed state;
- no email yet; a future sync run will catch the publication.
- - `state == "REJECTED"` → **surface as a blocker**: the record was withdrawn post-publication.
- Do not draft a reporter email; flag it to the security team.
- - lookup error (non-zero exit from `vetted-op-read` — exit 4) → record *"cve.org lookup failed — <short error> — try again next sync"*.
- Do not propose notification on an absent response.
-
-**Idempotence.** If the tracker's comment trail has a prior *"Sync YYYY-MM-DD — CVE-published reporter notification drafted"* status-change comment
-and the reporter thread already carries a corresponding sent message,
-skip the proposal and record *"CVE-published notification already sent on <date>"*.
+3. Write `{cve_id, cve_org_state, date_published, lookup_failed, notification_sent_on_thread, body, comments, public_advisory_url, advisory_records_configured, planner_exit, planner_result, planner_reason}`
+ and run `<sm> closed-announced`. Its `cve_org` is the move:
+ `notify` (propose the *CVE-published* reporter email in Step 2b), `already_notified` (a prior *"CVE-published reporter notification drafted"* comment and a sent message on the thread — record it),
+ `wait` (*"cve.org shows RESERVED; propagation not complete yet"*), `blocker` (the record was rejected after publication: no email, flag it to the security team),
+ `retry` (*"cve.org lookup failed — try again next sync"*; never propose a notification on an absent response) or `skip` (no CVE ID: a closing disposition, dropped from the sweep).
**Gmail-budget.** The cve.org check is one HTTP call per tracker, not metered against the Gmail budget,
but within the "≤ 1 extra HTTP round-trip per tracker" soft limit for closed-bucket scans:
@@ -568,31 +554,19 @@ with several closed trackers in scope, run the checks in parallel via the subage
**Security-pages checklist.** The same closed-`announced` bucket is the only place the post-announcement security-pages step can be observed:
the Step 14 close-out adds `announced` and closes the tracker in one apply, so an open tracker never carries the trigger state.
-For each closed-`announced` tracker whose *"Public advisory URL"* body field is populated,
-scan the issue body and every comment for the ticked checklist item
-```html
-- [x] <!-- apache-magpie: security-pages-checklist v1 -->
-```
-and, separately, for a reminder comment carrying
-```html
-<!-- apache-magpie: security-pages-reminder v1 -->
-```
-on its first line.
-Record `security_pages_reminder_pending: true` in the observed state only when **no** ticked checklist item exists **and** no reminder comment exists;
-an unticked box in the hand-off comment is the pending state, not the satisfied one.
+The same command scans the body and every comment the 1a fetch returned — no extra API call — for the ticked checklist item
+`- [x] <!-- apache-magpie: security-pages-checklist v1 -->` and for a comment opening with `<!-- apache-magpie: security-pages-reminder v1 -->`,
+and sets `security_pages_reminder_pending` only when the *Public advisory URL* is populated and neither exists
+(an unticked box in the hand-off comment is the pending state, not the satisfied one).
Step 2b turns a pending flag into the *Security-pages reminder comment* proposal ([`signals-to-actions.md`](signals-to-actions.md)).
-The scan reads the body and comments the 1a fetch already returned — no extra API call.
**Advisory records.** When [`<project-config>/project.md`](../../../../<project-config>/project.md#advisory-records) declares an `advisory_records` block,
sync also records the CVE in the project's own security records — a site repository over Git or Subversion
([`docs/security/advisory-records.md`](../../../../docs/security/advisory-records.md)).
Without the block, record *"advisory records: not configured — skipped"* in the observed state and do nothing else.
With it, `security_pages_reminder_pending` is always `false`: sync proposes the edit itself instead of reminding the release manager.
-For each closed-`announced` tracker:
+`closed-announced` prints the step in `advisory_records`: `done` (an `advisory-records v1` comment exists), `waiting_for_cve_org` (not `PUBLISHED` yet; the tool refuses before publication too — never work around it) or `run_planner`, then:
-1. Skip when a comment carrying `<!-- apache-magpie: advisory-records v1 -->` exists: record *"advisory records: done on <date>"*.
-2. Skip unless the cve.org check above returned `PUBLISHED`: record *"advisory records: waiting for cve.org"*.
- The tool refuses before publication too; never work around it.
3. Write the record to the session scratch directory:
`uv run --project <framework>/tools/cve-tool-vulnogram/generate-cve-json generate-cve-json <N> --output <scratch>/<CVE-ID>.json`.
4. Run the planner as a plain command, without `--apply`.
@@ -605,12 +579,10 @@ For each closed-`announced` tracker:
For `release-page`, pass `--reported-date` (the date of the first message on the *Security mailing list thread*).
When the output names a missing input (`--release-date` for a new release section, `--fix-commit`, `--affects`), find it on the tracker,
in [`release-trains.md`](../../../../<project-config>/release-trains.md) or in the release's announcement, and run it again; ask the user when none of them has it.
-5. Read the exit code:
- `0` with `proposed` → `advisory_records_pending: true`, keeping the tool's diff and numbered commands for Step 2b;
- `0` with `unchanged` → *"advisory records: already recorded"*;
- `4` → `advisory_records_pending: true` with the single `checkout` command;
- `3` → *"advisory records: refused — <reason>"*, surfaced as a blocker when the page cannot be parsed or already holds the CVE under another release;
- `2` → a configuration error to show the user.
+5. Pass the planner's exit code (and `proposed` / `unchanged`, and a refusal's reason) back to `closed-announced`:
+ `pending` → `advisory_records_pending: true`, keeping the tool's diff and numbered commands for Step 2b;
+ `already_recorded`; `pending_checkout` → `advisory_records_pending: true` with the single `checkout` command;
+ `refused` → a blocker when the page cannot be parsed or already holds the CVE under another release; `config_error` → show the user.
**When the tracker has no CVE ID.** Closed trackers without a `CVE-YYYY-NNNNN` in the *CVE tool link* body field
are closing dispositions (`invalid` / `duplicate` / `wontfix`):
@@ -663,26 +635,12 @@ mcp__ponymail__search_list(
)
```
-**Matching against the tracker's fix milestone.** For each returned thread:
-
-1. Extract the version from the subject line.
- ASF [VOTE] subjects follow `[VOTE] Release <Project> <X.Y.Z> from <X.Y.Z>rcN`
- (sometimes with sibling artifacts on the same vote — `& Task SDK <A.B.C> from <A.B.C>rcM`).
- Parse the **first** `<X.Y.Z>` token after the project name.
-2. Compare it with the fix-PR milestone — the milestone on the PR, not on the tracker:
- the tracker's may be a wave-month label like `<scope-b> 2026-04-21`, while the PR's carries the actual carrier version like `<product> 3.2.3`.
- For wave-based component milestones, also check that the vote's release ships the right *component wave*:
- the tracker's wave milestone (e.g. *<scope-b> 2026-04-21*) maps to a `[VOTE] Release <scope-b> …` thread.
-3. Check the thread's most recent message: a `[RESULT][VOTE]` reply is a closed vote; open votes have **no** `[RESULT]` reply yet.
- The `[VOTE]` query above already returns the `[RESULT][VOTE]` messages (their subjects contain `[VOTE]`), so decide open / closed from that one result set — no per-thread fetch.
- Closed votes get no label add: either the release shipped (the `fix released` flow takes over)
- or the vote failed (the team cuts a fresh RC, and the next sync picks up the new `[VOTE]` thread).
-
-**Record in the per-tracker state bag:**
-
-- `release_vote_in_progress`: `true` when a tracker in the `pr merged` window has a matching open `[VOTE]` thread on the right list.
-- `release_vote_thread_url`: the PonyMail thread URL if found (`<mail-archive-url>/thread/<hash>?<list>@<domain>`),
- used as the rationale in the Step 2b proposal.
+**Matching against the tracker's fix milestone.** Write `{gating, labels, fix_milestone, release_product, messages}` — the search results' `subject` and `url`,
+the milestone on the fix **PR** (it carries the carrier version, e.g. `<product> 3.2.3`, where the tracker's may be a wave label like `<scope-b> 2026-04-21`) —
+and run `<sm> release-vote`. It takes the first version after `Release` in each `[VOTE] Release <Project> <X.Y.Z> from <X.Y.Z>rcN` subject,
+treats a `[RESULT][VOTE]` message in the same result set as a closed vote (no label add: the release shipped or the vote failed and a fresh RC follows),
+and records `release_vote_in_progress` and `release_vote_thread_url` for a tracker in the `pr merged` window with a matching open vote;
+`release_product` keeps a component wave's vote to its own `[VOTE] Release <scope-b> …` thread.
**Hard rule — no auto-apply.** The result feeds Step 2b's proposal and applies only after the user confirms.
Never add `rc voting` silently: the next regen flips `CNA_private.state` to `REVIEW`,
diff --git a/plugins/magpie-security/skills/issue-sync/outcomes/scope-split.md b/plugins/magpie-security/skills/issue-sync/outcomes/scope-split.md
new file mode 100644
index 00000000..9fdf6819
--- /dev/null
+++ b/plugins/magpie-security/skills/issue-sync/outcomes/scope-split.md
@@ -0,0 +1,38 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+
+# security-issue-sync — splitting a multi-scope report
+
+The Guardrails rule in [`SKILL.md`](../SKILL.md#guardrails), with the split procedure.
+Relative links are rooted one level deeper than in `SKILL.md`.
+
+- **Multi-scope reports must be split into one tracking issue per
+ scope.** When a report affects more than one scope (for example a root cause in a shared core module
+ whose vector also exists in a plugin/extension component), never apply two scope labels to one issue;
+ propose the split instead:
+
+ 1. Keep the original issue on the scope whose milestone family ships *first*
+ (usually core, whose patch releases cut faster), and drop the extra scope label from it.
+ 2. Create one new issue per remaining scope via `gh issue create
+ --repo <tracker>`, copying the report body
+ verbatim but with a one-line preamble that says *"Split from
+ [#NNN](https://github.com/<tracker>/issues/<N>) for the `<scope>` scope — see that issue for the
+ full discussion history."*
+ 3. Apply to each split issue:
+ - exactly one scope label (see
+ [`<project-config>/scope-labels.md`](../../../../../<project-config>/scope-labels.md));
+ - the same `cve allocated` label if a CVE is shared across
+ scopes — CVE reuse is correct when the same upstream bug
+ affects multiple products, with one `affected[]` entry per
+ product in the CVE record;
+ - the PR / advisory labels (`pr created` / `pr merged` /
+ `fix released`) derived independently per scope from the same
+ fix PR, because each scope rides a different release train;
+ - the matching milestone for that scope (see
+ [`<project-config>/milestones.md`](../../../../../<project-config>/milestones.md));
+ - the same assignee set as the anchor issue.
+ 4. Post a cross-link comment on **each** issue pointing at the other(s).
+ 5. Update the open reporter email draft, if any, to mention the split and link every tracker.
+
+ Do **not** silently drop a scope label without splitting: each scope's release managers need the issue on their own milestone.
+ A single issue with two scope labels is a process bug — flag it as a **blocker** and propose the split as a concrete numbered item.
diff --git a/tools/security-management/README.md b/tools/security-management/README.md
index 7b552d4e..6c702fc5 100644
--- a/tools/security-management/README.md
+++ b/tools/security-management/README.md
@@ -255,6 +255,10 @@ Every command takes `--input F`, a JSON file.
| `draft-backend` | `backend` (`maildir`, `oauth_curl`, `claude_ai_mcp`), `decided_by` (`user.md`, `project.md`, `default`); no credential with links in the body exits `2`. |
| `ghsa-route` | `access_tier`, `action`, `channel`, and for an admin hand-off the draft tool, addressee, thread, rollup marker and three parts. |
| `push-decision` | `action` (`skip`, `block`, `hand-off`, `push`, `pending` with the `next` command or gate), `gate_failed`, `handoff_variant`, `state_advance`, `recap_blocker`. |
+| `disclosure` | `issue_age_days`, `overdue_for_disclosure`, `approaching_window_end`, `grace_period_expired`, `distributor_notify_pending`, `expected_disclosure_by`. |
+| `staleness` | `stale` and `days`: the team's latest message older than the timeout with no reporter reply since. |
+| `closed-announced` | `cve_org` (`notify`, `already_notified`, `wait`, `blocker`, `retry`, `skip`), `security_pages_reminder_pending`, `advisory_records` and its next step, `blockers`. |
+| `release-vote` | `release_vote_in_progress` and `release_vote_thread_url` for an open `[VOTE]` matching the fix-PR milestone. |
| `render-recap` | The recap `text`, `bare_refs`. |
## Tests
@@ -274,5 +278,6 @@ it pins the `step-26-precedent-search` signal rule, which stays a model eval.
`tests/test_issue_invalidate.py` holds the cases of the retired `security-issue-invalidate` suites `step-2-import-path`, `step-5a-labels`, `step-5b-closing-comment`, `step-5d-email-draft`, `step-5e-rollup-entry`, `step-5f-confirm` and `step-7-recap`.
`tests/test_cve_allocate.py` holds the cases of the retired `security-cve-allocate` suites `step-1-blocker-checks`, `step-2-title-normalize`, `step-3-allocation-recipe`, `step-4-propose-updates`, `step-5-confirm` and `step-7-recap`.
`tests/test_issue_deduplicate.py` holds the cases of the retired `security-issue-deduplicate` suites `step-1-classify`, `step-2-extract-fields`, `step-3-merge-body`, `step-4-rollup-entries`, `step-5-confirm` and `step-6-recap`.
-`tests/test_issue_sync.py` and `tests/test_issue_sync_rules.py` hold the cases of the retired `security-issue-sync` suites `step-bulk-selectors`, `step-bulk-orchestration`, `step-3-confirm`, `step-6-recap`, `step-guardrails`, `step-security-cc`, `step-draft-backend`, `step-ghsa-access-tier` and `step-5b-push-decision`.
+`tests/test_issue_sync_state.py` pins the `security-issue-sync` Step 1 observations;
+`tests/test_issue_sync.py` and `tests/test_issue_sync_rules.py` hold the cases of its retired suites `step-bulk-selectors`, `step-bulk-orchestration`, `step-3-confirm`, `step-6-recap`, `step-guardrails`, `step-security-cc`, `step-draft-backend`, `step-ghsa-access-tier` and `step-5b-push-decision`.
`tests/test_body.py` checks that the `tracker-body-template.md` files of the sibling import skills carry exactly the rendered headings.
diff --git a/tools/security-management/src/security_management/issue_sync/cli.py b/tools/security-management/src/security_management/issue_sync/cli.py
index af7b4a9e..0d9cf2eb 100644
--- a/tools/security-management/src/security_management/issue_sync/cli.py
+++ b/tools/security-management/src/security_management/issue_sync/cli.py
@@ -30,7 +30,7 @@ from collections.abc import Mapping
from typing import Any
from security_management.cli import emit, read_object, refuse
-from security_management.issue_sync import advisory, bulk, preflight, push, render, select
+from security_management.issue_sync import advisory, bulk, preflight, push, render, select, state
from security_management.shared import drafting, numbered, recipients
FAMILY = "issue-sync"
@@ -48,6 +48,10 @@ COMMANDS = {
"draft-backend": "Drafts: Gmail or Maildir, then the Gmail backend (JSON: user_backend, project_backend, oauth_credentials)",
"ghsa-route": "GHSA trackers: the access tier and how a change is delivered",
"push-decision": "Step 5b: skip, block, hand-off or push, and the next command",
+ "disclosure": "Step 1a: the issue's age and the disclosure-window flags",
+ "staleness": "Step 1c: whether the reporter thread is stale (JSON: now, messages, timeout_days)",
+ "closed-announced": "Step 1g: the cve.org state, the security-pages markers and the advisory records",
+ "release-vote": "Step 1h: an open [VOTE] thread matching the fix-PR milestone",
"render-recap": "Step 6: the recap",
}
@@ -91,6 +95,10 @@ def dispatch(args: argparse.Namespace) -> int:
"draft-backend": _backend,
"ghsa-route": advisory.route,
"push-decision": push.decide,
+ "disclosure": state.disclosure,
+ "staleness": state.staleness,
+ "closed-announced": state.closed_announced,
+ "release-vote": state.release_vote,
"render-recap": render.recap,
}
result = handlers[args.command](data)
diff --git a/tools/security-management/src/security_management/issue_sync/state.py b/tools/security-management/src/security_management/issue_sync/state.py
new file mode 100644
index 00000000..55949af9
--- /dev/null
+++ b/tools/security-management/src/security_management/issue_sync/state.py
@@ -0,0 +1,225 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Step 1 observations that are arithmetic on dates, labels and markers.
+
+- `disclosure` (1a): `issue_age_days` from `createdAt`, and the flags
+ `overdue_for_disclosure` (past `window_days`, not `announced`),
+ `approaching_window_end` (past 80 % of it), `grace_period_expired` (`pr
+ merged` / `fix released`, past `window_days + grace_period_days`) and
+ `distributor_notify_pending` (`pre_announce_distributors`, `pr merged` /
+ `fix released`, not `announced`). Defaults 90 / 14 / false.
+- `staleness` (1c step 5): stale when the team's latest outbound message is
+ older than `reporter_response_timeout_days` (default 14) and the
+ reporter has not replied since; the clock never runs while the latest
+ message is the reporter's.
+- `closed_announced` (1g): the cve.org state as the next move (`PUBLISHED`
+ → the CVE-published email unless a prior notification is recorded,
+ `RESERVED` → wait, `REJECTED` → blocker, lookup error → retry), the
+ security-pages checklist and reminder markers in the body and comments,
+ and the advisory-records step, including the planner's exit code.
+- `release_vote` (1h): an open `[VOTE]` thread on the dev list whose first
+ version matches the fix-PR milestone, for a tracker in the `pr merged`
+ window; a `[RESULT]` message closes a vote.
+"""
+
+from __future__ import annotations
+
+import re
+from collections.abc import Mapping
+from datetime import datetime, timedelta
+from typing import Any
+
+from security_management.issue_sync.select import _labels, _time
+from security_management.shared.rollup import one_line
+
+CHECKLIST_DONE = re.compile(
+ r"^\s*[-*] \[[xX]\] <!-- apache-magpie: security-pages-checklist v1 -->", re.MULTILINE
+)
+REMINDER = "<!-- apache-magpie: security-pages-reminder v1 -->"
+RECORDS = "<!-- apache-magpie: advisory-records v1 -->"
+NOTIFIED = re.compile(r"CVE-published reporter notification drafted", re.IGNORECASE)
+_VERSION = re.compile(r"\b(\d+\.\d+(?:\.\d+)?)\b")
+_VOTE = re.compile(r"\[VOTE\]", re.IGNORECASE)
+_RESULT = re.compile(r"\[RESULT\]", re.IGNORECASE)
+
+
+def _days(start: datetime, end: datetime) -> int:
+ return (end.date() - start.date()).days
+
+
+def disclosure(data: Mapping[str, Any]) -> dict[str, Any]:
+ now = _time(data["now"])
+ age = _days(_time(data["created_at"]), now)
+ window = int(data.get("window_days") or 90)
+ grace = int(data.get("grace_period_days") or 14)
+ labels = set(_labels(data))
+ announced = bool(labels & {"announced", "announced - emails sent"})
+ shipping = bool(labels & {"pr merged", "fix released"})
+ expected = _time(data["created_at"]) + timedelta(days=window + grace)
+ return {
+ "ok": True,
+ "issue_age_days": age,
+ "overdue_for_disclosure": age > window and not announced,
+ "approaching_window_end": age > window * 0.8 and not announced,
+ "grace_period_expired": shipping and age > window + grace and not announced,
+ "distributor_notify_pending": bool(data.get("pre_announce_distributors"))
+ and shipping
+ and not announced,
+ "expected_disclosure_by": expected.date().isoformat(),
+ }
+
+
+def staleness(data: Mapping[str, Any]) -> dict[str, Any]:
+ now = _time(data["now"])
+ timeout = int(data.get("timeout_days") or 14)
+ messages = sorted((_time(m["at"]), bool(m.get("from_team"))) for m in data.get("messages") or [])
+ if not messages or not messages[-1][1]:
+ return {
+ "ok": True,
+ "stale": False,
+ "days": None,
+ "reason": "the latest message is the reporter's, or there is none",
+ }
+ days = _days(messages[-1][0], now)
+ stale = days > timeout
+ return {
+ "ok": True,
+ "stale": stale,
+ "days": days,
+ "reason": f"the team's latest message is {days} days old (timeout {timeout})"
+ + ("; propose proceeding without the reporter's sign-off" if stale else ""),
+ }
+
+
+def _texts(data: Mapping[str, Any]) -> list[str]:
+ comments = data.get("comments") or []
+ if isinstance(comments, Mapping):
+ comments = comments.get("nodes") or []
+ return [str(data.get("body") or "")] + [
+ str(c.get("body") if isinstance(c, Mapping) else c) for c in comments
+ ]
+
+
+RECORD_EXITS = {
+ 0: "pending",
+ 4: "pending_checkout",
+ 3: "refused",
+ 2: "config_error",
+}
+
+
+def closed_announced(data: Mapping[str, Any]) -> dict[str, Any]:
+ texts = _texts(data)
+ out: dict[str, Any] = {"ok": True, "blockers": []}
+ if not one_line(data.get("cve_id")):
+ out["cve_org"] = "skip"
+ out["note"] = "no CVE ID: a closing disposition, dropped from the closed-bucket sweep"
+ return out
+ state = one_line(data.get("cve_org_state")).upper()
+ if data.get("lookup_failed"):
+ out["cve_org"] = "retry"
+ out["note"] = "cve.org lookup failed — try again next sync"
+ elif state == "PUBLISHED":
+ notified = any(NOTIFIED.search(t) for t in texts) and bool(data.get("notification_sent_on_thread"))
+ out["cve_org"] = "already_notified" if notified else "notify"
+ out["date_published"] = data.get("date_published")
+ elif state == "RESERVED":
+ out["cve_org"] = "wait"
+ out["note"] = "cve.org shows RESERVED; propagation not complete yet"
+ elif state == "REJECTED":
+ out["cve_org"] = "blocker"
+ out["blockers"].append(
+ "the CVE record was rejected after publication: flag it to the security team, no email"
+ )
+ else:
+ return {"ok": False, "refused": f"cve_org_state {state!r} is none of PUBLISHED, RESERVED, REJECTED"}
+ records = data.get("advisory_records_configured")
+ if records:
+ out["security_pages_reminder_pending"] = False
+ if any(RECORDS in t for t in texts):
+ out["advisory_records"] = "done"
+ elif state != "PUBLISHED":
+ out["advisory_records"] = "waiting_for_cve_org"
+ elif data.get("planner_exit") is None:
+ out["advisory_records"] = "run_planner"
+ else:
+ code = int(data["planner_exit"])
+ unchanged = code == 0 and one_line(data.get("planner_result")).lower() == "unchanged"
+ out["advisory_records"] = (
+ "already_recorded" if unchanged else RECORD_EXITS.get(code, "config_error")
+ )
+ if out["advisory_records"] == "refused":
+ out["blockers"].append(
+ f"advisory records refused: {one_line(data.get('planner_reason')) or 'see the planner output'}"
+ )
+ else:
+ out["advisory_records"] = "not_configured"
+ populated = bool(one_line(data.get("public_advisory_url")))
+ ticked = any(CHECKLIST_DONE.search(t) for t in texts)
+ reminded = any(t.lstrip().startswith(REMINDER) for t in texts[1:])
+ out["security_pages_reminder_pending"] = populated and not ticked and not reminded
+ return out
+
+
+def _first_version(subject: str) -> str | None:
+ match = _VERSION.search(subject)
+ return match.group(1) if match else None
+
+
+def release_vote(data: Mapping[str, Any]) -> dict[str, Any]:
+ if not data.get("gating"):
+ return {"ok": True, "release_vote_in_progress": False, "reason": "release-vote gating is off"}
+ labels = set(_labels(data))
+ if "pr merged" not in labels or "fix released" in labels:
+ return {
+ "ok": True,
+ "release_vote_in_progress": False,
+ "reason": "outside the pr merged → fix released window",
+ }
+ target = _first_version(str(data.get("fix_milestone") or ""))
+ if not target:
+ return {
+ "ok": True,
+ "release_vote_in_progress": False,
+ "reason": "no parseable fix-PR milestone version",
+ }
+ threads: dict[str, dict[str, Any]] = {}
+ for message in data.get("messages") or []:
+ subject = one_line(message.get("subject"))
+ if not _VOTE.search(subject):
+ continue
+ key = re.sub(r"^(?:re:\s*|\[result\]\s*)+", "", subject, flags=re.IGNORECASE).strip().lower()
+ thread = threads.setdefault(key, {"subject": subject, "url": message.get("url"), "closed": False})
+ if _RESULT.search(subject):
+ thread["closed"] = True
+ elif not thread.get("url"):
+ thread["url"] = message.get("url")
+ product = one_line(data.get("release_product")).lower()
+ for thread in threads.values():
+ subject = thread["subject"]
+ version = _first_version(re.sub(r"^.*?release\s+", "", subject, flags=re.IGNORECASE))
+ if version != target or thread["closed"]:
+ continue
+ if product and product not in subject.lower():
+ continue
+ return {
+ "ok": True,
+ "release_vote_in_progress": True,
+ "release_vote_thread_url": thread["url"],
+ "reason": f"open vote for {version}, matching the fix-PR milestone",
+ }
+ return {"ok": True, "release_vote_in_progress": False, "reason": f"no open [VOTE] thread for {target}"}
diff --git a/tools/security-management/tests/test_issue_sync_state.py b/tools/security-management/tests/test_issue_sync_state.py
new file mode 100644
index 00000000..c5793ebb
--- /dev/null
+++ b/tools/security-management/tests/test_issue_sync_state.py
@@ -0,0 +1,256 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""`security-management issue-sync` Step 1 observations: disclosure window, staleness, 1g and 1h."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+import pytest
+
+from security_management.cli import main
+
+NOW = "2026-10-19T12:00:00Z"
+
+
+def ok(capsys, tmp_path: Path, command: str, data: dict[str, Any]) -> dict[str, Any]:
+ path = tmp_path / f"{command}.json"
+ path.write_text(json.dumps(data), encoding="utf-8")
+ code = main(["issue-sync", command, "--input", str(path)])
+ out = json.loads(capsys.readouterr().out)
+ assert code == 0, out
+ return out
+
+
+# --- 1a: the disclosure window -----------------------------------------------
+
+
+@pytest.mark.parametrize(
+ ("created", "labels", "extra", "expected"),
+ [
+ (
+ "2026-09-01T00:00:00Z",
+ ["airflow"],
+ {},
+ {"overdue_for_disclosure": False, "approaching_window_end": False},
+ ),
+ (
+ "2026-08-01T00:00:00Z",
+ ["airflow"],
+ {},
+ {"overdue_for_disclosure": False, "approaching_window_end": True},
+ ),
+ (
+ "2026-07-01T00:00:00Z",
+ ["airflow"],
+ {},
+ {"overdue_for_disclosure": True, "grace_period_expired": False},
+ ),
+ (
+ "2026-06-01T00:00:00Z",
+ ["airflow", "pr merged"],
+ {},
+ {"overdue_for_disclosure": True, "grace_period_expired": True},
+ ),
+ (
+ "2026-06-01T00:00:00Z",
+ ["airflow", "fix released", "announced"],
+ {},
+ {"overdue_for_disclosure": False, "grace_period_expired": False},
+ ),
+ (
+ "2026-09-01T00:00:00Z",
+ ["airflow", "pr merged"],
+ {"pre_announce_distributors": True},
+ {"distributor_notify_pending": True},
+ ),
+ (
+ "2026-09-01T00:00:00Z",
+ ["airflow"],
+ {"pre_announce_distributors": True},
+ {"distributor_notify_pending": False},
+ ),
+ ("2026-08-01T00:00:00Z", ["airflow"], {"window_days": 60}, {"overdue_for_disclosure": True}),
+ ],
+)
+def test_disclosure_flags(capsys, tmp_path, created, labels, extra, expected):
+ out = ok(capsys, tmp_path, "disclosure", {"now": NOW, "created_at": created, "labels": labels, **extra})
+ for key, value in expected.items():
+ assert out[key] is value, key
+
+
+def test_disclosure_age_and_expected_date(capsys, tmp_path):
+ out = ok(capsys, tmp_path, "disclosure", {"now": NOW, "created_at": "2026-07-21T23:59:00Z", "labels": []})
+ assert out["issue_age_days"] == 90
+ assert out["overdue_for_disclosure"] is False
+ assert out["expected_disclosure_by"] == "2026-11-02"
+
+
+# --- 1c: reporter staleness --------------------------------------------------
+
+
+@pytest.mark.parametrize(
+ ("messages", "stale", "days"),
+ [
+ (
+ [
+ {"from_team": False, "at": "2026-09-01T00:00:00Z"},
+ {"from_team": True, "at": "2026-10-01T00:00:00Z"},
+ ],
+ True,
+ 18,
+ ),
+ ([{"from_team": True, "at": "2026-10-10T00:00:00Z"}], False, 9),
+ (
+ [
+ {"from_team": True, "at": "2026-09-01T00:00:00Z"},
+ {"from_team": False, "at": "2026-09-05T00:00:00Z"},
+ ],
+ False,
+ None,
+ ),
+ ([], False, None),
+ ],
+)
+def test_staleness(capsys, tmp_path, messages, stale, days):
+ out = ok(capsys, tmp_path, "staleness", {"now": NOW, "messages": messages})
+ assert (out["stale"], out["days"]) == (stale, days)
+
+
+# --- 1g: closed announced trackers -------------------------------------------
+
+
+def closed(**data):
+ return {"cve_id": "CVE-2026-1", "cve_org_state": "PUBLISHED", "body": "", "comments": [], **data}
+
+
+@pytest.mark.parametrize(
+ ("data", "move"),
+ [
+ (closed(), "notify"),
+ (
+ closed(
+ comments=[{"body": "Sync 2026-10-01 — CVE-published reporter notification drafted"}],
+ notification_sent_on_thread=True,
+ ),
+ "already_notified",
+ ),
+ (closed(cve_org_state="RESERVED"), "wait"),
+ (closed(cve_org_state="REJECTED"), "blocker"),
+ (closed(lookup_failed=True), "retry"),
+ (closed(cve_id=""), "skip"),
+ ],
+)
+def test_cve_org_state(capsys, tmp_path, data, move):
+ out = ok(capsys, tmp_path, "closed-announced", data)
+ assert out["cve_org"] == move
+ assert bool(out["blockers"]) is (move == "blocker")
+
+
+@pytest.mark.parametrize(
+ ("data", "pending"),
+ [
+ (closed(public_advisory_url="https://lists.apache.org/thread/x"), True),
+ (closed(public_advisory_url=""), False),
+ (
+ closed(
+ public_advisory_url="u",
+ comments=[{"body": "- [x] <!-- apache-magpie: security-pages-checklist v1 --> pages"}],
+ ),
+ False,
+ ),
+ (
+ closed(
+ public_advisory_url="u",
+ comments=[{"body": "- [ ] <!-- apache-magpie: security-pages-checklist v1 --> pages"}],
+ ),
+ True,
+ ),
+ (
+ closed(
+ public_advisory_url="u",
+ comments=[{"body": "<!-- apache-magpie: security-pages-reminder v1 -->\nplease"}],
+ ),
+ False,
+ ),
+ ],
+)
+def test_security_pages_reminder(capsys, tmp_path, data, pending):
+ out = ok(capsys, tmp_path, "closed-announced", data)
+ assert out["security_pages_reminder_pending"] is pending
+ assert out["advisory_records"] == "not_configured"
+
+
+@pytest.mark.parametrize(
+ ("data", "step"),
+ [
+ (closed(advisory_records_configured=True), "run_planner"),
+ (closed(advisory_records_configured=True, cve_org_state="RESERVED"), "waiting_for_cve_org"),
+ (
+ closed(
+ advisory_records_configured=True,
+ comments=[{"body": "<!-- apache-magpie: advisory-records v1 -->"}],
+ ),
+ "done",
+ ),
+ (closed(advisory_records_configured=True, planner_exit=0, planner_result="proposed"), "pending"),
+ (
+ closed(advisory_records_configured=True, planner_exit=0, planner_result="unchanged"),
+ "already_recorded",
+ ),
+ (closed(advisory_records_configured=True, planner_exit=4), "pending_checkout"),
+ (
+ closed(advisory_records_configured=True, planner_exit=3, planner_reason="page cannot be parsed"),
+ "refused",
+ ),
+ (closed(advisory_records_configured=True, planner_exit=2), "config_error"),
+ ],
+)
+def test_advisory_records(capsys, tmp_path, data, step):
+ out = ok(capsys, tmp_path, "closed-announced", data)
+ assert out["advisory_records"] == step
+ assert out["security_pages_reminder_pending"] is False
+
+
+# --- 1h: release votes -------------------------------------------------------
+
+VOTE = "[VOTE] Release Apache Airflow 3.2.3 from 3.2.3rc1"
+
+
+def vote(messages, labels=("airflow", "pr merged"), milestone="Airflow 3.2.3", gating=True):
+ return {"gating": gating, "labels": list(labels), "fix_milestone": milestone, "messages": messages}
+
+
+@pytest.mark.parametrize(
+ ("data", "in_progress"),
+ [
+ (vote([{"subject": VOTE, "url": "https://lists/t1"}]), True),
+ (vote([{"subject": VOTE, "url": "https://lists/t1"}, {"subject": "[RESULT]" + VOTE}]), False),
+ (vote([{"subject": "[VOTE] Release Apache Airflow 3.2.4 from 3.2.4rc1"}]), False),
+ (vote([{"subject": VOTE}], labels=("airflow", "pr merged", "fix released")), False),
+ (vote([{"subject": VOTE}], labels=("airflow", "cve allocated")), False),
+ (vote([{"subject": VOTE}], gating=False), False),
+ (vote([{"subject": VOTE}], milestone="Providers 2026-04-21"), False),
+ ],
+)
+def test_release_vote(capsys, tmp_path, data, in_progress):
+ out = ok(capsys, tmp_path, "release-vote", data)
+ assert out["release_vote_in_progress"] is in_progress
+ if in_progress:
+ assert out["release_vote_thread_url"] == "https://lists/t1"
diff --git a/tools/spec-loop/specs/security-issue-lifecycle.md b/tools/spec-loop/specs/security-issue-lifecycle.md
index 6b52d058..7694835d 100644
--- a/tools/spec-loop/specs/security-issue-lifecycle.md
+++ b/tools/spec-loop/specs/security-issue-lifecycle.md
@@ -179,7 +179,10 @@ publication, with a human gate and an audit-log entry at every step.
admin-only change at the collaborator tier, or after a `403`, is a draft
relay, never a tracker comment), and walks the Step 5b push flow,
checking the title, upgrade-target and CWE gates itself and taking the
- four judged gates as the model's verdicts; mining signals, gathering
+ four judged gates as the model's verdicts. The Step 1 observations
+ that are arithmetic are code too: the disclosure-window flags, reporter
+ staleness, the closed-`announced` cve.org move with the security-pages
+ markers and the advisory-records step, and the release-vote match; mining signals, gathering
state, judging those gates and writing comments and drafts stay with
the model.
- Shared recipes instead of per-skill copies (#1437): the safe-create