You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@magpie.apache.org by po...@apache.org on 2026/10/19 17:59:04 UTC
(magpie) branch main updated: perf(security-issue-sync): compute the Step 1 date and marker observations as code (#1711)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/magpie.git
The following commit(s) were added to refs/heads/main by this push:
new c819c902 perf(security-issue-sync): compute the Step 1 date and marker observations as code (#1711)
c819c902 is described below
commit c819c9023e87fbbf220436c953cd0f564a6d31e4
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Mon Oct 19 19:58:57 2026 +0200
perf(security-issue-sync): compute the Step 1 date and marker observations as code (#1711)
The Step 1 observations that are arithmetic move into the issue-sync
family:
- disclosure (1a): the issue's age and the overdue, approaching,
grace-expired and distributor-pending flags from the configured window;
- staleness (1c): whether the reporter thread is stale against the
response timeout, the clock stopped while the reporter has the last word;
- closed-announced (1g): the cve.org move (notify, already notified,
wait, blocker, retry, skip), the security-pages checklist and reminder
marker scan, and the advisory-records step and planner exit;
- release-vote (1h): an open [VOTE] thread whose version matches the
fix-PR milestone, a [RESULT] closing it.
gather.md shrinks by 42 lines, and the multi-scope split procedure moves
from the SKILL.md Guardrails to outcomes/scope-split.md behind a one-line
rule, bringing SKILL.md to 6,948 tokens (7,006 before layer 4b). The 38
remaining model cases of the skill pass.
Part of #1672
Generated-by: Claude Opus 5
---
plugins/magpie-security/skills/issue-sync/SKILL.md | 34 +--
.../magpie-security/skills/issue-sync/gather.md | 106 +++------
.../skills/issue-sync/outcomes/scope-split.md | 38 +++
tools/security-management/README.md | 7 +-
.../src/security_management/issue_sync/cli.py | 10 +-
.../src/security_management/issue_sync/state.py | 225 ++++++++++++++++++
.../tests/test_issue_sync_state.py | 256 +++++++++++++++++++++
tools/spec-loop/specs/security-issue-lifecycle.md | 5 +-
8 files changed, 574 insertions(+), 107 deletions(-)
diff --git a/plugins/magpie-security/skills/issue-sync/SKILL.md b/plugins/magpie-security/skills/issue-sync/SKILL.md
index 65030bed..4bf40253 100644
--- a/plugins/magpie-security/skills/issue-sync/SKILL.md
+++ b/plugins/magpie-security/skills/issue-sync/SKILL.md
@@ -25,7 +25,7 @@ argument-hint: "[issue-number]"
capability: capability:intake
surface_hash: sha256:5c4aad61d458d98f
license: Apache-2.0
-measured_tokens: 7282
+measured_tokens: 6948
---
<!-- Placeholder convention (see AGENTS.md#placeholder-convention-used-in-skill-files):
@@ -437,35 +437,9 @@ Fix every hit and check again; it is the floor, not the whole review, so still r
release-train state in
[`<project-config>/release-trains.md`](../../../../<project-config>/release-trains.md).
- **Multi-scope reports must be split into one tracking issue per
- scope.** When a report affects more than one scope (for example a root cause in a shared core module
- whose vector also exists in a plugin/extension component), never apply two scope labels to one issue;
- propose the split instead:
-
- 1. Keep the original issue on the scope whose milestone family ships *first*
- (usually core, whose patch releases cut faster), and drop the extra scope label from it.
- 2. Create one new issue per remaining scope via `gh issue create
- --repo <tracker>`, copying the report body
- verbatim but with a one-line preamble that says *"Split from
- [#NNN](https://github.com/<tracker>/issues/<N>) for the `<scope>` scope — see that issue for the
- full discussion history."*
- 3. Apply to each split issue:
- - exactly one scope label (see
- [`<project-config>/scope-labels.md`](../../../../<project-config>/scope-labels.md));
- - the same `cve allocated` label if a CVE is shared across
- scopes — CVE reuse is correct when the same upstream bug
- affects multiple products, with one `affected[]` entry per
- product in the CVE record;
- - the PR / advisory labels (`pr created` / `pr merged` /
- `fix released`) derived independently per scope from the same
- fix PR, because each scope rides a different release train;
- - the matching milestone for that scope (see
- [`<project-config>/milestones.md`](../../../../<project-config>/milestones.md));
- - the same assignee set as the anchor issue.
- 4. Post a cross-link comment on **each** issue pointing at the other(s).
- 5. Update the open reporter email draft, if any, to mention the split and link every tracker.
-
- Do **not** silently drop a scope label without splitting: each scope's release managers need the issue on their own milestone.
- A single issue with two scope labels is a process bug — flag it as a **blocker** and propose the split as a concrete numbered item.
+ scope.** Never apply two scope labels to one issue: a single issue with two is a process bug, flagged as a **blocker** with the split proposed as a concrete numbered item
+ (keep the original on the scope whose release ships first; one new issue per remaining scope with the report verbatim, its own scope label, labels and milestone; cross-links on each).
+ Do **not** silently drop a scope label without splitting. The procedure: [`outcomes/scope-split.md`](outcomes/scope-split.md).
---
diff --git a/plugins/magpie-security/skills/issue-sync/gather.md b/plugins/magpie-security/skills/issue-sync/gather.md
index 97ca5bfb..6d0877e1 100644
--- a/plugins/magpie-security/skills/issue-sync/gather.md
+++ b/plugins/magpie-security/skills/issue-sync/gather.md
@@ -48,15 +48,11 @@ Record:
- CVE ID (if allocated),
- link to the fixing PR(s);
- the discussion so far (comments), paying attention to the most recent activity and any stalled-for-30-days state.
-- **Disclosure deadline check.** Compute `issue_age_days` as calendar days from `createdAt` to today,
- compare it with `window_days` and `grace_period_days` from Step 0.6, and set these flags in the observed-state bag:
- - `overdue_for_disclosure: true` — `issue_age_days > window_days` **and** no `announced` label (the advisory has not been sent).
- - `approaching_window_end: true` — `issue_age_days > window_days * 0.8` (≥ 80 % of the window elapsed) and not yet `announced`;
- a yellow-flag caution in the observed-state dump, with no proposal item.
- - `grace_period_expired: true` — the issue carries `pr merged` or `fix released` **and** `issue_age_days > window_days + grace_period_days`,
- and no `announced` label.
- - `distributor_notify_pending: true` — `pre_announce_distributors: true` (Step 0.6) **and** the issue carries `pr merged` or `fix released`,
- and no `announced` label; Step 2b then proposes a pre-announcement draft for the distributor embargo list.
+- **Disclosure deadline check.** Write `{now, created_at, labels, window_days, grace_period_days, pre_announce_distributors}` (the Step 0.6 values) and run `<sm> disclosure`;
+ record its `issue_age_days` and flags in the observed-state bag:
+ `overdue_for_disclosure` (past the window, no `announced`), `approaching_window_end` (past 80 % of it — a yellow flag with no proposal item),
+ `grace_period_expired` (`pr merged` / `fix released` and past window plus grace) and `distributor_notify_pending` (Step 2b then proposes the distributor pre-announcement).
+ `<sm>` is the `security-management issue-sync` command of [`SKILL.md`](SKILL.md#prerequisites).
The flags are informational observations; they do not stop the sync.
A missing `announced` label on a tracker whose body has the `announced` date populated is a missing label (propose adding it), not overdue.
@@ -189,16 +185,13 @@ Process for finding the real reporter and the original thread:
- **every status update the security team has already sent the reporter** — this decides whether a new one is needed (Step 2b);
- the latest message, *who* sent it, and whether the ball is in our court.
-5. **Check for reporter staleness.** From the step 4 timestamps and senders, find the team's **most recent outbound message** on this thread
- and the reporter's **most recent reply**, if any.
- If the team's latest outbound message is older than `security_inbox.reporter_response_timeout_days`
- (resolved per the [configuration resolution order](../../../../AGENTS.md#configuration-resolution-order)
- — `<project-config>/project.md`, else the project's organization; both shipped organizations default to 14)
- **and** no reporter reply has landed since, mark the thread **stale** and surface it in Step 2b per the *Reporter unresponsiveness* row in
- [`signals-to-actions.md`](signals-to-actions.md#step-2b--proposed-changes-signal-to-action-lookup-table).
- It is a **proposal, not an automatic action** (policy: [`SKILL.md`](SKILL.md) Step 2b):
- the user confirms before the team proceeds without the reporter.
- Skip the check when the latest message is *from* the reporter — the clock runs only while we wait on a reply.
+5. **Check for reporter staleness.** Write the step 4 messages as `{at, from_team}` with `now` and `timeout_days`
+ (`security_inbox.reporter_response_timeout_days`, resolved per the [configuration resolution order](../../../../AGENTS.md#configuration-resolution-order); 14 in both shipped organizations)
+ and run `<sm> staleness`: the thread is **stale** when the team's latest outbound message is older than the timeout with no reporter reply since,
+ and the clock never runs while the latest message is the reporter's.
+ A stale thread is surfaced in Step 2b per the *Reporter unresponsiveness* row in
+ [`signals-to-actions.md`](signals-to-actions.md#step-2b--proposed-changes-signal-to-action-lookup-table) —
+ a **proposal, not an automatic action** (policy: [`SKILL.md`](SKILL.md) Step 2b): the user confirms before the team proceeds without the reporter.
6. **Sync a reporter-confirmed credit line into the issue body** whenever the thread holds a clear credit confirmation from the reporter
not yet reflected in the tracker's *"Reporter credited as"* field;
@@ -549,18 +542,11 @@ For each closed-`announced` tracker in this run:
```bash
uv run --project ~/.claude/magpie/vetted-ops vetted-op-read --caller security-issue-sync cve-check-published <CVE-ID>
```
-3. Interpret:
- - `state == "PUBLISHED"` → capture `datePublished` and propose the *CVE-published* reporter email in Step 2b.
- - `state == "RESERVED"` → record *"cve.org shows RESERVED; propagation not complete yet"* in the observed state;
- no email yet; a future sync run will catch the publication.
- - `state == "REJECTED"` → **surface as a blocker**: the record was withdrawn post-publication.
- Do not draft a reporter email; flag it to the security team.
- - lookup error (non-zero exit from `vetted-op-read` — exit 4) → record *"cve.org lookup failed — <short error> — try again next sync"*.
- Do not propose notification on an absent response.
-
-**Idempotence.** If the tracker's comment trail has a prior *"Sync YYYY-MM-DD — CVE-published reporter notification drafted"* status-change comment
-and the reporter thread already carries a corresponding sent message,
-skip the proposal and record *"CVE-published notification already sent on <date>"*.
+3. Write `{cve_id, cve_org_state, date_published, lookup_failed, notification_sent_on_thread, body, comments, public_advisory_url, advisory_records_configured, planner_exit, planner_result, planner_reason}`
+ and run `<sm> closed-announced`. Its `cve_org` is the move:
+ `notify` (propose the *CVE-published* reporter email in Step 2b), `already_notified` (a prior *"CVE-published reporter notification drafted"* comment and a sent message on the thread — record it),
+ `wait` (*"cve.org shows RESERVED; propagation not complete yet"*), `blocker` (the record was rejected after publication: no email, flag it to the security team),
+ `retry` (*"cve.org lookup failed — try again next sync"*; never propose a notification on an absent response) or `skip` (no CVE ID: a closing disposition, dropped from the sweep).
**Gmail-budget.** The cve.org check is one HTTP call per tracker, not metered against the Gmail budget,
but within the "≤ 1 extra HTTP round-trip per tracker" soft limit for closed-bucket scans:
@@ -568,31 +554,19 @@ with several closed trackers in scope, run the checks in parallel via the subage
**Security-pages checklist.** The same closed-`announced` bucket is the only place the post-announcement security-pages step can be observed:
the Step 14 close-out adds `announced` and closes the tracker in one apply, so an open tracker never carries the trigger state.
-For each closed-`announced` tracker whose *"Public advisory URL"* body field is populated,
-scan the issue body and every comment for the ticked checklist item
-```html
-- [x] <!-- apache-magpie: security-pages-checklist v1 -->
-```
-and, separately, for a reminder comment carrying
-```html
-<!-- apache-magpie: security-pages-reminder v1 -->
-```
-on its first line.
-Record `security_pages_reminder_pending: true` in the observed state only when **no** ticked checklist item exists **and** no reminder comment exists;
-an unticked box in the hand-off comment is the pending state, not the satisfied one.
+The same command scans the body and every comment the 1a fetch returned — no extra API call — for the ticked checklist item
+`- [x] <!-- apache-magpie: security-pages-checklist v1 -->` and for a comment opening with `<!-- apache-magpie: security-pages-reminder v1 -->`,
+and sets `security_pages_reminder_pending` only when the *Public advisory URL* is populated and neither exists
+(an unticked box in the hand-off comment is the pending state, not the satisfied one).
Step 2b turns a pending flag into the *Security-pages reminder comment* proposal ([`signals-to-actions.md`](signals-to-actions.md)).
-The scan reads the body and comments the 1a fetch already returned — no extra API call.
**Advisory records.** When [`<project-config>/project.md`](../../../../<project-config>/project.md#advisory-records) declares an `advisory_records` block,
sync also records the CVE in the project's own security records — a site repository over Git or Subversion
([`docs/security/advisory-records.md`](../../../../docs/security/advisory-records.md)).
Without the block, record *"advisory records: not configured — skipped"* in the observed state and do nothing else.
With it, `security_pages_reminder_pending` is always `false`: sync proposes the edit itself instead of reminding the release manager.
-For each closed-`announced` tracker:
+`closed-announced` prints the step in `advisory_records`: `done` (an `advisory-records v1` comment exists), `waiting_for_cve_org` (not `PUBLISHED` yet; the tool refuses before publication too — never work around it) or `run_planner`, then:
-1. Skip when a comment carrying `<!-- apache-magpie: advisory-records v1 -->` exists: record *"advisory records: done on <date>"*.
-2. Skip unless the cve.org check above returned `PUBLISHED`: record *"advisory records: waiting for cve.org"*.
- The tool refuses before publication too; never work around it.
3. Write the record to the session scratch directory:
`uv run --project <framework>/tools/cve-tool-vulnogram/generate-cve-json generate-cve-json <N> --output <scratch>/<CVE-ID>.json`.
4. Run the planner as a plain command, without `--apply`.
@@ -605,12 +579,10 @@ For each closed-`announced` tracker:
For `release-page`, pass `--reported-date` (the date of the first message on the *Security mailing list thread*).
When the output names a missing input (`--release-date` for a new release section, `--fix-commit`, `--affects`), find it on the tracker,
in [`release-trains.md`](../../../../<project-config>/release-trains.md) or in the release's announcement, and run it again; ask the user when none of them has it.
-5. Read the exit code:
- `0` with `proposed` → `advisory_records_pending: true`, keeping the tool's diff and numbered commands for Step 2b;
- `0` with `unchanged` → *"advisory records: already recorded"*;
- `4` → `advisory_records_pending: true` with the single `checkout` command;
- `3` → *"advisory records: refused — <reason>"*, surfaced as a blocker when the page cannot be parsed or already holds the CVE under another release;
- `2` → a configuration error to show the user.
+5. Pass the planner's exit code (and `proposed` / `unchanged`, and a refusal's reason) back to `closed-announced`:
+ `pending` → `advisory_records_pending: true`, keeping the tool's diff and numbered commands for Step 2b;
+ `already_recorded`; `pending_checkout` → `advisory_records_pending: true` with the single `checkout` command;
+ `refused` → a blocker when the page cannot be parsed or already holds the CVE under another release; `config_error` → show the user.
**When the tracker has no CVE ID.** Closed trackers without a `CVE-YYYY-NNNNN` in the *CVE tool link* body field
are closing dispositions (`invalid` / `duplicate` / `wontfix`):
@@ -663,26 +635,12 @@ mcp__ponymail__search_list(
)
```
-**Matching against the tracker's fix milestone.** For each returned thread:
-
-1. Extract the version from the subject line.
- ASF [VOTE] subjects follow `[VOTE] Release <Project> <X.Y.Z> from <X.Y.Z>rcN`
- (sometimes with sibling artifacts on the same vote — `& Task SDK <A.B.C> from <A.B.C>rcM`).
- Parse the **first** `<X.Y.Z>` token after the project name.
-2. Compare it with the fix-PR milestone — the milestone on the PR, not on the tracker:
- the tracker's may be a wave-month label like `<scope-b> 2026-04-21`, while the PR's carries the actual carrier version like `<product> 3.2.3`.
- For wave-based component milestones, also check that the vote's release ships the right *component wave*:
- the tracker's wave milestone (e.g. *<scope-b> 2026-04-21*) maps to a `[VOTE] Release <scope-b> …` thread.
-3. Check the thread's most recent message: a `[RESULT][VOTE]` reply is a closed vote; open votes have **no** `[RESULT]` reply yet.
- The `[VOTE]` query above already returns the `[RESULT][VOTE]` messages (their subjects contain `[VOTE]`), so decide open / closed from that one result set — no per-thread fetch.
- Closed votes get no label add: either the release shipped (the `fix released` flow takes over)
- or the vote failed (the team cuts a fresh RC, and the next sync picks up the new `[VOTE]` thread).
-
-**Record in the per-tracker state bag:**
-
-- `release_vote_in_progress`: `true` when a tracker in the `pr merged` window has a matching open `[VOTE]` thread on the right list.
-- `release_vote_thread_url`: the PonyMail thread URL if found (`<mail-archive-url>/thread/<hash>?<list>@<domain>`),
- used as the rationale in the Step 2b proposal.
+**Matching against the tracker's fix milestone.** Write `{gating, labels, fix_milestone, release_product, messages}` — the search results' `subject` and `url`,
+the milestone on the fix **PR** (it carries the carrier version, e.g. `<product> 3.2.3`, where the tracker's may be a wave label like `<scope-b> 2026-04-21`) —
+and run `<sm> release-vote`. It takes the first version after `Release` in each `[VOTE] Release <Project> <X.Y.Z> from <X.Y.Z>rcN` subject,
+treats a `[RESULT][VOTE]` message in the same result set as a closed vote (no label add: the release shipped or the vote failed and a fresh RC follows),
+and records `release_vote_in_progress` and `release_vote_thread_url` for a tracker in the `pr merged` window with a matching open vote;
+`release_product` keeps a component wave's vote to its own `[VOTE] Release <scope-b> …` thread.
**Hard rule — no auto-apply.** The result feeds Step 2b's proposal and applies only after the user confirms.
Never add `rc voting` silently: the next regen flips `CNA_private.state` to `REVIEW`,
diff --git a/plugins/magpie-security/skills/issue-sync/outcomes/scope-split.md b/plugins/magpie-security/skills/issue-sync/outcomes/scope-split.md
new file mode 100644
index 00000000..9fdf6819
--- /dev/null
+++ b/plugins/magpie-security/skills/issue-sync/outcomes/scope-split.md
@@ -0,0 +1,38 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+
+# security-issue-sync — splitting a multi-scope report
+
+The Guardrails rule in [`SKILL.md`](../SKILL.md#guardrails), with the split procedure.
+Relative links are rooted one level deeper than in `SKILL.md`.
+
+- **Multi-scope reports must be split into one tracking issue per
+ scope.** When a report affects more than one scope (for example a root cause in a shared core module
+ whose vector also exists in a plugin/extension component), never apply two scope labels to one issue;
+ propose the split instead:
+
+ 1. Keep the original issue on the scope whose milestone family ships *first*
+ (usually core, whose patch releases cut faster), and drop the extra scope label from it.
+ 2. Create one new issue per remaining scope via `gh issue create
+ --repo <tracker>`, copying the report body
+ verbatim but with a one-line preamble that says *"Split from
+ [#NNN](https://github.com/<tracker>/issues/<N>) for the `<scope>` scope — see that issue for the
+ full discussion history."*
+ 3. Apply to each split issue:
+ - exactly one scope label (see
+ [`<project-config>/scope-labels.md`](../../../../../<project-config>/scope-labels.md));
+ - the same `cve allocated` label if a CVE is shared across
+ scopes — CVE reuse is correct when the same upstream bug
+ affects multiple products, with one `affected[]` entry per
+ product in the CVE record;
+ - the PR / advisory labels (`pr created` / `pr merged` /
+ `fix released`) derived independently per scope from the same
+ fix PR, because each scope rides a different release train;
+ - the matching milestone for that scope (see
+ [`<project-config>/milestones.md`](../../../../../<project-config>/milestones.md));
+ - the same assignee set as the anchor issue.
+ 4. Post a cross-link comment on **each** issue pointing at the other(s).
+ 5. Update the open reporter email draft, if any, to mention the split and link every tracker.
+
+ Do **not** silently drop a scope label without splitting: each scope's release managers need the issue on their own milestone.
+ A single issue with two scope labels is a process bug — flag it as a **blocker** and propose the split as a concrete numbered item.
diff --git a/tools/security-management/README.md b/tools/security-management/README.md
index 7b552d4e..6c702fc5 100644
--- a/tools/security-management/README.md
+++ b/tools/security-management/README.md
@@ -255,6 +255,10 @@ Every command takes `--input F`, a JSON file.
| `draft-backend` | `backend` (`maildir`, `oauth_curl`, `claude_ai_mcp`), `decided_by` (`user.md`, `project.md`, `default`); no credential with links in the body exits `2`. |
| `ghsa-route` | `access_tier`, `action`, `channel`, and for an admin hand-off the draft tool, addressee, thread, rollup marker and three parts. |
| `push-decision` | `action` (`skip`, `block`, `hand-off`, `push`, `pending` with the `next` command or gate), `gate_failed`, `handoff_variant`, `state_advance`, `recap_blocker`. |
+| `disclosure` | `issue_age_days`, `overdue_for_disclosure`, `approaching_window_end`, `grace_period_expired`, `distributor_notify_pending`, `expected_disclosure_by`. |
+| `staleness` | `stale` and `days`: the team's latest message older than the timeout with no reporter reply since. |
+| `closed-announced` | `cve_org` (`notify`, `already_notified`, `wait`, `blocker`, `retry`, `skip`), `security_pages_reminder_pending`, `advisory_records` and its next step, `blockers`. |
+| `release-vote` | `release_vote_in_progress` and `release_vote_thread_url` for an open `[VOTE]` matching the fix-PR milestone. |
| `render-recap` | The recap `text`, `bare_refs`. |
## Tests
@@ -274,5 +278,6 @@ it pins the `step-26-precedent-search` signal rule, which stays a model eval.
`tests/test_issue_invalidate.py` holds the cases of the retired `security-issue-invalidate` suites `step-2-import-path`, `step-5a-labels`, `step-5b-closing-comment`, `step-5d-email-draft`, `step-5e-rollup-entry`, `step-5f-confirm` and `step-7-recap`.
`tests/test_cve_allocate.py` holds the cases of the retired `security-cve-allocate` suites `step-1-blocker-checks`, `step-2-title-normalize`, `step-3-allocation-recipe`, `step-4-propose-updates`, `step-5-confirm` and `step-7-recap`.
`tests/test_issue_deduplicate.py` holds the cases of the retired `security-issue-deduplicate` suites `step-1-classify`, `step-2-extract-fields`, `step-3-merge-body`, `step-4-rollup-entries`, `step-5-confirm` and `step-6-recap`.
-`tests/test_issue_sync.py` and `tests/test_issue_sync_rules.py` hold the cases of the retired `security-issue-sync` suites `step-bulk-selectors`, `step-bulk-orchestration`, `step-3-confirm`, `step-6-recap`, `step-guardrails`, `step-security-cc`, `step-draft-backend`, `step-ghsa-access-tier` and `step-5b-push-decision`.
+`tests/test_issue_sync_state.py` pins the `security-issue-sync` Step 1 observations;
+`tests/test_issue_sync.py` and `tests/test_issue_sync_rules.py` hold the cases of its retired suites `step-bulk-selectors`, `step-bulk-orchestration`, `step-3-confirm`, `step-6-recap`, `step-guardrails`, `step-security-cc`, `step-draft-backend`, `step-ghsa-access-tier` and `step-5b-push-decision`.
`tests/test_body.py` checks that the `tracker-body-template.md` files of the sibling import skills carry exactly the rendered headings.
diff --git a/tools/security-management/src/security_management/issue_sync/cli.py b/tools/security-management/src/security_management/issue_sync/cli.py
index af7b4a9e..0d9cf2eb 100644
--- a/tools/security-management/src/security_management/issue_sync/cli.py
+++ b/tools/security-management/src/security_management/issue_sync/cli.py
@@ -30,7 +30,7 @@ from collections.abc import Mapping
from typing import Any
from security_management.cli import emit, read_object, refuse
-from security_management.issue_sync import advisory, bulk, preflight, push, render, select
+from security_management.issue_sync import advisory, bulk, preflight, push, render, select, state
from security_management.shared import drafting, numbered, recipients
FAMILY = "issue-sync"
@@ -48,6 +48,10 @@ COMMANDS = {
"draft-backend": "Drafts: Gmail or Maildir, then the Gmail backend (JSON: user_backend, project_backend, oauth_credentials)",
"ghsa-route": "GHSA trackers: the access tier and how a change is delivered",
"push-decision": "Step 5b: skip, block, hand-off or push, and the next command",
+ "disclosure": "Step 1a: the issue's age and the disclosure-window flags",
+ "staleness": "Step 1c: whether the reporter thread is stale (JSON: now, messages, timeout_days)",
+ "closed-announced": "Step 1g: the cve.org state, the security-pages markers and the advisory records",
+ "release-vote": "Step 1h: an open [VOTE] thread matching the fix-PR milestone",
"render-recap": "Step 6: the recap",
}
@@ -91,6 +95,10 @@ def dispatch(args: argparse.Namespace) -> int:
"draft-backend": _backend,
"ghsa-route": advisory.route,
"push-decision": push.decide,
+ "disclosure": state.disclosure,
+ "staleness": state.staleness,
+ "closed-announced": state.closed_announced,
+ "release-vote": state.release_vote,
"render-recap": render.recap,
}
result = handlers[args.command](data)
diff --git a/tools/security-management/src/security_management/issue_sync/state.py b/tools/security-management/src/security_management/issue_sync/state.py
new file mode 100644
index 00000000..55949af9
--- /dev/null
+++ b/tools/security-management/src/security_management/issue_sync/state.py
@@ -0,0 +1,225 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Step 1 observations that are arithmetic on dates, labels and markers.
+
+- `disclosure` (1a): `issue_age_days` from `createdAt`, and the flags
+ `overdue_for_disclosure` (past `window_days`, not `announced`),
+ `approaching_window_end` (past 80 % of it), `grace_period_expired` (`pr
+ merged` / `fix released`, past `window_days + grace_period_days`) and
+ `distributor_notify_pending` (`pre_announce_distributors`, `pr merged` /
+ `fix released`, not `announced`). Defaults 90 / 14 / false.
+- `staleness` (1c step 5): stale when the team's latest outbound message is
+ older than `reporter_response_timeout_days` (default 14) and the
+ reporter has not replied since; the clock never runs while the latest
+ message is the reporter's.
+- `closed_announced` (1g): the cve.org state as the next move (`PUBLISHED`
+ → the CVE-published email unless a prior notification is recorded,
+ `RESERVED` → wait, `REJECTED` → blocker, lookup error → retry), the
+ security-pages checklist and reminder markers in the body and comments,
+ and the advisory-records step, including the planner's exit code.
+- `release_vote` (1h): an open `[VOTE]` thread on the dev list whose first
+ version matches the fix-PR milestone, for a tracker in the `pr merged`
+ window; a `[RESULT]` message closes a vote.
+"""
+
+from __future__ import annotations
+
+import re
+from collections.abc import Mapping
+from datetime import datetime, timedelta
+from typing import Any
+
+from security_management.issue_sync.select import _labels, _time
+from security_management.shared.rollup import one_line
+
+CHECKLIST_DONE = re.compile(
+ r"^\s*[-*] \[[xX]\] <!-- apache-magpie: security-pages-checklist v1 -->", re.MULTILINE
+)
+REMINDER = "<!-- apache-magpie: security-pages-reminder v1 -->"
+RECORDS = "<!-- apache-magpie: advisory-records v1 -->"
+NOTIFIED = re.compile(r"CVE-published reporter notification drafted", re.IGNORECASE)
+_VERSION = re.compile(r"\b(\d+\.\d+(?:\.\d+)?)\b")
+_VOTE = re.compile(r"\[VOTE\]", re.IGNORECASE)
+_RESULT = re.compile(r"\[RESULT\]", re.IGNORECASE)
+
+
+def _days(start: datetime, end: datetime) -> int:
+ return (end.date() - start.date()).days
+
+
+def disclosure(data: Mapping[str, Any]) -> dict[str, Any]:
+ now = _time(data["now"])
+ age = _days(_time(data["created_at"]), now)
+ window = int(data.get("window_days") or 90)
+ grace = int(data.get("grace_period_days") or 14)
+ labels = set(_labels(data))
+ announced = bool(labels & {"announced", "announced - emails sent"})
+ shipping = bool(labels & {"pr merged", "fix released"})
+ expected = _time(data["created_at"]) + timedelta(days=window + grace)
+ return {
+ "ok": True,
+ "issue_age_days": age,
+ "overdue_for_disclosure": age > window and not announced,
+ "approaching_window_end": age > window * 0.8 and not announced,
+ "grace_period_expired": shipping and age > window + grace and not announced,
+ "distributor_notify_pending": bool(data.get("pre_announce_distributors"))
+ and shipping
+ and not announced,
+ "expected_disclosure_by": expected.date().isoformat(),
+ }
+
+
+def staleness(data: Mapping[str, Any]) -> dict[str, Any]:
+ now = _time(data["now"])
+ timeout = int(data.get("timeout_days") or 14)
+ messages = sorted((_time(m["at"]), bool(m.get("from_team"))) for m in data.get("messages") or [])
+ if not messages or not messages[-1][1]:
+ return {
+ "ok": True,
+ "stale": False,
+ "days": None,
+ "reason": "the latest message is the reporter's, or there is none",
+ }
+ days = _days(messages[-1][0], now)
+ stale = days > timeout
+ return {
+ "ok": True,
+ "stale": stale,
+ "days": days,
+ "reason": f"the team's latest message is {days} days old (timeout {timeout})"
+ + ("; propose proceeding without the reporter's sign-off" if stale else ""),
+ }
+
+
+def _texts(data: Mapping[str, Any]) -> list[str]:
+ comments = data.get("comments") or []
+ if isinstance(comments, Mapping):
+ comments = comments.get("nodes") or []
+ return [str(data.get("body") or "")] + [
+ str(c.get("body") if isinstance(c, Mapping) else c) for c in comments
+ ]
+
+
+RECORD_EXITS = {
+ 0: "pending",
+ 4: "pending_checkout",
+ 3: "refused",
+ 2: "config_error",
+}
+
+
+def closed_announced(data: Mapping[str, Any]) -> dict[str, Any]:
+ texts = _texts(data)
+ out: dict[str, Any] = {"ok": True, "blockers": []}
+ if not one_line(data.get("cve_id")):
+ out["cve_org"] = "skip"
+ out["note"] = "no CVE ID: a closing disposition, dropped from the closed-bucket sweep"
+ return out
+ state = one_line(data.get("cve_org_state")).upper()
+ if data.get("lookup_failed"):
+ out["cve_org"] = "retry"
+ out["note"] = "cve.org lookup failed — try again next sync"
+ elif state == "PUBLISHED":
+ notified = any(NOTIFIED.search(t) for t in texts) and bool(data.get("notification_sent_on_thread"))
+ out["cve_org"] = "already_notified" if notified else "notify"
+ out["date_published"] = data.get("date_published")
+ elif state == "RESERVED":
+ out["cve_org"] = "wait"
+ out["note"] = "cve.org shows RESERVED; propagation not complete yet"
+ elif state == "REJECTED":
+ out["cve_org"] = "blocker"
+ out["blockers"].append(
+ "the CVE record was rejected after publication: flag it to the security team, no email"
+ )
+ else:
+ return {"ok": False, "refused": f"cve_org_state {state!r} is none of PUBLISHED, RESERVED, REJECTED"}
+ records = data.get("advisory_records_configured")
+ if records:
+ out["security_pages_reminder_pending"] = False
+ if any(RECORDS in t for t in texts):
+ out["advisory_records"] = "done"
+ elif state != "PUBLISHED":
+ out["advisory_records"] = "waiting_for_cve_org"
+ elif data.get("planner_exit") is None:
+ out["advisory_records"] = "run_planner"
+ else:
+ code = int(data["planner_exit"])
+ unchanged = code == 0 and one_line(data.get("planner_result")).lower() == "unchanged"
+ out["advisory_records"] = (
+ "already_recorded" if unchanged else RECORD_EXITS.get(code, "config_error")
+ )
+ if out["advisory_records"] == "refused":
+ out["blockers"].append(
+ f"advisory records refused: {one_line(data.get('planner_reason')) or 'see the planner output'}"
+ )
+ else:
+ out["advisory_records"] = "not_configured"
+ populated = bool(one_line(data.get("public_advisory_url")))
+ ticked = any(CHECKLIST_DONE.search(t) for t in texts)
+ reminded = any(t.lstrip().startswith(REMINDER) for t in texts[1:])
+ out["security_pages_reminder_pending"] = populated and not ticked and not reminded
+ return out
+
+
+def _first_version(subject: str) -> str | None:
+ match = _VERSION.search(subject)
+ return match.group(1) if match else None
+
+
+def release_vote(data: Mapping[str, Any]) -> dict[str, Any]:
+ if not data.get("gating"):
+ return {"ok": True, "release_vote_in_progress": False, "reason": "release-vote gating is off"}
+ labels = set(_labels(data))
+ if "pr merged" not in labels or "fix released" in labels:
+ return {
+ "ok": True,
+ "release_vote_in_progress": False,
+ "reason": "outside the pr merged → fix released window",
+ }
+ target = _first_version(str(data.get("fix_milestone") or ""))
+ if not target:
+ return {
+ "ok": True,
+ "release_vote_in_progress": False,
+ "reason": "no parseable fix-PR milestone version",
+ }
+ threads: dict[str, dict[str, Any]] = {}
+ for message in data.get("messages") or []:
+ subject = one_line(message.get("subject"))
+ if not _VOTE.search(subject):
+ continue
+ key = re.sub(r"^(?:re:\s*|\[result\]\s*)+", "", subject, flags=re.IGNORECASE).strip().lower()
+ thread = threads.setdefault(key, {"subject": subject, "url": message.get("url"), "closed": False})
+ if _RESULT.search(subject):
+ thread["closed"] = True
+ elif not thread.get("url"):
+ thread["url"] = message.get("url")
+ product = one_line(data.get("release_product")).lower()
+ for thread in threads.values():
+ subject = thread["subject"]
+ version = _first_version(re.sub(r"^.*?release\s+", "", subject, flags=re.IGNORECASE))
+ if version != target or thread["closed"]:
+ continue
+ if product and product not in subject.lower():
+ continue
+ return {
+ "ok": True,
+ "release_vote_in_progress": True,
+ "release_vote_thread_url": thread["url"],
+ "reason": f"open vote for {version}, matching the fix-PR milestone",
+ }
+ return {"ok": True, "release_vote_in_progress": False, "reason": f"no open [VOTE] thread for {target}"}
diff --git a/tools/security-management/tests/test_issue_sync_state.py b/tools/security-management/tests/test_issue_sync_state.py
new file mode 100644
index 00000000..c5793ebb
--- /dev/null
+++ b/tools/security-management/tests/test_issue_sync_state.py
@@ -0,0 +1,256 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""`security-management issue-sync` Step 1 observations: disclosure window, staleness, 1g and 1h."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+import pytest
+
+from security_management.cli import main
+
+NOW = "2026-10-19T12:00:00Z"
+
+
+def ok(capsys, tmp_path: Path, command: str, data: dict[str, Any]) -> dict[str, Any]:
+ path = tmp_path / f"{command}.json"
+ path.write_text(json.dumps(data), encoding="utf-8")
+ code = main(["issue-sync", command, "--input", str(path)])
+ out = json.loads(capsys.readouterr().out)
+ assert code == 0, out
+ return out
+
+
+# --- 1a: the disclosure window -----------------------------------------------
+
+
+@pytest.mark.parametrize(
+ ("created", "labels", "extra", "expected"),
+ [
+ (
+ "2026-09-01T00:00:00Z",
+ ["airflow"],
+ {},
+ {"overdue_for_disclosure": False, "approaching_window_end": False},
+ ),
+ (
+ "2026-08-01T00:00:00Z",
+ ["airflow"],
+ {},
+ {"overdue_for_disclosure": False, "approaching_window_end": True},
+ ),
+ (
+ "2026-07-01T00:00:00Z",
+ ["airflow"],
+ {},
+ {"overdue_for_disclosure": True, "grace_period_expired": False},
+ ),
+ (
+ "2026-06-01T00:00:00Z",
+ ["airflow", "pr merged"],
+ {},
+ {"overdue_for_disclosure": True, "grace_period_expired": True},
+ ),
+ (
+ "2026-06-01T00:00:00Z",
+ ["airflow", "fix released", "announced"],
+ {},
+ {"overdue_for_disclosure": False, "grace_period_expired": False},
+ ),
+ (
+ "2026-09-01T00:00:00Z",
+ ["airflow", "pr merged"],
+ {"pre_announce_distributors": True},
+ {"distributor_notify_pending": True},
+ ),
+ (
+ "2026-09-01T00:00:00Z",
+ ["airflow"],
+ {"pre_announce_distributors": True},
+ {"distributor_notify_pending": False},
+ ),
+ ("2026-08-01T00:00:00Z", ["airflow"], {"window_days": 60}, {"overdue_for_disclosure": True}),
+ ],
+)
+def test_disclosure_flags(capsys, tmp_path, created, labels, extra, expected):
+ out = ok(capsys, tmp_path, "disclosure", {"now": NOW, "created_at": created, "labels": labels, **extra})
+ for key, value in expected.items():
+ assert out[key] is value, key
+
+
+def test_disclosure_age_and_expected_date(capsys, tmp_path):
+ out = ok(capsys, tmp_path, "disclosure", {"now": NOW, "created_at": "2026-07-21T23:59:00Z", "labels": []})
+ assert out["issue_age_days"] == 90
+ assert out["overdue_for_disclosure"] is False
+ assert out["expected_disclosure_by"] == "2026-11-02"
+
+
+# --- 1c: reporter staleness --------------------------------------------------
+
+
+@pytest.mark.parametrize(
+ ("messages", "stale", "days"),
+ [
+ (
+ [
+ {"from_team": False, "at": "2026-09-01T00:00:00Z"},
+ {"from_team": True, "at": "2026-10-01T00:00:00Z"},
+ ],
+ True,
+ 18,
+ ),
+ ([{"from_team": True, "at": "2026-10-10T00:00:00Z"}], False, 9),
+ (
+ [
+ {"from_team": True, "at": "2026-09-01T00:00:00Z"},
+ {"from_team": False, "at": "2026-09-05T00:00:00Z"},
+ ],
+ False,
+ None,
+ ),
+ ([], False, None),
+ ],
+)
+def test_staleness(capsys, tmp_path, messages, stale, days):
+ out = ok(capsys, tmp_path, "staleness", {"now": NOW, "messages": messages})
+ assert (out["stale"], out["days"]) == (stale, days)
+
+
+# --- 1g: closed announced trackers -------------------------------------------
+
+
+def closed(**data):
+ return {"cve_id": "CVE-2026-1", "cve_org_state": "PUBLISHED", "body": "", "comments": [], **data}
+
+
+@pytest.mark.parametrize(
+ ("data", "move"),
+ [
+ (closed(), "notify"),
+ (
+ closed(
+ comments=[{"body": "Sync 2026-10-01 — CVE-published reporter notification drafted"}],
+ notification_sent_on_thread=True,
+ ),
+ "already_notified",
+ ),
+ (closed(cve_org_state="RESERVED"), "wait"),
+ (closed(cve_org_state="REJECTED"), "blocker"),
+ (closed(lookup_failed=True), "retry"),
+ (closed(cve_id=""), "skip"),
+ ],
+)
+def test_cve_org_state(capsys, tmp_path, data, move):
+ out = ok(capsys, tmp_path, "closed-announced", data)
+ assert out["cve_org"] == move
+ assert bool(out["blockers"]) is (move == "blocker")
+
+
+@pytest.mark.parametrize(
+ ("data", "pending"),
+ [
+ (closed(public_advisory_url="https://lists.apache.org/thread/x"), True),
+ (closed(public_advisory_url=""), False),
+ (
+ closed(
+ public_advisory_url="u",
+ comments=[{"body": "- [x] <!-- apache-magpie: security-pages-checklist v1 --> pages"}],
+ ),
+ False,
+ ),
+ (
+ closed(
+ public_advisory_url="u",
+ comments=[{"body": "- [ ] <!-- apache-magpie: security-pages-checklist v1 --> pages"}],
+ ),
+ True,
+ ),
+ (
+ closed(
+ public_advisory_url="u",
+ comments=[{"body": "<!-- apache-magpie: security-pages-reminder v1 -->\nplease"}],
+ ),
+ False,
+ ),
+ ],
+)
+def test_security_pages_reminder(capsys, tmp_path, data, pending):
+ out = ok(capsys, tmp_path, "closed-announced", data)
+ assert out["security_pages_reminder_pending"] is pending
+ assert out["advisory_records"] == "not_configured"
+
+
+@pytest.mark.parametrize(
+ ("data", "step"),
+ [
+ (closed(advisory_records_configured=True), "run_planner"),
+ (closed(advisory_records_configured=True, cve_org_state="RESERVED"), "waiting_for_cve_org"),
+ (
+ closed(
+ advisory_records_configured=True,
+ comments=[{"body": "<!-- apache-magpie: advisory-records v1 -->"}],
+ ),
+ "done",
+ ),
+ (closed(advisory_records_configured=True, planner_exit=0, planner_result="proposed"), "pending"),
+ (
+ closed(advisory_records_configured=True, planner_exit=0, planner_result="unchanged"),
+ "already_recorded",
+ ),
+ (closed(advisory_records_configured=True, planner_exit=4), "pending_checkout"),
+ (
+ closed(advisory_records_configured=True, planner_exit=3, planner_reason="page cannot be parsed"),
+ "refused",
+ ),
+ (closed(advisory_records_configured=True, planner_exit=2), "config_error"),
+ ],
+)
+def test_advisory_records(capsys, tmp_path, data, step):
+ out = ok(capsys, tmp_path, "closed-announced", data)
+ assert out["advisory_records"] == step
+ assert out["security_pages_reminder_pending"] is False
+
+
+# --- 1h: release votes -------------------------------------------------------
+
+VOTE = "[VOTE] Release Apache Airflow 3.2.3 from 3.2.3rc1"
+
+
+def vote(messages, labels=("airflow", "pr merged"), milestone="Airflow 3.2.3", gating=True):
+ return {"gating": gating, "labels": list(labels), "fix_milestone": milestone, "messages": messages}
+
+
+@pytest.mark.parametrize(
+ ("data", "in_progress"),
+ [
+ (vote([{"subject": VOTE, "url": "https://lists/t1"}]), True),
+ (vote([{"subject": VOTE, "url": "https://lists/t1"}, {"subject": "[RESULT]" + VOTE}]), False),
+ (vote([{"subject": "[VOTE] Release Apache Airflow 3.2.4 from 3.2.4rc1"}]), False),
+ (vote([{"subject": VOTE}], labels=("airflow", "pr merged", "fix released")), False),
+ (vote([{"subject": VOTE}], labels=("airflow", "cve allocated")), False),
+ (vote([{"subject": VOTE}], gating=False), False),
+ (vote([{"subject": VOTE}], milestone="Providers 2026-04-21"), False),
+ ],
+)
+def test_release_vote(capsys, tmp_path, data, in_progress):
+ out = ok(capsys, tmp_path, "release-vote", data)
+ assert out["release_vote_in_progress"] is in_progress
+ if in_progress:
+ assert out["release_vote_thread_url"] == "https://lists/t1"
diff --git a/tools/spec-loop/specs/security-issue-lifecycle.md b/tools/spec-loop/specs/security-issue-lifecycle.md
index 6b52d058..7694835d 100644
--- a/tools/spec-loop/specs/security-issue-lifecycle.md
+++ b/tools/spec-loop/specs/security-issue-lifecycle.md
@@ -179,7 +179,10 @@ publication, with a human gate and an audit-log entry at every step.
admin-only change at the collaborator tier, or after a `403`, is a draft
relay, never a tracker comment), and walks the Step 5b push flow,
checking the title, upgrade-target and CWE gates itself and taking the
- four judged gates as the model's verdicts; mining signals, gathering
+ four judged gates as the model's verdicts. The Step 1 observations
+ that are arithmetic are code too: the disclosure-window flags, reporter
+ staleness, the closed-`announced` cve.org move with the security-pages
+ markers and the advisory-records step, and the release-vote match; mining signals, gathering
state, judging those gates and writing comments and drafts stay with
the model.
- Shared recipes instead of per-skill copies (#1437): the safe-create