You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@magpie.apache.org by po...@apache.org on 2026/10/16 04:58:30 UTC
(magpie) 01/01: feat(security-issue-sync): record a published CVE in the project's own security records
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch feat/advisory-records
in repository https://gitbox.apache.org/repos/asf/magpie.git
commit b5feea2f04f52ed1fcb9eb15a5f428d173de3395
Author: Jarek Potiuk <po...@apache.org>
AuthorDate: Fri Oct 16 05:57:38 2026 +0100
feat(security-issue-sync): record a published CVE in the project's own security records
Once a CVE is public, projects record it where their site builds its
security pages, and Magpie stopped at a reminder. A new advisory-records
planner in generate-cve-json proposes that change, over Git or
Subversion, for the two layouts projects use:
- cve-json-file: one CVE 5.x JSON file per CVE in a site repository,
rendered from the generate-cve-json record (as httpd does).
- release-page: an entry in a per-release-line page (XML or Markdown),
added to the fixed release's section, created newest-first when
missing, from adopter-configured templates (as Tomcat's site does).
It refuses until cve.org reports the record PUBLISHED with a
vendor-advisory reference, escapes all tracker text, is idempotent, and
prints the file change plus the magpie-vcs commands as a proposal;
--apply writes only the file. security-issue-sync proposes it for
announced trackers when project.md configures `advisory_records`;
unset, nothing changes. tools/vcs gains `update` and `checkout`.
Generated-by: Claude Opus 5
---
docs/security/README.md | 4 +
docs/security/advisory-records.md | 117 ++
docs/security/process.md | 4 +
.../skills/issue-sync/apply-and-push.md | 9 +
.../magpie-security/skills/issue-sync/gather.md | 29 +
.../skills/issue-sync/signals-to-actions.md | 23 +
plugins/magpie-setup/templates/project.md | 51 +-
.../cve-tool-vulnogram/generate-cve-json/README.md | 54 +
.../generate-cve-json/pyproject.toml | 1 +
.../src/generate_cve_json/advisory_records.py | 1228 ++++++++++++++++++++
.../advisory_templates/release-page-entry.md.tmpl | 10 +
.../advisory_templates/release-page-entry.xml.tmpl | 13 +
.../release-page-section.md.tmpl | 7 +
.../release-page-section.xml.tmpl | 7 +
.../tests/test_advisory_records.py | 464 ++++++++
tools/github/source-control.md | 1 +
tools/skill-evals/README.md | 2 +-
.../evals/security-issue-sync/README.md | 3 +-
.../expected.json | 17 +
.../case-11-advisory-records-proposal/report.md | 39 +
.../expected.json | 10 +
.../report.md | 28 +
.../fixtures/output-spec.md | 3 +-
tools/spec-loop/specs/security-issue-lifecycle.md | 13 +
tools/vcs/README.md | 4 +
tools/vcs/src/magpie_vcs/__init__.py | 66 ++
tools/vcs/tests/test_vcs.py | 37 +
27 files changed, 2240 insertions(+), 4 deletions(-)
diff --git a/docs/security/README.md b/docs/security/README.md
index 7c422466..71d13302 100644
--- a/docs/security/README.md
+++ b/docs/security/README.md
@@ -255,6 +255,10 @@ a private GitHub repository (default), [Markdown files](../../tools/local-tracke
how the team operates and how a new member lands: the triager
rotation that runs the skills and delegates, tracker access, mail
list subscription, expected reading, first triage shadow.
+- [**`advisory-records.md`**](advisory-records.md) — recording a
+ published CVE in the project's own security records (a CVE JSON
+ file or a per-release-line page in its site repository, over Git
+ or Subversion), proposed by sync once cve.org shows it published.
- [**`threat-model.md`**](threat-model.md) — release-blocking
threat model for the security skill family: trust boundaries,
adversary personas, STRIDE matrix per skill, mitigation cross-
diff --git a/docs/security/advisory-records.md b/docs/security/advisory-records.md
new file mode 100644
index 00000000..a57ff7b8
--- /dev/null
+++ b/docs/security/advisory-records.md
@@ -0,0 +1,117 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [Recording published CVEs in the project's security records](#recording-published-cves-in-the-projects-security-records)
+ - [When it runs](#when-it-runs)
+ - [The two layouts](#the-two-layouts)
+ - [Configuration](#configuration)
+ - [What the user confirms](#what-the-user-confirms)
+ - [Safety properties](#safety-properties)
+ - [Limits](#limits)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# Recording published CVEs in the project's security records
+
+**Capability:** `capability:resolve` (see [`docs/labels-and-capabilities.md`](../labels-and-capabilities.md)).
+
+Most projects list their published vulnerabilities on their own website, built from a site repository they keep in Git or Subversion.
+Once an advisory is public, the ASF security-committers policy asks the project to update those pages.
+Without configuration, `security-issue-sync` only tracks that step: a checklist item in the release-manager hand-off comment, and at most one reminder.
+With [`advisory_records`](../../plugins/magpie-setup/templates/project.md#advisory-records) configured, sync proposes the edit itself:
+the file change, and the version-control commands that publish it.
+
+## When it runs
+
+In the [Step 1g](../../skills/security-issue-sync/gather.md#1g-recently-closed-trackers--check-cveorg-publication-state) closed-bucket scan,
+for a closed `announced` tracker, once cve.org reports the CVE `PUBLISHED` —
+the same condition sync uses for the final *"CVE is live"* reporter email.
+The planner checks the condition itself and refuses before it, so a mistaken run cannot record an embargoed CVE:
+it needs cve.org's `PUBLISHED` state *and* a `vendor-advisory` reference in the record (the archived public advisory).
+
+A tracker whose CVE is already recorded gets no proposal, and neither does one that carries the `advisory-records v1` marker comment sync posts after a successful run.
+
+## The two layouts
+
+| Layout | What it changes | Example of a project laid out this way |
+|---|---|---|
+| `cve-json-file` | Adds `<CVE-ID>.json` to a directory of the site repository: the CVE 5.x record [`generate-cve-json`](../../tools/cve-tool-vulnogram/generate-cve-json/README.md) emits, without the CNA tool's internal `CNA_private` block. The site builds its vulnerability pages from these files. | Apache HTTP Server keeps them under `content/security/json/` in its site repository |
+| `release-page` | Inserts an entry into the page for the fixed release's line, inside the section for that release; a missing section is created in newest-first order. | Apache Tomcat keeps one `xdocs/security-<line>.xml` page per release line in its Subversion site |
+
+The shipped `release-page` example templates follow that XML shape:
+
+```xml
+<section name="Fixed in <Product> <version>" rtext="<release date>">
+
+ <p><strong><Impact>: <title></strong>
+ <cve><CVE-ID></cve></p>
+
+ <p><description></p>
+
+ <p>This was fixed with commit <hashlink hash="<commit>"/>.</p>
+
+ <p>This issue was reported to the <Product> security team on <date>.
+ The issue was made public on <date>.</p>
+
+ <p>Affects: <range></p>
+
+</section>
+```
+
+The page path, the section name and both templates are the adopter's, so another XML page format works by changing the templates,
+and `format: markdown` handles a Markdown page whose sections are headings.
+The template reference is in the
+[tool README](../../tools/cve-tool-vulnogram/generate-cve-json/README.md#record-a-published-cve-in-the-projects-security-records--advisory-records).
+
+A CVE fixed in several release lines is recorded on each line's page in one commit.
+
+## Configuration
+
+The `advisory_records` block in `<project-config>/project.md`
+([template](../../plugins/magpie-setup/templates/project.md#advisory-records)):
+`vcs` (`git` or `svn`, defaulting to the project's source-control backend), `repository` (used to propose the first checkout), `working_copy`,
+`branch` and `remote` (Git), `commit_message`, `layout`, and the layout's own block (`cve_json_file` or `release_page`).
+The working copy is a per-user path, so `user.md → environment.advisory_records_clone` overrides `working_copy`.
+
+Leave the block out and nothing changes: sync notes *"advisory records: not configured — skipped"* and keeps the checklist reminder.
+
+## What the user confirms
+
+One numbered item in sync's Step 2b proposal: the diff, and the [`magpie-vcs`](../../tools/vcs/README.md) commands in order.
+
+| Step | Git | Subversion |
+|---|---|---|
+| 1 | `clean` — stop on unrelated changes | `clean` |
+| 2 | `branch` — must be the configured branch | — |
+| 3 | `update` (`pull --ff-only`) | `update` |
+| 4 | the planner again with `--apply`, on the updated copy | same |
+| 5 | `stage <files>` | `stage <files>` (`svn add` for a new file) |
+| 6 | `diff --cached <files>` | `diff <files>` |
+| 7 | `commit -m "<message>"` | `commit -m "<message>"` — this publishes |
+| 8 | `push <remote> <branch>` | — |
+
+With no working copy yet, the proposal is the single `checkout` command; the change is proposed on the next sync.
+After the commit (and the push on Git) succeeds, sync posts a status comment with the `advisory-records v1` marker and the ticked security-pages checklist item.
+Committing or pushing needs the user's credentials for the site repository; under the [secure agent setup](../setup/secure-agent-setup.md) the user may have to run those commands from their own terminal.
+
+## Safety properties
+
+- **Never before publication.** Refused unless cve.org reports `PUBLISHED` and the record carries the public advisory.
+- **Idempotent.** A re-run finds the entry (the CVE ID as an element's whole text or attribute value, or in the Markdown section) and proposes nothing.
+ A CVE recorded under a different release is refused, not duplicated.
+- **External content is escaped.** Title, description, product and affected-versions text come from the tracker;
+ they are escaped for the page format, and identifiers (CVE ID, versions, dates, commit hashes) are validated.
+- **Pages are spliced, not rewritten.** The planner refuses a page that is not well-formed XML, never fetches external entities,
+ and keeps the page's formatting, comments, entity references and line endings.
+- **No unconfirmed VCS command.** The planner runs none; sync runs the confirmed ones one by one and stops at the first failure.
+
+## Limits
+
+- The first page of a new release line is not created; the planner refuses and asks for it to be added by hand.
+- Inputs the record does not carry come from the tracker, the release train or the user:
+ the release date (only to create a section), the date the report arrived, and fix commits when no `patch` reference is a commit URL.
+- The default *Affects* text is the record's version range (`>= 9.0.0, < 9.0.99`); a project that writes it differently passes `--affects`.
diff --git a/docs/security/process.md b/docs/security/process.md
index 2810daf2..ebb2d685 100644
--- a/docs/security/process.md
+++ b/docs/security/process.md
@@ -424,6 +424,10 @@ hand-off comment, all of it single clicks in the CVE tool —
that sees the advisory shipped but the box still unticked posts
a one-line reminder comment — at most once per tracker, never
blocking the close-out.
+ A project that configures `advisory_records` gets the edit proposed instead:
+ once cve.org shows the CVE published, sync proposes the change to the site repository
+ (a CVE JSON file, or an entry in a per-release-line page) and the Git or Subversion commands that publish it —
+ see [`advisory-records.md`](advisory-records.md).
The severity score follows the
[ASF severity rating](https://security.apache.org/blog/severityrating)
diff --git a/plugins/magpie-security/skills/issue-sync/apply-and-push.md b/plugins/magpie-security/skills/issue-sync/apply-and-push.md
index faa850b8..a7939c6c 100644
--- a/plugins/magpie-security/skills/issue-sync/apply-and-push.md
+++ b/plugins/magpie-security/skills/issue-sync/apply-and-push.md
@@ -145,6 +145,15 @@ Use:
Idempotency keys on the marker `<!-- apache-magpie: release-manager-wrap-up v1 -->`:
if it is already on the tracker, skip the post entirely.
Apply the same bare-name → `@handle` scrub before posting.
+- **Advisory-records update:** run the confirmed commands from the 2b item one at a time, in order, each as a plain command,
+ and stop at the first that fails —
+ the clean check exits `1` when the working copy has unrelated changes, and a branch other than the configured one is a stop too.
+ The `--apply` re-run recomputes the change on the updated working copy; if its file list differs from the confirmed one, stop and show the new diff.
+ After the `commit` (and on Git the `push`) succeeds, post a status comment from a temp file:
+ line 1 is the marker `<!-- apache-magpie: advisory-records v1 -->`, then one sentence naming the file(s) and the commit,
+ then the ticked item `- [x] <!-- apache-magpie: security-pages-checklist v1 --> Project security pages updated with CVE_ID`,
+ which records the security-pages step as done.
+ A failed command posts nothing; report it in the recap with the command's output.
- **Vulnogram state transition (`REVIEW → PUBLIC`):** invoke the
[`vulnogram-api-record-publish`](../../../../tools/cve-tool-vulnogram/oauth-api/README.md)
CLI to flip the record's `CNA_private.state` over the OAuth API.
diff --git a/plugins/magpie-security/skills/issue-sync/gather.md b/plugins/magpie-security/skills/issue-sync/gather.md
index ee3d10ae..97ca5bfb 100644
--- a/plugins/magpie-security/skills/issue-sync/gather.md
+++ b/plugins/magpie-security/skills/issue-sync/gather.md
@@ -583,6 +583,35 @@ an unticked box in the hand-off comment is the pending state, not the satisfied
Step 2b turns a pending flag into the *Security-pages reminder comment* proposal ([`signals-to-actions.md`](signals-to-actions.md)).
The scan reads the body and comments the 1a fetch already returned — no extra API call.
+**Advisory records.** When [`<project-config>/project.md`](../../../../<project-config>/project.md#advisory-records) declares an `advisory_records` block,
+sync also records the CVE in the project's own security records — a site repository over Git or Subversion
+([`docs/security/advisory-records.md`](../../../../docs/security/advisory-records.md)).
+Without the block, record *"advisory records: not configured — skipped"* in the observed state and do nothing else.
+With it, `security_pages_reminder_pending` is always `false`: sync proposes the edit itself instead of reminding the release manager.
+For each closed-`announced` tracker:
+
+1. Skip when a comment carrying `<!-- apache-magpie: advisory-records v1 -->` exists: record *"advisory records: done on <date>"*.
+2. Skip unless the cve.org check above returned `PUBLISHED`: record *"advisory records: waiting for cve.org"*.
+ The tool refuses before publication too; never work around it.
+3. Write the record to the session scratch directory:
+ `uv run --project <framework>/tools/cve-tool-vulnogram/generate-cve-json generate-cve-json <N> --output <scratch>/<CVE-ID>.json`.
+4. Run the planner as a plain command, without `--apply`.
+ Each `advisory_records` key maps to the flag of the same name (`layout`, `vcs`, `repository`, `branch`, `remote`, `commit_message`,
+ and the layout's `path`, `indent` → `--json-indent`, `keep_cna_private`, `format`, `release_line_parts`, `section_name`, `section_template`, `entry_template`, `fix_commit_template`, `section_parent`);
+ the working copy is `user.md → environment.advisory_records_clone`, else `advisory_records.working_copy`:
+ ```bash
+ uv run --project <framework>/tools/cve-tool-vulnogram/generate-cve-json advisory-records --layout <layout> --working-copy <path> --path '<path template>' --record <scratch>/<CVE-ID>.json --cve-org-state PUBLISHED --date-published <datePublished> [--reported-date <date>] [layout flags]
+ ```
+ For `release-page`, pass `--reported-date` (the date of the first message on the *Security mailing list thread*).
+ When the output names a missing input (`--release-date` for a new release section, `--fix-commit`, `--affects`), find it on the tracker,
+ in [`release-trains.md`](../../../../<project-config>/release-trains.md) or in the release's announcement, and run it again; ask the user when none of them has it.
+5. Read the exit code:
+ `0` with `proposed` → `advisory_records_pending: true`, keeping the tool's diff and numbered commands for Step 2b;
+ `0` with `unchanged` → *"advisory records: already recorded"*;
+ `4` → `advisory_records_pending: true` with the single `checkout` command;
+ `3` → *"advisory records: refused — <reason>"*, surfaced as a blocker when the page cannot be parsed or already holds the CVE under another release;
+ `2` → a configuration error to show the user.
+
**When the tracker has no CVE ID.** Closed trackers without a `CVE-YYYY-NNNNN` in the *CVE tool link* body field
are closing dispositions (`invalid` / `duplicate` / `wontfix`):
skip the cve.org check entirely and drop the tracker from the closed-bucket sweep.
diff --git a/plugins/magpie-security/skills/issue-sync/signals-to-actions.md b/plugins/magpie-security/skills/issue-sync/signals-to-actions.md
index edbe62d8..b45b9a96 100644
--- a/plugins/magpie-security/skills/issue-sync/signals-to-actions.md
+++ b/plugins/magpie-security/skills/issue-sync/signals-to-actions.md
@@ -684,9 +684,32 @@ Group them by category:
After a PATCH, note which variant the body now carries
(*"flipped to OAuth-pushed variant after this sync's auto-push succeeded"* or vice-versa).
+- **Advisory-records update** — records the published CVE in the project's own security records
+ (the site repository its security pages are built from), when the project configures
+ [`advisory_records`](../../../../<project-config>/project.md#advisory-records)
+ ([`docs/security/advisory-records.md`](../../../../docs/security/advisory-records.md)).
+ It replaces the security-pages reminder for those projects: sync makes the edit instead of asking the release manager for it.
+
+ **Trigger.** Step 1g recorded `advisory_records_pending: true` for a closed-`announced` tracker:
+ cve.org reports the CVE `PUBLISHED`, no comment carries the `advisory-records v1` marker,
+ and the `advisory-records` planner returned a change (or, with no working copy yet, a `checkout` command).
+ Before publication there is no proposal — the planner refuses, and the observed state says *"waiting for cve.org"*.
+ Without `advisory_records` there is no proposal either; the observed state says *"not configured — skipped"*.
+
+ **Proposed action.** One numbered item showing the planner's file change (the diff it printed) and its numbered
+ [`magpie-vcs`](../../../../tools/vcs/README.md) commands, verbatim and in order:
+ the clean check, the branch check (Git), `update`, the planner re-run with `--apply`, `stage`, `diff`, `commit`, and `push` (Git).
+ On Subversion, say that the `commit` publishes the change.
+ With no working copy, the item is the single `checkout` command; the change is proposed on the next sync.
+
+ **Guard.** Never run a command the user has not confirmed, never edit the page or the JSON file by hand,
+ and never push to another branch or remote than the configured ones.
+ The planner is idempotent: an already-recorded CVE yields `unchanged` and no proposal.
+
- **Security-pages reminder comment** — the ASF security-committers policy's post-announcement step *"The project team updates the
project's security pages"* is a website edit, which sync cannot perform or gate the close-out on
(the close-out fires on the archive-URL signal).
+ Projects that configure `advisory_records` never get this reminder: the *Advisory-records update* above makes the edit.
Sync carries the checklist item that records it: both hand-off comment variants embed a
`- [ ] <!-- apache-magpie: security-pages-checklist v1 --> Project
security pages updated with CVE_ID` box, and ticking it records the completion marker on the tracker.
diff --git a/plugins/magpie-setup/templates/project.md b/plugins/magpie-setup/templates/project.md
index 0d263a56..c123d9f6 100644
--- a/plugins/magpie-setup/templates/project.md
+++ b/plugins/magpie-setup/templates/project.md
@@ -29,6 +29,7 @@
- [Project metadata](#project-metadata)
- [Security cross-reference](#security-cross-reference)
- [Tracker](#tracker)
+ - [Advisory records](#advisory-records)
- [Scope detection](#scope-detection)
- [Release process](#release-process)
- [Roster](#roster)
@@ -93,7 +94,7 @@ produces.
| `upstream_contributing_docs_url` | TODO | |
| `upstream_genai_disclosure_anchor` | TODO: URL + anchor for the project's Gen-AI disclosure guideline | |
| `upstream_security_policy_url` | TODO: `https://github.com/<upstream>/security/policy` | |
-| `security_pages_url` | TODO: e.g. `https://foo.apache.org/security` | Public security pages on the project website — updated by the release manager after each advisory (ASF security-committers policy step); read by `security-issue-sync` for the security-pages checklist reminder. Leave unset if the project has no security pages — the skill then uses the plain phrase without a link |
+| `security_pages_url` | TODO: e.g. `https://foo.apache.org/security` | Public security pages on the project website — updated by the release manager after each advisory (ASF security-committers policy step); read by `security-issue-sync` for the security-pages checklist reminder. Leave unset if the project has no security pages — the skill then uses the plain phrase without a link. To have sync propose the edit itself, also configure [Advisory records](#advisory-records) |
## Mailing lists
@@ -529,6 +530,54 @@ single `<!-- rejection-backfill v1 count: N -->` comment. The
ledger issue from all tracker classification. Closes handled by
`security-issue-invalidate` are **not** ledger entries.
+### Advisory records
+
+Optional, per-project — where the project records each published CVE in its own published security records
+(the site repository its security pages are built from),
+so that `security-issue-sync` can propose the change once cve.org shows the CVE published.
+Leave the block out and the step is skipped with a one-line note; the security-pages checklist reminder stays as it is.
+The deterministic part is `advisory-records` in
+[`tools/cve-tool-vulnogram/generate-cve-json/`](../../../tools/cve-tool-vulnogram/generate-cve-json/README.md#record-a-published-cve-in-the-projects-security-records--advisory-records);
+the step is described in [`docs/security/advisory-records.md`](../../../docs/security/advisory-records.md).
+
+```yaml
+advisory_records:
+ # git | svn. Default: the project's source-control backend (Tools enabled → Source control).
+ vcs: git
+ # Repository URL, used only to propose the first checkout.
+ repository: https://github.com/<org>/<project>-site.git
+ # Local working copy. A per-user path: `user.md → environment.advisory_records_clone` wins over this.
+ working_copy: ~/code/<project>-site
+ # Git only: the branch and remote the commit is pushed to.
+ branch: main
+ remote: origin
+ # One line; placeholder {cve_id}.
+ commit_message: "Add {cve_id} to the security pages"
+
+ # cve-json-file — one CVE 5.x JSON file per CVE, the record generate-cve-json emits
+ # (e.g. Apache HTTP Server keeps these under content/security/json/).
+ # release-page — an entry in a per-release-line page, one section per fixed release, newest first
+ # (e.g. Apache Tomcat's xdocs/security-<line>.xml).
+ layout: cve-json-file
+
+ cve_json_file:
+ path: content/security/json/{cve_id}.json # placeholders {cve_id}, {year}
+ indent: 4
+ keep_cna_private: false # drop the CNA tool's internal workflow block
+
+ release_page:
+ format: xml # xml | markdown
+ path: xdocs/security-{line}.xml # placeholder {line}
+ release_line_parts: 1 # version components that name a line: 1 → "9" for 9.0.99
+ section_name: "Fixed in {product} {version}"
+ # Template files, relative to <project-config>; omit to use the shipped examples
+ # (tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/advisory_templates/).
+ # section_template: advisory-records/section.xml
+ # entry_template: advisory-records/entry.xml
+ # fix_commit_template: '<hashlink hash="{hash}"/>'
+ # section_parent: body # xml: where the first release section goes
+```
+
### Scope detection
Per-project — whether this project distinguishes scope sub-products, and
diff --git a/tools/cve-tool-vulnogram/generate-cve-json/README.md b/tools/cve-tool-vulnogram/generate-cve-json/README.md
index df8356e7..2f8c5dd5 100644
--- a/tools/cve-tool-vulnogram/generate-cve-json/README.md
+++ b/tools/cve-tool-vulnogram/generate-cve-json/README.md
@@ -7,6 +7,7 @@
- [generate-cve-json](#generate-cve-json)
- [Run](#run)
+ - [Record a published CVE in the project's security records — `advisory-records`](#record-a-published-cve-in-the-projects-security-records--advisory-records)
- [Test](#test)
- [Lint / type-check](#lint--type-check)
@@ -73,6 +74,59 @@ uv run generate-cve-json <ISSUE-NUMBER>
Flags are documented in `generate-cve-json --help` and in [`SKILL.md`](SKILL.md).
+## Record a published CVE in the project's security records — `advisory-records`
+
+A second console script in the same project.
+After the advisory is public, it proposes adding the CVE to the project's *own* published security records:
+a site repository the project keeps over Git or Subversion.
+It prints the file change and the [`magpie-vcs`](../../vcs/README.md) commands that would publish it, and runs no VCS command;
+`--apply` writes the file change and nothing else.
+`security-issue-sync` calls it from its closed-bucket scan and runs the commands only after the user confirms them
+(see [`docs/security/advisory-records.md`](../../../docs/security/advisory-records.md)).
+
+Two layouts, chosen with `--layout`:
+
+| Layout | What changes | Example of a project laid out this way |
+|---|---|---|
+| `cve-json-file` | `--path` (e.g. `content/security/json/{cve_id}.json`) gets the CVE 5.x record `generate-cve-json --output` wrote, without the CNA tool's `CNA_private` block unless `--keep-cna-private` | Apache HTTP Server's site repository, which builds its vulnerability pages from these files |
+| `release-page` | the entry is inserted into the page for the fixed release's line (`--path xdocs/security-{line}.xml`, the line being the first `--release-line-parts` version components), in the section named by `--section-name` (default `Fixed in {product} {version}`); a missing section is created in newest-first order | Apache Tomcat's site, one `xdocs/security-<line>.xml` page per release line |
+
+The release-page templates are files: `--entry-template` and `--section-template`
+(defaults: the shipped examples in [`src/generate_cve_json/advisory_templates/`](src/generate_cve_json/advisory_templates/),
+an XML page with `<section name="..." rtext="<release date>">` sections and `<p>` entries carrying a `<cve>` element and `<hashlink>` fix commits).
+`--format markdown` switches to the Markdown examples: sections are headings at the level of the section template's first line.
+Placeholders are `{name}` only; an unknown one is an error.
+Entry: `{impact}` `{title}` `{cve_id}` `{description}` `{fix_commits}` `{fix_commit_noun}` `{reported_date}` `{public_date}` `{affects}` `{product}` `{version}` `{release_date}` `{line}`.
+Section: `{section_name}` `{release_date}` `{version}` `{line}` `{product}`, and a line holding only `{entries}`.
+An XML entry template must carry `{cve_id}` as the whole text of an element or a whole attribute value: that is how a re-run finds the entry.
+
+Behaviour that does not depend on configuration:
+
+- **Gate.** It refuses (exit 3) unless cve.org reports the record `PUBLISHED` (`--cve-org-state`, or `--cve-org-record` with the saved `cve-check-published` output)
+ and the record carries a `vendor-advisory` reference — the condition sync uses for the *"CVE is live"* step.
+- **Idempotent.** A CVE already in the right file or section is reported `unchanged` with no commands;
+ one recorded under a *different* release section is refused rather than duplicated.
+- **External text is escaped** for the page format (title, description, product, affected-versions text),
+ identifiers are validated (CVE ID, versions, dates, commit hashes), and a rendered XML entry must be well-formed.
+- **The page is edited in place**, never re-serialised: expat checks it is well-formed (or the run is refused) and locates the sections,
+ and the change is a splice that keeps the page's formatting, comments, entity references and line endings.
+- **Inputs it cannot derive** come as flags: `--release-date VERSION=DATE` (only to create a section), `--reported-date`,
+ `--fix-commit` (added to the commits read from the record's `patch` references), `--fixed-in`, `--affects VERSION=TEXT`;
+ `--public-date` defaults to the date cve.org published the record.
+ A template placeholder with no value is refused with the flag that supplies it.
+- **Commands** (Git): `clean`, `branch`, `update`, the same command with `--apply`, `stage`, `diff --cached`, `commit -m`, `push <remote> <branch>`.
+ On Subversion there is no `branch` or `push`, and the `commit` publishes.
+ A missing working copy yields one `checkout` command (exit 4).
+
+```bash
+uv run --project <framework>/tools/cve-tool-vulnogram/generate-cve-json advisory-records \
+ --layout release-page --working-copy ~/code/foo-site --path 'xdocs/security-{line}.xml' \
+ --record /tmp/CVE-2026-00001.json --cve-org-state PUBLISHED --date-published 2026-04-02 \
+ --reported-date 2026-02-14 --release-date 9.0.99=2026-03-30
+```
+
+Exit codes: `0` proposal (or `unchanged`, or `applied`), `2` usage or configuration error, `3` refused, `4` working copy missing.
+
## Test
```bash
diff --git a/tools/cve-tool-vulnogram/generate-cve-json/pyproject.toml b/tools/cve-tool-vulnogram/generate-cve-json/pyproject.toml
index 5a760b91..77944e5a 100644
--- a/tools/cve-tool-vulnogram/generate-cve-json/pyproject.toml
+++ b/tools/cve-tool-vulnogram/generate-cve-json/pyproject.toml
@@ -32,6 +32,7 @@ dependencies = []
[project.scripts]
generate-cve-json = "generate_cve_json:main"
+advisory-records = "generate_cve_json.advisory_records:main"
[tool.hatch.build.targets.wheel]
packages = ["src/generate_cve_json"]
diff --git a/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/advisory_records.py b/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/advisory_records.py
new file mode 100644
index 00000000..10ed3f7c
--- /dev/null
+++ b/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/advisory_records.py
@@ -0,0 +1,1228 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Record a published CVE in the project's own published security records.
+
+After the advisory is public and cve.org shows the record ``PUBLISHED``,
+a project usually also records the CVE in its own site repository. Two
+layouts are supported, both configured by the adopter (the
+``advisory_records`` block of ``<project-config>/project.md``):
+
+``cve-json-file``
+ One CVE 5.x JSON file per CVE in a directory of the site repository
+ (the record ``generate-cve-json`` emits), from which the site builds
+ its vulnerability pages.
+
+``release-page``
+ One page per release line (XML or Markdown), with one section per
+ fixed release, newest first, and one entry per vulnerability in it.
+ The page path, the section name and the entry are adopter templates;
+ the shipped example templates follow a ``<section name=... rtext=...>``
+ XML page.
+
+The tool computes the file change and the version-control commands
+(``magpie-vcs``, Git or Subversion) that would publish it, and prints
+them as a **proposal**. It never runs a VCS command. ``--apply`` writes
+the file change and nothing else; the calling skill runs the commands
+only after the user confirms them.
+
+Every text value taken from the CVE record (title, description,
+product, affected-versions text) is tracker content — external data —
+and is escaped for the page format before it is inserted. Identifiers
+(CVE ID, versions, dates, commit hashes) are validated instead.
+"""
+
+from __future__ import annotations
+
+import argparse
+import copy
+import difflib
+import json
+import re
+import shlex
+import sys
+import xml.parsers.expat
+from collections.abc import Callable, Mapping, Sequence
+from dataclasses import dataclass, field
+from pathlib import Path
+from xml.sax.saxutils import escape as _xml_escape
+
+from generate_cve_json.cve_json import format_version_range
+
+EXIT_OK = 0
+EXIT_USAGE = 2
+EXIT_REFUSED = 3
+EXIT_NEEDS_CHECKOUT = 4
+
+LAYOUTS = ("cve-json-file", "release-page")
+PAGE_FORMATS = ("xml", "markdown")
+VCS_CHOICES = ("auto", "git", "svn")
+
+CVE_ID_RE = re.compile(r"CVE-\d{4}-\d{4,}")
+VERSION_RE = re.compile(r"[0-9A-Za-z][0-9A-Za-z.+_-]*")
+DATE_RE = re.compile(r"\d{4}-\d{2}-\d{2}")
+HASH_RE = re.compile(r"[0-9a-f]{7,64}")
+_PLACEHOLDER_RE = re.compile(r"\{([a-z_]+)\}")
+# Commit URLs on the common forges and gitweb / cgit front ends.
+_COMMIT_URL_RE = re.compile(r"(?:/commits?/|[?;&](?:h|id)=)([0-9a-f]{7,64})(?![0-9a-z])")
+# Characters XML 1.0 does not allow at all.
+_XML_INVALID_RE = re.compile(r"[\x00-\x08\x0b\x0c\x0e-\x1f]")
+
+TEMPLATES_DIR = Path(__file__).resolve().parent / "advisory_templates"
+# src/generate_cve_json/advisory_records.py -> the generate-cve-json project, and tools/.
+_TOOL_PROJECT = Path(__file__).resolve().parents[2]
+_DEFAULT_VCS_PROJECT = Path(__file__).resolve().parents[4] / "vcs"
+
+DEFAULT_FIX_COMMIT_TEMPLATES = {"xml": '<hashlink hash="{hash}"/>', "markdown": "`{hash}`"}
+DEFAULT_COMMIT_MESSAGE = "Add {cve_id} to the security pages"
+
+# Placeholder -> the flag that supplies it, for "missing input" refusals.
+_INPUT_FLAGS = {
+ "reported_date": "--reported-date",
+ "release_date": "--release-date VERSION=YYYY-MM-DD",
+ "public_date": "--public-date (or --date-published / --cve-org-record)",
+ "fix_commits": "--fix-commit HASH (no commit URL among the record's `patch` references)",
+ "fix_commit_noun": "--fix-commit HASH",
+ "affects": "--affects VERSION=TEXT",
+ "impact": "a severity in the record's `metrics`",
+ "title": "a title in the record",
+ "description": "a description in the record",
+ "product": "a product in the record's `affected[]`",
+}
+ENTRY_PLACEHOLDERS = frozenset(
+ {
+ "affects",
+ "cve_id",
+ "description",
+ "fix_commit_noun",
+ "fix_commits",
+ "impact",
+ "line",
+ "product",
+ "public_date",
+ "release_date",
+ "reported_date",
+ "title",
+ "version",
+ }
+)
+SECTION_PLACEHOLDERS = frozenset({"section_name", "release_date", "entries", "version", "line", "product"})
+SECTION_NAME_PLACEHOLDERS = frozenset({"version", "line", "product"})
+PATH_PLACEHOLDERS_JSON = frozenset({"cve_id", "year"})
+PATH_PLACEHOLDERS_PAGE = frozenset({"line"})
+
+
+class UsageError(Exception):
+ """Bad invocation or configuration (exit 2)."""
+
+
+class Refusal(Exception):
+ """The tool will not propose a change (exit 3)."""
+
+
+# -----------------------------------------------------------------------------
+# Templates and escaping
+# -----------------------------------------------------------------------------
+
+
+def render_template(
+ template: str, values: Mapping[str, str | None], *, allowed: frozenset[str], what: str
+) -> str:
+ """Substitute ``{name}`` placeholders; refuse unknown names and missing values.
+
+ Only ``{lowercase_name}`` is a placeholder, so literal braces in a page
+ format survive. Values are inserted as given: callers escape them first.
+ """
+ unknown = sorted({m for m in _PLACEHOLDER_RE.findall(template) if m not in allowed})
+ if unknown:
+ raise UsageError(
+ f"{what}: unknown placeholder(s) {', '.join('{' + u + '}' for u in unknown)}; "
+ f"allowed: {', '.join('{' + a + '}' for a in sorted(allowed))}"
+ )
+ missing = sorted({m for m in _PLACEHOLDER_RE.findall(template) if values.get(m) in (None, "")})
+ if missing:
+ hints = "; ".join(f"{{{m}}}: {_INPUT_FLAGS.get(m, 'not available')}" for m in missing)
+ raise Refusal(f"{what} needs a value the inputs do not provide — {hints}")
+ return _PLACEHOLDER_RE.sub(lambda m: str(values[m.group(1)]), template)
+
+
+def _one_line(text: str) -> str:
+ return " ".join(text.split())
+
+
+def escape_xml(text: str) -> str:
+ """Escape external text for XML character data *and* attribute values."""
+ cleaned = _XML_INVALID_RE.sub(" ", _one_line(text))
+ return _xml_escape(cleaned, {'"': """, "'": "'"})
+
+
+_MD_SPECIAL_RE = re.compile(r"([\\`*_\[\]#|~!])")
+
+
+def escape_markdown(text: str) -> str:
+ """Escape external text so Markdown renders it literally (no markup, no raw HTML)."""
+ cleaned = _one_line(text).replace("&", "&").replace("<", "<").replace(">", ">")
+ cleaned = _MD_SPECIAL_RE.sub(r"\\\1", cleaned)
+ # A value that lands at the start of a line must not open a list or a heading.
+ return re.sub(r"^([-+=]|\d+[.)])", r"\\\1", cleaned)
+
+
+ESCAPERS: dict[str, Callable[[str], str]] = {"xml": escape_xml, "markdown": escape_markdown}
+
+
+_LEADING_COMMENT_RE = re.compile(r"\A\s*<!--.*?-->[ \t]*\n?", re.DOTALL)
+
+
+def _load_template(value: str | None, default_name: str) -> str:
+ """Read a template file; leading ``<!-- ... -->`` comments (a licence header) are dropped."""
+ path = TEMPLATES_DIR / default_name if value is None else Path(value).expanduser()
+ if not path.is_file():
+ raise UsageError(f"template file not found: {path}")
+ text = path.read_text(encoding="utf-8")
+ while _LEADING_COMMENT_RE.match(text):
+ text = _LEADING_COMMENT_RE.sub("", text, count=1)
+ return text
+
+
+# -----------------------------------------------------------------------------
+# Settings and inputs
+# -----------------------------------------------------------------------------
+
+
+@dataclass
+class Settings:
+ layout: str
+ vcs: str
+ working_copy: Path
+ repository: str | None
+ branch: str
+ remote: str
+ commit_message: str
+ path: str
+ json_indent: int = 4
+ keep_cna_private: bool = False
+ page_format: str = "xml"
+ release_line_parts: int = 1
+ section_name: str = "Fixed in {product} {version}"
+ section_template: str = ""
+ entry_template: str = ""
+ fix_commit_template: str = ""
+ section_parent: str = "body"
+ vcs_project: Path = _DEFAULT_VCS_PROJECT
+
+
+@dataclass
+class Inputs:
+ record: dict
+ cve_org_state: str | None
+ date_published: str | None
+ release_dates: dict[str, str] = field(default_factory=dict)
+ reported_date: str | None = None
+ public_date: str | None = None
+ fix_commits: list[str] = field(default_factory=list)
+ fixed_in: list[str] = field(default_factory=list)
+ affects: dict[str, str] = field(default_factory=dict)
+
+
+@dataclass
+class FileChange:
+ path: str
+ action: str # create | update | unchanged
+ summary: str
+ old: str | None
+ new: str
+
+ def diff(self) -> str:
+ old_lines = (self.old or "").splitlines(keepends=True)
+ new_lines = self.new.splitlines(keepends=True)
+ return "".join(
+ difflib.unified_diff(
+ old_lines,
+ new_lines,
+ fromfile="/dev/null" if self.old is None else f"a/{self.path}",
+ tofile=f"b/{self.path}",
+ )
+ )
+
+
+@dataclass
+class Release:
+ version: str
+ product: str
+ affects: str
+
+
+def record_cve_id(record: dict) -> str:
+ cve_id = str((record.get("cveMetadata") or {}).get("cveId") or "")
+ if not CVE_ID_RE.fullmatch(cve_id):
+ raise Refusal(f"the CVE record has no valid cveMetadata.cveId (got {cve_id!r})")
+ return cve_id
+
+
+def _cna(record: dict) -> dict:
+ cna = (record.get("containers") or {}).get("cna")
+ if not isinstance(cna, dict):
+ raise Refusal("the CVE record has no containers.cna block")
+ return cna
+
+
+def check_publication_gate(record: dict, cve_org_state: str | None) -> None:
+ """Refuse unless the advisory is public: the same condition sync uses for the
+ *"CVE is live"* step — cve.org reports the record ``PUBLISHED`` — plus a
+ ``vendor-advisory`` reference in the record (the public advisory URL is captured).
+ """
+ cve_id = record_cve_id(record)
+ if (cve_org_state or "").upper() != "PUBLISHED":
+ raise Refusal(
+ f"{cve_id} is not public yet: cve.org state is {cve_org_state or 'unknown'!s}, "
+ f"and the security records are only updated once cve.org reports PUBLISHED"
+ )
+ references = _cna(record).get("references") or []
+ if not any("vendor-advisory" in (ref.get("tags") or []) for ref in references):
+ raise Refusal(
+ f"{cve_id} has no public advisory yet: the record carries no `vendor-advisory` "
+ f"reference (populate the tracker's Public advisory URL field and regenerate the record)"
+ )
+
+
+# -----------------------------------------------------------------------------
+# Paths
+# -----------------------------------------------------------------------------
+
+
+def _safe_relative(path_text: str, working_copy: Path) -> str:
+ rel = Path(path_text)
+ if rel.is_absolute() or ".." in rel.parts or not rel.parts:
+ raise UsageError(f"record path {path_text!r} must be relative to the working copy, without '..'")
+ resolved = (working_copy / rel).resolve()
+ if not resolved.is_relative_to(working_copy.resolve()):
+ raise UsageError(f"record path {path_text!r} leaves the working copy")
+ return rel.as_posix()
+
+
+# -----------------------------------------------------------------------------
+# Layout A — one CVE JSON file per CVE
+# -----------------------------------------------------------------------------
+
+
+def render_cve_json_file(record: dict, *, keep_cna_private: bool) -> dict:
+ out = copy.deepcopy(record)
+ if not keep_cna_private:
+ # CNA_private is the CNA tool's internal workflow state, not record content.
+ out.pop("CNA_private", None)
+ return out
+
+
+def plan_cve_json_file(settings: Settings, inputs: Inputs) -> list[FileChange]:
+ cve_id = record_cve_id(inputs.record)
+ rel = _safe_relative(
+ render_template(
+ settings.path,
+ {"cve_id": cve_id, "year": cve_id.split("-")[1]},
+ allowed=PATH_PLACEHOLDERS_JSON,
+ what="cve-json-file path",
+ ),
+ settings.working_copy,
+ )
+ content = render_cve_json_file(inputs.record, keep_cna_private=settings.keep_cna_private)
+ text = json.dumps(content, indent=settings.json_indent, sort_keys=True, ensure_ascii=False) + "\n"
+ target = settings.working_copy / rel
+ if not target.exists():
+ return [FileChange(rel, "create", f"add {rel}", None, text)]
+ old = target.read_text(encoding="utf-8")
+ try:
+ same = json.loads(old) == content
+ except json.JSONDecodeError:
+ same = False
+ if same:
+ return [FileChange(rel, "unchanged", f"{rel} already holds this record", old, old)]
+ return [FileChange(rel, "update", f"replace {rel} with the published record", old, text)]
+
+
+# -----------------------------------------------------------------------------
+# Layout B — an entry in a per-release-line page
+# -----------------------------------------------------------------------------
+
+
+def version_key(version: str) -> list[tuple[int, int | str]]:
+ """Sort key for release versions: numeric parts numerically, a qualifier
+ (``M1``, ``beta2``) below the plain release it qualifies."""
+ key: list[tuple[int, int | str]] = []
+ for part in re.findall(r"\d+|[A-Za-z]+", version):
+ key.append((0, int(part)) if part.isdigit() else (-1, part.lower()))
+ return key
+
+
+def _version_lt(a: str, b: str) -> bool:
+ ka, kb = version_key(a), version_key(b)
+ width = max(len(ka), len(kb))
+ pad: tuple[int, int | str] = (0, 0)
+ return ka + [pad] * (width - len(ka)) < kb + [pad] * (width - len(kb))
+
+
+def release_line(version: str, parts: int) -> str:
+ return ".".join(version.split(".")[:parts])
+
+
+def _impact(cna: dict) -> str | None:
+ for metric in cna.get("metrics") or []:
+ text = ((metric.get("other") or {}).get("content") or {}).get("text")
+ if not text:
+ for key in ("cvssV4_0", "cvssV3_1", "cvssV3_0"):
+ text = (metric.get(key) or {}).get("baseSeverity")
+ if text:
+ break
+ if text and str(text).strip().lower() not in {"unknown", "_no response_"}:
+ words = str(text).strip().split()
+ return " ".join(w[:1].upper() + w[1:].lower() for w in words)
+ return None
+
+
+def _description(cna: dict) -> str | None:
+ descriptions = cna.get("descriptions") or []
+ for desc in descriptions:
+ if desc.get("lang", "en").startswith("en") and desc.get("value"):
+ return str(desc["value"])
+ return str(descriptions[0]["value"]) if descriptions and descriptions[0].get("value") else None
+
+
+def fix_commits(cna: dict, extra: Sequence[str]) -> list[str]:
+ hashes: list[str] = []
+ for ref in cna.get("references") or []:
+ if "patch" not in (ref.get("tags") or []):
+ continue
+ match = _COMMIT_URL_RE.search(str(ref.get("url") or ""))
+ if match:
+ hashes.append(match.group(1))
+ hashes.extend(extra)
+ return list(dict.fromkeys(hashes))
+
+
+def fixed_releases(cna: dict, inputs: Inputs) -> list[Release]:
+ derived: dict[str, Release] = {}
+ for entry in cna.get("affected") or []:
+ product = str(entry.get("product") or "")
+ for rng in entry.get("versions") or []:
+ if rng.get("status", "affected") != "affected" or not rng.get("lessThan"):
+ continue
+ fixed = str(rng["lessThan"]).strip()
+ derived.setdefault(fixed, Release(fixed, product, format_version_range([rng])))
+ if inputs.fixed_in:
+ default_product = next((r.product for r in derived.values()), "") or str(
+ ((cna.get("affected") or [{}])[0]).get("product") or ""
+ )
+ releases = [derived.get(v) or Release(v, default_product, "") for v in inputs.fixed_in]
+ else:
+ releases = list(derived.values())
+ if not releases:
+ raise Refusal(
+ "the record's affected versions give no fixed release (no `lessThan` bound); "
+ "name it with --fixed-in VERSION"
+ )
+ for release in releases:
+ if not VERSION_RE.fullmatch(release.version):
+ raise Refusal(f"fixed version {release.version!r} is not a plain version string")
+ if release.version in inputs.affects:
+ release.affects = inputs.affects[release.version]
+ return releases
+
+
+@dataclass
+class Section:
+ name: str
+ start: int # offset of the section's first character (tag / heading)
+ close: int # offset where content may be appended (closing tag / end of section)
+ end: int # offset just past the section, including its line ending
+ parent: int # index of the enclosing element (xml) or -1
+ raw: str
+
+
+@dataclass
+class PageModel:
+ sections: list[Section]
+ cve_owners: dict[str, list[int]] # cve id -> indexes of the sections holding it
+ parent_close: int | None # where to append a section when none matches
+ parent_has_children: bool
+
+
+def _line_start(text: str, offset: int) -> int:
+ return text.rfind("\n", 0, offset) + 1
+
+
+def _line_end(text: str, offset: int) -> int:
+ nl = text.find("\n", offset)
+ return len(text) if nl < 0 else nl + 1
+
+
+def _indent_at(text: str, offset: int) -> str | None:
+ """The whitespace before ``offset`` on its line, or None if anything else precedes it."""
+ prefix = text[_line_start(text, offset) : offset]
+ return prefix if prefix.strip() == "" else None
+
+
+class _XmlScanner:
+ """Locate elements in an XML page by offset, without re-serialising it.
+
+ expat validates well-formedness (a page it cannot parse is refused) and
+ reports where every element starts and ends, so the change is a textual
+ splice that keeps the page's own formatting, comments and entities.
+ """
+
+ def __init__(self, text: str, section_element: str, name_attr: str, cve_id: str) -> None:
+ self.text = text
+ self.data = text.encode("utf-8")
+ self.section_element = section_element
+ self.name_attr = name_attr
+ self.cve_id = cve_id
+ self.elements: list[dict] = []
+ self.stack: list[int] = []
+ self.chars: list[list[str]] = []
+ self.cve_elements: list[int] = []
+ self._char_offsets: dict[int, int] = {}
+
+ def _char(self, byte_offset: int) -> int:
+ if byte_offset not in self._char_offsets:
+ self._char_offsets[byte_offset] = len(self.data[:byte_offset].decode("utf-8"))
+ return self._char_offsets[byte_offset]
+
+ def scan(self) -> None:
+ parser = xml.parsers.expat.ParserCreate()
+ # Never fetch external entities or the external DTD subset: a page's
+ # `<!ENTITY x SYSTEM "...">` include is kept as-is, not resolved.
+ parser.SetParamEntityParsing(xml.parsers.expat.XML_PARAM_ENTITY_PARSING_NEVER)
+ parser.ExternalEntityRefHandler = lambda *_args: 1
+
+ def start(name: str, attrs: dict[str, str]) -> None:
+ index = len(self.elements)
+ self.elements.append(
+ {
+ "name": name,
+ "attrs": attrs,
+ "start": parser.CurrentByteIndex,
+ "close": None,
+ "parent": self.stack[-1] if self.stack else -1,
+ "children": 0,
+ }
+ )
+ if self.stack:
+ self.elements[self.stack[-1]]["children"] += 1
+ if self.cve_id in attrs.values():
+ self.cve_elements.append(index)
+ self.stack.append(index)
+ self.chars.append([])
+
+ def end(_name: str) -> None:
+ index = self.stack.pop()
+ text = "".join(self.chars.pop())
+ if text.strip() == self.cve_id:
+ self.cve_elements.append(index)
+ offset = parser.CurrentByteIndex
+ element = self.elements[index]
+ # For `<x/>` expat reports the end at the start tag; there is no closing tag.
+ element["close"] = offset if self.data.startswith(b"</", offset) else None
+ element["end_byte"] = offset
+
+ def chardata(data: str) -> None:
+ if self.chars:
+ self.chars[-1].append(data)
+
+ parser.StartElementHandler = start
+ parser.EndElementHandler = end
+ parser.CharacterDataHandler = chardata
+ try:
+ parser.Parse(self.data, True)
+ except xml.parsers.expat.ExpatError as exc:
+ raise Refusal(f"the page is not well-formed XML ({exc}); not editing it") from exc
+
+ def model(self, section_matches: Callable[[str], bool], parent_name: str) -> PageModel:
+ sections: list[Section] = []
+ element_to_section: dict[int, int] = {}
+ for index, element in enumerate(self.elements):
+ if element["name"] != self.section_element:
+ continue
+ name = element["attrs"].get(self.name_attr)
+ if name is None or not section_matches(name):
+ continue
+ if element["close"] is None:
+ raise Refusal(f"section {name!r} is an empty element (`<.../>`); expand it by hand first")
+ start = self._char(element["start"])
+ close = self._char(element["close"])
+ close_end = self.text.index(">", close) + 1
+ element_to_section[index] = len(sections)
+ sections.append(
+ Section(
+ name=name,
+ start=start,
+ close=close,
+ end=_line_end(self.text, close_end),
+ parent=element["parent"],
+ raw=self.text[start:close_end],
+ )
+ )
+ owners: dict[str, list[int]] = {}
+ for index in self.cve_elements:
+ cursor = index
+ while cursor >= 0 and cursor not in element_to_section:
+ cursor = self.elements[cursor]["parent"]
+ owners.setdefault(self.cve_id, []).append(element_to_section.get(cursor, -1))
+ parent_close = None
+ parent_has_children = False
+ if sections:
+ parent = self.elements[sections[0].parent] if sections[0].parent >= 0 else None
+ else:
+ parent = next((e for e in self.elements if e["name"] == parent_name), None)
+ if parent is not None and parent.get("close") is not None:
+ parent_close = self._char(parent["close"])
+ parent_has_children = parent["children"] > 0
+ return PageModel(sections, owners, parent_close, parent_has_children)
+
+
+_MD_HEADING_RE = re.compile(r"^(#{1,6})[ \t]+(.*?)[ \t]*#*[ \t]*$")
+_MD_FENCE_RE = re.compile(r"^[ \t]{0,3}(`{3,}|~{3,})")
+
+
+def _markdown_model(text: str, level: int, cve_id: str, section_matches: Callable[[str], bool]) -> PageModel:
+ lines = text.splitlines(keepends=True)
+ headings: list[tuple[int, int, str]] = [] # (offset, level, title)
+ offset = 0
+ fence: str | None = None
+ for line in lines:
+ stripped = line.rstrip("\r\n")
+ fence_match = _MD_FENCE_RE.match(stripped)
+ if fence_match:
+ marker = fence_match.group(1)
+ if fence is None:
+ fence = marker[0] * 3
+ elif marker.startswith(fence):
+ fence = None
+ elif fence is None:
+ heading = _MD_HEADING_RE.match(stripped)
+ if heading:
+ headings.append((offset, len(heading.group(1)), heading.group(2)))
+ offset += len(line)
+ if fence is not None:
+ raise Refusal("the page has an unterminated code fence; not editing it")
+ sections: list[Section] = []
+ for i, (start, heading_level, title) in enumerate(headings):
+ if heading_level != level or not section_matches(title):
+ continue
+ end = next((h[0] for h in headings[i + 1 :] if h[1] <= level), len(text))
+ body = text[start:end]
+ trimmed = start + len(body.rstrip())
+ sections.append(Section(title, start, trimmed, end, -1, body))
+ owners: dict[str, list[int]] = {}
+ for match in re.finditer(re.escape(cve_id) + r"(?!\d)", text):
+ owner = next((i for i, s in enumerate(sections) if s.start <= match.start() < s.end), -1)
+ owners.setdefault(cve_id, []).append(owner)
+ return PageModel(sections, owners, len(text.rstrip()), bool(text.strip()))
+
+
+def _section_matcher(settings: Settings, release: Release, line: str) -> tuple[re.Pattern[str], str]:
+ """Regex matching every release section of this page, and this release's section name."""
+ values = {"version": release.version, "line": line, "product": release.product}
+ target = render_template(
+ settings.section_name, values, allowed=SECTION_NAME_PLACEHOLDERS, what="section name template"
+ )
+ pattern = ""
+ for piece in re.split(r"(\{[a-z_]+\})", settings.section_name):
+ if piece == "{version}":
+ pattern += r"(?P<version>[0-9A-Za-z][0-9A-Za-z.+_-]*)"
+ elif piece in ("{line}", "{product}"):
+ pattern += re.escape(values[piece[1:-1]])
+ else:
+ pattern += re.escape(piece)
+ return re.compile(pattern), target
+
+
+def _indent_block(block: str, indent: str) -> str:
+ return "\n".join(indent + line if line.strip() else "" for line in block.split("\n"))
+
+
+def _render_section(section_template: str, values: dict[str, str | None], entry: str) -> str:
+ """Render the section template; its ``{entries}`` line becomes the indented entry."""
+ lines = section_template.strip("\n").split("\n")
+ if not any(line.strip() == "{entries}" for line in lines):
+ raise UsageError("section template: needs a line holding only {entries}")
+ out: list[str] = []
+ for line in lines:
+ if line.strip() == "{entries}":
+ out.append(_indent_block(entry, line[: len(line) - len(line.lstrip())]))
+ else:
+ out.append(
+ render_template(
+ line, values, allowed=SECTION_PLACEHOLDERS - {"entries"}, what="section template"
+ )
+ )
+ return "\n".join(out)
+
+
+def insert_into_page(
+ text: str,
+ *,
+ settings: Settings,
+ release: Release,
+ line: str,
+ cve_id: str,
+ entry: str,
+ release_date: str | None,
+) -> tuple[str, str]:
+ """Insert ``entry`` (already rendered and escaped) into the page.
+
+ Returns ``(new_text, summary)``; ``new_text == text`` when the CVE is
+ already recorded in this release's section (idempotent re-run).
+ """
+ if "\r\n" in text:
+ # Work on LF line endings and restore the page's CRLF afterwards.
+ new_text, summary = insert_into_page(
+ text.replace("\r\n", "\n"),
+ settings=settings,
+ release=release,
+ line=line,
+ cve_id=cve_id,
+ entry=entry,
+ release_date=release_date,
+ )
+ return (text if new_text == text.replace("\r\n", "\n") else new_text.replace("\n", "\r\n")), summary
+
+ pattern, target_name = _section_matcher(settings, release, line)
+
+ def matches(name: str) -> bool:
+ return pattern.fullmatch(name) is not None
+
+ if settings.page_format == "xml":
+ first_tag = re.match(r"\s*<([A-Za-z_][\w.:-]*)\b", settings.section_template)
+ name_attr = re.search(r"([A-Za-z_][\w.:-]*)\s*=\s*\"\{section_name\}\"", settings.section_template)
+ if not first_tag or not name_attr:
+ raise UsageError(
+ 'section template: must start with an element whose attribute is "{section_name}", '
+ 'e.g. <section name="{section_name}" ...>'
+ )
+ scanner = _XmlScanner(text, first_tag.group(1), name_attr.group(1), cve_id)
+ scanner.scan()
+ model = scanner.model(matches, settings.section_parent)
+ else:
+ heading = re.match(r"\s*(#{1,6})[ \t]+\{section_name\}", settings.section_template)
+ if not heading:
+ raise UsageError("section template: must start with a heading line `## {section_name}`")
+ model = _markdown_model(text, len(heading.group(1)), cve_id, matches)
+
+ target_index = next((i for i, s in enumerate(model.sections) if s.name == target_name), None)
+ owners = model.cve_owners.get(cve_id, [])
+ if owners:
+ if target_index is not None and target_index in owners:
+ return text, f"{cve_id} is already recorded under {target_name!r}"
+ elsewhere = sorted(
+ {model.sections[i].name if i >= 0 else "(outside any release section)" for i in owners}
+ )
+ raise Refusal(
+ f"{cve_id} already appears in this page under {', '.join(repr(n) for n in elsewhere)}, "
+ f"not under {target_name!r}; not adding a second entry"
+ )
+
+ if target_index is not None:
+ section = model.sections[target_index]
+ if settings.page_format == "xml":
+ close_indent = _indent_at(text, section.close)
+ section_indent = _indent_at(text, section.start) or ""
+ child_indent = _first_child_indent(section.raw) or section_indent + " "
+ # After the section's last content, one blank line apart; the
+ # whitespace before the closing tag stays as the page had it.
+ insert_at = len(text[: section.close].rstrip())
+ insertion = "\n\n" + _indent_block(entry, child_indent)
+ if close_indent is None:
+ insertion += "\n" + section_indent
+ else:
+ insert_at, insertion = section.close, "\n\n" + entry
+ new_text = text[:insert_at] + insertion + text[insert_at:]
+ return new_text, f"add the {cve_id} entry to the existing section {target_name!r}"
+
+ if not release_date:
+ raise Refusal(
+ f"the page has no section {target_name!r} yet, and creating it needs the release date: "
+ f"{_INPUT_FLAGS['release_date']}"
+ )
+ values: dict[str, str | None] = {
+ "section_name": ESCAPERS[settings.page_format](target_name),
+ "release_date": release_date,
+ "version": release.version,
+ "line": line,
+ "product": ESCAPERS[settings.page_format](release.product),
+ }
+ section_block = _render_section(settings.section_template, values, entry)
+ older = next((s for s in model.sections if _section_version_lt(pattern, s.name, release.version)), None)
+ if older is not None:
+ indent = _indent_at(text, older.start) or ""
+ insert_at = (
+ _line_start(text, older.start) if _indent_at(text, older.start) is not None else older.start
+ )
+ insertion = (
+ _indent_block(section_block, indent) + "\n\n" + (indent if insert_at == older.start else "")
+ )
+ where = f"before {older.name!r}"
+ elif model.sections:
+ last = model.sections[-1]
+ if settings.page_format == "xml":
+ indent = _indent_at(text, last.start) or ""
+ insert_at = last.end
+ insertion = "\n" + _indent_block(section_block, indent) + "\n"
+ else:
+ insert_at = last.close
+ insertion = "\n\n" + section_block
+ where = f"after {last.name!r}"
+ elif model.parent_close is not None:
+ if settings.page_format == "xml":
+ parent_indent = _indent_at(text, model.parent_close)
+ indent = (parent_indent or "") + " "
+ insert_at = (
+ _line_start(text, model.parent_close) if parent_indent is not None else model.parent_close
+ )
+ lead = "\n" if model.parent_has_children else ""
+ insertion = lead + _indent_block(section_block, indent) + "\n"
+ where = f"as the first release section inside <{settings.section_parent}>"
+ else:
+ insert_at = model.parent_close
+ insertion = ("\n\n" if model.parent_has_children else "") + section_block + "\n"
+ where = "as the first release section, at the end of the page"
+ else:
+ raise Refusal(
+ f"the page has no release section and no <{settings.section_parent}> element to add one to; "
+ f"add the first section by hand"
+ )
+ new_text = text[:insert_at] + insertion + text[insert_at:]
+ return new_text, f"create section {target_name!r} ({release_date}) {where}, holding the {cve_id} entry"
+
+
+def _first_child_indent(raw_section: str) -> str | None:
+ match = re.search(r"\n([ \t]*)<", raw_section)
+ if match and not raw_section[match.end() - 1 :].startswith("</"):
+ return match.group(1)
+ return None
+
+
+def _section_version_lt(pattern: re.Pattern[str], name: str, version: str) -> bool:
+ match = pattern.fullmatch(name)
+ if not match or "version" not in match.groupdict():
+ return False
+ return _version_lt(match.group("version"), version)
+
+
+def render_entry(settings: Settings, inputs: Inputs, release: Release, line: str, cve_id: str) -> str:
+ cna = _cna(inputs.record)
+ esc = ESCAPERS[settings.page_format]
+ commits = fix_commits(cna, inputs.fix_commits)
+ rendered_commits = [
+ render_template(
+ settings.fix_commit_template, {"hash": h}, allowed=frozenset({"hash"}), what="fix-commit template"
+ )
+ for h in commits
+ ]
+ if len(rendered_commits) > 1:
+ joined = ", ".join(rendered_commits[:-1]) + " and " + rendered_commits[-1]
+ else:
+ joined = "".join(rendered_commits)
+
+ def opt(value: str | None) -> str | None:
+ return esc(value) if value else None
+
+ values: dict[str, str | None] = {
+ "affects": opt(release.affects),
+ "cve_id": cve_id,
+ "description": opt(_description(cna)),
+ "fix_commit_noun": ("commits" if len(commits) > 1 else "commit") if commits else None,
+ "fix_commits": joined or None,
+ "impact": opt(_impact(cna)),
+ "line": line,
+ "product": opt(release.product),
+ "public_date": inputs.public_date,
+ "release_date": inputs.release_dates.get(release.version),
+ "reported_date": inputs.reported_date,
+ "title": opt(str(cna.get("title") or "")),
+ "version": release.version,
+ }
+ entry = render_template(
+ settings.entry_template, values, allowed=ENTRY_PLACEHOLDERS, what="entry template"
+ )
+ entry = entry.strip("\n")
+ if settings.page_format == "xml":
+ _check_entry_xml(entry, cve_id)
+ return entry
+
+
+def _check_entry_xml(entry: str, cve_id: str) -> None:
+ """The rendered entry must be well-formed and carry the CVE ID as a whole
+ element text or attribute value — that is how a re-run recognises it."""
+ scanner = _XmlScanner(f"<entry>{entry}</entry>", "\0", "\0", cve_id)
+ try:
+ scanner.scan()
+ except Refusal as exc:
+ raise UsageError(f"entry template renders to malformed XML: {exc}") from exc
+ if not [i for i in scanner.cve_elements if i != 0]:
+ raise UsageError(
+ "entry template: put {cve_id} as the whole text of an element (e.g. <cve>{cve_id}</cve>) "
+ "or as a whole attribute value, so a re-run can recognise the entry"
+ )
+
+
+def plan_release_page(settings: Settings, inputs: Inputs) -> list[FileChange]:
+ cve_id = record_cve_id(inputs.record)
+ cna = _cna(inputs.record)
+ releases = fixed_releases(cna, inputs)
+ pages: dict[str, tuple[str | None, str, list[str]]] = {}
+ for release in releases:
+ line = release_line(release.version, settings.release_line_parts)
+ rel = _safe_relative(
+ render_template(settings.path, {"line": line}, allowed=PATH_PLACEHOLDERS_PAGE, what="page path"),
+ settings.working_copy,
+ )
+ if rel not in pages:
+ target = settings.working_copy / rel
+ if not target.is_file():
+ raise Refusal(
+ f"page {rel} does not exist in the working copy; create it (with its first "
+ f"release section) by hand, or check the page path template"
+ )
+ try:
+ original = target.read_bytes().decode("utf-8")
+ except UnicodeDecodeError as exc:
+ raise Refusal(f"page {rel} is not UTF-8; not editing it") from exc
+ pages[rel] = (original, original, [])
+ old, current, summaries = pages[rel]
+ entry = render_entry(settings, inputs, release, line, cve_id)
+ new, summary = insert_into_page(
+ current,
+ settings=settings,
+ release=release,
+ line=line,
+ cve_id=cve_id,
+ entry=entry,
+ release_date=inputs.release_dates.get(release.version),
+ )
+ pages[rel] = (old, new, [*summaries, summary])
+ changes = []
+ for rel, (old, new, summaries) in pages.items():
+ action = "unchanged" if new == old else "update"
+ changes.append(FileChange(rel, action, "; ".join(summaries), old, new))
+ return changes
+
+
+# -----------------------------------------------------------------------------
+# VCS commands
+# -----------------------------------------------------------------------------
+
+
+def detect_vcs(working_copy: Path) -> str | None:
+ for directory in (working_copy, *working_copy.parents):
+ if (directory / ".svn").exists():
+ return "svn"
+ if (directory / ".git").exists():
+ return "git"
+ return None
+
+
+def _cmd(purpose: str, argv: list[str]) -> dict:
+ return {"purpose": purpose, "argv": argv, "command": shlex.join(argv)}
+
+
+def vcs_commands(settings: Settings, changed: list[str], cve_id: str, apply_argv: list[str]) -> list[dict]:
+ def vcs(*args: str) -> list[str]:
+ return [
+ "uv",
+ "run",
+ "--project",
+ str(settings.vcs_project),
+ "magpie-vcs",
+ "-C",
+ str(settings.working_copy),
+ *args,
+ ]
+
+ message = render_template(
+ settings.commit_message, {"cve_id": cve_id}, allowed=frozenset({"cve_id"}), what="commit message"
+ )
+ if "\n" in message:
+ raise UsageError("commit message: must be one line")
+ commands = [_cmd("check the working copy has no unrelated changes (must exit 0)", vcs("clean"))]
+ if settings.vcs == "git":
+ commands.append(_cmd(f"check the branch (must print {settings.branch})", vcs("branch")))
+ commands += [
+ _cmd("bring the working copy up to date", vcs("update")),
+ _cmd("write the change shown above (re-computed on the updated copy)", apply_argv),
+ _cmd("add the changed file(s)", vcs("stage", *changed)),
+ ]
+ if settings.vcs == "git":
+ commands += [
+ _cmd("show the staged change", vcs("diff", "--cached", *changed)),
+ _cmd("commit", vcs("commit", "-m", message)),
+ _cmd(
+ f"publish to {settings.remote}/{settings.branch}",
+ vcs("push", settings.remote, settings.branch),
+ ),
+ ]
+ else:
+ commands += [
+ _cmd("show the change", vcs("diff", *changed)),
+ _cmd("commit — on Subversion this publishes the change", vcs("commit", "-m", message)),
+ ]
+ return commands
+
+
+# -----------------------------------------------------------------------------
+# CLI
+# -----------------------------------------------------------------------------
+
+
+def _key_value(items: Sequence[str], flag: str, check: Callable[[str], bool], what: str) -> dict[str, str]:
+ out: dict[str, str] = {}
+ for item in items:
+ key, sep, value = item.partition("=")
+ key, value = key.strip(), value.strip()
+ if not sep or not VERSION_RE.fullmatch(key) or not check(value):
+ raise UsageError(f"{flag} expects VERSION={what}, got {item!r}")
+ out[key] = value
+ return out
+
+
+def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace:
+ parser = argparse.ArgumentParser(
+ prog="advisory-records",
+ description=(
+ "Propose recording a published CVE in the project's own security records "
+ "(a site repository, over Git or Subversion). Prints the file change and the "
+ "magpie-vcs commands; runs no VCS command. --apply writes the file change only."
+ ),
+ )
+ g = parser.add_argument_group("configuration (the advisory_records block of project.md)")
+ g.add_argument("--layout", required=True, choices=LAYOUTS)
+ g.add_argument(
+ "--vcs", default="auto", choices=VCS_CHOICES, help="default: detected from the working copy"
+ )
+ g.add_argument(
+ "--working-copy", required=True, type=Path, help="local working copy of the site repository"
+ )
+ g.add_argument("--repository", help="repository URL, used only to propose the first checkout")
+ g.add_argument("--branch", default="main", help="Git only: the branch to publish to (default: main)")
+ g.add_argument("--remote", default="origin", help="Git only: the remote to push to (default: origin)")
+ g.add_argument("--commit-message", default=DEFAULT_COMMIT_MESSAGE, help="one line; placeholder {cve_id}")
+ g.add_argument(
+ "--path",
+ required=True,
+ help="file path in the working copy: {cve_id}/{year} for cve-json-file, {line} for release-page",
+ )
+ g.add_argument("--json-indent", type=int, default=4)
+ g.add_argument("--keep-cna-private", action="store_true", help="keep the CNA tool's CNA_private block")
+ g.add_argument("--format", dest="page_format", default="xml", choices=PAGE_FORMATS)
+ g.add_argument("--release-line-parts", type=int, default=1, help="version components naming a line")
+ g.add_argument("--section-name", default="Fixed in {product} {version}")
+ g.add_argument("--section-template", help="file; default: the shipped example for --format")
+ g.add_argument("--entry-template", help="file; default: the shipped example for --format")
+ g.add_argument(
+ "--fix-commit-template", help='default: <hashlink hash="{hash}"/> (xml), `{hash}` (markdown)'
+ )
+ g.add_argument("--section-parent", default="body", help="xml: element a first release section goes into")
+ g.add_argument("--vcs-project", type=Path, default=_DEFAULT_VCS_PROJECT, help="path of tools/vcs")
+
+ i = parser.add_argument_group("inputs")
+ i.add_argument(
+ "--record", required=True, type=Path, help="CVE 5.x JSON record (generate-cve-json --output)"
+ )
+ i.add_argument("--cve-org-record", type=Path, help="the cve-check-published JSON output, saved to a file")
+ i.add_argument("--cve-org-state", help="cve.org cveMetadata.state, when --cve-org-record is not given")
+ i.add_argument("--date-published", help="cve.org cveMetadata.datePublished")
+ i.add_argument("--release-date", action="append", default=[], metavar="VERSION=YYYY-MM-DD")
+ i.add_argument("--reported-date", help="YYYY-MM-DD the report reached the security team")
+ i.add_argument("--public-date", help="YYYY-MM-DD; default: the date part of --date-published")
+ i.add_argument("--fix-commit", action="append", default=[], metavar="HASH")
+ i.add_argument("--fixed-in", action="append", default=[], metavar="VERSION")
+ i.add_argument("--affects", action="append", default=[], metavar="VERSION=TEXT")
+
+ o = parser.add_argument_group("output")
+ o.add_argument("--apply", action="store_true", help="write the file change (never runs a VCS command)")
+ o.add_argument("--json", action="store_true", help="print the proposal as JSON")
+ return parser.parse_args(argv)
+
+
+def _load_json(path: Path, what: str) -> dict:
+ try:
+ data = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise UsageError(f"cannot read {what} {path}: {exc}") from exc
+ if not isinstance(data, dict):
+ raise UsageError(f"{what} {path} is not a JSON object")
+ return data
+
+
+def build(args: argparse.Namespace) -> tuple[Settings, Inputs]:
+ record = _load_json(args.record, "CVE record")
+ cve_id = record_cve_id(record)
+ state, published = args.cve_org_state, args.date_published
+ if args.cve_org_record is not None:
+ meta = _load_json(args.cve_org_record, "cve.org record").get("cveMetadata") or {}
+ if meta.get("cveId") != cve_id:
+ raise Refusal(f"the cve.org record is for {meta.get('cveId')!r}, not {cve_id}")
+ state, published = meta.get("state"), published or meta.get("datePublished")
+ public_date = args.public_date or (str(published)[:10] if published else None)
+ for flag, value in (("--reported-date", args.reported_date), ("--public-date", public_date)):
+ if value and not DATE_RE.fullmatch(value):
+ raise UsageError(f"{flag} expects YYYY-MM-DD, got {value!r}")
+ for commit in args.fix_commit:
+ if not HASH_RE.fullmatch(commit):
+ raise UsageError(f"--fix-commit expects a lowercase hex commit hash, got {commit!r}")
+ for version in args.fixed_in:
+ if not VERSION_RE.fullmatch(version):
+ raise UsageError(f"--fixed-in expects a version, got {version!r}")
+ inputs = Inputs(
+ record=record,
+ cve_org_state=state,
+ date_published=published,
+ release_dates=_key_value(
+ args.release_date, "--release-date", lambda v: bool(DATE_RE.fullmatch(v)), "YYYY-MM-DD"
+ ),
+ reported_date=args.reported_date,
+ public_date=public_date,
+ fix_commits=list(args.fix_commit),
+ fixed_in=list(args.fixed_in),
+ affects=_key_value(args.affects, "--affects", bool, "TEXT"),
+ )
+ working_copy = args.working_copy.expanduser().resolve()
+ vcs = args.vcs
+ if vcs == "auto":
+ vcs = detect_vcs(working_copy) if working_copy.exists() else None
+ if vcs is None:
+ raise UsageError(f"cannot tell the VCS of {working_copy}; pass --vcs git|svn")
+ if args.release_line_parts < 1:
+ raise UsageError("--release-line-parts must be at least 1")
+ fmt = args.page_format
+ settings = Settings(
+ layout=args.layout,
+ vcs=vcs,
+ working_copy=working_copy,
+ repository=args.repository,
+ branch=args.branch,
+ remote=args.remote,
+ commit_message=args.commit_message,
+ path=args.path,
+ json_indent=args.json_indent,
+ keep_cna_private=args.keep_cna_private,
+ page_format=fmt,
+ release_line_parts=args.release_line_parts,
+ section_name=args.section_name,
+ section_template=_load_template(args.section_template, f"release-page-section.{_ext(fmt)}.tmpl"),
+ entry_template=_load_template(args.entry_template, f"release-page-entry.{_ext(fmt)}.tmpl"),
+ fix_commit_template=args.fix_commit_template or DEFAULT_FIX_COMMIT_TEMPLATES[fmt],
+ section_parent=args.section_parent,
+ vcs_project=args.vcs_project.expanduser().resolve(),
+ )
+ return settings, inputs
+
+
+def _ext(page_format: str) -> str:
+ return "xml" if page_format == "xml" else "md"
+
+
+def plan(settings: Settings, inputs: Inputs) -> list[FileChange]:
+ check_publication_gate(inputs.record, inputs.cve_org_state)
+ if settings.layout == "cve-json-file":
+ return plan_cve_json_file(settings, inputs)
+ return plan_release_page(settings, inputs)
+
+
+def _apply_argv(argv: Sequence[str]) -> list[str]:
+ rest = [a for a in argv if a not in ("--json", "--apply")]
+ return ["uv", "run", "--project", str(_TOOL_PROJECT), "advisory-records", *rest, "--apply"]
+
+
+def run(argv: Sequence[str]) -> tuple[int, dict]:
+ args = parse_args(argv)
+ settings, inputs = build(args)
+ cve_id = record_cve_id(inputs.record)
+ result: dict = {
+ "cve_id": cve_id,
+ "layout": settings.layout,
+ "vcs": settings.vcs,
+ "working_copy": str(settings.working_copy),
+ "changes": [],
+ "commands": [],
+ }
+ if not settings.working_copy.exists():
+ check_publication_gate(inputs.record, inputs.cve_org_state)
+ if not settings.repository:
+ raise UsageError(
+ f"working copy {settings.working_copy} does not exist and no --repository is set"
+ )
+ argv_checkout = [
+ "uv", "run", "--project", str(settings.vcs_project), "magpie-vcs",
+ "--backend", settings.vcs, "-C", str(settings.working_copy.parent),
+ "checkout", settings.repository, settings.working_copy.name,
+ ] # fmt: skip
+ result["status"] = "needs-checkout"
+ result["commands"] = [_cmd("create the working copy, then re-run this command", argv_checkout)]
+ return EXIT_NEEDS_CHECKOUT, result
+
+ changes = plan(settings, inputs)
+ result["changes"] = [
+ {"path": c.path, "action": c.action, "summary": c.summary, "diff": c.diff()} for c in changes
+ ]
+ changed = [c.path for c in changes if c.action != "unchanged"]
+ if not changed:
+ result["status"] = "unchanged"
+ return EXIT_OK, result
+ if args.apply:
+ for change in changes:
+ if change.action == "unchanged":
+ continue
+ target = settings.working_copy / change.path
+ target.parent.mkdir(parents=True, exist_ok=True)
+ target.write_bytes(change.new.encode("utf-8"))
+ result["status"] = "applied"
+ return EXIT_OK, result
+ result["status"] = "proposed"
+ result["commands"] = vcs_commands(settings, changed, cve_id, _apply_argv(argv))
+ return EXIT_OK, result
+
+
+def _print_text(result: dict) -> None:
+ status = result.get("status")
+ print(f"advisory-records: {result['cve_id']} ({result['layout']}, {result['vcs']}) — {status}")
+ print(f"working copy: {result['working_copy']}")
+ for change in result["changes"]:
+ print(f"- {change['path']}: {change['action']} — {change['summary']}")
+ if change["diff"]:
+ sys.stdout.write(change["diff"])
+ if result["commands"]:
+ print("Commands, in order, after the user confirms:")
+ for n, cmd in enumerate(result["commands"], 1):
+ print(f" {n}. {cmd['purpose']}\n {cmd['command']}")
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+ argv = list(sys.argv[1:] if argv is None else argv)
+ try:
+ code, result = run(argv)
+ except UsageError as exc:
+ print(f"advisory-records: error: {exc}", file=sys.stderr)
+ return EXIT_USAGE
+ except Refusal as exc:
+ print(f"advisory-records: refusing: {exc}", file=sys.stderr)
+ return EXIT_REFUSED
+ if "--json" in argv:
+ print(json.dumps(result, indent=2, ensure_ascii=False))
+ else:
+ _print_text(result)
+ return code
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/advisory_templates/release-page-entry.md.tmpl b/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/advisory_templates/release-page-entry.md.tmpl
new file mode 100644
index 00000000..47877d02
--- /dev/null
+++ b/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/advisory_templates/release-page-entry.md.tmpl
@@ -0,0 +1,10 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+### {impact}: {title} ({cve_id})
+
+{description}
+
+Fixed with {fix_commit_noun} {fix_commits}.
+Reported to the {product} security team on {reported_date}; made public on {public_date}.
+
+Affects: {affects}
diff --git a/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/advisory_templates/release-page-entry.xml.tmpl b/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/advisory_templates/release-page-entry.xml.tmpl
new file mode 100644
index 00000000..44181f11
--- /dev/null
+++ b/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/advisory_templates/release-page-entry.xml.tmpl
@@ -0,0 +1,13 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+<p><strong>{impact}: {title}</strong>
+<cve>{cve_id}</cve></p>
+
+<p>{description}</p>
+
+<p>This was fixed with {fix_commit_noun} {fix_commits}.</p>
+
+<p>This issue was reported to the {product} security team on {reported_date}.
+The issue was made public on {public_date}.</p>
+
+<p>Affects: {affects}</p>
diff --git a/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/advisory_templates/release-page-section.md.tmpl b/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/advisory_templates/release-page-section.md.tmpl
new file mode 100644
index 00000000..174a06f4
--- /dev/null
+++ b/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/advisory_templates/release-page-section.md.tmpl
@@ -0,0 +1,7 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+## {section_name}
+
+Released {release_date}.
+
+{entries}
diff --git a/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/advisory_templates/release-page-section.xml.tmpl b/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/advisory_templates/release-page-section.xml.tmpl
new file mode 100644
index 00000000..9bde78f2
--- /dev/null
+++ b/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/advisory_templates/release-page-section.xml.tmpl
@@ -0,0 +1,7 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+<section name="{section_name}" rtext="{release_date}">
+
+ {entries}
+
+</section>
diff --git a/tools/cve-tool-vulnogram/generate-cve-json/tests/test_advisory_records.py b/tools/cve-tool-vulnogram/generate-cve-json/tests/test_advisory_records.py
new file mode 100644
index 00000000..505b2ae1
--- /dev/null
+++ b/tools/cve-tool-vulnogram/generate-cve-json/tests/test_advisory_records.py
@@ -0,0 +1,464 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Tests for ``advisory-records`` — recording a published CVE in the project's
+own security records. Every CVE ID, product and commit here is made up."""
+
+from __future__ import annotations
+
+import json
+import xml.parsers.expat
+from pathlib import Path
+
+import pytest
+
+from generate_cve_json import advisory_records as ar
+
+CVE = "CVE-2026-00001"
+COMMIT = "0123456789abcdef0123456789abcdef01234567"
+
+
+def make_record(
+ *,
+ cve_id: str = CVE,
+ title: str = "Path traversal in the static file handler",
+ description: str = "A crafted request could read files outside the web root.",
+ ranges: list[tuple[str, str]] | None = None,
+ advisory: bool = True,
+) -> dict:
+ references = [{"tags": ["patch"], "url": f"https://github.com/example/server/commit/{COMMIT}"}]
+ if advisory:
+ references.append({"tags": ["vendor-advisory"], "url": "https://lists.example.org/thread/abc"})
+ versions = [
+ {"lessThan": fixed, "status": "affected", "version": start, "versionType": "semver"}
+ for start, fixed in (ranges or [("9.0.0", "9.0.99")])
+ ]
+ return {
+ "CNA_private": {"owner": "example", "state": "PUBLIC"},
+ "containers": {
+ "cna": {
+ "affected": [
+ {
+ "defaultStatus": "unaffected",
+ "product": "Example Server",
+ "vendor": "Example",
+ "versions": versions,
+ }
+ ],
+ "descriptions": [{"lang": "en", "value": description}],
+ "metrics": [
+ {"other": {"content": {"text": "important"}, "type": "Textual description of severity"}}
+ ],
+ "references": references,
+ "title": title,
+ }
+ },
+ "cveMetadata": {"cveId": cve_id, "state": "PUBLISHED"},
+ "dataType": "CVE_RECORD",
+ "dataVersion": "5.1",
+ }
+
+
+PAGE = """<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE document [
+ <!ENTITY project SYSTEM "project.xml">
+]>
+<document url="security-9.html">
+ &project;
+ <properties>
+ <title>Example Server 9.x vulnerabilities</title>
+ </properties>
+<body>
+ <section name="Example Server 9.x vulnerabilities">
+ <p>Introductory text.</p>
+ </section>
+
+ <section name="Fixed in Example Server 9.0.98" rtext="2026-03-01">
+
+ <p><strong>Low: An older issue</strong>
+ <cve>CVE-2026-00002</cve></p>
+
+ </section>
+
+ <section name="Fixed in Example Server 9.0.96" rtext="2026-01-10">
+
+ <p><strong>Moderate: Another older issue</strong>
+ <cve>CVE-2026-00003</cve></p>
+
+ </section>
+</body>
+</document>
+"""
+
+
+def write(tmp_path: Path, name: str, data: object) -> Path:
+ path = tmp_path / name
+ path.write_text(data if isinstance(data, str) else json.dumps(data), encoding="utf-8")
+ return path
+
+
+@pytest.fixture
+def site(tmp_path: Path) -> Path:
+ wc = tmp_path / "site"
+ (wc / ".git").mkdir(parents=True)
+ (wc / "xdocs").mkdir()
+ (wc / "xdocs" / "security-9.xml").write_text(PAGE, encoding="utf-8")
+ return wc
+
+
+def page_args(wc: Path, record: Path, *extra: str) -> list[str]:
+ return [
+ "--layout", "release-page",
+ "--working-copy", str(wc),
+ "--path", "xdocs/security-{line}.xml",
+ "--record", str(record),
+ "--cve-org-state", "PUBLISHED",
+ "--date-published", "2026-04-02T10:00:00.000Z",
+ "--reported-date", "2026-02-14",
+ "--release-date", "9.0.99=2026-03-30",
+ *extra,
+ ] # fmt: skip
+
+
+def run_json(argv: list[str], capsys: pytest.CaptureFixture[str]) -> tuple[int, dict]:
+ code = ar.main([*argv, "--json"])
+ out = capsys.readouterr().out
+ return code, (json.loads(out) if out.strip() else {})
+
+
+def parses(text: str) -> None:
+ parser = xml.parsers.expat.ParserCreate()
+ parser.SetParamEntityParsing(xml.parsers.expat.XML_PARAM_ENTITY_PARSING_NEVER)
+ parser.ExternalEntityRefHandler = lambda *_args: 1
+ parser.Parse(text.encode("utf-8"), True)
+
+
+# -- publication gate ----------------------------------------------------------
+
+
+def test_refuses_before_cve_org_publishes(
+ site: Path, tmp_path: Path, capsys: pytest.CaptureFixture[str]
+) -> None:
+ record = write(tmp_path, "r.json", make_record())
+ argv = [a if a != "PUBLISHED" else "RESERVED" for a in page_args(site, record)]
+ assert ar.main(argv) == ar.EXIT_REFUSED
+ assert "not public yet" in capsys.readouterr().err
+ assert (site / "xdocs" / "security-9.xml").read_text() == PAGE
+
+
+def test_refuses_without_public_advisory(
+ site: Path, tmp_path: Path, capsys: pytest.CaptureFixture[str]
+) -> None:
+ record = write(tmp_path, "r.json", make_record(advisory=False))
+ assert ar.main(page_args(site, record)) == ar.EXIT_REFUSED
+ assert "vendor-advisory" in capsys.readouterr().err
+
+
+def test_refuses_apply_before_publication(site: Path, tmp_path: Path) -> None:
+ record = write(tmp_path, "r.json", make_record())
+ argv = [a if a != "PUBLISHED" else "RESERVED" for a in page_args(site, record, "--apply")]
+ assert ar.main(argv) == ar.EXIT_REFUSED
+ assert (site / "xdocs" / "security-9.xml").read_text() == PAGE
+
+
+def test_cve_org_record_file_is_the_gate(
+ site: Path, tmp_path: Path, capsys: pytest.CaptureFixture[str]
+) -> None:
+ record = write(tmp_path, "r.json", make_record())
+ other = write(tmp_path, "org.json", {"cveMetadata": {"cveId": "CVE-2026-00009", "state": "PUBLISHED"}})
+ argv = [*page_args(site, record), "--cve-org-record", str(other)]
+ assert ar.main(argv) == ar.EXIT_REFUSED
+ assert "not CVE-2026-00001" in capsys.readouterr().err
+ good = write(
+ tmp_path, "org2.json", {"cveMetadata": {"cveId": CVE, "state": "RESERVED", "datePublished": None}}
+ )
+ argv = [a for a in page_args(site, record) if a not in ("--cve-org-state", "PUBLISHED")]
+ assert ar.main([*argv, "--cve-org-record", str(good)]) == ar.EXIT_REFUSED
+
+
+# -- Layout A: one CVE JSON file per CVE ---------------------------------------
+
+
+def json_args(wc: Path, record: Path, *extra: str) -> list[str]:
+ return [
+ "--layout", "cve-json-file",
+ "--working-copy", str(wc),
+ "--path", "content/security/json/{cve_id}.json",
+ "--record", str(record),
+ "--cve-org-state", "PUBLISHED",
+ *extra,
+ ] # fmt: skip
+
+
+def test_cve_json_file_git_proposal_apply_and_rerun(
+ site: Path, tmp_path: Path, capsys: pytest.CaptureFixture[str]
+) -> None:
+ record = write(tmp_path, "r.json", make_record())
+ code, result = run_json(json_args(site, record), capsys)
+ assert code == 0
+ assert result["status"] == "proposed"
+ assert result["vcs"] == "git"
+ assert [c["action"] for c in result["changes"]] == ["create"]
+ rel = f"content/security/json/{CVE}.json"
+ assert result["changes"][0]["path"] == rel
+ ops = [
+ c["argv"][c["argv"].index("-C") + 2 :] if "-C" in c["argv"] else c["argv"] for c in result["commands"]
+ ]
+ assert ops[0] == ["clean"]
+ assert ops[1] == ["branch"]
+ assert ops[2] == ["update"]
+ assert ops[3][-1] == "--apply" and "advisory-records" in ops[3] and "--json" not in ops[3]
+ assert ops[4] == ["stage", rel]
+ assert ops[5] == ["diff", "--cached", rel]
+ assert ops[6] == ["commit", "-m", f"Add {CVE} to the security pages"]
+ assert ops[7] == ["push", "origin", "main"]
+
+ assert ar.main([*json_args(site, record), "--apply"]) == 0
+ capsys.readouterr()
+ written = json.loads((site / rel).read_text())
+ assert "CNA_private" not in written
+ assert written["cveMetadata"]["cveId"] == CVE
+
+ code, result = run_json(json_args(site, record), capsys)
+ assert code == 0
+ assert result["status"] == "unchanged"
+ assert result["commands"] == []
+
+
+def test_cve_json_file_svn_commands(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ wc = tmp_path / "site-svn"
+ (wc / ".svn").mkdir(parents=True)
+ record = write(tmp_path, "r.json", make_record())
+ code, result = run_json(json_args(wc, record, "--commit-message", "Publish {cve_id}"), capsys)
+ assert code == 0
+ assert result["vcs"] == "svn"
+ commands = [c["command"] for c in result["commands"]]
+ assert not any(" push " in c or c.endswith(" branch") for c in commands)
+ assert any(c.endswith(f"diff content/security/json/{CVE}.json") for c in commands)
+ assert commands[-1].endswith(f"commit -m 'Publish {CVE}'")
+ assert "publishes" in result["commands"][-1]["purpose"]
+
+
+def test_cve_json_file_keep_cna_private_and_update(
+ site: Path, tmp_path: Path, capsys: pytest.CaptureFixture[str]
+) -> None:
+ rel = site / "content" / "security" / "json" / f"{CVE}.json"
+ rel.parent.mkdir(parents=True)
+ rel.write_text('{"stale": true}\n')
+ record = write(tmp_path, "r.json", make_record())
+ _code, result = run_json(json_args(site, record, "--keep-cna-private"), capsys)
+ assert result["changes"][0]["action"] == "update"
+ assert '"CNA_private"' in result["changes"][0]["diff"]
+
+
+def test_needs_checkout(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ record = write(tmp_path, "r.json", make_record())
+ wc = tmp_path / "absent"
+ argv = json_args(wc, record, "--vcs", "svn", "--repository", "https://svn.example.org/repos/site")
+ code, result = run_json(argv, capsys)
+ assert code == ar.EXIT_NEEDS_CHECKOUT
+ assert result["status"] == "needs-checkout"
+ checkout = result["commands"][0]["argv"]
+ assert checkout[-3:] == ["checkout", "https://svn.example.org/repos/site", "absent"]
+ assert ["--backend", "svn"] == checkout[checkout.index("--backend") : checkout.index("--backend") + 2]
+
+
+def test_rejects_path_escaping_working_copy(
+ site: Path, tmp_path: Path, capsys: pytest.CaptureFixture[str]
+) -> None:
+ record = write(tmp_path, "r.json", make_record())
+ argv = json_args(site, record)
+ argv[argv.index("content/security/json/{cve_id}.json")] = "../{cve_id}.json"
+ assert ar.main(argv) == ar.EXIT_USAGE
+
+
+# -- Layout B: an entry in a per-release-line page -----------------------------
+
+
+def test_new_section_goes_first_when_newest(
+ site: Path, tmp_path: Path, capsys: pytest.CaptureFixture[str]
+) -> None:
+ record = write(tmp_path, "r.json", make_record())
+ code, result = run_json(page_args(site, record), capsys)
+ assert code == 0, result
+ assert result["changes"][0]["action"] == "update"
+ assert ar.main([*page_args(site, record), "--apply"]) == 0
+ capsys.readouterr()
+ text = (site / "xdocs" / "security-9.xml").read_text()
+ parses(text)
+ new = text.index('<section name="Fixed in Example Server 9.0.99" rtext="2026-03-30">')
+ assert text.index("9.x vulnerabilities</title>") < new < text.index("Fixed in Example Server 9.0.98")
+ assert f" <cve>{CVE}</cve></p>" in text
+ assert f'This was fixed with commit <hashlink hash="{COMMIT}"/>.' in text
+ assert "reported to the Example Server security team on 2026-02-14." in text
+ assert "The issue was made public on 2026-04-02.</p>" in text
+ assert "<p>Affects: >= 9.0.0, < 9.0.99</p>" in text
+ assert "<strong>Important: Path traversal in the static file handler</strong>" in text
+
+ # Idempotent: a second run proposes nothing and a second apply changes nothing.
+ code, result = run_json(page_args(site, record), capsys)
+ assert result["status"] == "unchanged"
+ assert ar.main([*page_args(site, record), "--apply"]) == 0
+ assert (site / "xdocs" / "security-9.xml").read_text() == text
+
+
+@pytest.mark.parametrize(
+ ("fixed", "before", "after"),
+ [
+ ("9.0.97", "Fixed in Example Server 9.0.98", "Fixed in Example Server 9.0.96"),
+ ("9.0.90", "Fixed in Example Server 9.0.96", "</body>"),
+ ],
+)
+def test_new_section_keeps_newest_first_order(
+ site: Path, tmp_path: Path, capsys: pytest.CaptureFixture[str], fixed: str, before: str, after: str
+) -> None:
+ record = write(tmp_path, "r.json", make_record(ranges=[("9.0.0", fixed)]))
+ argv = [*page_args(site, record), "--release-date", f"{fixed}=2026-02-01", "--apply"]
+ assert ar.main(argv) == 0, capsys.readouterr().err
+ text = (site / "xdocs" / "security-9.xml").read_text()
+ parses(text)
+ new = text.index(f'name="Fixed in Example Server {fixed}"')
+ assert text.index(before) < new < text.index(after)
+
+
+def test_entry_joins_existing_section(site: Path, tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ record = write(tmp_path, "r.json", make_record(ranges=[("9.0.0", "9.0.98")]))
+ # No --release-date for 9.0.98: the section exists, so none is needed.
+ argv = [a for a in page_args(site, record) if a not in ("--release-date", "9.0.99=2026-03-30")]
+ assert ar.main([*argv, "--apply"]) == 0, capsys.readouterr().err
+ text = (site / "xdocs" / "security-9.xml").read_text()
+ parses(text)
+ section = text[
+ text.index("Fixed in Example Server 9.0.98") : text.index("Fixed in Example Server 9.0.96")
+ ]
+ assert section.index("CVE-2026-00002") < section.index(CVE)
+ assert text.count('name="Fixed in Example Server 9.0.98"') == 1
+
+
+def test_missing_release_date_refuses(site: Path, tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ record = write(tmp_path, "r.json", make_record())
+ argv = [a for a in page_args(site, record) if a not in ("--release-date", "9.0.99=2026-03-30")]
+ assert ar.main(argv) == ar.EXIT_REFUSED
+ assert "--release-date" in capsys.readouterr().err
+
+
+def test_tracker_text_is_xml_escaped(site: Path, tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ hostile = 'Bad <script>alert("x")</script> & </section><section name="Fixed in Example Server 99">'
+ record = write(tmp_path, "r.json", make_record(title=hostile, description=hostile + "\x07"))
+ assert ar.main([*page_args(site, record), "--apply"]) == 0, capsys.readouterr().err
+ text = (site / "xdocs" / "security-9.xml").read_text()
+ parses(text)
+ assert "<script>" not in text
+ assert "<script>alert("x")</script> & </section>" in text
+ assert text.count("<section ") == 4
+
+
+def test_refuses_unparsable_page(site: Path, tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ (site / "xdocs" / "security-9.xml").write_text(PAGE.replace("</body>", ""))
+ record = write(tmp_path, "r.json", make_record())
+ assert ar.main(page_args(site, record)) == ar.EXIT_REFUSED
+ assert "not well-formed" in capsys.readouterr().err
+
+
+def test_refuses_cve_recorded_under_another_release(
+ site: Path, tmp_path: Path, capsys: pytest.CaptureFixture[str]
+) -> None:
+ record = write(tmp_path, "r.json", make_record(cve_id="CVE-2026-00002"))
+ assert ar.main(page_args(site, record)) == ar.EXIT_REFUSED
+ assert "already appears" in capsys.readouterr().err
+
+
+def test_missing_page_refuses(site: Path, tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ record = write(tmp_path, "r.json", make_record(ranges=[("11.0.0", "11.0.5")]))
+ assert ar.main(page_args(site, record)) == ar.EXIT_REFUSED
+ assert "security-11.xml does not exist" in capsys.readouterr().err
+
+
+def test_several_release_lines_change_several_pages(
+ site: Path, tmp_path: Path, capsys: pytest.CaptureFixture[str]
+) -> None:
+ (site / "xdocs" / "security-10.xml").write_text(PAGE.replace("9.", "10."))
+ record = write(tmp_path, "r.json", make_record(ranges=[("9.0.0", "9.0.99"), ("10.1.0", "10.1.40")]))
+ argv = [*page_args(site, record), "--release-date", "10.1.40=2026-03-31"]
+ code, result = run_json(argv, capsys)
+ assert code == 0
+ assert sorted(c["path"] for c in result["changes"]) == ["xdocs/security-10.xml", "xdocs/security-9.xml"]
+ stage = next(c["argv"] for c in result["commands"] if "stage" in c["argv"])
+ assert stage[-2:] == ["xdocs/security-9.xml", "xdocs/security-10.xml"]
+
+
+def test_svn_release_page_and_crlf(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ wc = tmp_path / "site"
+ (wc / ".svn").mkdir(parents=True)
+ (wc / "xdocs").mkdir()
+ (wc / "xdocs" / "security-9.xml").write_bytes(PAGE.replace("\n", "\r\n").encode())
+ record = write(tmp_path, "r.json", make_record())
+ _code, result = run_json(page_args(wc, record), capsys)
+ assert result["vcs"] == "svn"
+ assert result["commands"][-1]["argv"][-3:-1] == ["commit", "-m"]
+ assert ar.main([*page_args(wc, record), "--apply"]) == 0
+ data = (wc / "xdocs" / "security-9.xml").read_bytes().decode()
+ assert "\r\n" in data and "\n" not in data.replace("\r\n", "")
+ parses(data)
+
+
+def test_fix_commit_flag_and_plural(site: Path, tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ record = write(tmp_path, "r.json", make_record())
+ assert ar.main([*page_args(site, record), "--fix-commit", "fedcba9876543", "--apply"]) == 0
+ text = (site / "xdocs" / "security-9.xml").read_text()
+ assert f'commits <hashlink hash="{COMMIT}"/> and <hashlink hash="fedcba9876543"/>.' in text
+ assert ar.main([*page_args(site, record), "--fix-commit", "NOT-HEX"]) == ar.EXIT_USAGE
+
+
+def test_markdown_page(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ wc = tmp_path / "site"
+ (wc / ".git").mkdir(parents=True)
+ page = "# Security\n\nIntro.\n\n## Fixed in Example Server 9.0.98\n\nReleased 2026-03-01.\n\n### Low: old (CVE-2026-00002)\n"
+ (wc / "security-9.md").write_text(page)
+ record = write(tmp_path, "r.json", make_record(title="Use of *eval* in [templates]"))
+ argv = page_args(wc, record, "--format", "markdown")
+ argv[argv.index("xdocs/security-{line}.xml")] = "security-{line}.md"
+ assert ar.main([*argv, "--apply"]) == 0, capsys.readouterr().err
+ text = (wc / "security-9.md").read_text()
+ assert text.index("## Fixed in Example Server 9.0.99") < text.index("## Fixed in Example Server 9.0.98")
+ assert r"### Important: Use of \*eval\* in \[templates\] (CVE-2026-00001)" in text
+ assert "Affects: >= 9.0.0, < 9.0.99" in text
+ assert ar.main([*argv, "--apply"]) == 0
+ assert (wc / "security-9.md").read_text() == text
+
+
+def test_entry_template_must_mark_the_cve(
+ site: Path, tmp_path: Path, capsys: pytest.CaptureFixture[str]
+) -> None:
+ template = write(tmp_path, "entry.xml", "<p>{title} {cve_id}</p>\n")
+ record = write(tmp_path, "r.json", make_record())
+ assert ar.main([*page_args(site, record), "--entry-template", str(template)]) == ar.EXIT_USAGE
+ assert "whole text of an element" in capsys.readouterr().err
+
+
+def test_unknown_placeholder_is_a_usage_error(
+ site: Path, tmp_path: Path, capsys: pytest.CaptureFixture[str]
+) -> None:
+ record = write(tmp_path, "r.json", make_record())
+ assert ar.main([*page_args(site, record), "--section-name", "Fixed in {nope}"]) == ar.EXIT_USAGE
+
+
+def test_version_ordering() -> None:
+ assert ar._version_lt("9.0.0.M1", "9.0.0")
+ assert ar._version_lt("9.0.9", "9.0.10")
+ assert not ar._version_lt("10.0.0", "9.0.99")
+ assert ar.release_line("10.1.40", 1) == "10"
+ assert ar.release_line("10.1.40", 2) == "10.1"
diff --git a/tools/github/source-control.md b/tools/github/source-control.md
index b3bccb48..a2731ac8 100644
--- a/tools/github/source-control.md
+++ b/tools/github/source-control.md
@@ -53,6 +53,7 @@ the same abstract operation.
| History read | `git log` (`--oneline` / `--grep` / `--author` / `--since`), `git show` | reproducer, reassess, nomination |
| Determine divergence base | `git merge-base`, `git rev-parse` | code-review (diff against base) |
| Sync with the forge | `git fetch [origin]`, `git push [-u]` | fix-workflow hand-off |
+| Bring the working copy up to date / first checkout | `git pull --ff-only`, `git clone` | security-issue-sync (advisory records) |
| Re-apply onto an updated base | `git rebase` | triage rebase action |
| Isolated checkouts | `git worktree add` / `list --porcelain` | `magpie-setup` worktree flow |
| Park uncommitted work | `git stash --include-untracked` | fix-workflow safety |
diff --git a/tools/skill-evals/README.md b/tools/skill-evals/README.md
index 8d83e3de..0813b717 100644
--- a/tools/skill-evals/README.md
+++ b/tools/skill-evals/README.md
@@ -21,7 +21,7 @@ Suites are currently implemented for:
- **security-issue-triage** — 33 cases across 9 steps
- **security-issue-deduplicate** — 18 cases across 6 steps (steps 1, 2, 3, 4, 5, 6)
- **security-cve-allocate** — 22 cases across 6 steps (steps 1, 2, 3, 4, 5, 7)
-- **security-issue-sync**: 80 cases across 14 steps (1d, 1f, 2a, 2b, 2c, 3, 5b, 6, bulk-orchestration, bulk-selectors, draft-backend, ghsa-access-tier, guardrails, security-cc)
+- **security-issue-sync**: 82 cases across 14 steps (1d, 1f, 2a, 2b, 2c, 3, 5b, 6, bulk-orchestration, bulk-selectors, draft-backend, ghsa-access-tier, guardrails, security-cc)
- **security-issue-fix** — 50 cases across 15 steps (2, 3, 5a–5g, 6, 7, 11, and the flow choice plus Subversion steps 8 and 9)
- **security-issue-invalidate** — 24 cases across 9 steps (2, 3, 4, 5a, 5b, 5d, 5e, 5f, 7)
- **security-issue-import-from-md** — 11 cases across 4 steps (1, 2, 4, 6)
diff --git a/tools/skill-evals/evals/security-issue-sync/README.md b/tools/skill-evals/evals/security-issue-sync/README.md
index 78b32eca..b90cc010 100644
--- a/tools/skill-evals/evals/security-issue-sync/README.md
+++ b/tools/skill-evals/evals/security-issue-sync/README.md
@@ -12,7 +12,7 @@ Steps 0 (pre-flight), 1a–1c and 1e (data gathering), 1g (cve.org API check), 4
|------|------|-------|-------|
| 1f | Process step identification | 7 | Decision table covering steps 1-2 through 14 |
| 2a | Observed state | 3 | Step 11 (PR merged), step 6 (CVE needed), step 2 (stale) |
-| 2b | Proposed changes | 10 | Label swap, milestone create, providers wave, disclosure overdue, distributor pre-announce, reporter unresponsive + replied, security-pages reminder |
+| 2b | Proposed changes | 12 | Label swap, milestone create, providers wave, disclosure overdue, distributor pre-announce, reporter unresponsive + replied, security-pages reminder, advisory-records update (proposed once published, held while cve.org is not) |
| 2c | Next-step recommendation | 4 | Steps 3, 6, 11, 13 |
| Guardrails | Guardrail violation detection | 3 | CVSS propagation, ASF project naming, clean pass |
| 3 | Confirm with user | 3 | apply-all, selective, cancel |
@@ -34,6 +34,7 @@ Steps 0 (pre-flight), 1a–1c and 1e (data gathering), 1g (cve.org API check), 4
- **CVE allocate skill explicitly named and linked**: Step-6 next-step must name and link `security-cve-allocate` (2c case-2).
- **No-action parking**: Step-11 produces `has_concrete_action: false` (2c case-3).
- **Security-pages reminder fires only post-advisory, once**: on a closed-`announced` tracker with *Public advisory URL* populated, the hand-off comment's checklist item still unticked and no reminder marker posted, the reminder status comment is proposed with `security_pages_reminder_proposed: true`; a ticked checklist item or an already-posted reminder proposes nothing (2b cases 8-10).
+- **Advisory records only after publication**: with `advisory_records` configured, a closed-`announced` tracker whose CVE cve.org reports `PUBLISHED` gets one `advisory_records` item carrying the planner's change and commands (`advisory_records_proposed: true`) and no security-pages reminder; while cve.org does not report it `PUBLISHED`, nothing is proposed (2b cases 11-12).
- **Golden rule 2 — no bare issue numbers**: `has_bare_issue_numbers` must always be false.
- **CVE-affecting trackers walked individually**: in bulk mode, any tracker whose `proposed_body_field_updates` names a CVE-publication field (Title, Short public summary for publish, CWE, Severity, Affected versions, Reporter credited as, Remediation developer, PR with the fix, Public advisory URL) must end up in `cve_affecting` and the `walk_order` is ascending by tracker number (bulk-orchestration cases 2 + 3).
- **Non-CVE-affecting trackers bundled**: trackers whose proposals are limited to label flips, milestone touches, assignee swaps, project-board moves, status-rollup entries, reporter Gmail drafts, or RM hand-off comments end up in `non_cve_affecting` (bulk-orchestration case 1 + the two label-only trackers in case 2).
diff --git a/tools/skill-evals/evals/security-issue-sync/step-2b-proposed-changes/fixtures/case-11-advisory-records-proposal/expected.json b/tools/skill-evals/evals/security-issue-sync/step-2b-proposed-changes/fixtures/case-11-advisory-records-proposal/expected.json
new file mode 100644
index 00000000..76b7ce39
--- /dev/null
+++ b/tools/skill-evals/evals/security-issue-sync/step-2b-proposed-changes/fixtures/case-11-advisory-records-proposal/expected.json
@@ -0,0 +1,17 @@
+{
+ "items": [
+ {
+ "number": 1,
+ "category": "advisory_records",
+ "action": "Record CVE-2026-00001 in the project's security records: show the planner's change to xdocs/security-9.xml (a new section 'Fixed in My Project 9.0.99' dated 2026-03-30, placed before the 9.0.98 section, holding the entry) and, on confirmation, run its magpie-vcs commands in order — clean check, update, the --apply re-run, stage, diff, commit (on Subversion the commit publishes the change) — then post the advisory-records v1 status comment with the ticked security-pages c [...]
+ "reason": "advisory_records_pending is true: cve.org reports CVE-2026-00001 PUBLISHED, the project configures advisory_records, no advisory-records marker comment exists, and the planner returned a change."
+ }
+ ],
+ "labels_to_add": [],
+ "labels_to_remove": [],
+ "milestone_create_needed": false,
+ "assignee_proposed": null,
+ "has_bare_issue_numbers": false,
+ "security_pages_reminder_proposed": false,
+ "advisory_records_proposed": true
+}
diff --git a/tools/skill-evals/evals/security-issue-sync/step-2b-proposed-changes/fixtures/case-11-advisory-records-proposal/report.md b/tools/skill-evals/evals/security-issue-sync/step-2b-proposed-changes/fixtures/case-11-advisory-records-proposal/report.md
new file mode 100644
index 00000000..4c52dc91
--- /dev/null
+++ b/tools/skill-evals/evals/security-issue-sync/step-2b-proposed-changes/fixtures/case-11-advisory-records-proposal/report.md
@@ -0,0 +1,39 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+
+## Observed tracker state
+
+**Tracker:** myproject-s/myproject-s#61 (closed as `completed`)
+**Current labels:** security issue, cve allocated, announced - emails sent, announced
+**Current milestone:** 9.0.99 (closed)
+**Current assignee:** jsnow-rm
+
+**Body fields:** *CVE tool link* = `https://cveprocess.apache.org/cve5/CVE-2026-00001`; *Public advisory URL* = `https://lists.apache.org/thread/q8w1e4r7t2y5u9i3o6p0a` (populated by the Step 14 combined apply)
+
+## Gathered state from Step 1
+
+**Process step:** post-15 (closed-`announced` bucket)
+
+**cve.org (Step 1g):** `CVE-2026-00001` is `PUBLISHED`, `datePublished` 2026-04-02.
+The *CVE-published* reporter notification was already sent on 2026-04-03 (status comment and sent message on the thread) — no new email.
+
+**Security-pages checklist (Step 1g closed-bucket scan):**
+- Hand-off comment carries `- [ ] <!-- apache-magpie: security-pages-checklist v1 --> Project security pages updated with CVE-2026-00001` — still unticked
+- No comment carries `<!-- apache-magpie: security-pages-reminder v1 -->`
+- `security_pages_reminder_pending: false` (the project configures `advisory_records`, so sync makes the edit itself)
+
+**Advisory records (Step 1g):**
+- `advisory_records` is configured in `projects/myproject/project.md`: `layout: release-page`, `vcs: svn`, `format: xml`, `path: xdocs/security-{line}.xml`
+- No comment carries `<!-- apache-magpie: advisory-records v1 -->`
+- The `advisory-records` planner (run without `--apply`) exited `0` with status `proposed`:
+ - `xdocs/security-9.xml`: update — create section `Fixed in My Project 9.0.99` (2026-03-30) before `Fixed in My Project 9.0.98`, holding the CVE-2026-00001 entry
+ - Commands, in order, after confirmation:
+ 1. `magpie-vcs -C ~/code/myproject-site clean`
+ 2. `magpie-vcs -C ~/code/myproject-site update`
+ 3. `advisory-records … --apply`
+ 4. `magpie-vcs -C ~/code/myproject-site stage xdocs/security-9.xml`
+ 5. `magpie-vcs -C ~/code/myproject-site diff xdocs/security-9.xml`
+ 6. `magpie-vcs -C ~/code/myproject-site commit -m 'Add CVE-2026-00001 to the security pages'` (on Subversion this publishes the change)
+- `advisory_records_pending: true`
+
+Produce the numbered proposal for this tracker.
diff --git a/tools/skill-evals/evals/security-issue-sync/step-2b-proposed-changes/fixtures/case-12-advisory-records-waiting-for-cve-org/expected.json b/tools/skill-evals/evals/security-issue-sync/step-2b-proposed-changes/fixtures/case-12-advisory-records-waiting-for-cve-org/expected.json
new file mode 100644
index 00000000..428b5064
--- /dev/null
+++ b/tools/skill-evals/evals/security-issue-sync/step-2b-proposed-changes/fixtures/case-12-advisory-records-waiting-for-cve-org/expected.json
@@ -0,0 +1,10 @@
+{
+ "items": [],
+ "labels_to_add": [],
+ "labels_to_remove": [],
+ "milestone_create_needed": false,
+ "assignee_proposed": null,
+ "has_bare_issue_numbers": false,
+ "security_pages_reminder_proposed": false,
+ "advisory_records_proposed": false
+}
diff --git a/tools/skill-evals/evals/security-issue-sync/step-2b-proposed-changes/fixtures/case-12-advisory-records-waiting-for-cve-org/report.md b/tools/skill-evals/evals/security-issue-sync/step-2b-proposed-changes/fixtures/case-12-advisory-records-waiting-for-cve-org/report.md
new file mode 100644
index 00000000..f47fcfeb
--- /dev/null
+++ b/tools/skill-evals/evals/security-issue-sync/step-2b-proposed-changes/fixtures/case-12-advisory-records-waiting-for-cve-org/report.md
@@ -0,0 +1,28 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+
+## Observed tracker state
+
+**Tracker:** myproject-s/myproject-s#62 (closed as `completed`)
+**Current labels:** security issue, cve allocated, announced - emails sent, announced
+**Current milestone:** 9.0.99 (closed)
+**Current assignee:** jsnow-rm
+
+**Body fields:** *CVE tool link* = `https://cveprocess.apache.org/cve5/CVE-2026-00004`; *Public advisory URL* = `https://lists.apache.org/thread/z5x8c1v4b7n0m3l6k9j2h` (populated by the Step 14 combined apply)
+
+## Gathered state from Step 1
+
+**Process step:** 15 (closed-`announced` bucket; CVE record `PUBLIC` on the CVE tool, cve.org still shows `RESERVED` — no reporter email yet)
+
+**Security-pages checklist (Step 1g closed-bucket scan):**
+- Hand-off comment carries `- [ ] <!-- apache-magpie: security-pages-checklist v1 --> Project security pages updated with CVE-2026-00004` — still unticked
+- No comment carries `<!-- apache-magpie: security-pages-reminder v1 -->`
+- `security_pages_reminder_pending: false` (the project configures `advisory_records`, so sync makes the edit itself)
+
+**Advisory records (Step 1g):**
+- `advisory_records` is configured in `projects/myproject/project.md`: `layout: cve-json-file`, `vcs: git`, `path: content/security/json/{cve_id}.json`
+- No comment carries `<!-- apache-magpie: advisory-records v1 -->`
+- *"advisory records: waiting for cve.org"* — the planner was not run, because cve.org does not report the CVE `PUBLISHED` yet
+- `advisory_records_pending: false`
+
+Produce the numbered proposal for this tracker.
diff --git a/tools/skill-evals/evals/security-issue-sync/step-2b-proposed-changes/fixtures/output-spec.md b/tools/skill-evals/evals/security-issue-sync/step-2b-proposed-changes/fixtures/output-spec.md
index d0c4ee65..d1a03c64 100644
--- a/tools/skill-evals/evals/security-issue-sync/step-2b-proposed-changes/fixtures/output-spec.md
+++ b/tools/skill-evals/evals/security-issue-sync/step-2b-proposed-changes/fixtures/output-spec.md
@@ -12,7 +12,7 @@ The observed state (current tracker) vs. gathered state (PR, milestone, assignee
"items": [
{
"number": 1,
- "category": "labels" | "milestone" | "assignee" | "body_field" | "status_comment" | "reporter_unresponsive",
+ "category": "labels" | "milestone" | "assignee" | "body_field" | "status_comment" | "reporter_unresponsive" | "advisory_records",
"action": "<string>",
"reason": "<string>"
}
@@ -41,3 +41,4 @@ Field rules:
- `distributor_notify_proposed`: `true` when `items` contains a distributor pre-announcement draft proposal. Optional field — omit (or set `false`) when `pre_announce_distributors` is `false` or the fix is not yet in a pending release.
- `reporter_unresponsive_proposed`: `true` when `items` contains a `reporter_unresponsive` item. Optional field — omit (or set `false`) when the reporter thread is not stale.
- `security_pages_reminder_proposed`: `true` when `items` contains the security-pages reminder status comment. Optional field — omit (or set `false`) unless the gathered state reports `security_pages_reminder_pending: true` — in particular when a ticked `- [x]` security-pages checklist item exists anywhere on the tracker, or when a comment carrying the `security-pages-reminder v1` marker has already been posted (the reminder fires at most once per tracker).
+- `advisory_records_proposed`: `true` when `items` contains an `advisory_records` item — the advisory-records update that records the published CVE in the project's own security records (the planner's file change plus its version-control commands). Propose it only when the gathered state reports `advisory_records_pending: true`; omit (or set `false`) when the project does not configure `advisory_records`, when cve.org does not report the CVE `PUBLISHED` yet, or when the CVE is already recorded.
diff --git a/tools/spec-loop/specs/security-issue-lifecycle.md b/tools/spec-loop/specs/security-issue-lifecycle.md
index c7ce7ca7..5890e94d 100644
--- a/tools/spec-loop/specs/security-issue-lifecycle.md
+++ b/tools/spec-loop/specs/security-issue-lifecycle.md
@@ -277,6 +277,19 @@ publication, with a human gate and an audit-log entry at every step.
A re-rendered hand-off comment keeps the release manager's tick, and a
ticked box keeps later runs quiet. The reminder links the project's
pages through the optional `security_pages_url` manifest key (#1355).
+- **With `advisory_records` configured, sync proposes the security-pages edit.**
+ The optional `advisory_records` manifest block names the site repository
+ (Git or Subversion), its working copy and a layout: one CVE 5.x JSON file
+ per CVE, or an entry in a per-release-line page whose path, section name
+ and entry come from adopter templates. In the Step 1g closed-tracker pass,
+ once cve.org reports the CVE `PUBLISHED`, the deterministic
+ `advisory-records` planner (in `generate-cve-json`) computes the change and
+ the `magpie-vcs` commands; sync proposes them as one item and runs them only
+ on confirmation, then posts an `advisory-records` marker comment with the
+ ticked checklist item. The planner refuses before publication, is
+ idempotent, escapes tracker text, and refuses a page it cannot parse. The
+ reminder never fires for such a project. `magpie-vcs` gains `update` and
+ `checkout` for it.
## Out of scope
diff --git a/tools/vcs/README.md b/tools/vcs/README.md
index ec8185f4..322916c1 100644
--- a/tools/vcs/README.md
+++ b/tools/vcs/README.md
@@ -106,6 +106,8 @@ editing every skill.
| Stage | `stage <paths…>` | write |
| Commit | `commit (-m <msg> \| -F <file>) [--scrub]` | write |
| Sync from forge | `fetch [remote] [ref]` | write |
+| Update the working copy (never merges) | `update` | write |
+| First checkout | `--backend <name> checkout <url> <dest>` | write |
| Publish | `push [-u] <remote> <ref>` | write |
| Reset working copy | `reset-worktree` | write |
| Fix flow for this backend | `fix-flow [--declared VALUE]` | read |
@@ -147,6 +149,8 @@ documented in [`tools/asf-svn/source-control.md`](../asf-svn/source-control.md#c
| `log` | server history up to `HEAD`, one `r<rev> <subject>` line per revision; `--grep` / `--author` are regexes, `--since` takes an ISO 8601 date. With `--grep` / `--author` / `--since`, `-n` caps the output only: the path's whole history (back to `--since`, if given) is fetched before filtering, which is slow on a large shared repository such as svn.apache.org |
| `cat <rev> <path>` | the working-copy path at `<rev>`, following copy history (so a revision from before a branch was created resolves through trunk). A path no longer in the working copy is read through the revision before the commit that deleted it on this line of work, which reads the line's log (slow on a large repository) |
| `fetch` | no-op: refs live on the server and every history read queries it |
+| `update` | `svn update`; a commit from an out-of-date working copy fails, so run it before editing (Git: `pull --ff-only`, hg: `pull -u`, fossil: `update`) |
+| `checkout <url> <dest>` | `svn checkout` (Git and hg: `clone`; not supported on fossil). There is no working copy to detect yet, so it needs `--backend` |
| `push` | error: `commit` already published the change |
A caller using the Python API tells the models apart with
diff --git a/tools/vcs/src/magpie_vcs/__init__.py b/tools/vcs/src/magpie_vcs/__init__.py
index d275413f..a2a1c3f6 100644
--- a/tools/vcs/src/magpie_vcs/__init__.py
+++ b/tools/vcs/src/magpie_vcs/__init__.py
@@ -171,6 +171,15 @@ class VCSBackend(ABC):
def fetch(self, remote: str | None = None, ref: str | None = None) -> None:
"""Sync refs from the forge without changing the working copy."""
+ @abstractmethod
+ def update(self) -> None:
+ """Bring the working copy up to date with its upstream, never by merging."""
+
+ @classmethod
+ def checkout(cls, url: str, dest: Path) -> None:
+ """Create a new working copy of ``url`` at ``dest`` (which must not exist)."""
+ raise VCSError(f"{cls.name}: checkout is not supported on this backend")
+
@abstractmethod
def push(self, remote: str, ref: str, set_upstream: bool = False) -> None:
"""Publish a ref to the forge."""
@@ -217,6 +226,14 @@ def _run(
return proc.stdout if capture else ""
+def _require_new_dest(dest: Path) -> None:
+ """Refuse a checkout onto an existing path, or under a missing parent."""
+ if dest.exists():
+ raise VCSError(f"checkout: {dest} already exists")
+ if not dest.parent.is_dir():
+ raise VCSError(f"checkout: parent directory {dest.parent} does not exist")
+
+
class GitBackend(VCSBackend):
"""Git binding of the source-control capability (GitHub's native VCS)."""
@@ -297,6 +314,15 @@ class GitBackend(VCSBackend):
args.append(ref)
_run(args, self.root, capture=False)
+ def update(self) -> None:
+ # Fast-forward only: a diverged local branch is an error, never a merge.
+ _run(["git", "pull", "--ff-only"], self.root, capture=False)
+
+ @classmethod
+ def checkout(cls, url: str, dest: Path) -> None:
+ _require_new_dest(dest)
+ _run(["git", "clone", "--", url, str(dest)], dest.parent, capture=False)
+
def push(self, remote: str, ref: str, set_upstream: bool = False) -> None:
args = ["git", "push"]
if set_upstream:
@@ -400,6 +426,15 @@ class MercurialBackend(VCSBackend):
args.append(remote)
_run(args, self.root, capture=False)
+ def update(self) -> None:
+ # `pull -u` updates only when it can do so without a merge.
+ _run(["hg", "pull", "-u"], self.root, capture=False)
+
+ @classmethod
+ def checkout(cls, url: str, dest: Path) -> None:
+ _require_new_dest(dest)
+ _run(["hg", "clone", "--", url, str(dest)], dest.parent, capture=False)
+
def push(self, remote: str, ref: str, set_upstream: bool = False) -> None:
args = ["hg", "push", "-B", ref, remote]
_run(args, self.root, capture=False)
@@ -533,6 +568,10 @@ class FossilBackend(VCSBackend):
args.append(remote)
_run(args, self.root, capture=False)
+ def update(self) -> None:
+ # With autosync (fossil's default) `update` pulls before it updates.
+ _run(["fossil", "update"], self.root, capture=False)
+
def push(self, remote: str, ref: str, set_upstream: bool = False) -> None:
args = ["fossil", "push"]
if remote:
@@ -807,6 +846,15 @@ class SubversionBackend(VCSBackend):
# change the working copy, which fetch must not do.
return None
+ def update(self) -> None:
+ # A commit from an out-of-date working copy fails, so callers update first.
+ self._svn("update", capture=False)
+
+ @classmethod
+ def checkout(cls, url: str, dest: Path) -> None:
+ _require_new_dest(dest)
+ _run(["svn", "--non-interactive", "checkout", "--", url, str(dest)], dest.parent, capture=False)
+
def push(self, remote: str, ref: str, set_upstream: bool = False) -> None:
raise VCSError(
"svn is centralized: `commit` already published the change to the server; there is nothing to push"
@@ -1130,6 +1178,12 @@ def _build_parser() -> argparse.ArgumentParser:
p.add_argument("remote", nargs="?")
p.add_argument("ref", nargs="?")
+ sub.add_parser("update", help="bring the working copy up to date (never merges)")
+
+ p = sub.add_parser("checkout", help="create a new working copy (needs --backend)")
+ p.add_argument("url")
+ p.add_argument("dest")
+
p = sub.add_parser("push", help="publish a ref to the forge")
p.add_argument("-u", "--set-upstream", action="store_true")
p.add_argument("remote")
@@ -1219,6 +1273,16 @@ def main(argv: Sequence[str] | None = None) -> int:
print(json.dumps(flow, indent=2))
return 0
+ if ns.op == "checkout":
+ # There is no working copy to detect yet, so the backend is named.
+ forced = ns.backend or os.environ.get(BACKEND_ENV) or ""
+ checkout_cls = _BACKENDS_BY_NAME.get(forced)
+ if checkout_cls is None:
+ known = ", ".join(sorted(_BACKENDS_BY_NAME))
+ raise VCSError(f"checkout needs --backend (or ${BACKEND_ENV}) naming one of: {known}")
+ checkout_cls.checkout(ns.url, (start / ns.dest).resolve())
+ return 0
+
backend = get_backend(start, ns.backend)
if ns.op == "detect":
@@ -1290,6 +1354,8 @@ def main(argv: Sequence[str] | None = None) -> int:
return 0 if proposal["ready"] else 1
elif ns.op == "fetch":
backend.fetch(ns.remote, ns.ref)
+ elif ns.op == "update":
+ backend.update()
elif ns.op == "push":
backend.push(ns.remote, ns.ref, set_upstream=ns.set_upstream)
elif ns.op == "reset-worktree":
diff --git a/tools/vcs/tests/test_vcs.py b/tools/vcs/tests/test_vcs.py
index 59a3a35a..a575ff17 100644
--- a/tools/vcs/tests/test_vcs.py
+++ b/tools/vcs/tests/test_vcs.py
@@ -485,6 +485,22 @@ def test_svn_fetch_leaves_wc_and_push_refuses(svn_wc: Path) -> None:
backend.push("origin", "trunk")
+@svn_required
+def test_svn_checkout_and_update(svn_wc: Path, tmp_path: Path) -> None:
+ url = subprocess.run(
+ ["svn", "info", "--show-item", "url"], cwd=svn_wc, check=True, capture_output=True, text=True
+ ).stdout.strip()
+ second = tmp_path / "second"
+ assert main(["--backend", "svn", "-C", str(tmp_path), "checkout", url, "second"]) == 0
+ assert (second / "file.txt").read_text() == "hello\n"
+ (svn_wc / "file.txt").write_text("from the first copy\n")
+ _svn(svn_wc, "commit", "-m", "change file.txt")
+ assert main(["-C", str(second), "update"]) == 0
+ assert (second / "file.txt").read_text() == "from the first copy\n"
+ with pytest.raises(VCSError, match="already exists"):
+ SubversionBackend.checkout(url, second)
+
+
# -- hg backend operations -------------------------------------------------
@@ -578,6 +594,27 @@ def test_cli_svn_detect_and_status(svn_wc: Path, capsys: pytest.CaptureFixture[s
assert "already published" in capsys.readouterr().err
+@git_required
+def test_git_checkout_and_update(git_repo: Path, tmp_path: Path) -> None:
+ clone = tmp_path / "clone"
+ assert main(["--backend", "git", "-C", str(tmp_path), "checkout", str(git_repo), "clone"]) == 0
+ assert (clone / "file.txt").read_text() == "hello\n"
+ (git_repo / "file.txt").write_text("upstream change\n")
+ _git(git_repo, "commit", "-q", "-am", "upstream change")
+ assert main(["-C", str(clone), "update"]) == 0
+ assert (clone / "file.txt").read_text() == "upstream change\n"
+
+
+def test_cli_checkout_needs_backend(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ assert main(["-C", str(tmp_path), "checkout", "https://example.org/repo", "dest"]) == 2
+ assert "checkout needs --backend" in capsys.readouterr().err
+
+
+def test_fossil_checkout_unsupported(tmp_path: Path) -> None:
+ with pytest.raises(VCSError, match="not supported"):
+ FossilBackend.checkout("https://example.org/repo", tmp_path / "dest")
+
+
@git_required
def test_cli_cat(git_repo: Path, capsys: pytest.CaptureFixture[str]) -> None:
assert main(["-C", str(git_repo), "cat", "HEAD", "file.txt"]) == 0