You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@magpie.apache.org by po...@apache.org on 2026/10/04 23:45:10 UTC
(magpie) 02/03: perf(release-verify-rc): move the deterministic checks into tools/release-verify (#1511)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/magpie.git
commit 20d1238e67bc0ca24e84b586c89dc6f603a14aa2
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Mon Oct 5 01:44:42 2026 +0200
perf(release-verify-rc): move the deterministic checks into tools/release-verify (#1511)
* perf(release-verify-rc): move the deterministic checks into tools/release-verify
verify-rc's artefact inventory, signature and KEYS classification,
checksums, NOTICE/LICENSE presence, binary exclusion, symlink check,
version consistency and verdict roll-up were procedures the model
followed step by step. They now run in tools/release-verify, a
read-only tool that imports KEYS into a temporary GNUPGHOME, never
the user's keyring, and prints one JSON object per check. The skill
keeps the invocation and how to read each status; RAT, the
NOTICE/LICENSE materiality call, adopter validators and the report
stay with the model.
Rules settled by the maintainer while coding them:
- an artefact is required unless the build template marks it
optional;
- only sha512 is required; md5 never fails alone (WARN);
- a symlink must resolve inside the unpacked archive;
- SKIP is neutral in the verdict, and skipped steps are listed.
The eval cases for steps 2, 3, 5, 6 and 8 now feed the tool's JSON;
the classification itself is covered by the tool's tests.
Generated-by: Claude Opus 5
* fix(release-verify): quote paste recipes, keep local paths out, fail stale signatures
From a security review of the new tool:
- Every value interpolated into a paste_recipe is now shell-quoted.
Artefact names come from the staging area, so a crafted name such
as x$(...).tar.gz ran on the voter's machine when pasted.
- A signature made by a revoked or expired key, or an expired
signature, is FAIL. gpg still exits 0 with VALIDSIG for those, so
the tool reads REVKEYSIG / EXPKEYSIG / EXPSIG from the status output.
- gpg messages copied into a result's detail no longer carry the
temporary GNUPGHOME path, and a local --keys file appears in the
recipe by name only; the report can be posted to the planning issue.
Tests cover each case; the eval fixtures carry the quoted recipes.
Generated-by: Claude Opus 5
* build(rat): exclude the release-verify signature fixtures
Apache RAT flagged the OpenPGP keys, detached signatures and stand-in
artefacts under tools/release-verify/tests/fixtures/gpg/. Key and
signature data cannot carry a licence header; generate.py, which wrote
them, does and stays scanned.
Generated-by: Claude Opus 5
---
.github/labeler.yml | 1 +
.rat-excludes | 8 +
.typos.toml | 3 +
docs/labels-and-capabilities.md | 3 +-
docs/vendor-neutrality.md | 5 +-
.../skills/verify-rc/SKILL.md | 465 +++++------
plugins/magpie-setup/templates/release-build.md | 16 +-
pyproject.toml | 1 +
tools/release-verify/README.md | 99 +++
tools/release-verify/pyproject.toml | 89 +++
.../release-verify/src/release_verify/__init__.py | 857 +++++++++++++++++++++
.../release-verify/tests/fixtures/gpg/FINGERPRINTS | 2 +
tools/release-verify/tests/fixtures/gpg/KEYS | 9 +
tools/release-verify/tests/fixtures/gpg/OTHER.pub | 9 +
.../tests/fixtures/gpg/artefact.tar.gz | 1 +
.../tests/fixtures/gpg/artefact.tar.gz.asc | 7 +
.../release-verify/tests/fixtures/gpg/generate.py | 133 ++++
.../release-verify/tests/fixtures/gpg/other.tar.gz | 1 +
.../tests/fixtures/gpg/other.tar.gz.asc | 7 +
tools/release-verify/tests/test_release_verify.py | 639 +++++++++++++++
.../skill-evals/evals/release-verify-rc/README.md | 8 +-
.../fixtures/case-1-all-sigs-pass/expected.json | 6 +-
.../fixtures/case-1-all-sigs-pass/report.md | 53 +-
.../fixtures/case-2-tampered-sig/expected.json | 4 +-
.../fixtures/case-2-tampered-sig/report.md | 56 +-
.../fixtures/output-spec.md | 3 +-
.../fixtures/case-1-checksums-pass/expected.json | 2 +-
.../fixtures/case-1-checksums-pass/report.md | 56 +-
.../case-2-checksum-mismatch/expected.json | 2 +-
.../fixtures/case-2-checksum-mismatch/report.md | 57 +-
.../fixtures/output-spec.md | 9 +-
.../fixtures/case-1-both-present/report.md | 32 +-
.../fixtures/case-2-missing-notice/report.md | 31 +-
.../step-5-notice-license/fixtures/output-spec.md | 3 +-
.../case-1-no-prohibited-binaries/expected.json | 2 +-
.../case-1-no-prohibited-binaries/report.md | 26 +-
.../case-2-prohibited-binary-found/expected.json | 2 +-
.../case-2-prohibited-binary-found/report.md | 29 +-
.../fixtures/case-1-version-consistent/report.md | 35 +-
.../fixtures/case-2-version-mismatch/report.md | 36 +-
uv.lock | 16 +
41 files changed, 2433 insertions(+), 390 deletions(-)
diff --git a/.github/labeler.yml b/.github/labeler.yml
index b16b35f3..a4745cc6 100644
--- a/.github/labeler.yml
+++ b/.github/labeler.yml
@@ -182,6 +182,7 @@ substrate:release:
- any-glob-to-any-file:
- 'tools/maven-artifact-verify/**'
- 'tools/release-config/**'
+ - 'tools/release-verify/**'
- 'tools/reproducible-archive/**'
substrate:review:
diff --git a/.rat-excludes b/.rat-excludes
index 4bd83c34..366c7831 100644
--- a/.rat-excludes
+++ b/.rat-excludes
@@ -54,3 +54,11 @@
# non-Apache (GPL / EPL / …) licence text as eval input, so RAT must not
# treat them as source files needing an Apache header.
tools/skill-evals/evals/pr-management-code-review/step-4-third-party-license/fixtures/**
+# tools/release-verify signature fixtures: OpenPGP public keys, detached
+# signatures and stand-in artefacts the signature tests run against. Key and
+# signature data cannot carry a header; generate.py, which wrote them, does.
+tools/release-verify/tests/fixtures/gpg/KEYS
+tools/release-verify/tests/fixtures/gpg/OTHER.pub
+tools/release-verify/tests/fixtures/gpg/FINGERPRINTS
+tools/release-verify/tests/fixtures/gpg/*.tar.gz
+tools/release-verify/tests/fixtures/gpg/*.tar.gz.asc
diff --git a/.typos.toml b/.typos.toml
index 5dd52d80..354790ef 100644
--- a/.typos.toml
+++ b/.typos.toml
@@ -56,6 +56,9 @@ Nd = "Nd"
# `mis-` prefix in `mis-allocation` is a legitimate compound, not a
# typo of "miss" / "mist".
mis = "mis"
+# `fpr` is GnuPG's `--with-colons` fingerprint record type, parsed by
+# tools/release-verify and the keys-sync key check; not a typo of "for".
+fpr = "fpr"
# `pre-empted` is a real word; typos flags `empted` as a typo of
# `emptied` only because of how it splits hyphenated words.
empted = "empted"
diff --git a/docs/labels-and-capabilities.md b/docs/labels-and-capabilities.md
index 3a4cec6c..7abfbda5 100644
--- a/docs/labels-and-capabilities.md
+++ b/docs/labels-and-capabilities.md
@@ -147,7 +147,7 @@ framework substrate:
| `substrate:action-guard` | substrate | Deterministic pre-tool-use command guards. |
| `substrate:privacy` | substrate | PII redaction / approved-LLM gating. |
| `substrate:framework-dev` | substrate | Build / validate / eval the framework itself. |
-| `substrate:release` | substrate | Release-artefact helpers an adopter's release process runs: reproducible-archive build, lint and comparison; maven-artifact-verify for staged JVM artefacts; release-config loading and pre-flight checks. |
+| `substrate:release` | substrate | Release-artefact helpers an adopter's release process runs: reproducible-archive build, lint and comparison; maven-artifact-verify for staged JVM artefacts; release-config loading and pre-flight checks; release-verify checks of a staged release candidate. |
| `substrate:setup` | substrate | Adopter-side setup state an agent resolves at runtime rather than at development time: lock parsing, floor comparison, reconciliation fingerprints. |
| `substrate:review` | substrate | A second model's read of a change before it is published: other models' CLIs run read-only, findings merged. |
@@ -365,6 +365,7 @@ or a contract-free mix of substrates (e.g. `tools/spec-inventory` is
| [`tools/release-config`](../tools/release-config/) | `substrate:release` | `release-config` — loads the release-management config (`<project-config>` layered lookup, defaults, derived values) and runs each `release-*` skill's deterministic Step 0 pre-flight checks as JSON; used by every `release-*` skill's Step 0 and the Step 1 metadata of `rc-cut`, `vote-draft`, `announce-draft` and `promote` |
| [`tools/reproducible-archive`](../tools/reproducible-archive/) | `substrate:release` | `repro-archive` — build, lint and compare reproducible source archives from a git ref (`git archive` + `.gitattributes` `export-ignore`, every reproducible-builds.org archive rule applied); used by `release-rc-cut` Step 2/2b and `release-verify-rc` Step 9 |
| [`tools/maven-artifact-verify`](../tools/maven-artifact-verify/) | `substrate:release` | `maven-artifact-verify` — verify locally staged JVM RC artefacts: POM ALv2 licence / developers / scm (with parent-POM inheritance resolution), podling incubation disclaimer in `<description>`, companion `-sources.jar` / `-javadoc.jar` presence with their own signatures and checksums; used by `release-verify-rc` Step 6b |
+| [`tools/release-verify`](../tools/release-verify/) | `substrate:release` | `release-verify` — deterministic, read-only checks of a staged RC (inventory, signatures against the project KEYS only, checksums, NOTICE/LICENSE, prohibited binaries, dangling symlinks, version strings) and the verdict roll-up, each as JSON; used by `release-verify-rc` Steps 1–3, 5–8 and 10 |
| [`tools/symlink-lint`](../tools/symlink-lint/) | `substrate:framework-dev` | Self-adoption symlink hygiene — rejects cyclic symlinks, misdirected skill relays (canonical/relay target-correctness), and incomplete self-adoption symlink sets |
| [`tools/pilot-report-validator`](../tools/pilot-report-validator/) | `substrate:framework-dev` | Adopter pilot-report validator — required frontmatter keys, no unfilled placeholders, valid profile, and required body sections; counterpart to `spec-validator` for `docs/pilot-report-template.md` |
| [`tools/skill-reconciler-diff`](../tools/skill-reconciler-diff/) | `substrate:framework-dev` | Deterministic structural diff between two skill trees — parses frontmatter, section headings, step inventory, placeholders, support files, and safety-baseline clauses into a JSON diff object for the `skill-reconciler` skill |
diff --git a/docs/vendor-neutrality.md b/docs/vendor-neutrality.md
index c124f957..2169dab5 100644
--- a/docs/vendor-neutrality.md
+++ b/docs/vendor-neutrality.md
@@ -603,7 +603,7 @@ Organization scope (declared, orthogonal to vendor): ASF = 16, agnostic = 62.
**LLM / agent-integration neutrality**
-**Agent harness: 31/31 substrate tools run under any harness unchanged (100%).** Substrate tools are Magpie's own machinery; each declares the agent harness it integrates with (`**Harness:**`), or `agnostic`. A tool is neutral when it is harness-agnostic or supports two or more harnesses; *coupled* when it targets a single harness.
+**Agent harness: 32/32 substrate tools run under any harness unchanged (100%).** Substrate tools are Magpie's own machinery; each declares the agent harness it integrates with (`**Harness:**`), or `agnostic`. A tool is neutral when it is harness-agnostic or supports two or more harnesses; *coupled* when it targets a single harness.
| Substrate tool | Substrate | Harness support | Verdict |
|---|---|---|---|
@@ -623,6 +623,7 @@ Organization scope (declared, orthogonal to vendor): ASF = 16, agnostic = 62.
| `privacy-llm` | privacy | any | ✅ agnostic |
| `probe-templates` | sandbox | any | ✅ agnostic |
| `release-config` | release | any | ✅ agnostic |
+| `release-verify` | release | any | ✅ agnostic |
| `reproducible-archive` | release | any | ✅ agnostic |
| `sandbox-lint` | sandbox | Claude Code, Codex, Cursor, Gemini CLI, Kiro, OpenCode | ✅ portable |
| `security-tracker-stats-dashboard` | analytics | any | ✅ agnostic |
@@ -647,7 +648,7 @@ Harness → substrate tools it supports:
- **Gemini CLI** (3): `agent-guard`, `sandbox-lint`, `spec-loop`
- **Kiro** (3): `agent-guard`, `sandbox-lint`, `spec-loop`
- **OpenCode** (3): `agent-guard`, `sandbox-lint`, `spec-loop`
-- **any harness** (28): `adversarial-review`, `agent-isolation`, `container-gateway`, `contributor-metrics`, `dashboard-generator`, `dev`, `egress-gateway`, `maven-artifact-verify`, `permission-audit`, `pilot-report-validator`, `pr-management-stats`, `preflight-audit`, `privacy-llm`, `probe-templates`, `release-config`, `reproducible-archive`, `security-tracker-stats-dashboard`, `setup-preflight`, `skill-and-tool-validator`, `skill-evals`, `skill-reconciler-diff`, `skill-token-count`, `s [...]
+- **any harness** (29): `adversarial-review`, `agent-isolation`, `container-gateway`, `contributor-metrics`, `dashboard-generator`, `dev`, `egress-gateway`, `maven-artifact-verify`, `permission-audit`, `pilot-report-validator`, `pr-management-stats`, `preflight-audit`, `privacy-llm`, `probe-templates`, `release-config`, `release-verify`, `reproducible-archive`, `security-tracker-stats-dashboard`, `setup-preflight`, `skill-and-tool-validator`, `skill-evals`, `skill-reconciler-diff`, `skil [...]
**Model endpoint: neutral by construction — 4 default-approved endpoint classes across independent trust domains, plus adopter opt-in.** From the [`privacy-llm` registry](../tools/privacy-llm/models.md): the framework keys approval on *endpoint identity*, not on who hosts the model, so no single LLM vendor is privileged.
diff --git a/plugins/magpie-release-management/skills/verify-rc/SKILL.md b/plugins/magpie-release-management/skills/verify-rc/SKILL.md
index 153456a1..159d1a0f 100644
--- a/plugins/magpie-release-management/skills/verify-rc/SKILL.md
+++ b/plugins/magpie-release-management/skills/verify-rc/SKILL.md
@@ -37,7 +37,7 @@ argument-hint: "<version>-rcN [--post-to <planning-issue-url>] [--skip-repro] [-
capability: capability:triage
surface_hash: sha256:ed944a58facaae21
license: Apache-2.0
-measured_tokens: 12622
+measured_tokens: 12585
---
<!-- SPDX-License-Identifier: Apache-2.0
@@ -198,6 +198,10 @@ override file. Framework changes go via PR to
- **Network reachable** — the staging URL and the `KEYS` file URL
must be fetchable. If either is unreachable, the skill stops at
the inventory step and reports `FAIL` with the URL that failed.
+- **`gpg` and Python 3.11+** — Steps 1–3, 5–8 and 10 run
+ [`tools/release-verify`](../../../../tools/release-verify/README.md)
+ (stdlib only; without `uv`, run
+ `python3 <framework>/tools/release-verify/src/release_verify/__init__.py`).
- **A clone of `<upstream>` reachable** for Step 9 — the resolved
`user.md` local clone path, or a fresh `git clone` the recipe emits.
The rebuild happens in the voter's checkout, on the voter's machine.
@@ -268,147 +272,90 @@ passed; `null` otherwise.
## Step 1 — Fetch RC inventory
-Fetch the directory listing of `staging_url` (derived in Step 0).
-Match the listing against the expected artefact list from
-`release-build.md`.
+Download `staging_url` (derived in Step 0) into a scratch directory
+`<staging-copy>` (on dist.apache.org,
+`svn export <staging_url> <staging-copy>`) and `keys_file_url` into a
+file `<keys-file>` beside it. If either download fails, stop: the step
+is `FAIL`, reported with the URL that failed.
+
+Match the copy against the expected artefact list with the
+`release-verify` tool (`<framework>` is `.apache-magpie` in an adopting
+project, `.` in the framework checkout): one `--expect` per entry of
+`release-build.md § Expected artefact list` marked `required` (or
+unmarked), one `--expect-optional` per entry marked `optional`
+(filename pattern, `<version>` substituted, in the listed order), and
+one `--digest` per entry of `§ Digest set`. Pass the same pattern
+options to Steps 2 and 3.
-Classify each expected artefact as:
-
-- `FOUND` — present in the listing.
-- `MISSING` — absent from the listing.
-
-Classify each listing entry as:
-
-- `EXPECTED` — matches a pattern in the expected artefact list.
-- `UNEXPECTED` — not matched; surface for the RM to review.
+```bash
+uv run --project <framework>/tools/release-verify release-verify inventory \
+ --dir <staging-copy> --expect "<artefact-pattern>" … [--expect-optional "<artefact-pattern>"]… \
+ --digest sha512 [--digest sha256]
+```
-If any required artefact is `MISSING`, the overall classification for
-this step is `FAIL`. If `UNEXPECTED` entries appear, the classification
-is `WARN`.
+`status` is `FAIL` when a required artefact is `missing`; `WARN` when
+only an optional artefact is missing (`missing_optional`) or
+`unexpected` entries appear (surface them for the RM to review); else
+`PASS`.
-Return ONLY valid JSON with this structure:
-
-```json
-{
- "step": "inventory",
- "status": "PASS" | "WARN" | "FAIL",
- "found": ["<filename>"],
- "missing": ["<filename>"],
- "unexpected": ["<filename>"]
-}
-```
+Return ONLY the tool's JSON (`step`, `status`, `found`, `missing`,
+`missing_optional`, `unexpected`).
---
## Step 2 — Verify GPG signatures
-For each source artefact (and any convenience binary) listed as
-`FOUND` in Step 1, verify its `.asc` detached signature against the
-public keys in the project `KEYS` file.
-
-Emit the paste-ready shell recipe the voter or RM can run on their own
-machine:
+Verify the `.asc` signature of every artefact `FOUND` in Step 1 against
+the project `KEYS` file. The tool imports `KEYS` into a throwaway
+GNUPGHOME, never the user's keyring, and refuses one holding private-key
+material: then stop and flag it (golden rule 5).
```bash
-# Import project keys
-curl -s <keys-url> | gpg --import
-
-# Verify each artefact
-gpg --verify <artefact>.asc <artefact>
-```
-
-The `paste_recipe` must be directly runnable: resolve every placeholder
-to a concrete value before emitting it. Substitute `<keys-url>` with the
-project KEYS URL from `<project-config>/release-management-config.md` and
-`<artefact>` with each real artefact filename. Never leave a bracketed
-placeholder such as `<keys-url>` or `<artefact>` in the recipe.
-
-Classify each artefact as:
-
-- `PASS` — `gpg --verify` exits 0 and the signing key appears in the
- project `KEYS` file.
-- `KEY-NOT-IN-KEYS` — `gpg --verify` exits 0 but the signing
- fingerprint does not appear in `KEYS`.
-- `FAIL` — `gpg --verify` exits non-zero (bad or missing signature).
-
-`KEY-NOT-IN-KEYS` is a hard `FAIL` for the step: a key not in the
-project's trust anchor is treated equivalently to a bad signature. The
-RM must add the key via `release-keys-sync` (proposed) before
-proceeding.
-
-Return ONLY valid JSON with this structure:
-
-```json
-{
- "step": "signatures",
- "status": "PASS" | "FAIL",
- "results": [
- {
- "file": "<artefact filename>",
- "sig_file": "<artefact>.asc",
- "classification": "PASS" | "KEY-NOT-IN-KEYS" | "FAIL",
- "fingerprint": "<key fingerprint or null>",
- "key_in_keys": true | false
- }
- ],
- "paste_recipe": "<multi-line shell commands>"
-}
+uv run --project <framework>/tools/release-verify release-verify signatures \
+ --dir <staging-copy> --expect "<artefact-pattern>" … [--expect-optional "<artefact-pattern>"]… \
+ --keys <keys-file> --keys-url "<keys_file_url>" [--extra-key <signer-public-key-file>]
```
-`status` is `"FAIL"` if any `classification` is not `"PASS"`.
+Each `classification` is `PASS` (good signature, key in `KEYS`),
+`KEY-NOT-IN-KEYS` (good signature, key not in `KEYS`) or `FAIL` (bad or
+missing signature, or one made by a revoked or expired key, or an expired
+signature; `detail` says which). When `detail` names a key
+absent from `KEYS`, fetch that public key from `<keyserver>` and re-run
+with `--extra-key` to tell the two apart; it is never a trust anchor.
+Anything but `PASS` fails the step: a key outside the project's trust
+anchor counts as a bad signature, and the RM adds it via `release-keys-sync`
+(proposed). `paste_recipe` is the voter's own-machine recipe, fully
+resolved; pass it through unchanged.
+
+Return ONLY the tool's JSON (`step`, `status`, `results[]` with `file`,
+`sig_file`, `classification`, `fingerprint`, `key_in_keys`, and
+`paste_recipe`).
---
## Step 3 — Verify checksums
-For each artefact, verify every digest file (`.sha512`, `.sha256`)
-listed in the digest set from `release-build.md`.
-
-Emit the paste-ready verification recipe:
+Verify the digests of every staged artefact, one `--digest` per entry of
+`release-build.md § Digest set`:
```bash
-# sha512 example
-sha512sum --check <artefact>.sha512
-
-# sha256 example (when published)
-sha256sum --check <artefact>.sha256
+uv run --project <framework>/tools/release-verify release-verify checksums \
+ --dir <staging-copy> --expect "<artefact-pattern>" … [--expect-optional "<artefact-pattern>"]… \
+ --digest sha512 [--digest sha256]
```
-Note: `md5` digests are no longer accepted per ASF infrastructure
-guidance. If a `.md5` file appears in the staging directory, report it
-as `WARN` (deprecated digest present) but do not fail the step solely
-on that basis.
-
-Classify each artefact–digest pair as:
-
-- `PASS` — digest matches.
-- `MISMATCH` — digest does not match.
-- `MISSING-DIGEST` — digest file absent for a required digest type.
+Each artefact–digest pair is `PASS`, `MISMATCH` or `MISSING-DIGEST`.
+Only `sha512` is required: a missing `.sha512` is `MISSING-DIGEST` and
+`FAIL`s the step. Every other digest (`sha256`, …) is optional — checked
+when its file is staged, not listed when it is not — and a `MISMATCH` on
+one still `FAIL`s. md5 never fails alone: `md5` is no longer accepted per
+ASF infrastructure guidance, so a `.md5` file sets
+`deprecated_md5_present` and makes the step `WARN`, even when its digest
+mismatches. Pass `paste_recipe` through unchanged.
-Return ONLY valid JSON with this structure:
-
-```json
-{
- "step": "checksums",
- "status": "PASS" | "WARN" | "FAIL",
- "results": [
- {
- "file": "<artefact filename>",
- "digests": [
- {
- "type": "sha512" | "sha256" | "md5",
- "classification": "PASS" | "MISMATCH" | "MISSING-DIGEST"
- }
- ]
- }
- ],
- "deprecated_md5_present": true | false,
- "paste_recipe": "<multi-line shell commands>"
-}
-```
-
-`status` is `"FAIL"` if any `MISMATCH` or required `MISSING-DIGEST`
-appears. `status` is `"WARN"` if only deprecated `md5` is the anomaly.
+Return ONLY the tool's JSON (`step`, `status`, `results[]` with `file`
+and `digests[]` of `type` and `classification`,
+`deprecated_md5_present`, `paste_recipe`).
---
@@ -457,111 +404,68 @@ When `classification` is `"SKIP"`, `status` is `"WARN"` and
## Step 5 — NOTICE / LICENSE presence and diff
-Unpack the source artefact (or read its directory listing) and verify:
-
-1. A `NOTICE` file exists at the root.
-2. A `LICENSE` file exists at the root.
-3. If a previous promoted release exists in `dist/release/<project>/` (svnpubsub; see `release_dist_backend`),
- fetch its `NOTICE` and `LICENSE` and produce a diff against the
- current RC's files.
-
-Surface the diff to the RM for review. Material changes to `NOTICE`
-(e.g. added or removed third-party attributions) or `LICENSE` (e.g.
-added or removed full licence texts) are classified `WARN` — they
-require RM review before the vote opens, but do not hard-block the RC
-by themselves.
+Unpack the source artefact into `<unpacked-dir>`. If a previous promoted
+release exists in `dist/release/<project>/` (svnpubsub; see
+`release_dist_backend`), fetch its `NOTICE` and `LICENSE` into
+`<previous-dir>`.
-Return ONLY valid JSON with this structure:
-
-```json
-{
- "step": "notice-license",
- "status": "PASS" | "WARN" | "FAIL",
- "notice_present": true | false,
- "license_present": true | false,
- "notice_diff_lines": <integer | null>,
- "license_diff_lines": <integer | null>,
- "diff_summary": "<one-line description of changes or 'no diff — no previous release found' or 'no changes'>"
-}
+```bash
+uv run --project <framework>/tools/release-verify release-verify notice-license \
+ --tree <unpacked-dir> [--previous <previous-dir>]
```
-`status` is `"FAIL"` if either file is absent. `status` is `"WARN"` if
-both files are present but the diff shows material changes. `status` is
-`"PASS"` when both files are present and the diff is empty or trivially
-small (version-string-only changes).
+The tool's `status` is `FAIL` when either file is absent from the root
+of the current RC artefact — `notice_present` / `license_present` is
+`false` and `detail` names the missing file. That is a defect of this
+RC, whatever any previous release contains: the diff counts are `null`
+because there is nothing to diff, not because a previous release is
+missing. It is `PASS` when both are present and there is no previous
+release or no change. `REVIEW` means a diff exists and the call is
+yours: read `notice_diff` / `license_diff`, surface them to the RM, and
+decide.
+
+- `PASS` — version-string-only or trivially small changes.
+- `WARN` — material changes to `NOTICE` (added or removed third-party
+ attributions) or `LICENSE` (added or removed full licence texts). They
+ require RM review before the vote opens, but do not hard-block the RC
+ by themselves.
+
+Return ONLY valid JSON: the tool's `step`, `status` (with `REVIEW`
+resolved to `PASS` or `WARN`), `notice_present`, `license_present`,
+`notice_diff_lines`, `license_diff_lines`, plus `diff_summary` — a
+one-line description of the changes, or `"no diff — no previous release
+found"`, or `"no changes"`; on `FAIL`, which file the RC artefact lacks
+(e.g. `"NOTICE absent from the RC artefact root"`).
---
## Step 6 — Binary exclusion check
-Scan the unpacked source artefact for prohibited binaries. The
-paste-ready `find` always starts from a **fixed baseline** of
-patterns; it is not generated wholesale from adopter config.
-
-Using the Binary-exclude list from `release-build.md` (heading
-**Binary-exclude list** — same file Step 4 reads for RAT
-configuration), append any additional globs that list names beyond
-the baseline, then emit the recipe:
-
-**Baseline (always scanned):** `.class`, `.jar`, `.so`, `.dylib`,
-`.dll`, `.exe`, `.pyc`, and `__pycache__` directories.
-
-**Additions from `release-build.md`:** any extra globs under
-**Binary-exclude list** that the baseline does not already cover
-(for example `*.min.js` or `assets/vendor/**/*.min.js`). Translate
-each into a `-name` or `-path` predicate and OR it into the `find`
-below before emitting.
+Scan the unpacked source artefact for prohibited binaries: the tool's
+fixed baseline (`.class`, `.jar`, `.so`, `.dylib`, `.dll`, `.exe`,
+`.pyc`, `__pycache__`) plus `release-build.md § Binary-exclude list`,
+each additional prohibited glob as `--prohibit`, each known-and-accepted
+exception as `--accept`.
```bash
-# Fixed baseline. `.pyc` / `__pycache__` must NEVER appear in a
-# source release — their presence proves the artefact was zipped from
-# a working tree that had run tests rather than exported clean from
-# the tag (build via `git archive <tag>`, never `zip -r`).
-# Append -name / -path OR-predicates for any extra globs named under
-# <project-config>/release-build.md § Binary-exclude list that the
-# baseline does not already cover.
-find <unpacked-dir> \( -type f \( -name "*.class" -o -name "*.jar" \
- -o -name "*.so" -o -name "*.dylib" -o -name "*.dll" -o -name "*.exe" \
- -o -name "*.pyc" \) -o -type d -name "__pycache__" \) -print
+uv run --project <framework>/tools/release-verify release-verify binaries \
+ --tree <unpacked-dir> [--prohibit "<glob>"]… [--accept "<glob>"]…
```
-Emit the bare `find` with no `grep` post-filtering: the recipe must
-surface every matching file so nothing is hidden from the voter. Do
-not drop baseline predicates when the Binary-exclude list is empty or
-only restates the baseline — the baseline is mandatory.
-
-`<unpacked-dir>` is the source artefact filename with its archive
-extension removed: `<artefact-source-release>.tar.gz` unpacks
-to `<artefact-source-release>`. Do not drop the
-`-source-release` suffix or substitute a shortened name. Resolve
-`<unpacked-dir>` to this concrete directory before emitting the recipe.
+`<unpacked-dir>` is the source artefact filename without its archive
+extension (`<artefact-source-release>.tar.gz` →
+`<artefact-source-release>`; keep the `-source-release` suffix).
-The same Binary-exclude list is then applied in the JSON
-classification below. A found path the list marks as a
-known-and-accepted binary is `EXPECTED-BINARY` (`expected_binaries`);
-any other baseline or addition hit is `PROHIBITED-BINARY`
-(`prohibited_found`). Classification does not filter the command.
-
-A file that matches a prohibited pattern but is NOT marked
-known-and-accepted in the Binary-exclude list is classified
-`PROHIBITED-BINARY` and causes a hard `FAIL`.
-
-Return ONLY valid JSON with this structure:
+`expected_binaries` are the known-and-accepted hits; any path in
+`prohibited_found` is a hard `FAIL`. A `.pyc` or `__pycache__` is never
+accepted: it proves the tarball was zipped from a working tree that ran
+tests rather than exported clean from the tag (build via
+`git archive <tag>`, never `zip -r`). `paste_recipe` is the voter's bare
+`find`, baseline included and nothing filtered; pass it through
+unchanged.
-```json
-{
- "step": "binary-exclusion",
- "status": "PASS" | "FAIL",
- "prohibited_found": ["<path>"],
- "expected_binaries": ["<path>"],
- "paste_recipe": "<multi-line shell commands>"
-}
-```
-
-`status` is `"FAIL"` if `prohibited_found` is non-empty. Any `.pyc`
-file or `__pycache__` directory found is a hard `FAIL` (never an
-`EXPECTED-BINARY`): it is both a prohibited binary and proof the
-tarball was not exported clean from the tag.
+Return ONLY the tool's JSON (`step`, `status`, `prohibited_found`,
+`expected_binaries`, `paste_recipe`).
---
@@ -668,47 +572,35 @@ becomes `FAIL` when `release-build.md § JVM artefact checks` declares
## Step 7 — Source-tree integrity (dangling symlinks + broken references)
-A source archive can be signed, checksummed and licence-clean and
-still be broken: a committed symlink whose target was stripped by
-`export-ignore`, or a shipped file that links to a path the release
-no longer contains. The framework's own first RC failed on exactly
-this (relay symlinks into a stripped directory, docs linking stripped
-templates), so this step runs the project's own integrity checks
-**against the unpacked archive**, where a packaging regression fails
-the RC before the `[VOTE]` rather than during it.
-
-Read `source_tree_validators` from
-`<project-config>/release-build.md § Source-tree validators` — the
-project's own commands, run from the unpacked directory (the adopter
-chooses them; the framework does not assume any). Emit:
+A signed, checksummed, licence-clean archive can still be broken: a
+symlink whose target `export-ignore` stripped, a symlink pointing out of
+the archive, or a shipped file linking to a path the release no longer
+contains. Catch that **in the unpacked
+archive**, before the `[VOTE]`. Pass each command of
+`release-build.md § Source-tree validators` (the adopter's own; the
+framework assumes none) as `--validator`:
```bash
-cd <unpacked-dir>
-
-# 1. Dangling symlinks — every symlink must resolve inside the archive.
-find . -type l ! -exec test -e {} \; -print # any output = FAIL
-
-# 2. Internal reference / link integrity — the project's own validators
-# from release-build.md § Source-tree validators, one per line:
-<source_tree_validators[0]>
-<source_tree_validators[1]>
+uv run --project <framework>/tools/release-verify release-verify symlinks \
+ --tree <unpacked-dir> [--validator "<source_tree_validators[i]>"]…
```
-Classify:
-
-- `PASS` — no dangling symlinks and every validator exits 0.
-- `FAIL` — any dangling symlink, or any validator reports a broken
- internal link / missing referenced file. This is a hard `FAIL`:
- a release whose own files reference content that was stripped from
- the artefact is incomplete.
-- `SKIP` — the project ships no symlinks and declares no validators
- (state this explicitly; do not silently pass — the dangling-symlink
- scan still runs whenever the archive contains a symlink).
-
-Do **not** post-filter the `find`; surface every dangling link so the
-voter sees the full set. When a validator is not shippable in the
-tarball, run it from a checkout of the *same tag* against the unpacked
-dir instead, and note that in the report.
+Every symlink must resolve to an existing path inside the unpacked
+archive. The tool lists each one whose target does not exist in
+`dangling_symlinks`, and each one that resolves outside the archive (an
+absolute path, `..` past the root, or a chain through either) in
+`outside_symlinks`, even when that target exists. It puts the validators in
+`paste_recipe` without running them; run each from `<unpacked-dir>` (or,
+when it is not shippable, from a checkout of the *same tag* against the
+unpacked dir, and say so). Read the tool's `status`:
+
+- `FAIL` — a dangling symlink or one resolving outside the archive; final.
+- `REVIEW` — yours to resolve: `PASS` when every validator exits 0,
+ `FAIL` when any reports a broken internal link or missing referenced
+ file (one `validator_failures` entry each).
+- `PASS` — every symlink resolves inside the archive and no validators
+ are declared.
+- `SKIP` — no symlinks and no validators; state this explicitly.
Return ONLY valid JSON with this structure:
@@ -717,49 +609,33 @@ Return ONLY valid JSON with this structure:
"step": "source-tree-integrity",
"status": "PASS" | "FAIL" | "SKIP",
"dangling_symlinks": ["<path>"],
+ "outside_symlinks": ["<path>"],
"validator_failures": [
{"validator": "<name>", "detail": "<broken link / missing target>"}
],
- "paste_recipe": "<multi-line shell commands>"
+ "paste_recipe": "<the tool's paste_recipe>"
}
```
-`status` is `"FAIL"` if `dangling_symlinks` is non-empty or any
-validator failed.
-
---
## Step 8 — Version string consistency
-Read each file listed in `version_manifest_files` from
-`release-management-config.md` (e.g. `setup.cfg`,
-`airflow/__init__.py`, `pom.xml`). Extract the version string from
-each file using the canonical extraction pattern for that file type.
-
-Compare every extracted version against the `<version>` from the RC
-tag (without the `-rcN` suffix). An exact string match is required.
-Any deviation (wrong version, dev suffix present, snapshot suffix
-present) is a hard `FAIL`.
-
-Return ONLY valid JSON with this structure:
+Check the version in every file of `version_manifest_files` from
+`release-management-config.md`, one `--manifest` each:
-```json
-{
- "step": "version-consistency",
- "status": "PASS" | "FAIL",
- "expected_version": "<version>",
- "results": [
- {
- "file": "<manifest file path>",
- "extracted": "<version string found or null>",
- "match": true | false
- }
- ]
-}
+```bash
+uv run --project <framework>/tools/release-verify release-verify version \
+ --tree <unpacked-dir> --rc-tag <version>-rcN --manifest <file> …
```
-`status` is `"FAIL"` if any `match` is `false` or any `extracted` is
-`null`.
+For a file type the tool has no canonical pattern for, `extracted` is
+`null` and `detail` says so: re-run with `--manifest <file>=<regex>`
+(first group = the version). Exact match only: a wrong version, a dev or
+snapshot suffix, or a `null` extraction is a hard `FAIL`.
+
+Return ONLY the tool's JSON (`step`, `status`, `expected_version`,
+`results[]` with `file`, `extracted`, `match`).
---
@@ -911,12 +787,24 @@ the computed value so the RM can add it.
Aggregate the per-step results into a final report.
-**Overall classification rules:**
+**Overall verdict.** Compute it with the tool, not by hand. Pass the
+JSON result of every step, Step 6b's included when it ran, plus the status
+of each step the tool does not decide: Step 4, Step 9, and any `REVIEW`
+resolved in Steps 5 and 7.
+
+```bash
+uv run --project <framework>/tools/release-verify release-verify verdict <step-result>.json … \
+ --status rat-license-headers=<status> --status reproducibility=<status> \
+ [--status notice-license=<PASS|WARN>] [--status source-tree-integrity=<PASS|FAIL>]
+```
-- `FAIL` — any step that itself classifies as `FAIL`.
-- `PASS-WITH-WARNINGS` — no `FAIL` steps, but one or more `WARN`
- steps.
-- `PASS` — all steps are `PASS`.
+`overall` is `FAIL` if any step fails, else `PASS-WITH-WARNINGS` if any
+warns, else `PASS`. `SKIP` is neutral — it neither passes nor warns —
+and `skip_steps` lists every skipped step; name each one in the report.
+A tool-computed status is final (`ignored_overrides`
+lists attempts to change one); `overall: null` means a step is still
+`unresolved`. `release-verify all` runs Steps 1–3, 5–8 and this roll-up
+in one call with the same options.
**Report sections:**
@@ -929,7 +817,8 @@ Aggregate the per-step results into a final report.
> test the candidate on their own hardware before posting a binding
> `+1`.*
3. **Per-step summary table** — one row per step with status
- (`PASS` / `WARN` / `FAIL` / `SKIP`) and a one-line finding.
+ (`PASS` / `WARN` / `FAIL` / `SKIP`) and a one-line finding; every
+ step in `skip_steps` appears with the reason it was skipped.
4. **FAIL detail** — for each failing step, the exact file or check
that failed and the RM remediation action.
5. **WARN detail** — for each warning step, the observation and the
@@ -1021,9 +910,12 @@ the RM has not yet confirmed posting.
|---|---|---|
| Pre-flight blocked — config key missing | `release-management-config.md` or `release-build.md` lacks a required key | Add the missing key per the adopter scaffold |
| Step 1 FAIL — artefact missing | RC was staged incompletely | RM re-stages the missing artefact |
+| Step 1 WARN — optional artefact missing | An artefact `release-build.md` marks `optional` was not staged | RM confirms the omission is intended, or stages it |
| Step 2 FAIL — bad signature | Artefact was corrupted or signed with wrong key | RM re-signs and re-stages |
| Step 2 FAIL — key not in KEYS | Signing key not yet published | RM adds key via `release-keys-sync` (proposed) |
-| Step 3 FAIL — checksum mismatch | Artefact was corrupted or digest file is wrong | RM regenerates artefact + digest files |
+| Step 3 FAIL — checksum mismatch | Artefact was corrupted or a sha512 / sha256 digest file is wrong | RM regenerates artefact + digest files |
+| Step 3 FAIL — `.sha512` missing | Required sha512 digest not staged | RM generates and stages the `.sha512` file |
+| Step 3 WARN — `.md5` present | Deprecated md5 digest staged (matching or not) | RM drops the `.md5` file for the next RC |
| Step 4 WARN — RAT config absent | `release-build.md` has no RAT config section | RM adds RAT config; do not proceed to vote without it |
| Step 4 FAIL — unapproved headers | Source file missing or incorrect licence header | RM fixes headers and cuts a new RC |
| Step 5 FAIL — NOTICE or LICENSE absent | Source artefact build skipped packaging | RM fixes build process and cuts a new RC |
@@ -1036,6 +928,7 @@ the RM has not yet confirmed posting.
| Step 6b WARN — `INHERITED-UNVERIFIED` | POM element inherited from a parent POM that is not staged locally | Verify against the effective POM (`mvn help:effective-pom`); if correct, no action |
| Step 6b FAIL — jar absent but `jvm_companion_location: staged` | The RC was expected to stage its jars locally and did not | RM re-stages the jar set or corrects `release-build.md` |
| Step 7 FAIL — dangling symlink | A committed symlink's target was stripped by `export-ignore` (or is otherwise absent) | RM fixes `.gitattributes` to ship the target (or drops the symlink), cuts new RC |
+| Step 7 FAIL — symlink resolves outside the archive | A committed symlink is absolute or climbs out of the tree with `..` | RM replaces it with a relative link to shipped content (or drops it), cuts new RC |
| Step 7 FAIL — broken internal reference | A shipped file links to a path stripped from the artefact | RM stops stripping the referenced path, or repoints the reference at shipped content, cuts new RC |
| Step 8 FAIL — version mismatch | Version bump missed one manifest file | RM fixes the manifest and cuts a new RC |
| Step 9 WARN — `content-identical` | RM built with a plain `git archive` or a different tool version instead of `repro-archive build` | Accept for this RC in RM-key mode; RM switches to `repro-archive build` for the next one. Under automated signing this is `FAIL` |
@@ -1062,6 +955,8 @@ the RM has not yet confirmed posting.
- [`docs/release-management/reproducibility.md`](../../../../docs/release-management/reproducibility.md) —
Step 9 background: the source-archive contract, the reproducibility
checks, and the 🪶 ASF-specific automated-signing validation.
+- [`tools/release-verify`](../../../../tools/release-verify/README.md) —
+ the Steps 1–3, 5–8 and 10 checks and their JSON.
- [`tools/reproducible-archive`](../../../../tools/reproducible-archive/README.md) —
`repro-archive check` / `build` / `compare`.
- [`tools/maven-artifact-verify`](../../../../tools/maven-artifact-verify/README.md) —
diff --git a/plugins/magpie-setup/templates/release-build.md b/plugins/magpie-setup/templates/release-build.md
index 9f2a1c47..01e5d966 100644
--- a/plugins/magpie-setup/templates/release-build.md
+++ b/plugins/magpie-setup/templates/release-build.md
@@ -182,7 +182,12 @@ Example shape:
TODO: list the artefacts the build invocation produces and the
release ships. Each entry: filename pattern, content type, whether
-it is the canonical source artefact or a convenience binary.
+it is the canonical source artefact or a convenience binary, and
+`required` or `optional` in its parenthesised attributes.
+`release-verify-rc` Step 1 fails the RC when a `required` artefact
+is not staged and only warns when an `optional` one is not; an entry
+with no marking is `required`. Every staged artefact, of either kind,
+must be signed and checksummed.
Example shape:
@@ -198,9 +203,12 @@ signature regime.
## Digest set
TODO: list which digests the project publishes alongside each
-artefact. ASF baseline is `sha512`; many projects also publish
-`sha256` for older downstream tools. `md5` is no longer accepted
-per ASF infrastructure guidance.
+artefact. ASF baseline is `sha512`, the only digest
+`release-verify-rc` Step 3 requires; many projects also publish
+`sha256` for older downstream tools, which Step 3 checks whenever it
+is staged (a mismatch fails the RC). `md5` is no longer accepted per
+ASF infrastructure guidance: a staged `.md5` is a warning, never a
+failure on its own.
Example shape:
diff --git a/pyproject.toml b/pyproject.toml
index d95a11ea..0459b2e8 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -138,6 +138,7 @@ members = [
"tools/privacy-llm/checker",
"tools/privacy-llm/redactor",
"tools/release-config",
+ "tools/release-verify",
"tools/reproducible-archive",
"tools/sandbox-lint",
"tools/security-tracker-stats-dashboard",
diff --git a/tools/release-verify/README.md b/tools/release-verify/README.md
new file mode 100644
index 00000000..56c184a9
--- /dev/null
+++ b/tools/release-verify/README.md
@@ -0,0 +1,99 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+
+- [`release-verify`](#release-verify)
+ - [Prerequisites](#prerequisites)
+ - [How to use](#how-to-use)
+ - [Output](#output)
+ - [Wiring](#wiring)
+ - [Tests](#tests)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# `release-verify`
+
+**Capability:** substrate:release
+
+**Harness:** agnostic
+
+Deterministic, read-only checks of a staged release candidate, one subcommand per mechanical step of the [`release-verify-rc`](../../plugins/magpie-release-management/skills/verify-rc/SKILL.md) skill.
+Each prints one JSON object; the skill reads it and reports.
+The rules are the skill's, implemented once in code so the model does not re-derive them from prose on every run.
+Judgement stays with the reader: whether a NOTICE / LICENSE diff is material, and whether the project's own source-tree validators passed, come back as status `REVIEW`.
+
+Guarantees: nothing is written outside a throwaway `GNUPGHOME`; the user's keyring is never read or changed; nothing is signed; a `KEYS` file containing private-key material is refused; nothing touches the network — the caller downloads the staged files and `KEYS` first.
+
+## Prerequisites
+
+- **Runtime:** Python 3.11+ (stdlib only); run via `uv run --project tools/release-verify` or as a plain `python3 <framework>/tools/release-verify/src/release_verify/__init__.py`.
+- **CLIs:** `gpg` (GnuPG 2.2+) on `PATH` for `signatures` and `all`; every other subcommand works on local files alone. No `gpg-agent` is needed — verification uses public keys only.
+- **Credentials / auth:** None.
+- **Network:** None — runs fully offline on a local copy of the staging directory, the `KEYS` file and the unpacked source artefact.
+
+## How to use
+
+From the framework root (`<framework>` is `.apache-magpie/` in an adopting project, `.` in the framework checkout).
+`<staging>` is a local copy of the staged RC directory (e.g. `svn export <staging-url> <staging>`), `KEYS` a local copy of the project `KEYS` file, `<tree>` the unpacked source artefact.
+Patterns and paths come from the adopter's `release-build.md` and `release-management-config.md`.
+
+```bash
+RV="uv run --project <framework>/tools/release-verify release-verify"
+
+$RV inventory --dir <staging> --expect '<pattern>'... [--expect-optional '<pattern>']... --digest sha512 [--digest sha256]
+$RV signatures --dir <staging> --expect '<pattern>'... [--expect-optional '<pattern>']... --keys KEYS --keys-url <keys-url> [--extra-key <signer.pub>]
+$RV checksums --dir <staging> --expect '<pattern>'... [--expect-optional '<pattern>']... --digest sha512 [--digest sha256]
+$RV notice-license --tree <tree> [--previous <dir with the previous release's NOTICE and LICENSE>]
+$RV binaries --tree <tree> [--prohibit '<glob>']... [--accept '<glob>']...
+$RV symlinks --tree <tree> [--validator '<command>']...
+$RV version --tree <tree> --rc-tag <version>-rcN --manifest <path>[=<regex>]...
+$RV verdict <step.json>... [--status <step>=PASS|WARN|FAIL|SKIP]...
+$RV all --dir <staging> --tree <tree> <the options above> [--status <step>=<status>]...
+```
+
+`--expect` takes each artefact `release-build.md § Expected artefact list` marks `required` (the default), `--expect-optional` each one it marks `optional`; `signatures` and `checksums` check every staged artefact of either kind.
+`inventory` also takes `--listing` (a file of names, e.g. `svn ls <staging-url>` output) in place of `--dir`. `--keys-url` is only printed in the voter's `paste_recipe`.
+Exit status is `0` whenever a JSON result is printed, whatever the verdict; `2` with `{"error": …}` for unusable input.
+
+### Output
+
+Every step object carries `step`, `status` and the step's fields as the skill's hand-back schema names them, plus a few extra diagnostic fields (marked † below).
+
+| Subcommand | `step` | `status` | Fields |
+|---|---|---|---|
+| `inventory` | `inventory` | `FAIL` a required pattern unmatched · `WARN` an optional pattern unmatched, or an unexpected entry · else `PASS` | `found`, `missing` (required patterns), `missing_optional`†, `unexpected`, `source`† |
+| `signatures` | `signatures` | `PASS` only when every artefact is `PASS`, else `FAIL` | `results[]` (`file`, `sig_file`, `classification` `PASS` / `KEY-NOT-IN-KEYS` / `FAIL`, `fingerprint`, `key_in_keys`, `detail`†), `keys_fingerprints`†, `paste_recipe` |
+| `checksums` | `checksums` | `FAIL` a sha512 `MISSING-DIGEST`, or a `MISMATCH` on any digest but md5 · `WARN` a `.md5` file is present (matching or not) · else `PASS` | `results[].digests[]` (`type`, `classification`, `detail`†), `deprecated_md5_present`, `paste_recipe` |
+| `notice-license` | `notice-license` | `FAIL` either file absent from the RC artefact · `PASS` no previous release, or both diffs empty · else `REVIEW` | `notice_present`, `license_present`, `notice_diff_lines`, `license_diff_lines`, `notice_diff`†, `license_diff`†, `previous`†, `detail`† (on `FAIL`: which file the current RC artefact lacks) |
+| `binaries` | `binary-exclusion` | `FAIL` when `prohibited_found` is non-empty | `prohibited_found`, `expected_binaries`, `paste_recipe` |
+| `symlinks` | `source-tree-integrity` | `FAIL` a symlink that dangles or resolves outside the archive · `REVIEW` validators declared (run them) · `PASS` symlinks, all resolve inside · `SKIP` no symlinks, no validators | `dangling_symlinks`, `outside_symlinks`†, `symlinks_present`†, `validator_failures` (always `[]`), `validators_to_run`†, `paste_recipe` |
+| `version` | `version-consistency` | `FAIL` any `extracted` is `null` or differs from the expected version | `expected_version`, `results[]` (`file`, `extracted`, `match`, `detail`†) |
+| `verdict` | `verdict` | — | `overall` (`FAIL` > `PASS-WITH-WARNINGS` > `PASS`; `null` while a step is unresolved), `step_summary`, `fail_steps`, `warn_steps`, `skip_steps`, `unresolved`, `ignored_overrides`† |
+
+Rules the subcommands apply:
+
+- **Signatures.** The `KEYS` file is imported into a temporary `GNUPGHOME` with no agent, keyboxd or key retrieval. `PASS` when `gpg --verify` exits 0 and the signing key (primary or subkey fingerprint) is in `KEYS`. `KEY-NOT-IN-KEYS` when the signature is good but the key is not in `KEYS`; telling that apart needs the signer's public key, passed with `--extra-key` (never a trust anchor). Without it such a signature cannot be checked and is `FAIL` with `detail` naming the missing key. A [...]
+- **Inventory.** A missing required artefact is `FAIL`; a missing optional one is `WARN` (listed in `missing_optional`), never `FAIL`.
+- **Checksums.** Only `sha512` is required: a missing `.sha512` is `MISSING-DIGEST`, a `FAIL`. Every other digest (`sha256`, whether or not passed as `--digest`) is optional — checked when its file is staged, absent otherwise without effect — and a `MISMATCH` on one still `FAIL`s. md5 never fails alone: a `.md5` file, matching or mismatched, is listed and makes the step `WARN`. Digest files in `sha512sum`, BSD-tag (`SHA512 (f) = …`) and `gpg --print-md` layouts are read; an unparsable on [...]
+- **Binaries.** Fixed baseline `*.class`, `*.jar`, `*.so`, `*.dylib`, `*.dll`, `*.exe`, `*.pyc` (files) and `__pycache__` (directories), plus every `--prohibit` glob. A hit matching an `--accept` glob is `expected_binaries`, any other is `prohibited_found`; `.pyc` and `__pycache__` are always prohibited. A glob without `/` matches the file name (`find -name`), one with `/` the path inside the tree (`find -path`, `**` = `*`).
+- **Notice / licence.** On `FAIL`, `detail` names the file the current RC artefact lacks; the diff counts are `null` because there is nothing to diff, not because a previous release is missing.
+- **Symlinks.** Every symlink must resolve to an existing path inside the unpacked archive. One whose target does not exist is dangling; one that resolves outside the archive (`..` past the root, an absolute path, or a chain through either) is listed in `outside_symlinks` and also `FAIL`s, even when the target exists. `paste_recipe` repeats both checks with `realpath`. Validators are printed in `paste_recipe`, never run.
+- **Version.** `--rc-tag` drops the `-rcN` suffix. Built-in extraction for `setup.cfg` (`[metadata] version`), `pyproject.toml` (`[project]` or `[tool.poetry]`), `*.py` (`__version__ = "…"`; `version="…"` in `setup.py`), `pom.xml` (`<project><version>`), `package.json`, `Cargo.toml`, `*.properties`, `build.gradle(.kts)`, `Chart.yaml`, `VERSION` / `version.txt`. Anything else needs `PATH=REGEX` (first group). Dynamic versions (`attr:`, `dynamic = ["version"]`) are `null`.
+- **Verdict.** A status the tool computed is final; `--status` only fills a step it did not run (`rat-license-headers`, `reproducibility`) or resolves a `REVIEW`. `SKIP` neither passes nor warns, and `skip_steps` lists every skipped step so the report names it.
+
+## Wiring
+
+- `release-verify-rc` Steps 1–3 and 5–8 run the matching subcommand, and Step 10 runs `verdict` (or the skill runs `all` once). Step 4 (Apache RAT) is build-system specific and Step 9 uses [`reproducible-archive`](../reproducible-archive/README.md); both report their status into `verdict` with `--status`.
+- The workspace pre-commit hooks (`ruff`, `mypy`, `pytest`) pick the project up from the root `pyproject.toml` `[tool.uv.workspace] members` list.
+
+## Tests
+
+```bash
+uv run --project tools/release-verify python -m pytest
+```
+
+Every subcommand is exercised on its pass case and on each failure class the skill names.
+The signature tests use static fixtures in `tests/fixtures/gpg/` — public keys and detached signatures only, written by `generate.py` without `gpg-agent` — so they need `gpg` but run in a sandbox; they skip cleanly when `gpg` is absent.
+One further test generates a throwaway key in a temporary `GNUPGHOME` and skips where no agent can start.
diff --git a/tools/release-verify/pyproject.toml b/tools/release-verify/pyproject.toml
new file mode 100644
index 00000000..2d06c424
--- /dev/null
+++ b/tools/release-verify/pyproject.toml
@@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "release-verify"
+version = "0.1.0"
+description = "Deterministic, read-only checks of a staged release candidate (inventory, signatures, checksums, NOTICE/LICENSE, binaries, symlinks, version), each emitting JSON for the release-verify-rc skill."
+readme = "README.md"
+requires-python = ">=3.11"
+license = { text = "Apache-2.0" }
+# stdlib-only on purpose — hashlib, difflib, tomllib and subprocess (gpg)
+# are all it needs, so the module runs as a plain `python3 <file>` too.
+dependencies = []
+
+[project.scripts]
+release-verify = "release_verify:main"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/release_verify"]
+
+[tool.ruff]
+line-length = 160
+target-version = "py311"
+src = ["src", "tests"]
+
+[tool.ruff.lint]
+select = [
+ "E", # pycodestyle errors
+ "W", # pycodestyle warnings
+ "F", # pyflakes
+ "I", # isort
+ "B", # flake8-bugbear
+ "UP", # pyupgrade
+ "SIM", # flake8-simplify
+ "C4", # flake8-comprehensions
+ "RUF", # ruff-specific
+]
+ignore = [
+ "E501", # line-too-long — the limit above is already generous
+]
+
+[tool.ruff.lint.per-file-ignores]
+"tests/**" = ["B", "SIM"]
+
+[tool.mypy]
+python_version = "3.11"
+files = ["src", "tests"]
+warn_unused_ignores = true
+warn_redundant_casts = true
+warn_unreachable = true
+check_untyped_defs = true
+no_implicit_optional = true
+disallow_untyped_defs = true
+disallow_incomplete_defs = true
+
+[[tool.mypy.overrides]]
+module = "tests.*"
+disallow_untyped_defs = false
+disallow_incomplete_defs = false
+
+[tool.pytest.ini_options]
+minversion = "8.0"
+addopts = "-ra -q"
+testpaths = ["tests"]
+
+[dependency-groups]
+# The shared toolchain (mypy, pytest, ruff) comes from `magpie-dev`
+# (tools/dev), declared once for the whole workspace. The checks run each tool
+# via `uv run --directory <member> --project . python -m <tool>` — see
+# tools/dev/run-workspace-check.sh.
+dev = ["magpie-dev"]
diff --git a/tools/release-verify/src/release_verify/__init__.py b/tools/release-verify/src/release_verify/__init__.py
new file mode 100644
index 00000000..0c2222a3
--- /dev/null
+++ b/tools/release-verify/src/release_verify/__init__.py
@@ -0,0 +1,857 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Deterministic checks of a staged release candidate, as JSON.
+
+One subcommand per mechanical step of the `release-verify-rc` skill.
+Every subcommand prints a single JSON object on stdout; the skill reads
+it and reports. The rules are the skill's, implemented once:
+
+- ``inventory`` — Step 1: staging listing vs the expected artefact
+ patterns → FOUND / MISSING / UNEXPECTED; a missing *optional*
+ artefact is a WARN, a missing required one a FAIL.
+- ``signatures`` — Step 2: ``gpg --verify`` in a throwaway GNUPGHOME
+ holding only the project's *public* KEYS → PASS / KEY-NOT-IN-KEYS / FAIL.
+- ``checksums`` — Step 3: digest compare → PASS / MISMATCH /
+ MISSING-DIGEST. Only sha512 is required; other digests are checked
+ when present; md5 never fails alone (present or mismatched → WARN).
+- ``notice-license`` — Step 5: NOTICE / LICENSE presence and diff line
+ counts. Whether a non-empty diff is *material* is left to the reader
+ (status ``REVIEW``).
+- ``binaries`` — Step 6: fixed baseline + configured globs;
+ ``.pyc`` / ``__pycache__`` are always PROHIBITED.
+- ``symlinks`` — Step 7: symlinks in the unpacked tree that dangle or
+ resolve outside it. The
+ project's own validators are echoed in the recipe, never run.
+- ``version`` — Step 8: version string per manifest, exact match.
+- ``verdict`` — Step 10: FAIL > WARN > PASS roll-up of step results.
+- ``all`` — the above in one report.
+
+Read-only throughout: no file outside a temporary GNUPGHOME is written,
+the user's keyring is never touched, nothing is signed, and nothing
+touches the network: every input is a local file. Stdlib only.
+"""
+
+from __future__ import annotations
+
+import argparse
+import configparser
+import contextlib
+import difflib
+import fnmatch
+import hashlib
+import json
+import os
+import re
+import shutil
+import subprocess
+import sys
+import tempfile
+import tomllib
+import xml.etree.ElementTree as ET
+from collections.abc import Iterator, Sequence
+from pathlib import Path
+from typing import Any
+
+STATUSES = ("PASS", "WARN", "FAIL", "SKIP")
+BASELINE_FILE_GLOBS = ("*.class", "*.jar", "*.so", "*.dylib", "*.dll", "*.exe", "*.pyc")
+DIGEST_TYPES = ("sha512", "sha256", "md5")
+REQUIRED_DIGEST = "sha512"
+HEX_LEN = {"sha512": 128, "sha256": 64, "md5": 32}
+PRIVATE_KEY_MARKER = "PRIVATE KEY BLOCK"
+STEP_ORDER = (
+ "inventory",
+ "signatures",
+ "checksums",
+ "rat-license-headers",
+ "notice-license",
+ "binary-exclusion",
+ "jvm-artefacts", # Step 6b, produced by maven-artifact-verify
+ "source-tree-integrity",
+ "version-consistency",
+ "reproducibility",
+)
+
+
+class InputError(Exception):
+ """Bad input: reported as ``{"error": ...}`` with exit code 2."""
+
+
+# ---------------------------------------------------------------------------
+# Shared helpers
+# ---------------------------------------------------------------------------
+
+
+def _match_artefact(name: str, patterns: Sequence[str]) -> bool:
+ return any(fnmatch.fnmatchcase(name, p) for p in patterns)
+
+
+def _found_artefacts(names: Sequence[str], patterns: Sequence[str]) -> list[str]:
+ """Matching names, in the order of the patterns that match them (the config's order)."""
+ found: list[str] = []
+ for pattern in patterns:
+ found += [n for n in sorted(names) if fnmatch.fnmatchcase(n, pattern) and n not in found]
+ return found
+
+
+def _patterns(args: argparse.Namespace) -> list[str]:
+ """Required then optional artefact patterns: every staged artefact is signed and checksummed."""
+ return [*args.expect, *(getattr(args, "expect_optional", None) or [])]
+
+
+def _dir_names(directory: Path) -> list[str]:
+ if not directory.is_dir():
+ raise InputError(f"not a directory: {directory}")
+ return sorted(p.name for p in directory.iterdir() if p.is_file())
+
+
+def _rel(path: Path, top: Path) -> str:
+ return path.relative_to(top).as_posix()
+
+
+def _translate_glob(pattern: str) -> str:
+ """`**/` and `**` collapse to `*`; `fnmatch` and `find -path` let `*` cross `/`."""
+ pattern = pattern.strip().lstrip("/")
+ if pattern.startswith("./"):
+ pattern = pattern[2:]
+ return pattern.replace("**/", "").replace("**", "*")
+
+
+def _glob_hits(rel: str, pattern: str) -> bool:
+ """`find -name` semantics for a bare glob, `find -path` for one with a `/`."""
+ if "/" not in pattern.strip().lstrip("/"):
+ return fnmatch.fnmatchcase(rel.rsplit("/", 1)[-1], pattern.strip())
+ return fnmatch.fnmatchcase(rel, _translate_glob(pattern))
+
+
+def _shq(value: str) -> str:
+ return "'" + value.replace("'", "'\"'\"'") + "'"
+
+
+# ---------------------------------------------------------------------------
+# Step 1 — inventory
+# ---------------------------------------------------------------------------
+
+
+def inventory(names: Sequence[str], patterns: Sequence[str], digests: Sequence[str], source: str, optional: Sequence[str] = ()) -> dict[str, Any]:
+ """Classify a staging listing against the expected artefact patterns.
+
+ ``patterns`` are required artefacts (missing → FAIL); ``optional`` ones
+ are the entries ``release-build.md`` marks optional (missing → WARN).
+ """
+ every = [*patterns, *optional]
+ companions = (".asc", *(f".{d}" for d in dict.fromkeys(["sha512", *digests])))
+ found = _found_artefacts(names, every)
+ missing = [p for p in patterns if not any(fnmatch.fnmatchcase(n, p) for n in names)]
+ missing_optional = [p for p in optional if not any(fnmatch.fnmatchcase(n, p) for n in names)]
+ unexpected = []
+ for name in sorted(names):
+ if name in found:
+ continue
+ base = next((name[: -len(s)] for s in companions if name.endswith(s)), None)
+ if base is not None and _match_artefact(base, every):
+ continue
+ unexpected.append(name)
+ status = "FAIL" if missing else "WARN" if unexpected or missing_optional else "PASS"
+ return {
+ "step": "inventory",
+ "status": status,
+ "found": found,
+ "missing": missing,
+ "missing_optional": missing_optional,
+ "unexpected": unexpected,
+ "source": source,
+ }
+
+
+def cmd_inventory(args: argparse.Namespace) -> dict[str, Any]:
+ if args.dir:
+ return inventory(_dir_names(Path(args.dir)), args.expect, args.digest, str(args.dir), args.expect_optional or [])
+ lines = Path(args.listing).read_text().splitlines()
+ names = [ln.strip().rsplit("/", 1)[-1] for ln in lines if ln.strip() and not ln.strip().endswith("/")]
+ return inventory(names, args.expect, args.digest, str(args.listing), args.expect_optional or [])
+
+
+# ---------------------------------------------------------------------------
+# Step 2 — signatures
+# ---------------------------------------------------------------------------
+
+
+def _read_public_keys(source: str) -> bytes:
+ data = Path(source).read_bytes()
+ if PRIVATE_KEY_MARKER in data.decode("utf-8", "replace"):
+ raise InputError(f"{source} contains private key material; refusing to continue (golden rule 5)")
+ return data
+
+
+@contextlib.contextmanager
+def _gnupghome() -> Iterator[Path]:
+ """A throwaway GNUPGHOME: public keys only, no agent, no keyboxd, no network."""
+ home = Path(tempfile.mkdtemp(prefix="rv-gpg-"))
+ try:
+ home.chmod(0o700)
+ (home / "common.conf").write_text("") # stops gpg >= 2.4.1 from enabling keyboxd
+ (home / "gpg.conf").write_text("no-auto-key-retrieve\nno-auto-key-import\n")
+ yield home
+ finally:
+ shutil.rmtree(home, ignore_errors=True)
+
+
+def _gpg(home: Path, *args: str) -> subprocess.CompletedProcess[str]:
+ gpg = shutil.which("gpg")
+ if gpg is None:
+ raise InputError("gpg not found on PATH")
+ proc = subprocess.run(
+ [gpg, "--batch", "--no-tty", "--no-autostart", "--homedir", str(home), *args],
+ capture_output=True,
+ text=True,
+ check=False,
+ env={**os.environ, "GNUPGHOME": str(home), "LC_ALL": "C"},
+ )
+ # gpg names its home directory in some messages; the report may be posted
+ # to the planning issue, so local paths never reach it.
+ proc.stderr = proc.stderr.replace(str(home), "<gnupghome>")
+ return proc
+
+
+def _import(home: Path, data: bytes, label: str) -> None:
+ src = home / "import.asc"
+ src.write_bytes(data)
+ proc = _gpg(home, "--import", str(src))
+ src.unlink()
+ if "imported:" not in proc.stderr and "unchanged:" not in proc.stderr:
+ raise InputError(f"could not import keys from {label}: {proc.stderr.strip()}")
+
+
+def _fingerprints(home: Path) -> set[str]:
+ proc = _gpg(home, "--with-colons", "--fingerprint", "--fingerprint", "--list-keys")
+ return {line.split(":")[9].upper() for line in proc.stdout.splitlines() if line.startswith("fpr:")}
+
+
+STALE_SIGNATURE_STATUS = {
+ "REVKEYSIG": "signature made by a revoked key",
+ "EXPKEYSIG": "signature made by an expired key",
+ "EXPSIG": "the signature itself has expired",
+}
+
+
+def verify_one(home: Path, artefact: Path, keys_fprs: set[str]) -> dict[str, Any]:
+ sig = artefact.with_name(artefact.name + ".asc")
+ result: dict[str, Any] = {
+ "file": artefact.name,
+ "sig_file": sig.name,
+ "classification": "FAIL",
+ "fingerprint": None,
+ "key_in_keys": False,
+ }
+ if not sig.is_file():
+ result["detail"] = "signature file missing"
+ return result
+ proc = _gpg(home, "--status-fd", "1", "--verify", str(sig), str(artefact))
+ status = [ln[len("[GNUPG:] ") :].split() for ln in proc.stdout.splitlines() if ln.startswith("[GNUPG:] ")]
+ validsig = next((s for s in status if s[0] == "VALIDSIG"), None)
+ # gpg still exits 0 with VALIDSIG for these; the release must not pass on them.
+ stale = next((s[0] for s in status if s[0] in STALE_SIGNATURE_STATUS), None)
+ if stale is not None:
+ result["fingerprint"] = validsig[-1].upper() if validsig else None
+ result["detail"] = STALE_SIGNATURE_STATUS[stale]
+ return result
+ if proc.returncode == 0 and validsig is not None:
+ signing, primary = validsig[1].upper(), validsig[-1].upper()
+ in_keys = signing in keys_fprs or primary in keys_fprs
+ result.update(fingerprint=primary, key_in_keys=in_keys, classification="PASS" if in_keys else "KEY-NOT-IN-KEYS")
+ if not in_keys:
+ result["detail"] = "good signature, but the signing key is not in KEYS"
+ return result
+ tokens = {s[0]: s for s in status}
+ if "BADSIG" in tokens:
+ result["detail"] = "BAD signature"
+ elif "NO_PUBKEY" in tokens:
+ result["detail"] = f"no public key for {tokens['NO_PUBKEY'][1]}: the signing key is not in KEYS"
+ else:
+ lines = [ln for ln in proc.stderr.strip().splitlines() if ln.strip()]
+ result["detail"] = lines[-1] if lines else f"gpg exited {proc.returncode}"
+ return result
+
+
+def signatures(directory: Path, patterns: Sequence[str], keys: str, keys_url: str | None, extra_keys: Sequence[str]) -> dict[str, Any]:
+ artefacts = _found_artefacts(_dir_names(directory), patterns)
+ keys_data = _read_public_keys(keys)
+ extra = [(k, _read_public_keys(k)) for k in extra_keys]
+ with _gnupghome() as home:
+ _import(home, keys_data, keys)
+ keys_fprs = _fingerprints(home)
+ if not keys_fprs:
+ raise InputError(f"no public keys found in {keys}")
+ for label, data in extra:
+ _import(home, data, label)
+ results = [verify_one(home, directory / name, keys_fprs) for name in artefacts]
+ lines = [f"curl -s {_shq(keys_url)} | gpg --import" if keys_url else f"gpg --import {_shq(Path(keys).name)}"]
+ lines += [f"gpg --verify {_shq(r['sig_file'])} {_shq(r['file'])}" for r in results]
+ status = "PASS" if results and all(r["classification"] == "PASS" for r in results) else "FAIL"
+ return {
+ "step": "signatures",
+ "status": status,
+ "results": results,
+ "keys_fingerprints": sorted(keys_fprs),
+ "paste_recipe": "\n".join(lines),
+ }
+
+
+def cmd_signatures(args: argparse.Namespace) -> dict[str, Any]:
+ return signatures(Path(args.dir), _patterns(args), args.keys, args.keys_url, args.extra_key or [])
+
+
+# ---------------------------------------------------------------------------
+# Step 3 — checksums
+# ---------------------------------------------------------------------------
+
+
+def parse_digest_file(text: str, kind: str) -> str | None:
+ """The hex digest in a ``sha512sum``, BSD-tag or ``gpg --print-md`` file."""
+ want = HEX_LEN[kind]
+ bsd = re.search(r"\)\s*=\s*([0-9A-Fa-f]+)\s*$", text.strip())
+ if bsd and len(bsd.group(1)) == want:
+ return bsd.group(1).lower()
+ first = text.strip().split()
+ if first and re.fullmatch(rf"[0-9A-Fa-f]{{{want}}}", first[0]):
+ return first[0].lower()
+ if ":" in text:
+ packed = re.sub(r"\s+", "", text.split(":", 1)[1])
+ if re.fullmatch(rf"[0-9A-Fa-f]{{{want}}}", packed):
+ return packed.lower()
+ return None
+
+
+def _digest_of(path: Path, kind: str) -> str:
+ h = hashlib.new(kind)
+ with path.open("rb") as fh:
+ for chunk in iter(lambda: fh.read(1 << 20), b""):
+ h.update(chunk)
+ return h.hexdigest()
+
+
+def checksums(directory: Path, patterns: Sequence[str], digests: Sequence[str]) -> dict[str, Any]:
+ """Compare digest files with the artefacts.
+
+ ``sha512`` is the only required digest: absent → MISSING-DIGEST, a FAIL.
+ Every other known digest is optional and checked when its file is
+ present; a MISMATCH on one FAILs, except ``md5``, which never fails
+ alone — its presence or its mismatch is a WARN.
+ """
+ names = _dir_names(directory)
+ artefacts = _found_artefacts(names, patterns)
+ md5_present = any(n.endswith(".md5") for n in names)
+ results: list[dict[str, Any]] = []
+ recipe: list[str] = []
+ status = "PASS" if artefacts else "FAIL"
+ for name in artefacts:
+ entries: list[dict[str, Any]] = []
+ present = [k for k in DIGEST_TYPES if (directory / f"{name}.{k}").is_file()]
+ kinds = list(dict.fromkeys([REQUIRED_DIGEST, *(d for d in digests if d in present), *present]))
+ for kind in kinds:
+ dfile = directory / f"{name}.{kind}"
+ if kind != "md5":
+ recipe.append(f"{kind}sum --check {_shq(dfile.name)}")
+ if not dfile.is_file():
+ entries.append({"type": kind, "classification": "MISSING-DIGEST"})
+ status = "FAIL"
+ continue
+ recorded = parse_digest_file(dfile.read_text(errors="replace"), kind)
+ entry: dict[str, Any] = {"type": kind, "classification": "PASS" if recorded == _digest_of(directory / name, kind) else "MISMATCH"}
+ if recorded is None:
+ entry["detail"] = "digest file could not be parsed"
+ if entry["classification"] == "MISMATCH":
+ if kind == "md5":
+ entry["detail"] = entry.get("detail", "md5 mismatch") + "; md5 never fails alone (WARN)"
+ else:
+ status = "FAIL"
+ entries.append(entry)
+ results.append({"file": name, "digests": entries})
+ if status == "PASS" and md5_present:
+ status = "WARN"
+ return {"step": "checksums", "status": status, "results": results, "deprecated_md5_present": md5_present, "paste_recipe": "\n".join(recipe)}
+
+
+def cmd_checksums(args: argparse.Namespace) -> dict[str, Any]:
+ return checksums(Path(args.dir), _patterns(args), args.digest)
+
+
+# ---------------------------------------------------------------------------
+# Step 5 — NOTICE / LICENSE
+# ---------------------------------------------------------------------------
+
+
+def _diff(current: Path, previous: Path, name: str) -> tuple[int, str]:
+ old = previous.read_text(errors="replace").splitlines(keepends=True) if previous.is_file() else []
+ new = current.read_text(errors="replace").splitlines(keepends=True)
+ lines = list(difflib.unified_diff(old, new, fromfile=f"previous/{name}", tofile=f"rc/{name}"))
+ changed = sum(1 for ln in lines if ln[:1] in "+-" and not ln.startswith(("+++", "---")))
+ return changed, "".join(lines)
+
+
+def notice_license(tree: Path, previous: Path | None) -> dict[str, Any]:
+ if not tree.is_dir():
+ raise InputError(f"not a directory: {tree}")
+ out: dict[str, Any] = {"step": "notice-license", "status": None}
+ present = {}
+ for name in ("NOTICE", "LICENSE"):
+ key = name.lower()
+ present[key] = (tree / name).is_file()
+ out[f"{key}_present"] = present[key]
+ for name in ("NOTICE", "LICENSE"):
+ key = name.lower()
+ if previous is not None and present[key]:
+ lines, text = _diff(tree / name, previous / name, name)
+ out[f"{key}_diff_lines"], out[f"{key}_diff"] = lines, text
+ else:
+ out[f"{key}_diff_lines"], out[f"{key}_diff"] = None, None
+ out["previous"] = str(previous) if previous is not None else None
+ if not all(present.values()):
+ out["status"] = "FAIL"
+ absent = " and ".join(n for n in ("NOTICE", "LICENSE") if not present[n.lower()])
+ out["detail"] = (
+ f"{absent} absent from the root of the current RC artefact ({tree.name}); "
+ "this RC is defective whatever any previous release contains. "
+ "Diff line counts are null because there is no file to diff."
+ )
+ elif previous is None or (out["notice_diff_lines"] == 0 and out["license_diff_lines"] == 0):
+ out["status"] = "PASS"
+ else:
+ out["status"] = "REVIEW"
+ return out
+
+
+def cmd_notice_license(args: argparse.Namespace) -> dict[str, Any]:
+ return notice_license(Path(args.tree), Path(args.previous) if args.previous else None)
+
+
+# ---------------------------------------------------------------------------
+# Step 6 — binary exclusion
+# ---------------------------------------------------------------------------
+
+
+def _walk(top: Path) -> Iterator[tuple[Path, list[str], list[str]]]:
+ for root, dirs, files in os.walk(top, followlinks=False):
+ dirs.sort()
+ yield Path(root), dirs, sorted(files)
+
+
+def binary_find_recipe(dirname: str, extra: Sequence[str]) -> str:
+ preds = [f"-name {_shq(g)}" for g in BASELINE_FILE_GLOBS]
+ for glob in extra:
+ g = glob.strip()
+ if g in BASELINE_FILE_GLOBS or not g:
+ continue
+ preds.append(f"-path {_shq(dirname + '/' + _translate_glob(g))}" if "/" in g.lstrip("/") else f"-name {_shq(g)}")
+ return f"find {_shq(dirname)} \\( -type f \\( " + " -o ".join(preds) + " \\) -o -type d -name '__pycache__' \\) -print"
+
+
+def binaries(tree: Path, prohibit: Sequence[str], accept: Sequence[str], recipe_dir: str | None) -> dict[str, Any]:
+ if not tree.is_dir():
+ raise InputError(f"not a directory: {tree}")
+ prohibited: list[str] = []
+ expected: list[str] = []
+ for root, dirs, files in _walk(tree):
+ for d in dirs:
+ if d == "__pycache__" and not (root / d).is_symlink():
+ prohibited.append(_rel(root / d, tree))
+ for f in files:
+ path = root / f
+ if path.is_symlink() or not path.is_file():
+ continue
+ rel = _rel(path, tree)
+ hit_baseline = any(fnmatch.fnmatchcase(f, g) for g in BASELINE_FILE_GLOBS)
+ if not hit_baseline and not any(_glob_hits(rel, g) for g in prohibit):
+ continue
+ if fnmatch.fnmatchcase(f, "*.pyc") or not any(_glob_hits(rel, g) for g in accept):
+ prohibited.append(rel)
+ else:
+ expected.append(rel)
+ return {
+ "step": "binary-exclusion",
+ "status": "FAIL" if prohibited else "PASS",
+ "prohibited_found": sorted(prohibited),
+ "expected_binaries": sorted(expected),
+ "paste_recipe": binary_find_recipe(recipe_dir or tree.name, prohibit),
+ }
+
+
+def cmd_binaries(args: argparse.Namespace) -> dict[str, Any]:
+ return binaries(Path(args.tree), args.prohibit or [], args.accept or [], args.recipe_dir)
+
+
+# ---------------------------------------------------------------------------
+# Step 7 — symlinks (dangling or escaping the archive)
+# ---------------------------------------------------------------------------
+
+
+SYMLINK_RECIPE = (
+ "top=$(pwd -P); find . -type l | while IFS= read -r l; do "
+ 'if [ ! -e "$l" ]; then echo "dangling: $l"; '
+ 'else case "$(realpath "$l")" in "$top"|"$top"/*) ;; *) echo "outside: $l";; esac; fi; done'
+)
+
+
+def symlinks(tree: Path, validators: Sequence[str], recipe_dir: str | None) -> dict[str, Any]:
+ """Every symlink must resolve to an existing path inside the unpacked archive."""
+ if not tree.is_dir():
+ raise InputError(f"not a directory: {tree}")
+ top = tree.resolve()
+ links: list[Path] = []
+ for root, dirs, files in _walk(tree):
+ links += [root / n for n in (*dirs, *files) if (root / n).is_symlink()]
+ dangling: list[str] = []
+ outside: list[str] = []
+ for link in links:
+ target = Path(os.path.realpath(link))
+ if not target.exists():
+ dangling.append(_rel(link, tree))
+ elif target != top and top not in target.parents:
+ outside.append(_rel(link, tree))
+ if dangling or outside:
+ status = "FAIL"
+ elif validators:
+ status = "REVIEW"
+ else:
+ status = "PASS" if links else "SKIP"
+ recipe = [f"cd {_shq(recipe_dir or tree.name)}", SYMLINK_RECIPE, *validators]
+ return {
+ "step": "source-tree-integrity",
+ "status": status,
+ "symlinks_present": len(links),
+ "dangling_symlinks": sorted(dangling),
+ "outside_symlinks": sorted(outside),
+ "validator_failures": [],
+ "validators_to_run": list(validators),
+ "paste_recipe": "\n".join(recipe),
+ }
+
+
+def cmd_symlinks(args: argparse.Namespace) -> dict[str, Any]:
+ return symlinks(Path(args.tree), args.validator or [], args.recipe_dir)
+
+
+# ---------------------------------------------------------------------------
+# Step 8 — version consistency
+# ---------------------------------------------------------------------------
+
+
+def expected_version(rc_tag: str) -> str:
+ m = re.fullmatch(r"(.+)-rc\d+", rc_tag)
+ return m.group(1) if m else rc_tag
+
+
+def _first_group(pattern: str, text: str, flags: int = re.MULTILINE) -> str | None:
+ m = re.search(pattern, text, flags)
+ if not m:
+ return None
+ return (m.group(1) if m.groups() else m.group(0)).strip()
+
+
+def extract_version(path: Path, regex: str | None = None) -> tuple[str | None, str | None]:
+ """(version, detail) for one manifest file; version is None when not found."""
+ if not path.is_file():
+ return None, "file not found"
+ text = path.read_text(errors="replace")
+ if regex:
+ found = _first_group(regex, text)
+ return found, None if found else "pattern did not match"
+ name = path.name
+ if name == "setup.cfg":
+ cfg = configparser.ConfigParser(interpolation=None)
+ cfg.read_string(text)
+ value = cfg.get("metadata", "version", fallback=None)
+ if value and value.strip().startswith(("attr:", "file:")):
+ return None, f"dynamic version ({value.strip()})"
+ return (value.strip(), None) if value else (None, "no [metadata] version")
+ if name in ("pyproject.toml", "Cargo.toml"):
+ data = tomllib.loads(text)
+ if name == "Cargo.toml":
+ value = data.get("package", {}).get("version")
+ else:
+ project = data.get("project", {})
+ if "version" in project.get("dynamic", []):
+ return None, "dynamic version ([project] dynamic)"
+ value = project.get("version") or data.get("tool", {}).get("poetry", {}).get("version")
+ return (value, None) if isinstance(value, str) else (None, "no version key")
+ if name == "pom.xml":
+ if "<!DOCTYPE" in text or "<!ENTITY" in text:
+ return None, "pom.xml declares a DOCTYPE/ENTITY; not parsed"
+ root = ET.fromstring(text)
+ ns = root.tag[: root.tag.index("}") + 1] if root.tag.startswith("{") else ""
+ node = root.find(f"{ns}version")
+ return (node.text.strip(), None) if node is not None and node.text else (None, "no <project><version> (inherited from parent?)")
+ if name == "package.json":
+ value = json.loads(text).get("version")
+ return (value, None) if isinstance(value, str) else (None, "no version key")
+ if name.endswith(".py"):
+ pattern = r"""^\s*version\s*=\s*['"]([^'"]+)['"]""" if name == "setup.py" else r"""^__version__\s*(?::\s*[\w.]+\s*)?=\s*['"]([^'"]+)['"]"""
+ found = _first_group(pattern, text)
+ return found, None if found else "no version assignment"
+ if name.endswith(".properties"):
+ found = _first_group(r"^\s*version\s*[=:]\s*(\S+)\s*$", text)
+ return found, None if found else "no version property"
+ if name in ("build.gradle", "build.gradle.kts"):
+ found = _first_group(r"""^\s*version\s*=?\s*['"]([^'"]+)['"]""", text)
+ return found, None if found else "no version assignment"
+ if name in ("Chart.yaml", "Chart.yml"):
+ found = _first_group(r"""^version:\s*['"]?([^'"\s]+)""", text)
+ return found, None if found else "no version key"
+ if name in ("VERSION", "version.txt", "VERSION.txt"):
+ line = next((ln.strip() for ln in text.splitlines() if ln.strip()), None)
+ return line, None if line else "empty file"
+ return None, "no known extraction pattern for this file type; pass PATH=REGEX"
+
+
+def version_consistency(tree: Path, expected: str, manifests: Sequence[str]) -> dict[str, Any]:
+ results = []
+ for spec in manifests:
+ rel, _, regex = spec.partition("=")
+ try:
+ found, detail = extract_version(tree / rel, regex or None)
+ except (ValueError, tomllib.TOMLDecodeError, ET.ParseError, configparser.Error) as exc:
+ found, detail = None, f"could not parse: {exc}"
+ entry: dict[str, Any] = {"file": rel, "extracted": found, "match": found == expected}
+ if detail:
+ entry["detail"] = detail
+ results.append(entry)
+ ok = bool(results) and all(r["match"] and r["extracted"] is not None for r in results)
+ return {"step": "version-consistency", "status": "PASS" if ok else "FAIL", "expected_version": expected, "results": results}
+
+
+def cmd_version(args: argparse.Namespace) -> dict[str, Any]:
+ expected = args.expected or expected_version(args.rc_tag)
+ return version_consistency(Path(args.tree), expected, args.manifest)
+
+
+# ---------------------------------------------------------------------------
+# Step 10 — verdict
+# ---------------------------------------------------------------------------
+
+
+def _parse_overrides(values: Sequence[str]) -> dict[str, str]:
+ out = {}
+ for v in values:
+ step, sep, status = v.partition("=")
+ if not sep or status not in STATUSES:
+ raise InputError(f"--status wants STEP=PASS|WARN|FAIL|SKIP, got {v!r}")
+ out[step] = status
+ return out
+
+
+def verdict(steps: Sequence[dict[str, Any]], overrides: dict[str, str]) -> dict[str, Any]:
+ """Roll step statuses up: any FAIL → FAIL; else any WARN → PASS-WITH-WARNINGS; else PASS.
+
+ SKIP is neutral — it neither passes nor warns — and every skipped step
+ is listed in ``skip_steps`` so the report can name it.
+
+ A tool-computed status is final; ``overrides`` only resolve steps the
+ tool could not decide (``REVIEW``) or did not run (RAT, reproducibility).
+ """
+ summary: dict[str, str] = {}
+ ignored = []
+ for s in steps:
+ summary[s["step"]] = s["status"]
+ for step, status in overrides.items():
+ if summary.get(step, "REVIEW") == "REVIEW":
+ summary[step] = status
+ else:
+ ignored.append(f"{step}={status} (tool status {summary[step]} is final)")
+ ordered = sorted(summary, key=lambda k: STEP_ORDER.index(k) if k in STEP_ORDER else len(STEP_ORDER))
+ statuses = [summary[k] for k in ordered]
+ unresolved = [k for k in ordered if summary[k] == "REVIEW"]
+ if "FAIL" in statuses:
+ overall: str | None = "FAIL"
+ elif unresolved:
+ overall = None
+ elif "WARN" in statuses:
+ overall = "PASS-WITH-WARNINGS"
+ else:
+ overall = "PASS"
+ out: dict[str, Any] = {
+ "step": "verdict",
+ "overall": overall,
+ "step_summary": [{"step": k, "status": summary[k]} for k in ordered],
+ "fail_steps": [k for k in ordered if summary[k] == "FAIL"],
+ "warn_steps": [k for k in ordered if summary[k] == "WARN"],
+ "skip_steps": [k for k in ordered if summary[k] == "SKIP"],
+ "unresolved": unresolved,
+ }
+ if ignored:
+ out["ignored_overrides"] = ignored
+ return out
+
+
+def _load_steps(paths: Sequence[str]) -> list[dict[str, Any]]:
+ steps: list[dict[str, Any]] = []
+ for p in paths:
+ data = json.loads(Path(p).read_text())
+ items = data.get("steps", [data]) if isinstance(data, dict) else data
+ steps += [s for s in items if isinstance(s, dict) and "step" in s and "status" in s and s["step"] != "verdict"]
+ return steps
+
+
+def cmd_verdict(args: argparse.Namespace) -> dict[str, Any]:
+ return verdict(_load_steps(args.results), _parse_overrides(args.status or []))
+
+
+# ---------------------------------------------------------------------------
+# all
+# ---------------------------------------------------------------------------
+
+
+def cmd_all(args: argparse.Namespace) -> dict[str, Any]:
+ staging, tree = Path(args.dir), Path(args.tree)
+ steps = [
+ inventory(_dir_names(staging), args.expect, args.digest, str(staging), args.expect_optional or []),
+ signatures(staging, _patterns(args), args.keys, args.keys_url, args.extra_key or []),
+ checksums(staging, _patterns(args), args.digest),
+ notice_license(tree, Path(args.previous) if args.previous else None),
+ binaries(tree, args.prohibit or [], args.accept or [], args.recipe_dir),
+ symlinks(tree, args.validator or [], args.recipe_dir),
+ version_consistency(tree, args.expected or expected_version(args.rc_tag), args.manifest),
+ ]
+ return {"tool": "release-verify", "steps": steps, "verdict": verdict(steps, _parse_overrides(args.status or []))}
+
+
+# ---------------------------------------------------------------------------
+# CLI
+# ---------------------------------------------------------------------------
+
+
+def _add_expect(p: argparse.ArgumentParser) -> None:
+ p.add_argument("--expect", action="append", required=True, metavar="GLOB", help="required artefact filename pattern (repeatable); missing → FAIL")
+ p.add_argument(
+ "--expect-optional",
+ action="append",
+ metavar="GLOB",
+ help="artefact release-build.md marks optional (repeatable); missing → WARN, staged → checked like the rest",
+ )
+
+
+def _add_digest(p: argparse.ArgumentParser) -> None:
+ p.add_argument(
+ "--digest",
+ action="append",
+ choices=DIGEST_TYPES[:2],
+ metavar="TYPE",
+ help="digest type of the Digest set (repeatable): sha512 is always required, others are checked when present",
+ )
+
+
+def _add_keys(p: argparse.ArgumentParser) -> None:
+ p.add_argument("--keys", required=True, help="local copy of the project KEYS file (public keys only)")
+ p.add_argument("--keys-url", help="KEYS URL the voter fetches in paste_recipe")
+ p.add_argument("--extra-key", action="append", metavar="FILE", help="public key that is NOT a trust anchor, to tell KEY-NOT-IN-KEYS from FAIL")
+
+
+def _add_tree_opts(p: argparse.ArgumentParser) -> None:
+ p.add_argument("--recipe-dir", help="directory name to print in paste_recipe (default: the tree's basename)")
+
+
+def _add_version(p: argparse.ArgumentParser) -> None:
+ g = p.add_mutually_exclusive_group(required=True)
+ g.add_argument("--expected", help="expected version string")
+ g.add_argument("--rc-tag", help="RC tag; the -rcN suffix is stripped")
+ p.add_argument("--manifest", action="append", required=True, metavar="PATH[=REGEX]", help="manifest file relative to --tree (repeatable)")
+
+
+def build_parser() -> argparse.ArgumentParser:
+ parser = argparse.ArgumentParser(prog="release-verify", description=(__doc__ or "").split("\n\n")[0])
+ sub = parser.add_subparsers(dest="cmd", required=True)
+
+ p = sub.add_parser("inventory", help="Step 1: staging listing vs expected artefacts")
+ src = p.add_mutually_exclusive_group(required=True)
+ src.add_argument("--dir", help="local copy of the staging directory")
+ src.add_argument("--listing", help="file with one staged filename per line (e.g. svn ls output)")
+ _add_expect(p)
+ _add_digest(p)
+ p.set_defaults(func=cmd_inventory)
+
+ p = sub.add_parser("signatures", help="Step 2: gpg --verify against the project KEYS only")
+ p.add_argument("--dir", required=True, help="local copy of the staging directory")
+ _add_expect(p)
+ _add_keys(p)
+ p.set_defaults(func=cmd_signatures)
+
+ p = sub.add_parser("checksums", help="Step 3: digest files vs the artefacts")
+ p.add_argument("--dir", required=True, help="local copy of the staging directory")
+ _add_expect(p)
+ _add_digest(p)
+ p.set_defaults(func=cmd_checksums)
+
+ p = sub.add_parser("notice-license", help="Step 5: NOTICE / LICENSE presence and diff")
+ p.add_argument("--tree", required=True, help="unpacked source artefact")
+ p.add_argument("--previous", help="directory holding the previous release's NOTICE and LICENSE")
+ p.set_defaults(func=cmd_notice_license)
+
+ p = sub.add_parser("binaries", help="Step 6: prohibited binaries in the unpacked source")
+ p.add_argument("--tree", required=True, help="unpacked source artefact")
+ p.add_argument("--prohibit", action="append", metavar="GLOB", help="additional prohibited glob from the Binary-exclude list")
+ p.add_argument("--accept", action="append", metavar="GLOB", help="known-accepted exception from the Binary-exclude list")
+ _add_tree_opts(p)
+ p.set_defaults(func=cmd_binaries)
+
+ p = sub.add_parser("symlinks", help="Step 7: symlinks that dangle or resolve outside the unpacked source")
+ p.add_argument("--tree", required=True, help="unpacked source artefact")
+ p.add_argument("--validator", action="append", metavar="CMD", help="source-tree validator; printed in the recipe, never run")
+ _add_tree_opts(p)
+ p.set_defaults(func=cmd_symlinks)
+
+ p = sub.add_parser("version", help="Step 8: version string in every manifest, exact match")
+ p.add_argument("--tree", required=True, help="unpacked source artefact")
+ _add_version(p)
+ p.set_defaults(func=cmd_version)
+
+ p = sub.add_parser("verdict", help="Step 10: roll step results up into the overall verdict")
+ p.add_argument("results", nargs="+", help="JSON files: step results or an `all` report")
+ p.add_argument("--status", action="append", metavar="STEP=STATUS", help="status of a step the tool did not decide (repeatable)")
+ p.set_defaults(func=cmd_verdict)
+
+ p = sub.add_parser("all", help="Steps 1-3, 5-8 and the verdict in one report")
+ p.add_argument("--dir", required=True, help="local copy of the staging directory")
+ p.add_argument("--tree", required=True, help="unpacked source artefact")
+ _add_expect(p)
+ _add_digest(p)
+ _add_keys(p)
+ p.add_argument("--previous", help="directory holding the previous release's NOTICE and LICENSE")
+ p.add_argument("--prohibit", action="append", metavar="GLOB")
+ p.add_argument("--accept", action="append", metavar="GLOB")
+ p.add_argument("--validator", action="append", metavar="CMD")
+ _add_tree_opts(p)
+ _add_version(p)
+ p.add_argument("--status", action="append", metavar="STEP=STATUS", help="status of a step the tool did not decide (repeatable)")
+ p.set_defaults(func=cmd_all)
+ return parser
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+ args = build_parser().parse_args(argv)
+ if hasattr(args, "digest") and not args.digest:
+ args.digest = ["sha512"]
+ try:
+ result = args.func(args)
+ except (InputError, OSError) as exc:
+ print(json.dumps({"error": str(exc)}, indent=2))
+ return 2
+ print(json.dumps(result, indent=2))
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tools/release-verify/tests/fixtures/gpg/FINGERPRINTS b/tools/release-verify/tests/fixtures/gpg/FINGERPRINTS
new file mode 100644
index 00000000..b7975bd5
--- /dev/null
+++ b/tools/release-verify/tests/fixtures/gpg/FINGERPRINTS
@@ -0,0 +1,2 @@
+rm 26389C7FF5982465406A1BDFB8E90D83CD36F86A
+other 54E469E39D3C9DFF82DAFED7982DD7A5D92850BD
diff --git a/tools/release-verify/tests/fixtures/gpg/KEYS b/tools/release-verify/tests/fixtures/gpg/KEYS
new file mode 100644
index 00000000..f90a368d
--- /dev/null
+++ b/tools/release-verify/tests/fixtures/gpg/KEYS
@@ -0,0 +1,9 @@
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+xjMEaM3ugBYJKwYBBAHaRw8BAQdA0qKjmnrlPITqg4eHS72/AiIuQrStSTg6NWUD
+E3aww8bNGFRlc3QgUk0gPHJtQGV4YW1wbGUub3JnPsJ4BBMWCAAgBQJoze6AFiEE
+Jjicf/WYJGVAahvfuOkNg802+GoCGwMACgkQuOkNg802+GqwAAD7B7aEx/bH1Qjc
+vJCsuvO4DQ7A2NmNgm0NPqOk3XlC+n0A/Axpzv8rqMPeS22KoWe00ODJjqXrrW9/
+ZbGwcMM6cZ0K
+=zSVX
+-----END PGP PUBLIC KEY BLOCK-----
diff --git a/tools/release-verify/tests/fixtures/gpg/OTHER.pub b/tools/release-verify/tests/fixtures/gpg/OTHER.pub
new file mode 100644
index 00000000..c9dc1922
--- /dev/null
+++ b/tools/release-verify/tests/fixtures/gpg/OTHER.pub
@@ -0,0 +1,9 @@
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+xjMEaM3ugBYJKwYBBAHaRw8BAQdAOQLrSFs0zGdgRMK/KDuThzALZUxo1aLXCgmU
+WvfWNrnNIE90aGVyIFNpZ25lciA8b3RoZXJAZXhhbXBsZS5vcmc+wngEExYIACAF
+AmjN7oAWIQRU5GnjnTyd/4La/teYLdel2ShQvQIbAwAKCRCYLdel2ShQvcphAQCj
+O/NfyWhbq7cVDRjt+ajOMKr1XjFxhggxKVAGP+a9MgEAmP/ewI4d4CHXe2xA3XfJ
+qzbbkEjviu89V/PmIDPodAA=
+=KnGk
+-----END PGP PUBLIC KEY BLOCK-----
diff --git a/tools/release-verify/tests/fixtures/gpg/artefact.tar.gz b/tools/release-verify/tests/fixtures/gpg/artefact.tar.gz
new file mode 100644
index 00000000..8ec1ca92
--- /dev/null
+++ b/tools/release-verify/tests/fixtures/gpg/artefact.tar.gz
@@ -0,0 +1 @@
+release-verify fixture payload
diff --git a/tools/release-verify/tests/fixtures/gpg/artefact.tar.gz.asc b/tools/release-verify/tests/fixtures/gpg/artefact.tar.gz.asc
new file mode 100644
index 00000000..7d62585b
--- /dev/null
+++ b/tools/release-verify/tests/fixtures/gpg/artefact.tar.gz.asc
@@ -0,0 +1,7 @@
+-----BEGIN PGP SIGNATURE-----
+
+wnUEABYIAB0FAmjN7oAWIQQmOJx/9ZgkZUBqG9+46Q2DzTb4agAKCRC46Q2DzTb4
+aq+FAP9aEsRgGHFkooe5W/cilZEQ8DBALnBvsOXQ+XCboBHy4AEArn7h24ogPl73
+3atbjhBenacj44wTxdfhLXbMERjvpgE=
+=cuIr
+-----END PGP SIGNATURE-----
diff --git a/tools/release-verify/tests/fixtures/gpg/generate.py b/tools/release-verify/tests/fixtures/gpg/generate.py
new file mode 100644
index 00000000..be26b395
--- /dev/null
+++ b/tools/release-verify/tests/fixtures/gpg/generate.py
@@ -0,0 +1,133 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# mypy: ignore-errors
+"""Regenerate the static OpenPGP fixtures in this directory.
+
+Not run by the test suite. The fixtures are produced without gpg-agent
+(which a sandboxed test run cannot start) by writing OpenPGP v4 EdDSA
+packets directly with the `cryptography` package. Both private keys
+live only in memory and are discarded; only the public keys and the
+detached signatures are written:
+
+- ``KEYS`` — public key of "Test RM" (the project trust anchor)
+- ``OTHER.pub`` — public key of "Other Signer" (not in KEYS)
+- ``artefact.tar.gz`` — the signed payload
+- ``artefact.tar.gz.asc`` — good signature by Test RM
+- ``other.tar.gz`` + ``.asc`` — good signature by Other Signer
+
+Run with: python3 generate.py (needs `cryptography`)
+"""
+
+from __future__ import annotations
+
+import base64
+import hashlib
+import struct
+from pathlib import Path
+
+from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
+from cryptography.hazmat.primitives.serialization import Encoding, PublicFormat
+
+HERE = Path(__file__).parent
+CREATED = 1758326400
+ED25519_OID = bytes.fromhex("2B06010401DA470F01")
+
+
+def packet(tag: int, body: bytes) -> bytes:
+ n = len(body)
+ if n < 192:
+ hdr = bytes([n])
+ elif n < 8384:
+ n -= 192
+ hdr = bytes([(n >> 8) + 192, n & 0xFF])
+ else:
+ hdr = b"\xff" + struct.pack(">I", n)
+ return bytes([0xC0 | tag]) + hdr + body
+
+
+def mpi(data: bytes) -> bytes:
+ data = data.lstrip(b"\x00") or b"\x00"
+ return struct.pack(">H", int.from_bytes(data, "big").bit_length()) + data
+
+
+def crc24(data: bytes) -> int:
+ crc = 0xB704CE
+ for b in data:
+ crc ^= b << 16
+ for _ in range(8):
+ crc <<= 1
+ if crc & 0x1000000:
+ crc ^= 0x1864CFB
+ return crc & 0xFFFFFF
+
+
+def armor(kind: str, data: bytes) -> str:
+ b64 = base64.b64encode(data).decode()
+ lines = [b64[i : i + 64] for i in range(0, len(b64), 64)]
+ crc = base64.b64encode(crc24(data).to_bytes(3, "big")).decode()
+ return f"-----BEGIN PGP {kind}-----\n\n" + "\n".join(lines) + f"\n={crc}\n-----END PGP {kind}-----\n"
+
+
+class Key:
+ def __init__(self, uid: str) -> None:
+ self.sk = Ed25519PrivateKey.generate()
+ raw = self.sk.public_key().public_bytes(Encoding.Raw, PublicFormat.Raw)
+ self.body = b"\x04" + struct.pack(">I", CREATED) + bytes([22, len(ED25519_OID)]) + ED25519_OID + mpi(b"\x40" + raw)
+ self.fpr = hashlib.sha1(b"\x99" + struct.pack(">H", len(self.body)) + self.body).digest()
+ self.uid = uid.encode()
+
+ def _sign(self, sigtype: int, prefix: bytes, extra_hashed: bytes = b"") -> bytes:
+ hashed_sub = (
+ bytes([5, 2])
+ + struct.pack(">I", CREATED) # signature creation time
+ + bytes([22, 33, 4])
+ + self.fpr # issuer fingerprint
+ + extra_hashed
+ )
+ hashed = bytes([4, sigtype, 22, 8]) + struct.pack(">H", len(hashed_sub)) + hashed_sub
+ trailer = b"\x04\xff" + struct.pack(">I", len(hashed))
+ digest = hashlib.sha256(prefix + hashed + trailer).digest()
+ sig = self.sk.sign(digest)
+ unhashed = bytes([9, 16]) + self.fpr[-8:] # issuer key id
+ body = hashed + struct.pack(">H", len(unhashed)) + unhashed + digest[:2] + mpi(sig[:32]) + mpi(sig[32:])
+ return packet(2, body)
+
+ def public_block(self) -> str:
+ key_prefix = b"\x99" + struct.pack(">H", len(self.body)) + self.body
+ uid_prefix = b"\xb4" + struct.pack(">I", len(self.uid)) + self.uid
+ selfsig = self._sign(0x13, key_prefix + uid_prefix, extra_hashed=bytes([2, 27, 0x03]))
+ return armor("PUBLIC KEY BLOCK", packet(6, self.body) + packet(13, self.uid) + selfsig)
+
+ def detach_sign(self, data: bytes) -> str:
+ return armor("SIGNATURE", self._sign(0x00, data))
+
+
+def main() -> None:
+ rm = Key("Test RM <rm...@example.org>")
+ other = Key("Other Signer <ot...@example.org>")
+ (HERE / "KEYS").write_text(rm.public_block())
+ (HERE / "OTHER.pub").write_text(other.public_block())
+ payload = b"release-verify fixture payload\n"
+ (HERE / "artefact.tar.gz").write_bytes(payload)
+ (HERE / "artefact.tar.gz.asc").write_text(rm.detach_sign(payload))
+ (HERE / "other.tar.gz").write_bytes(payload)
+ (HERE / "other.tar.gz.asc").write_text(other.detach_sign(payload))
+ (HERE / "FINGERPRINTS").write_text(f"rm {rm.fpr.hex().upper()}\nother {other.fpr.hex().upper()}\n")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tools/release-verify/tests/fixtures/gpg/other.tar.gz b/tools/release-verify/tests/fixtures/gpg/other.tar.gz
new file mode 100644
index 00000000..8ec1ca92
--- /dev/null
+++ b/tools/release-verify/tests/fixtures/gpg/other.tar.gz
@@ -0,0 +1 @@
+release-verify fixture payload
diff --git a/tools/release-verify/tests/fixtures/gpg/other.tar.gz.asc b/tools/release-verify/tests/fixtures/gpg/other.tar.gz.asc
new file mode 100644
index 00000000..b5a46179
--- /dev/null
+++ b/tools/release-verify/tests/fixtures/gpg/other.tar.gz.asc
@@ -0,0 +1,7 @@
+-----BEGIN PGP SIGNATURE-----
+
+wnUEABYIAB0FAmjN7oAWIQRU5GnjnTyd/4La/teYLdel2ShQvQAKCRCYLdel2ShQ
+vUONAQCH/gcFH3ZfQcCEU6ekuXRZcc/i9gWieA3zC4QqdW4kOQD8CDuSoHPdGhqV
+fkPcqtbmzxGJsuKSsokwaTyag8G7CQg=
+=SFTK
+-----END PGP SIGNATURE-----
diff --git a/tools/release-verify/tests/test_release_verify.py b/tools/release-verify/tests/test_release_verify.py
new file mode 100644
index 00000000..b11ba0f4
--- /dev/null
+++ b/tools/release-verify/tests/test_release_verify.py
@@ -0,0 +1,639 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Behavioural fixtures for `release-verify`.
+
+Each subcommand is exercised on the pass case and on every failure
+class the `release-verify-rc` skill names for its step. The signature
+tests use static fixtures (public keys and detached signatures only,
+see ``fixtures/gpg/generate.py``) so they need `gpg` but not
+`gpg-agent`; one extra test generates a throwaway key in a temporary
+GNUPGHOME and is skipped where an agent cannot start.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import shutil
+import subprocess
+from pathlib import Path
+
+import pytest
+
+import release_verify as rv
+
+GPG_FIXTURES = Path(__file__).parent / "fixtures" / "gpg"
+SRC = "apache-foo-2.11.0-source-release.tar.gz"
+BIN = "apache-foo-2.11.0-bin.tar.gz"
+PATTERNS = ["apache-foo-2.11.0-source-release.tar.gz", "apache-foo-2.11.0-bin.tar.gz"]
+needs_gpg = pytest.mark.skipif(shutil.which("gpg") is None, reason="gpg not installed")
+
+
+def run(capsys: pytest.CaptureFixture[str], *argv: str) -> dict:
+ rc = rv.main(list(argv))
+ out = json.loads(capsys.readouterr().out)
+ out["_rc"] = rc
+ return out
+
+
+def write_digests(directory: Path, name: str, kinds: tuple[str, ...] = ("sha512", "sha256")) -> None:
+ data = (directory / name).read_bytes()
+ for kind in kinds:
+ (directory / f"{name}.{kind}").write_text(f"{hashlib.new(kind, data).hexdigest()} {name}\n")
+
+
+@pytest.fixture
+def staging(tmp_path: Path) -> Path:
+ d = tmp_path / "staging"
+ d.mkdir()
+ for name in (SRC, BIN):
+ (d / name).write_bytes(name.encode())
+ (d / f"{name}.asc").write_text("sig")
+ write_digests(d, name)
+ return d
+
+
+# ---------------------------------------------------------------- inventory
+
+
+def test_inventory_pass(staging: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ out = run(capsys, "inventory", "--dir", str(staging), "--expect", PATTERNS[0], "--expect", PATTERNS[1], "--digest", "sha512", "--digest", "sha256")
+ assert out["status"] == "PASS"
+ assert out["found"] == [SRC, BIN] # config order, not alphabetical
+ assert out["missing"] == [] and out["missing_optional"] == [] and out["unexpected"] == []
+
+
+def test_inventory_missing_is_fail(staging: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ (staging / BIN).unlink()
+ out = run(capsys, "inventory", "--dir", str(staging), "--expect", PATTERNS[0], "--expect", PATTERNS[1], "--digest", "sha512", "--digest", "sha256")
+ assert out["status"] == "FAIL"
+ assert out["missing"] == [BIN]
+
+
+def test_inventory_missing_optional_is_warn(staging: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ (staging / BIN).unlink()
+ out = run(capsys, "inventory", "--dir", str(staging), "--expect", PATTERNS[0], "--expect-optional", PATTERNS[1], "--digest", "sha512", "--digest", "sha256")
+ assert out["status"] == "WARN"
+ assert out["missing"] == [] and out["missing_optional"] == [BIN]
+ assert out["unexpected"] == [] # the orphaned .asc / digests still belong to an expected pattern
+
+
+def test_inventory_optional_present_is_found(staging: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ out = run(capsys, "inventory", "--dir", str(staging), "--expect", PATTERNS[0], "--expect-optional", PATTERNS[1], "--digest", "sha512", "--digest", "sha256")
+ assert out["status"] == "PASS"
+ assert out["found"] == [SRC, BIN] and out["missing_optional"] == []
+
+
+def test_inventory_unexpected_is_warn(staging: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ (staging / "notes.txt").write_text("x")
+ (staging / f"{SRC}.md5").write_text("x")
+ out = run(capsys, "inventory", "--dir", str(staging), "--expect", "apache-foo-*.tar.gz", "--digest", "sha512", "--digest", "sha256")
+ assert out["status"] == "WARN"
+ assert out["unexpected"] == ["apache-foo-2.11.0-source-release.tar.gz.md5", "notes.txt"]
+
+
+def test_inventory_from_listing(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ listing = tmp_path / "ls.txt"
+ listing.write_text("a.tar.gz\na.tar.gz.asc\nsub/\n")
+ out = run(capsys, "inventory", "--listing", str(listing), "--expect", "a.tar.gz", "--expect", "b.zip")
+ assert out["status"] == "FAIL" and out["missing"] == ["b.zip"]
+ assert out["found"] == ["a.tar.gz"] and out["unexpected"] == []
+
+
+# ---------------------------------------------------------------- checksums
+
+
+def test_checksums_pass(staging: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ out = run(capsys, "checksums", "--dir", str(staging), "--expect", PATTERNS[0], "--expect", PATTERNS[1], "--digest", "sha512", "--digest", "sha256")
+ assert out["status"] == "PASS"
+ assert out["deprecated_md5_present"] is False
+ assert [d["classification"] for r in out["results"] for d in r["digests"]] == ["PASS"] * 4
+ assert "sha512sum --check 'apache-foo-2.11.0-bin.tar.gz.sha512'" in out["paste_recipe"]
+
+
+def test_checksums_mismatch_is_fail(staging: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ (staging / f"{BIN}.sha512").write_text("0" * 128 + f" {BIN}\n")
+ out = run(capsys, "checksums", "--dir", str(staging), "--expect", PATTERNS[0], "--expect", PATTERNS[1], "--digest", "sha512", "--digest", "sha256")
+ assert out["status"] == "FAIL"
+ bin_result = next(r for r in out["results"] if r["file"] == BIN)
+ assert bin_result["digests"] == [{"type": "sha512", "classification": "MISMATCH"}, {"type": "sha256", "classification": "PASS"}]
+
+
+def test_checksums_missing_sha512_is_fail(staging: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ (staging / f"{SRC}.sha512").unlink()
+ out = run(capsys, "checksums", "--dir", str(staging), "--expect", PATTERNS[0], "--digest", "sha256")
+ assert out["status"] == "FAIL"
+ assert out["results"][0]["digests"][0] == {"type": "sha512", "classification": "MISSING-DIGEST"}
+
+
+def test_checksums_missing_optional_digest_is_not_checked(staging: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ (staging / f"{SRC}.sha256").unlink()
+ out = run(capsys, "checksums", "--dir", str(staging), "--expect", PATTERNS[0], "--digest", "sha512", "--digest", "sha256")
+ assert out["status"] == "PASS"
+ assert out["results"][0]["digests"] == [{"type": "sha512", "classification": "PASS"}]
+
+
+def test_checksums_present_optional_digest_is_checked(staging: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ out = run(capsys, "checksums", "--dir", str(staging), "--expect", PATTERNS[0]) # sha256 not in the Digest set, but staged
+ assert out["status"] == "PASS"
+ assert [d["type"] for d in out["results"][0]["digests"]] == ["sha512", "sha256"]
+
+
+def test_checksums_sha256_mismatch_is_fail(staging: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ (staging / f"{SRC}.sha256").write_text("0" * 64 + f" {SRC}\n")
+ out = run(capsys, "checksums", "--dir", str(staging), "--expect", PATTERNS[0], "--digest", "sha512", "--digest", "sha256")
+ assert out["status"] == "FAIL"
+ assert out["results"][0]["digests"][1] == {"type": "sha256", "classification": "MISMATCH"}
+
+
+def test_checksums_md5_file_is_warn(staging: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ write_digests(staging, SRC, ("md5",))
+ out = run(capsys, "checksums", "--dir", str(staging), "--expect", PATTERNS[0], "--digest", "sha512")
+ assert out["status"] == "WARN"
+ assert out["deprecated_md5_present"] is True
+ assert {"type": "md5", "classification": "PASS"} in out["results"][0]["digests"]
+ assert "md5sum" not in out["paste_recipe"]
+
+
+def test_checksums_md5_mismatch_is_warn_never_fail(staging: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ (staging / f"{SRC}.md5").write_text("0" * 32 + f" {SRC}\n")
+ out = run(capsys, "checksums", "--dir", str(staging), "--expect", PATTERNS[0], "--digest", "sha512")
+ assert out["status"] == "WARN"
+ md5 = next(d for d in out["results"][0]["digests"] if d["type"] == "md5")
+ assert md5["classification"] == "MISMATCH" and "never fails alone" in md5["detail"]
+
+
+def test_checksums_optional_artefact_is_checked(staging: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ (staging / f"{BIN}.sha512").write_text("0" * 128 + f" {BIN}\n")
+ out = run(capsys, "checksums", "--dir", str(staging), "--expect", PATTERNS[0], "--expect-optional", PATTERNS[1])
+ assert out["status"] == "FAIL"
+ assert [r["file"] for r in out["results"]] == [SRC, BIN]
+
+
+def test_digest_file_formats() -> None:
+ hexd = hashlib.sha512(b"x").hexdigest()
+ upper = hexd.upper()
+ gpg_style = f"file.tar.gz: {' '.join(upper[i : i + 8] for i in range(0, 64, 8))}\n {' '.join(upper[i : i + 8] for i in range(64, 128, 8))}\n"
+ assert rv.parse_digest_file(f"{hexd} file.tar.gz\n", "sha512") == hexd
+ assert rv.parse_digest_file(f"SHA512 (file.tar.gz) = {hexd}\n", "sha512") == hexd
+ assert rv.parse_digest_file(gpg_style, "sha512") == hexd
+ assert rv.parse_digest_file("garbage", "sha512") is None
+
+
+# ---------------------------------------------------------------- notice / license
+
+
+def make_tree(root: Path, notice: str | None = "Apache Foo 2.11.0\nCopyright\n", license_text: str | None = "Apache License\n") -> Path:
+ root.mkdir(parents=True, exist_ok=True)
+ if notice is not None:
+ (root / "NOTICE").write_text(notice)
+ if license_text is not None:
+ (root / "LICENSE").write_text(license_text)
+ return root
+
+
+def test_notice_license_version_only_diff_needs_review(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ tree = make_tree(tmp_path / "rc")
+ prev = make_tree(tmp_path / "prev", notice="Apache Foo 2.10.0\nCopyright\n")
+ out = run(capsys, "notice-license", "--tree", str(tree), "--previous", str(prev))
+ assert out["status"] == "REVIEW"
+ assert (out["notice_diff_lines"], out["license_diff_lines"]) == (2, 0)
+ assert "-Apache Foo 2.10.0" in out["notice_diff"]
+
+
+def test_notice_license_no_previous_is_pass(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ out = run(capsys, "notice-license", "--tree", str(make_tree(tmp_path / "rc")))
+ assert out["status"] == "PASS"
+ assert out["notice_diff_lines"] is None and out["license_diff_lines"] is None
+
+
+def test_notice_license_identical_is_pass(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ out = run(capsys, "notice-license", "--tree", str(make_tree(tmp_path / "rc")), "--previous", str(make_tree(tmp_path / "p")))
+ assert out["status"] == "PASS" and out["notice_diff_lines"] == 0
+
+
+def test_notice_missing_is_fail(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ out = run(capsys, "notice-license", "--tree", str(make_tree(tmp_path / "rc", notice=None)))
+ assert out["status"] == "FAIL"
+ assert out["notice_present"] is False and out["license_present"] is True
+ assert out["detail"].startswith("NOTICE absent from the root of the current RC artefact (rc)")
+
+
+# ---------------------------------------------------------------- binaries
+
+
+def touch(path: Path) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_bytes(b"\0")
+
+
+def test_binaries_clean_tree_pass(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ tree = tmp_path / "apache-foo-2.11.0-source-release"
+ touch(tree / "src" / "foo.py")
+ out = run(capsys, "binaries", "--tree", str(tree), "--accept", "*.class", "--accept", "*.jar")
+ assert out["status"] == "PASS"
+ assert out["prohibited_found"] == [] and out["expected_binaries"] == []
+ assert out["paste_recipe"].startswith("find 'apache-foo-2.11.0-source-release' \\( -type f \\( -name '*.class'")
+ assert "-type d -name '__pycache__'" in out["paste_recipe"]
+
+
+def test_binaries_prohibited_and_accepted(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ tree = tmp_path / "rc"
+ touch(tree / "vendor" / "lib.jar")
+ touch(tree / "third-party" / "native.so")
+ touch(tree / "assets" / "vendor" / "x" / "app.min.js")
+ out = run(capsys, "binaries", "--tree", str(tree), "--accept", "third-party/native.so", "--prohibit", "assets/vendor/**/*.min.js")
+ assert out["status"] == "FAIL"
+ assert out["prohibited_found"] == ["assets/vendor/x/app.min.js", "vendor/lib.jar"]
+ assert out["expected_binaries"] == ["third-party/native.so"]
+ assert "-path 'rc/assets/vendor/*.min.js'" in out["paste_recipe"]
+
+
+def test_pyc_and_pycache_are_never_accepted(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ tree = tmp_path / "rc"
+ touch(tree / "foo" / "__pycache__" / "core.cpython-312.pyc")
+ out = run(capsys, "binaries", "--tree", str(tree), "--accept", "*.pyc", "--accept", "**/__pycache__")
+ assert out["status"] == "FAIL"
+ assert out["prohibited_found"] == ["foo/__pycache__", "foo/__pycache__/core.cpython-312.pyc"]
+ assert out["expected_binaries"] == []
+
+
+# ---------------------------------------------------------------- symlinks
+
+
+def test_symlinks_dangling_is_fail(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ tree = tmp_path / "rc"
+ touch(tree / "docs" / "real.md")
+ os.symlink("real.md", tree / "docs" / "ok.md")
+ os.symlink("../stripped/template.md", tree / "docs" / "broken.md")
+ out = run(capsys, "symlinks", "--tree", str(tree))
+ assert out["status"] == "FAIL"
+ assert out["dangling_symlinks"] == ["docs/broken.md"]
+ assert out["outside_symlinks"] == []
+ assert out["symlinks_present"] == 2
+
+
+def test_symlink_to_existing_path_outside_archive_is_fail(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ tree = tmp_path / "rc"
+ touch(tree / "a.md")
+ touch(tmp_path / "outside.md")
+ os.symlink("../outside.md", tree / "escape.md")
+ os.symlink(str(tmp_path / "outside.md"), tree / "absolute.md")
+ os.symlink("escape.md", tree / "chained.md")
+ os.symlink(".", tree / "self")
+ out = run(capsys, "symlinks", "--tree", str(tree))
+ assert out["status"] == "FAIL"
+ assert out["dangling_symlinks"] == []
+ assert out["outside_symlinks"] == ["absolute.md", "chained.md", "escape.md"]
+
+
+def test_symlinks_resolving_pass_and_none_skip(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ tree = tmp_path / "rc"
+ touch(tree / "a.md")
+ assert run(capsys, "symlinks", "--tree", str(tree))["status"] == "SKIP"
+ os.symlink("a.md", tree / "b.md")
+ assert run(capsys, "symlinks", "--tree", str(tree))["status"] == "PASS"
+
+
+def test_symlinks_validators_are_left_to_run(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ tree = tmp_path / "rc"
+ touch(tree / "a.md")
+ out = run(capsys, "symlinks", "--tree", str(tree), "--validator", "make check-links")
+ assert out["status"] == "REVIEW"
+ assert out["validators_to_run"] == ["make check-links"]
+ assert out["paste_recipe"].splitlines() == ["cd 'rc'", rv.SYMLINK_RECIPE, "make check-links"]
+
+
+def test_symlink_recipe_flags_dangling_and_outside(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ tree = tmp_path / "rc"
+ touch(tree / "a.md")
+ touch(tmp_path / "outside.md")
+ os.symlink("a.md", tree / "ok.md")
+ os.symlink("../outside.md", tree / "escape.md")
+ os.symlink("gone.md", tree / "broken.md")
+ recipe = run(capsys, "symlinks", "--tree", str(tree))["paste_recipe"].splitlines()[1]
+ out = subprocess.run(["sh", "-c", recipe], cwd=tree, capture_output=True, text=True, check=True).stdout
+ assert sorted(out.splitlines()) == ["dangling: ./broken.md", "outside: ./escape.md"]
+
+
+# ---------------------------------------------------------------- version
+
+
+def test_version_consistent(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ tree = tmp_path / "rc"
+ touch(tree / "x")
+ (tree / "setup.cfg").write_text("[metadata]\nname = foo\nversion = 2.11.0\n")
+ (tree / "foo").mkdir()
+ (tree / "foo" / "__init__.py").write_text('__version__ = "2.11.0"\n')
+ out = run(capsys, "version", "--tree", str(tree), "--rc-tag", "2.11.0-rc1", "--manifest", "setup.cfg", "--manifest", "foo/__init__.py")
+ assert out["status"] == "PASS"
+ assert out["expected_version"] == "2.11.0"
+ assert out["results"] == [
+ {"file": "setup.cfg", "extracted": "2.11.0", "match": True},
+ {"file": "foo/__init__.py", "extracted": "2.11.0", "match": True},
+ ]
+
+
+def test_version_dev_suffix_is_fail(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ tree = tmp_path / "rc"
+ (tree / "foo").mkdir(parents=True)
+ (tree / "foo" / "__init__.py").write_text('__version__ = "2.11.0.dev0"\n')
+ out = run(capsys, "version", "--tree", str(tree), "--expected", "2.11.0", "--manifest", "foo/__init__.py", "--manifest", "missing.cfg")
+ assert out["status"] == "FAIL"
+ assert out["results"][0]["extracted"] == "2.11.0.dev0" and out["results"][0]["match"] is False
+ assert out["results"][1]["extracted"] is None
+
+
+@pytest.mark.parametrize(
+ ("name", "content", "expected"),
+ [
+ ("pyproject.toml", '[project]\nname = "x"\nversion = "1.2.3"\n', "1.2.3"),
+ ("Cargo.toml", '[package]\nname = "x"\nversion = "1.2.3"\n', "1.2.3"),
+ ("package.json", '{"name": "x", "version": "1.2.3"}', "1.2.3"),
+ (
+ "pom.xml",
+ '<project xmlns="http://maven.apache.org/POM/4.0.0"><parent><version>9</version></parent><version>1.2.3-SNAPSHOT</version></project>',
+ "1.2.3-SNAPSHOT",
+ ),
+ ("gradle.properties", "group=x\nversion=1.2.3\n", "1.2.3"),
+ ("Chart.yaml", "apiVersion: v2\nversion: 1.2.3\n", "1.2.3"),
+ ("VERSION", "\n1.2.3\n", "1.2.3"),
+ ("setup.py", 'setup(\n name="x",\n version="1.2.3",\n)\n', "1.2.3"),
+ ],
+)
+def test_manifest_formats(tmp_path: Path, name: str, content: str, expected: str) -> None:
+ (tmp_path / name).write_text(content)
+ assert rv.extract_version(tmp_path / name) == (expected, None)
+
+
+def test_unknown_format_needs_pattern(tmp_path: Path) -> None:
+ (tmp_path / "meta.txt").write_text("release: 1.2.3\n")
+ assert rv.extract_version(tmp_path / "meta.txt")[0] is None
+ assert rv.extract_version(tmp_path / "meta.txt", r"release: (\S+)") == ("1.2.3", None)
+
+
+def test_dynamic_version_is_null(tmp_path: Path) -> None:
+ (tmp_path / "pyproject.toml").write_text('[project]\nname = "x"\ndynamic = ["version"]\n')
+ assert rv.extract_version(tmp_path / "pyproject.toml")[0] is None
+
+
+# ---------------------------------------------------------------- verdict
+
+
+def test_verdict_order() -> None:
+ steps = [{"step": "inventory", "status": "PASS"}, {"step": "checksums", "status": "WARN"}]
+ assert rv.verdict(steps, {})["overall"] == "PASS-WITH-WARNINGS"
+ assert rv.verdict([*steps, {"step": "signatures", "status": "FAIL"}], {})["overall"] == "FAIL"
+ out = rv.verdict([{"step": "inventory", "status": "PASS"}, {"step": "source-tree-integrity", "status": "SKIP"}], {"reproducibility": "SKIP"})
+ assert out["overall"] == "PASS"
+ assert out["skip_steps"] == ["source-tree-integrity", "reproducibility"]
+ assert out["warn_steps"] == [] and out["fail_steps"] == []
+
+
+def test_verdict_review_needs_resolution_and_fail_is_final() -> None:
+ steps = [{"step": "notice-license", "status": "REVIEW"}, {"step": "signatures", "status": "FAIL"}]
+ out = rv.verdict(steps[:1], {})
+ assert out["overall"] is None and out["unresolved"] == ["notice-license"]
+ assert rv.verdict(steps[:1], {"notice-license": "WARN"})["overall"] == "PASS-WITH-WARNINGS"
+ out = rv.verdict(steps, {"signatures": "PASS", "notice-license": "PASS"})
+ assert out["overall"] == "FAIL"
+ assert out["ignored_overrides"] == ["signatures=PASS (tool status FAIL is final)"]
+
+
+def test_verdict_reads_files(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ (tmp_path / "a.json").write_text(json.dumps({"step": "inventory", "status": "PASS"}))
+ (tmp_path / "b.json").write_text(json.dumps({"steps": [{"step": "checksums", "status": "PASS"}]}))
+ out = run(capsys, "verdict", str(tmp_path / "a.json"), str(tmp_path / "b.json"), "--status", "rat-license-headers=PASS")
+ assert out["overall"] == "PASS"
+ assert [s["step"] for s in out["step_summary"]] == ["inventory", "checksums", "rat-license-headers"]
+
+
+def test_bad_status_override_is_error(capsys: pytest.CaptureFixture[str]) -> None:
+ out = run(capsys, "verdict", "/nonexistent.json", "--status", "x=MAYBE")
+ assert out["_rc"] == 2
+
+
+# ---------------------------------------------------------------- signatures
+
+
+def gpg_staging(tmp_path: Path) -> Path:
+ d = tmp_path / "staging"
+ d.mkdir()
+ for name in ("artefact.tar.gz", "artefact.tar.gz.asc", "other.tar.gz", "other.tar.gz.asc"):
+ shutil.copy(GPG_FIXTURES / name, d / name)
+ return d
+
+
+def fingerprints() -> dict[str, str]:
+ return dict(line.split() for line in (GPG_FIXTURES / "FINGERPRINTS").read_text().splitlines())
+
+
+@needs_gpg
+def test_signature_pass(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ d = gpg_staging(tmp_path)
+ out = run(
+ capsys, "signatures", "--dir", str(d), "--expect", "artefact.tar.gz", "--keys", str(GPG_FIXTURES / "KEYS"), "--keys-url", "https://example.org/KEYS"
+ )
+ assert out["status"] == "PASS"
+ assert out["results"] == [
+ {"file": "artefact.tar.gz", "sig_file": "artefact.tar.gz.asc", "classification": "PASS", "fingerprint": fingerprints()["rm"], "key_in_keys": True}
+ ]
+ assert out["paste_recipe"] == "curl -s 'https://example.org/KEYS' | gpg --import\ngpg --verify 'artefact.tar.gz.asc' 'artefact.tar.gz'"
+
+
+@needs_gpg
+def test_tampered_artefact_is_fail(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ d = gpg_staging(tmp_path)
+ (d / "artefact.tar.gz").write_bytes(b"tampered\n")
+ out = run(capsys, "signatures", "--dir", str(d), "--expect", "artefact.tar.gz", "--keys", str(GPG_FIXTURES / "KEYS"))
+ assert out["status"] == "FAIL"
+ r = out["results"][0]
+ assert (r["classification"], r["fingerprint"], r["key_in_keys"]) == ("FAIL", None, False)
+ assert r["detail"] == "BAD signature"
+
+
+@needs_gpg
+def test_unknown_key_is_fail_and_known_outsider_is_key_not_in_keys(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ d = gpg_staging(tmp_path)
+ out = run(capsys, "signatures", "--dir", str(d), "--expect", "other.tar.gz", "--keys", str(GPG_FIXTURES / "KEYS"))
+ assert out["status"] == "FAIL"
+ assert out["results"][0]["classification"] == "FAIL"
+ assert "not in KEYS" in out["results"][0]["detail"]
+ out = run(
+ capsys, "signatures", "--dir", str(d), "--expect", "other.tar.gz", "--keys", str(GPG_FIXTURES / "KEYS"), "--extra-key", str(GPG_FIXTURES / "OTHER.pub")
+ )
+ r = out["results"][0]
+ assert out["status"] == "FAIL"
+ assert (r["classification"], r["fingerprint"], r["key_in_keys"]) == ("KEY-NOT-IN-KEYS", fingerprints()["other"], False)
+
+
+@needs_gpg
+def test_missing_signature_is_fail(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ d = gpg_staging(tmp_path)
+ (d / "artefact.tar.gz.asc").unlink()
+ out = run(capsys, "signatures", "--dir", str(d), "--expect", "artefact.tar.gz", "--keys", str(GPG_FIXTURES / "KEYS"))
+ assert out["results"][0]["detail"] == "signature file missing" and out["status"] == "FAIL"
+
+
+def test_private_key_material_is_refused(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ keys = tmp_path / "KEYS"
+ # Built at runtime so the repository's private-key detector does not flag the fixture.
+ block = "PGP " + "PRIVATE KEY BLOCK"
+ keys.write_text(f"-----BEGIN {block}-----\nxx\n-----END {block}-----\n")
+ out = run(capsys, "signatures", "--dir", str(tmp_path), "--expect", "x", "--keys", str(keys))
+ assert out["_rc"] == 2 and "private key" in out["error"]
+
+
+@needs_gpg
+def test_user_keyring_is_never_touched(tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch) -> None:
+ home = tmp_path / "user-gnupg"
+ home.mkdir(mode=0o700)
+ monkeypatch.setenv("GNUPGHOME", str(home))
+ run(capsys, "signatures", "--dir", str(gpg_staging(tmp_path)), "--expect", "artefact.tar.gz", "--keys", str(GPG_FIXTURES / "KEYS"))
+ assert list(home.iterdir()) == []
+
+
+@needs_gpg
+def test_freshly_generated_key(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ """End to end with a key gpg made itself; needs a gpg-agent, so skips where one cannot start."""
+ gen = Path(os.environ.get("TMPDIR", "/tmp")) / f"rvk-{os.getpid()}"
+ gen.mkdir(mode=0o700)
+ try:
+ base = ["gpg", "--batch", "--homedir", str(gen), "--pinentry-mode", "loopback", "--passphrase", ""]
+ made = subprocess.run([*base, "--quick-gen-key", "Throwaway <t...@example.org>", "ed25519", "sign", "never"], capture_output=True, check=False)
+ if made.returncode != 0:
+ pytest.skip(f"cannot generate a key here: {made.stderr.decode().strip().splitlines()[-1:]}")
+ d = tmp_path / "staging"
+ d.mkdir()
+ (d / "a.tar.gz").write_bytes(b"payload")
+ subprocess.run([*base, "--armor", "--detach-sign", "--output", str(d / "a.tar.gz.asc"), str(d / "a.tar.gz")], check=True, capture_output=True)
+ keys = tmp_path / "KEYS"
+ keys.write_bytes(subprocess.run(["gpg", "--homedir", str(gen), "--armor", "--export"], check=True, capture_output=True).stdout)
+ finally:
+ subprocess.run(["gpgconf", "--homedir", str(gen), "--kill", "all"], capture_output=True, check=False)
+ shutil.rmtree(gen, ignore_errors=True)
+ out = run(capsys, "signatures", "--dir", str(d), "--expect", "a.tar.gz", "--keys", str(keys))
+ assert out["status"] == "PASS"
+
+
+# ---------------------------------------------------------------- all
+
+
+@needs_gpg
+def test_all_report(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ d = gpg_staging(tmp_path)
+ write_digests(d, "artefact.tar.gz", ("sha512",))
+ tree = make_tree(tmp_path / "artefact")
+ (tree / "VERSION").write_text("2.11.0\n")
+ out = run(
+ capsys, "all", "--dir", str(d), "--tree", str(tree), "--expect", "artefact.tar.gz", "--keys", str(GPG_FIXTURES / "KEYS"),
+ "--rc-tag", "2.11.0-rc1", "--manifest", "VERSION", "--status", "rat-license-headers=PASS", "--status", "reproducibility=SKIP",
+ ) # fmt: skip
+ statuses = {s["step"]: s["status"] for s in out["steps"]}
+ # other.tar.gz / .asc are staged but not expected → inventory WARN
+ assert statuses == {
+ "inventory": "WARN",
+ "signatures": "PASS",
+ "checksums": "PASS",
+ "notice-license": "PASS",
+ "binary-exclusion": "PASS",
+ "source-tree-integrity": "SKIP",
+ "version-consistency": "PASS",
+ }
+ assert out["verdict"]["overall"] == "PASS-WITH-WARNINGS"
+ assert out["verdict"]["warn_steps"] == ["inventory"]
+ assert out["verdict"]["skip_steps"] == ["source-tree-integrity", "reproducibility"]
+
+
+def test_verdict_includes_jvm_artefacts_in_step_order() -> None:
+ steps = [
+ {"step": "version-consistency", "status": "PASS"},
+ {"step": "jvm-artefacts", "status": "FAIL"},
+ {"step": "binary-exclusion", "status": "PASS"},
+ ]
+ out = rv.verdict(steps, {})
+ assert out["overall"] == "FAIL"
+ assert out["fail_steps"] == ["jvm-artefacts"]
+ assert [s["step"] for s in out["step_summary"]] == ["binary-exclusion", "jvm-artefacts", "version-consistency"]
+
+
+# --- Security: recipes, local paths and stale signatures ---------------------
+
+
+def test_recipes_quote_hostile_file_names(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
+ hostile = "x$(touch pwned)`id`;rm -rf ~.tar.gz"
+ (tmp_path / hostile).write_bytes(b"payload")
+ (tmp_path / f"{hostile}.sha512").write_text(hashlib.sha512(b"payload").hexdigest() + " " + hostile + "\n")
+ out = run(capsys, "checksums", "--dir", str(tmp_path), "--expect", "x*.tar.gz", "--digest", "sha512")
+ line = next(ln for ln in out["paste_recipe"].splitlines() if "sha512sum" in ln)
+ assert line == "sha512sum --check " + rv._shq(f"{hostile}.sha512")
+ assert "$(" not in line.replace(rv._shq(f"{hostile}.sha512"), "")
+
+
+def test_binary_and_symlink_recipes_quote_the_directory() -> None:
+ assert rv.binary_find_recipe("rc;id", []).startswith("find 'rc;id' ")
+
+
+def test_local_keys_path_never_reaches_the_recipe() -> None:
+ assert rv._shq(Path("/Users/someone/secret/KEYS").name) == "'KEYS'"
+
+
+def _fake_verify(monkeypatch: pytest.MonkeyPatch, status: list[str], returncode: int = 0) -> None:
+ def fake(home: Path, *args: str) -> subprocess.CompletedProcess[str]:
+ stdout = "".join(f"[GNUPG:] {line}\n" for line in status)
+ return subprocess.CompletedProcess(args, returncode, stdout, "")
+
+ monkeypatch.setattr(rv, "_gpg", fake)
+
+
+VALIDSIG = "VALIDSIG " + "A" * 40 + " 2026-01-01 1767225600 0 4 0 22 10 00 " + "B" * 40
+
+
+@pytest.mark.parametrize(
+ ("token", "detail"),
+ [("REVKEYSIG", "revoked key"), ("EXPKEYSIG", "expired key"), ("EXPSIG", "signature itself has expired")],
+)
+def test_stale_signature_never_passes(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, token: str, detail: str) -> None:
+ artefact = tmp_path / "a.tar.gz"
+ artefact.write_bytes(b"x")
+ (tmp_path / "a.tar.gz.asc").write_text("sig")
+ _fake_verify(monkeypatch, [f"{token} " + "B" * 16 + " Someone", VALIDSIG])
+ result = rv.verify_one(tmp_path, artefact, {"B" * 40})
+ assert result["classification"] == "FAIL"
+ assert detail in result["detail"]
+
+
+def test_good_signature_still_passes_with_fake_gpg(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ artefact = tmp_path / "a.tar.gz"
+ artefact.write_bytes(b"x")
+ (tmp_path / "a.tar.gz.asc").write_text("sig")
+ _fake_verify(monkeypatch, ["GOODSIG " + "B" * 16 + " Someone", VALIDSIG])
+ assert rv.verify_one(tmp_path, artefact, {"B" * 40})["classification"] == "PASS"
+
+
+def test_gpg_home_path_is_redacted(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ home = tmp_path / "rv-gpg-abc"
+
+ def fake_run(cmd: list[str], **kwargs: object) -> subprocess.CompletedProcess[str]:
+ return subprocess.CompletedProcess(cmd, 2, "", f"gpg: keybox '{home}/pubring.kbx' created\n")
+
+ monkeypatch.setattr(rv.shutil, "which", lambda _: "/usr/bin/gpg")
+ monkeypatch.setattr(rv.subprocess, "run", fake_run)
+ proc = rv._gpg(home, "--list-keys")
+ assert str(home) not in proc.stderr
+ assert "<gnupghome>/pubring.kbx" in proc.stderr
diff --git a/tools/skill-evals/evals/release-verify-rc/README.md b/tools/skill-evals/evals/release-verify-rc/README.md
index 7036912c..3b96b6fc 100644
--- a/tools/skill-evals/evals/release-verify-rc/README.md
+++ b/tools/skill-evals/evals/release-verify-rc/README.md
@@ -36,7 +36,13 @@ uv run --project tools/skill-evals skill-eval --cli tools/skill-evals/evals/rele
## Grading methodology
-Steps 0, 2, 3, 5, 6, 8 and 9 all emit structured JSON. Cases use
+Steps 0, 2, 3, 5, 6, 8 and 9 all emit structured JSON.
+In Steps 2, 3, 5, 6 and 8 each case feeds the
+[`release-verify`](../../../release-verify/README.md) tool's JSON for the
+scenario (produced by running the tool on equivalent fixture data); the
+classification logic itself is covered by that tool's pytest suite, so
+these cases grade how the model reads and reports the tool output, and
+the judgement the tool leaves to it (Step 5's `REVIEW`). Cases use
`expected.json` for exact-field grading and `output-spec.md` to
document the allowed schema.
diff --git a/tools/skill-evals/evals/release-verify-rc/step-2-verify-signatures/fixtures/case-1-all-sigs-pass/expected.json b/tools/skill-evals/evals/release-verify-rc/step-2-verify-signatures/fixtures/case-1-all-sigs-pass/expected.json
index 60130e1b..0dbeb60a 100644
--- a/tools/skill-evals/evals/release-verify-rc/step-2-verify-signatures/fixtures/case-1-all-sigs-pass/expected.json
+++ b/tools/skill-evals/evals/release-verify-rc/step-2-verify-signatures/fixtures/case-1-all-sigs-pass/expected.json
@@ -6,16 +6,16 @@
"file": "apache-airflow-2.11.0-source-release.tar.gz",
"sig_file": "apache-airflow-2.11.0-source-release.tar.gz.asc",
"classification": "PASS",
- "fingerprint": "ABCD1234ABCD1234ABCD1234ABCD1234ABCD1234",
+ "fingerprint": "26389C7FF5982465406A1BDFB8E90D83CD36F86A",
"key_in_keys": true
},
{
"file": "apache-airflow-2.11.0-bin.tar.gz",
"sig_file": "apache-airflow-2.11.0-bin.tar.gz.asc",
"classification": "PASS",
- "fingerprint": "ABCD1234ABCD1234ABCD1234ABCD1234ABCD1234",
+ "fingerprint": "26389C7FF5982465406A1BDFB8E90D83CD36F86A",
"key_in_keys": true
}
],
- "paste_recipe": "curl -s https://dist.apache.org/repos/dist/release/airflow/KEYS | gpg --import\ngpg --verify apache-airflow-2.11.0-source-release.tar.gz.asc apache-airflow-2.11.0-source-release.tar.gz\ngpg --verify apache-airflow-2.11.0-bin.tar.gz.asc apache-airflow-2.11.0-bin.tar.gz"
+ "paste_recipe": "curl -s 'https://dist.apache.org/repos/dist/release/airflow/KEYS' | gpg --import\ngpg --verify 'apache-airflow-2.11.0-source-release.tar.gz.asc' 'apache-airflow-2.11.0-source-release.tar.gz'\ngpg --verify 'apache-airflow-2.11.0-bin.tar.gz.asc' 'apache-airflow-2.11.0-bin.tar.gz'"
}
diff --git a/tools/skill-evals/evals/release-verify-rc/step-2-verify-signatures/fixtures/case-1-all-sigs-pass/report.md b/tools/skill-evals/evals/release-verify-rc/step-2-verify-signatures/fixtures/case-1-all-sigs-pass/report.md
index 2be0350f..e74b1cdc 100644
--- a/tools/skill-evals/evals/release-verify-rc/step-2-verify-signatures/fixtures/case-1-all-sigs-pass/report.md
+++ b/tools/skill-evals/evals/release-verify-rc/step-2-verify-signatures/fixtures/case-1-all-sigs-pass/report.md
@@ -1,20 +1,49 @@
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
+RC: 2.11.0-rc1
+release-build.md expected artefacts: apache-airflow-2.11.0-source-release.tar.gz, apache-airflow-2.11.0-bin.tar.gz
+KEYS: https://dist.apache.org/repos/dist/release/airflow/KEYS (downloaded to KEYS)
+
RC inventory from Step 1 (PASS):
apache-airflow-2.11.0-source-release.tar.gz (FOUND)
- apache-airflow-2.11.0-source-release.tar.gz.asc (FOUND)
+ apache-airflow-2.11.0-source-release.tar.gz.asc
apache-airflow-2.11.0-bin.tar.gz (FOUND)
- apache-airflow-2.11.0-bin.tar.gz.asc (FOUND)
+ apache-airflow-2.11.0-bin.tar.gz.asc
+
+Tool run:
+
+```bash
+uv run --project .apache-magpie/tools/release-verify release-verify signatures \
+ --dir staging --expect apache-airflow-2.11.0-source-release.tar.gz --expect apache-airflow-2.11.0-bin.tar.gz \
+ --keys KEYS --keys-url https://dist.apache.org/repos/dist/release/airflow/KEYS
+```
-GPG verification results:
- apache-airflow-2.11.0-source-release.tar.gz.asc:
- gpg --verify exits 0
- Signing fingerprint: ABCD1234ABCD1234ABCD1234ABCD1234ABCD1234
- Fingerprint ABCD1234ABCD1234ABCD1234ABCD1234ABCD1234 IS present in the project KEYS file
- (https://dist.apache.org/repos/dist/release/airflow/KEYS)
+Tool output:
- apache-airflow-2.11.0-bin.tar.gz.asc:
- gpg --verify exits 0
- Signing fingerprint: ABCD1234ABCD1234ABCD1234ABCD1234ABCD1234
- Fingerprint ABCD1234ABCD1234ABCD1234ABCD1234ABCD1234 IS present in the project KEYS file
+```json
+{
+ "step": "signatures",
+ "status": "PASS",
+ "results": [
+ {
+ "file": "apache-airflow-2.11.0-source-release.tar.gz",
+ "sig_file": "apache-airflow-2.11.0-source-release.tar.gz.asc",
+ "classification": "PASS",
+ "fingerprint": "26389C7FF5982465406A1BDFB8E90D83CD36F86A",
+ "key_in_keys": true
+ },
+ {
+ "file": "apache-airflow-2.11.0-bin.tar.gz",
+ "sig_file": "apache-airflow-2.11.0-bin.tar.gz.asc",
+ "classification": "PASS",
+ "fingerprint": "26389C7FF5982465406A1BDFB8E90D83CD36F86A",
+ "key_in_keys": true
+ }
+ ],
+ "keys_fingerprints": [
+ "26389C7FF5982465406A1BDFB8E90D83CD36F86A"
+ ],
+ "paste_recipe": "curl -s 'https://dist.apache.org/repos/dist/release/airflow/KEYS' | gpg --import\ngpg --verify 'apache-airflow-2.11.0-source-release.tar.gz.asc' 'apache-airflow-2.11.0-source-release.tar.gz'\ngpg --verify 'apache-airflow-2.11.0-bin.tar.gz.asc' 'apache-airflow-2.11.0-bin.tar.gz'"
+}
+```
diff --git a/tools/skill-evals/evals/release-verify-rc/step-2-verify-signatures/fixtures/case-2-tampered-sig/expected.json b/tools/skill-evals/evals/release-verify-rc/step-2-verify-signatures/fixtures/case-2-tampered-sig/expected.json
index f9c90c4a..176f44af 100644
--- a/tools/skill-evals/evals/release-verify-rc/step-2-verify-signatures/fixtures/case-2-tampered-sig/expected.json
+++ b/tools/skill-evals/evals/release-verify-rc/step-2-verify-signatures/fixtures/case-2-tampered-sig/expected.json
@@ -6,7 +6,7 @@
"file": "apache-airflow-2.11.0-source-release.tar.gz",
"sig_file": "apache-airflow-2.11.0-source-release.tar.gz.asc",
"classification": "PASS",
- "fingerprint": "ABCD1234ABCD1234ABCD1234ABCD1234ABCD1234",
+ "fingerprint": "26389C7FF5982465406A1BDFB8E90D83CD36F86A",
"key_in_keys": true
},
{
@@ -17,5 +17,5 @@
"key_in_keys": false
}
],
- "paste_recipe": "curl -s https://dist.apache.org/repos/dist/release/airflow/KEYS | gpg --import\ngpg --verify apache-airflow-2.11.0-source-release.tar.gz.asc apache-airflow-2.11.0-source-release.tar.gz\ngpg --verify apache-airflow-2.11.0-bin.tar.gz.asc apache-airflow-2.11.0-bin.tar.gz"
+ "paste_recipe": "curl -s 'https://dist.apache.org/repos/dist/release/airflow/KEYS' | gpg --import\ngpg --verify 'apache-airflow-2.11.0-source-release.tar.gz.asc' 'apache-airflow-2.11.0-source-release.tar.gz'\ngpg --verify 'apache-airflow-2.11.0-bin.tar.gz.asc' 'apache-airflow-2.11.0-bin.tar.gz'"
}
diff --git a/tools/skill-evals/evals/release-verify-rc/step-2-verify-signatures/fixtures/case-2-tampered-sig/report.md b/tools/skill-evals/evals/release-verify-rc/step-2-verify-signatures/fixtures/case-2-tampered-sig/report.md
index 9ccf30a3..612f4cdf 100644
--- a/tools/skill-evals/evals/release-verify-rc/step-2-verify-signatures/fixtures/case-2-tampered-sig/report.md
+++ b/tools/skill-evals/evals/release-verify-rc/step-2-verify-signatures/fixtures/case-2-tampered-sig/report.md
@@ -1,20 +1,52 @@
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
+RC: 2.11.0-rc1
+release-build.md expected artefacts: apache-airflow-2.11.0-source-release.tar.gz, apache-airflow-2.11.0-bin.tar.gz
+KEYS: https://dist.apache.org/repos/dist/release/airflow/KEYS (downloaded to KEYS)
+
RC inventory from Step 1 (PASS):
apache-airflow-2.11.0-source-release.tar.gz (FOUND)
- apache-airflow-2.11.0-source-release.tar.gz.asc (FOUND)
+ apache-airflow-2.11.0-source-release.tar.gz.asc
apache-airflow-2.11.0-bin.tar.gz (FOUND)
- apache-airflow-2.11.0-bin.tar.gz.asc (FOUND)
+ apache-airflow-2.11.0-bin.tar.gz.asc
+
+Note: apache-airflow-2.11.0-bin.tar.gz was modified after it was signed.
+
+Tool run:
+
+```bash
+uv run --project .apache-magpie/tools/release-verify release-verify signatures \
+ --dir staging --expect apache-airflow-2.11.0-source-release.tar.gz --expect apache-airflow-2.11.0-bin.tar.gz \
+ --keys KEYS --keys-url https://dist.apache.org/repos/dist/release/airflow/KEYS
+```
-GPG verification results:
- apache-airflow-2.11.0-source-release.tar.gz.asc:
- gpg --verify exits 0
- Signing fingerprint: ABCD1234ABCD1234ABCD1234ABCD1234ABCD1234
- Fingerprint ABCD1234ABCD1234ABCD1234ABCD1234ABCD1234 IS present in the project KEYS file
+Tool output:
- apache-airflow-2.11.0-bin.tar.gz.asc:
- gpg --verify exits NON-ZERO (exit code 2)
- Output: "gpg: BAD signature from ..."
- The binary artefact appears to have been modified after signing, or the .asc file
- does not match this binary.
+```json
+{
+ "step": "signatures",
+ "status": "FAIL",
+ "results": [
+ {
+ "file": "apache-airflow-2.11.0-source-release.tar.gz",
+ "sig_file": "apache-airflow-2.11.0-source-release.tar.gz.asc",
+ "classification": "PASS",
+ "fingerprint": "26389C7FF5982465406A1BDFB8E90D83CD36F86A",
+ "key_in_keys": true
+ },
+ {
+ "file": "apache-airflow-2.11.0-bin.tar.gz",
+ "sig_file": "apache-airflow-2.11.0-bin.tar.gz.asc",
+ "classification": "FAIL",
+ "fingerprint": null,
+ "key_in_keys": false,
+ "detail": "BAD signature"
+ }
+ ],
+ "keys_fingerprints": [
+ "26389C7FF5982465406A1BDFB8E90D83CD36F86A"
+ ],
+ "paste_recipe": "curl -s 'https://dist.apache.org/repos/dist/release/airflow/KEYS' | gpg --import\ngpg --verify 'apache-airflow-2.11.0-source-release.tar.gz.asc' 'apache-airflow-2.11.0-source-release.tar.gz'\ngpg --verify 'apache-airflow-2.11.0-bin.tar.gz.asc' 'apache-airflow-2.11.0-bin.tar.gz'"
+}
+```
diff --git a/tools/skill-evals/evals/release-verify-rc/step-2-verify-signatures/fixtures/output-spec.md b/tools/skill-evals/evals/release-verify-rc/step-2-verify-signatures/fixtures/output-spec.md
index 6ef62fc7..8864b859 100644
--- a/tools/skill-evals/evals/release-verify-rc/step-2-verify-signatures/fixtures/output-spec.md
+++ b/tools/skill-evals/evals/release-verify-rc/step-2-verify-signatures/fixtures/output-spec.md
@@ -29,4 +29,5 @@ Grading rules:
- `paste_recipe` must be a non-empty string containing the gpg import and verify commands.
- A `KEY-NOT-IN-KEYS` classification must cause `status` to be `"FAIL"` — a key absent
from the project trust anchor is never downgraded to a warning.
-- No extra keys are permitted in the response.
+- The response is the `release-verify signatures` JSON; diagnostic fields the tool adds
+ beyond this schema (`detail`, `keys_fingerprints`) may be passed through.
diff --git a/tools/skill-evals/evals/release-verify-rc/step-3-verify-checksums/fixtures/case-1-checksums-pass/expected.json b/tools/skill-evals/evals/release-verify-rc/step-3-verify-checksums/fixtures/case-1-checksums-pass/expected.json
index 9ec76c3a..e78c5b43 100644
--- a/tools/skill-evals/evals/release-verify-rc/step-3-verify-checksums/fixtures/case-1-checksums-pass/expected.json
+++ b/tools/skill-evals/evals/release-verify-rc/step-3-verify-checksums/fixtures/case-1-checksums-pass/expected.json
@@ -18,5 +18,5 @@
}
],
"deprecated_md5_present": false,
- "paste_recipe": "sha512sum --check apache-airflow-2.11.0-source-release.tar.gz.sha512\nsha256sum --check apache-airflow-2.11.0-source-release.tar.gz.sha256\nsha512sum --check apache-airflow-2.11.0-bin.tar.gz.sha512\nsha256sum --check apache-airflow-2.11.0-bin.tar.gz.sha256"
+ "paste_recipe": "sha512sum --check 'apache-airflow-2.11.0-source-release.tar.gz.sha512'\nsha256sum --check 'apache-airflow-2.11.0-source-release.tar.gz.sha256'\nsha512sum --check 'apache-airflow-2.11.0-bin.tar.gz.sha512'\nsha256sum --check 'apache-airflow-2.11.0-bin.tar.gz.sha256'"
}
diff --git a/tools/skill-evals/evals/release-verify-rc/step-3-verify-checksums/fixtures/case-1-checksums-pass/report.md b/tools/skill-evals/evals/release-verify-rc/step-3-verify-checksums/fixtures/case-1-checksums-pass/report.md
index e4c2c7e7..4209d73f 100644
--- a/tools/skill-evals/evals/release-verify-rc/step-3-verify-checksums/fixtures/case-1-checksums-pass/report.md
+++ b/tools/skill-evals/evals/release-verify-rc/step-3-verify-checksums/fixtures/case-1-checksums-pass/report.md
@@ -1,14 +1,54 @@
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
+RC: 2.11.0-rc1
+release-build.md expected artefacts: apache-airflow-2.11.0-source-release.tar.gz, apache-airflow-2.11.0-bin.tar.gz
release-build.md digest set: sha512, sha256
+No .md5 files in the staging directory.
-Artefacts and digest files present:
- apache-airflow-2.11.0-source-release.tar.gz
- apache-airflow-2.11.0-source-release.tar.gz.sha512 — sha512sum --check: OK
- apache-airflow-2.11.0-source-release.tar.gz.sha256 — sha256sum --check: OK
- apache-airflow-2.11.0-bin.tar.gz
- apache-airflow-2.11.0-bin.tar.gz.sha512 — sha512sum --check: OK
- apache-airflow-2.11.0-bin.tar.gz.sha256 — sha256sum --check: OK
+Tool run:
-No .md5 files found in the staging directory.
+```bash
+uv run --project .apache-magpie/tools/release-verify release-verify checksums \
+ --dir staging --expect apache-airflow-2.11.0-source-release.tar.gz --expect apache-airflow-2.11.0-bin.tar.gz \
+ --digest sha512 --digest sha256
+```
+
+Tool output:
+
+```json
+{
+ "step": "checksums",
+ "status": "PASS",
+ "results": [
+ {
+ "file": "apache-airflow-2.11.0-source-release.tar.gz",
+ "digests": [
+ {
+ "type": "sha512",
+ "classification": "PASS"
+ },
+ {
+ "type": "sha256",
+ "classification": "PASS"
+ }
+ ]
+ },
+ {
+ "file": "apache-airflow-2.11.0-bin.tar.gz",
+ "digests": [
+ {
+ "type": "sha512",
+ "classification": "PASS"
+ },
+ {
+ "type": "sha256",
+ "classification": "PASS"
+ }
+ ]
+ }
+ ],
+ "deprecated_md5_present": false,
+ "paste_recipe": "sha512sum --check 'apache-airflow-2.11.0-source-release.tar.gz.sha512'\nsha256sum --check 'apache-airflow-2.11.0-source-release.tar.gz.sha256'\nsha512sum --check 'apache-airflow-2.11.0-bin.tar.gz.sha512'\nsha256sum --check 'apache-airflow-2.11.0-bin.tar.gz.sha256'"
+}
+```
diff --git a/tools/skill-evals/evals/release-verify-rc/step-3-verify-checksums/fixtures/case-2-checksum-mismatch/expected.json b/tools/skill-evals/evals/release-verify-rc/step-3-verify-checksums/fixtures/case-2-checksum-mismatch/expected.json
index 322c543b..b9445d91 100644
--- a/tools/skill-evals/evals/release-verify-rc/step-3-verify-checksums/fixtures/case-2-checksum-mismatch/expected.json
+++ b/tools/skill-evals/evals/release-verify-rc/step-3-verify-checksums/fixtures/case-2-checksum-mismatch/expected.json
@@ -18,5 +18,5 @@
}
],
"deprecated_md5_present": false,
- "paste_recipe": "sha512sum --check apache-airflow-2.11.0-source-release.tar.gz.sha512\nsha256sum --check apache-airflow-2.11.0-source-release.tar.gz.sha256\nsha512sum --check apache-airflow-2.11.0-bin.tar.gz.sha512\nsha256sum --check apache-airflow-2.11.0-bin.tar.gz.sha256"
+ "paste_recipe": "sha512sum --check 'apache-airflow-2.11.0-source-release.tar.gz.sha512'\nsha256sum --check 'apache-airflow-2.11.0-source-release.tar.gz.sha256'\nsha512sum --check 'apache-airflow-2.11.0-bin.tar.gz.sha512'\nsha256sum --check 'apache-airflow-2.11.0-bin.tar.gz.sha256'"
}
diff --git a/tools/skill-evals/evals/release-verify-rc/step-3-verify-checksums/fixtures/case-2-checksum-mismatch/report.md b/tools/skill-evals/evals/release-verify-rc/step-3-verify-checksums/fixtures/case-2-checksum-mismatch/report.md
index 0862c193..b144fd4b 100644
--- a/tools/skill-evals/evals/release-verify-rc/step-3-verify-checksums/fixtures/case-2-checksum-mismatch/report.md
+++ b/tools/skill-evals/evals/release-verify-rc/step-3-verify-checksums/fixtures/case-2-checksum-mismatch/report.md
@@ -1,15 +1,54 @@
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
+RC: 2.11.0-rc1
+release-build.md expected artefacts: apache-airflow-2.11.0-source-release.tar.gz, apache-airflow-2.11.0-bin.tar.gz
release-build.md digest set: sha512, sha256
+No .md5 files in the staging directory.
-Artefacts and digest files present:
- apache-airflow-2.11.0-source-release.tar.gz
- apache-airflow-2.11.0-source-release.tar.gz.sha512 — sha512sum --check: OK
- apache-airflow-2.11.0-source-release.tar.gz.sha256 — sha256sum --check: OK
- apache-airflow-2.11.0-bin.tar.gz
- apache-airflow-2.11.0-bin.tar.gz.sha512 — sha512sum --check: FAILED
- (reported hash: abc123...; computed hash: def456... — mismatch)
- apache-airflow-2.11.0-bin.tar.gz.sha256 — sha256sum --check: OK
+Tool run:
-No .md5 files found in the staging directory.
+```bash
+uv run --project .apache-magpie/tools/release-verify release-verify checksums \
+ --dir staging --expect apache-airflow-2.11.0-source-release.tar.gz --expect apache-airflow-2.11.0-bin.tar.gz \
+ --digest sha512 --digest sha256
+```
+
+Tool output:
+
+```json
+{
+ "step": "checksums",
+ "status": "FAIL",
+ "results": [
+ {
+ "file": "apache-airflow-2.11.0-source-release.tar.gz",
+ "digests": [
+ {
+ "type": "sha512",
+ "classification": "PASS"
+ },
+ {
+ "type": "sha256",
+ "classification": "PASS"
+ }
+ ]
+ },
+ {
+ "file": "apache-airflow-2.11.0-bin.tar.gz",
+ "digests": [
+ {
+ "type": "sha512",
+ "classification": "MISMATCH"
+ },
+ {
+ "type": "sha256",
+ "classification": "PASS"
+ }
+ ]
+ }
+ ],
+ "deprecated_md5_present": false,
+ "paste_recipe": "sha512sum --check 'apache-airflow-2.11.0-source-release.tar.gz.sha512'\nsha256sum --check 'apache-airflow-2.11.0-source-release.tar.gz.sha256'\nsha512sum --check 'apache-airflow-2.11.0-bin.tar.gz.sha512'\nsha256sum --check 'apache-airflow-2.11.0-bin.tar.gz.sha256'"
+}
+```
diff --git a/tools/skill-evals/evals/release-verify-rc/step-3-verify-checksums/fixtures/output-spec.md b/tools/skill-evals/evals/release-verify-rc/step-3-verify-checksums/fixtures/output-spec.md
index 48b1fef7..1d350a91 100644
--- a/tools/skill-evals/evals/release-verify-rc/step-3-verify-checksums/fixtures/output-spec.md
+++ b/tools/skill-evals/evals/release-verify-rc/step-3-verify-checksums/fixtures/output-spec.md
@@ -26,11 +26,12 @@ The model must return ONLY valid JSON matching this schema:
```
Grading rules:
-- `status` must be `"FAIL"` if any `classification` is `"MISMATCH"` or
- required `MISSING-DIGEST`.
+- `status` must be `"FAIL"` if any non-md5 `classification` is `"MISMATCH"`
+ or the required sha512 is `MISSING-DIGEST`.
- `status` must be `"WARN"` if the only anomaly is a deprecated `md5` file
- being present.
-- `status` must be `"PASS"` if all required digests match and no anomalies.
+ being present, matching or mismatched (md5 never fails alone).
+- `status` must be `"PASS"` if sha512 and every staged optional digest match
+ and no anomalies.
- `deprecated_md5_present` must be `true` when a `.md5` file appears in
the staging directory; `false` otherwise.
- `paste_recipe` must be a non-empty string with the sha512sum / sha256sum
diff --git a/tools/skill-evals/evals/release-verify-rc/step-5-notice-license/fixtures/case-1-both-present/report.md b/tools/skill-evals/evals/release-verify-rc/step-5-notice-license/fixtures/case-1-both-present/report.md
index 625d87bd..d01482e9 100644
--- a/tools/skill-evals/evals/release-verify-rc/step-5-notice-license/fixtures/case-1-both-present/report.md
+++ b/tools/skill-evals/evals/release-verify-rc/step-5-notice-license/fixtures/case-1-both-present/report.md
@@ -1,15 +1,31 @@
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
-RC source artefact: apache-airflow-2.11.0-source-release.tar.gz (unpacked)
-
-Files at artefact root:
- NOTICE — present
- LICENSE — present
+RC source artefact: apache-airflow-2.11.0-source-release.tar.gz (unpacked to apache-airflow-2.11.0-source-release/)
Previous release found at:
https://dist.apache.org/repos/dist/release/airflow/2.10.0/apache-airflow-2.10.0-source-release.tar.gz
+ (its NOTICE and LICENSE fetched into previous/)
+
+Tool run:
+
+```bash
+uv run --project .apache-magpie/tools/release-verify release-verify notice-license \
+ --tree apache-airflow-2.11.0-source-release --previous previous
+```
+
+Tool output:
-Diff results:
- NOTICE diff: 2 lines changed (version string updated from 2.10.0 to 2.11.0 in header)
- LICENSE diff: 0 lines changed (no changes)
+```json
+{
+ "step": "notice-license",
+ "status": "REVIEW",
+ "notice_present": true,
+ "license_present": true,
+ "notice_diff_lines": 2,
+ "notice_diff": "--- previous/NOTICE\n+++ rc/NOTICE\n@@ -1,4 +1,4 @@\n-Apache Airflow 2.10.0\n+Apache Airflow 2.11.0\n Copyright 2016-2026 The Apache Software Foundation\n \n This product includes software developed at\n",
+ "license_diff_lines": 0,
+ "license_diff": "",
+ "previous": "previous"
+}
+```
diff --git a/tools/skill-evals/evals/release-verify-rc/step-5-notice-license/fixtures/case-2-missing-notice/report.md b/tools/skill-evals/evals/release-verify-rc/step-5-notice-license/fixtures/case-2-missing-notice/report.md
index 81192f5a..54e3582e 100644
--- a/tools/skill-evals/evals/release-verify-rc/step-5-notice-license/fixtures/case-2-missing-notice/report.md
+++ b/tools/skill-evals/evals/release-verify-rc/step-5-notice-license/fixtures/case-2-missing-notice/report.md
@@ -1,10 +1,31 @@
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
-RC source artefact: apache-airflow-2.11.0-source-release.tar.gz (unpacked)
+RC source artefact: apache-airflow-2.11.0-source-release.tar.gz (unpacked to apache-airflow-2.11.0-source-release/)
-Files at artefact root:
- NOTICE — ABSENT (not found at root of unpacked artefact)
- LICENSE — present
+The unpacked RC artefact has a LICENSE at its root but no NOTICE.
+No previous release was fetched for this run.
-No diff possible: NOTICE is absent from the current RC.
+Tool run:
+
+```bash
+uv run --project .apache-magpie/tools/release-verify release-verify notice-license \
+ --tree apache-airflow-2.11.0-source-release
+```
+
+Tool output:
+
+```json
+{
+ "step": "notice-license",
+ "status": "FAIL",
+ "notice_present": false,
+ "license_present": true,
+ "notice_diff_lines": null,
+ "notice_diff": null,
+ "license_diff_lines": null,
+ "license_diff": null,
+ "previous": null,
+ "detail": "NOTICE absent from the root of the current RC artefact (apache-airflow-2.11.0-source-release); this RC is defective whatever any previous release contains. Diff line counts are null because there is no file to diff."
+}
+```
diff --git a/tools/skill-evals/evals/release-verify-rc/step-5-notice-license/fixtures/output-spec.md b/tools/skill-evals/evals/release-verify-rc/step-5-notice-license/fixtures/output-spec.md
index 609cd43d..61e1a13c 100644
--- a/tools/skill-evals/evals/release-verify-rc/step-5-notice-license/fixtures/output-spec.md
+++ b/tools/skill-evals/evals/release-verify-rc/step-5-notice-license/fixtures/output-spec.md
@@ -26,5 +26,6 @@ Grading rules:
- `notice_diff_lines` and `license_diff_lines` are the number of diff lines when a
previous release was found; `null` when no previous release was found.
- `diff_summary` must be a non-empty one-line description of the changes or
- "no diff — no previous release found" or "no changes".
+ "no diff — no previous release found" or "no changes"; on `FAIL` it names the
+ file the current RC artefact lacks, never the absence of a previous release.
- No extra keys are permitted in the response.
diff --git a/tools/skill-evals/evals/release-verify-rc/step-6-binary-exclusion/fixtures/case-1-no-prohibited-binaries/expected.json b/tools/skill-evals/evals/release-verify-rc/step-6-binary-exclusion/fixtures/case-1-no-prohibited-binaries/expected.json
index e038b864..bc5ad6a7 100644
--- a/tools/skill-evals/evals/release-verify-rc/step-6-binary-exclusion/fixtures/case-1-no-prohibited-binaries/expected.json
+++ b/tools/skill-evals/evals/release-verify-rc/step-6-binary-exclusion/fixtures/case-1-no-prohibited-binaries/expected.json
@@ -3,5 +3,5 @@
"status": "PASS",
"prohibited_found": [],
"expected_binaries": [],
- "paste_recipe": "find apache-airflow-2.11.0-source-release \\( -type f \\( -name '*.class' -o -name '*.jar' -o -name '*.so' -o -name '*.dylib' -o -name '*.dll' -o -name '*.exe' -o -name '*.pyc' \\) -o -type d -name '__pycache__' \\) -print"
+ "paste_recipe": "find 'apache-airflow-2.11.0-source-release' \\( -type f \\( -name '*.class' -o -name '*.jar' -o -name '*.so' -o -name '*.dylib' -o -name '*.dll' -o -name '*.exe' -o -name '*.pyc' \\) -o -type d -name '__pycache__' \\) -print"
}
diff --git a/tools/skill-evals/evals/release-verify-rc/step-6-binary-exclusion/fixtures/case-1-no-prohibited-binaries/report.md b/tools/skill-evals/evals/release-verify-rc/step-6-binary-exclusion/fixtures/case-1-no-prohibited-binaries/report.md
index cd64c4c6..0fa0b06a 100644
--- a/tools/skill-evals/evals/release-verify-rc/step-6-binary-exclusion/fixtures/case-1-no-prohibited-binaries/report.md
+++ b/tools/skill-evals/evals/release-verify-rc/step-6-binary-exclusion/fixtures/case-1-no-prohibited-binaries/report.md
@@ -2,9 +2,25 @@
https://www.apache.org/licenses/LICENSE-2.0 -->
release-build.md Binary-exclude list (known-accepted): *.class, *.jar
-(no additional scan globs beyond the fixed baseline)
+(no additional prohibited globs beyond the fixed baseline)
-Scan of unpacked apache-airflow-2.11.0-source-release.tar.gz:
- Files with prohibited extensions found: none
- Files listed as EXPECTED-BINARY in release-build.md: none
- No .class, .jar, .so, .dylib, .dll, .exe, .pyc, or __pycache__ detected in the source artefact.
+RC source artefact: apache-airflow-2.11.0-source-release.tar.gz (unpacked to apache-airflow-2.11.0-source-release/)
+
+Tool run:
+
+```bash
+uv run --project .apache-magpie/tools/release-verify release-verify binaries \
+ --tree apache-airflow-2.11.0-source-release --accept '*.class' --accept '*.jar'
+```
+
+Tool output:
+
+```json
+{
+ "step": "binary-exclusion",
+ "status": "PASS",
+ "prohibited_found": [],
+ "expected_binaries": [],
+ "paste_recipe": "find 'apache-airflow-2.11.0-source-release' \\( -type f \\( -name '*.class' -o -name '*.jar' -o -name '*.so' -o -name '*.dylib' -o -name '*.dll' -o -name '*.exe' -o -name '*.pyc' \\) -o -type d -name '__pycache__' \\) -print"
+}
+```
diff --git a/tools/skill-evals/evals/release-verify-rc/step-6-binary-exclusion/fixtures/case-2-prohibited-binary-found/expected.json b/tools/skill-evals/evals/release-verify-rc/step-6-binary-exclusion/fixtures/case-2-prohibited-binary-found/expected.json
index 8da76355..4c1e0f0f 100644
--- a/tools/skill-evals/evals/release-verify-rc/step-6-binary-exclusion/fixtures/case-2-prohibited-binary-found/expected.json
+++ b/tools/skill-evals/evals/release-verify-rc/step-6-binary-exclusion/fixtures/case-2-prohibited-binary-found/expected.json
@@ -3,5 +3,5 @@
"status": "FAIL",
"prohibited_found": ["airflow/vendor/some-lib/some-lib-1.0.jar"],
"expected_binaries": [],
- "paste_recipe": "find apache-airflow-2.11.0-source-release \\( -type f \\( -name '*.class' -o -name '*.jar' -o -name '*.so' -o -name '*.dylib' -o -name '*.dll' -o -name '*.exe' -o -name '*.pyc' \\) -o -type d -name '__pycache__' \\) -print"
+ "paste_recipe": "find 'apache-airflow-2.11.0-source-release' \\( -type f \\( -name '*.class' -o -name '*.jar' -o -name '*.so' -o -name '*.dylib' -o -name '*.dll' -o -name '*.exe' -o -name '*.pyc' \\) -o -type d -name '__pycache__' \\) -print"
}
diff --git a/tools/skill-evals/evals/release-verify-rc/step-6-binary-exclusion/fixtures/case-2-prohibited-binary-found/report.md b/tools/skill-evals/evals/release-verify-rc/step-6-binary-exclusion/fixtures/case-2-prohibited-binary-found/report.md
index a97259f6..23dc4a6c 100644
--- a/tools/skill-evals/evals/release-verify-rc/step-6-binary-exclusion/fixtures/case-2-prohibited-binary-found/report.md
+++ b/tools/skill-evals/evals/release-verify-rc/step-6-binary-exclusion/fixtures/case-2-prohibited-binary-found/report.md
@@ -2,14 +2,27 @@
https://www.apache.org/licenses/LICENSE-2.0 -->
release-build.md Binary-exclude list (known-accepted): *.class
-(no additional scan globs beyond the fixed baseline)
+(no additional prohibited globs beyond the fixed baseline)
-Note: *.jar is NOT marked known-and-accepted — jar files matching the
-fixed baseline are prohibited in this source artefact.
+RC source artefact: apache-airflow-2.11.0-source-release.tar.gz (unpacked to apache-airflow-2.11.0-source-release/)
-Scan of unpacked apache-airflow-2.11.0-source-release.tar.gz:
- Files with prohibited extensions found:
- airflow/vendor/some-lib/some-lib-1.0.jar (*.jar — prohibited, not known-accepted)
+Tool run:
- Files listed as EXPECTED-BINARY in release-build.md:
- (none with *.class extension found either)
+```bash
+uv run --project .apache-magpie/tools/release-verify release-verify binaries \
+ --tree apache-airflow-2.11.0-source-release --accept '*.class'
+```
+
+Tool output:
+
+```json
+{
+ "step": "binary-exclusion",
+ "status": "FAIL",
+ "prohibited_found": [
+ "airflow/vendor/some-lib/some-lib-1.0.jar"
+ ],
+ "expected_binaries": [],
+ "paste_recipe": "find 'apache-airflow-2.11.0-source-release' \\( -type f \\( -name '*.class' -o -name '*.jar' -o -name '*.so' -o -name '*.dylib' -o -name '*.dll' -o -name '*.exe' -o -name '*.pyc' \\) -o -type d -name '__pycache__' \\) -print"
+}
+```
diff --git a/tools/skill-evals/evals/release-verify-rc/step-8-version-consistency/fixtures/case-1-version-consistent/report.md b/tools/skill-evals/evals/release-verify-rc/step-8-version-consistency/fixtures/case-1-version-consistent/report.md
index c72ca859..7426e0d1 100644
--- a/tools/skill-evals/evals/release-verify-rc/step-8-version-consistency/fixtures/case-1-version-consistent/report.md
+++ b/tools/skill-evals/evals/release-verify-rc/step-8-version-consistency/fixtures/case-1-version-consistent/report.md
@@ -1,14 +1,35 @@
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
-RC tag: 2.11.0-rc1 → expected version: 2.11.0
+RC tag: 2.11.0-rc1
version_manifest_files (from release-management-config.md): setup.cfg, airflow/__init__.py
-Extracted version strings:
- setup.cfg:
- [metadata] section → version = 2.11.0
- airflow/__init__.py:
- __version__ = "2.11.0"
+Tool run:
-Both manifest files report 2.11.0, which matches the expected version exactly.
+```bash
+uv run --project .apache-magpie/tools/release-verify release-verify version \
+ --tree apache-airflow-2.11.0-source-release --rc-tag 2.11.0-rc1 --manifest setup.cfg --manifest airflow/__init__.py
+```
+
+Tool output:
+
+```json
+{
+ "step": "version-consistency",
+ "status": "PASS",
+ "expected_version": "2.11.0",
+ "results": [
+ {
+ "file": "setup.cfg",
+ "extracted": "2.11.0",
+ "match": true
+ },
+ {
+ "file": "airflow/__init__.py",
+ "extracted": "2.11.0",
+ "match": true
+ }
+ ]
+}
+```
diff --git a/tools/skill-evals/evals/release-verify-rc/step-8-version-consistency/fixtures/case-2-version-mismatch/report.md b/tools/skill-evals/evals/release-verify-rc/step-8-version-consistency/fixtures/case-2-version-mismatch/report.md
index a27369ce..443b8748 100644
--- a/tools/skill-evals/evals/release-verify-rc/step-8-version-consistency/fixtures/case-2-version-mismatch/report.md
+++ b/tools/skill-evals/evals/release-verify-rc/step-8-version-consistency/fixtures/case-2-version-mismatch/report.md
@@ -1,15 +1,35 @@
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
-RC tag: 2.11.0-rc1 → expected version: 2.11.0
+RC tag: 2.11.0-rc1
version_manifest_files (from release-management-config.md): setup.cfg, airflow/__init__.py
-Extracted version strings:
- setup.cfg:
- [metadata] section → version = 2.11.0
- airflow/__init__.py:
- __version__ = "2.11.0.dev0"
+Tool run:
-Note: airflow/__init__.py still carries the dev suffix "2.11.0.dev0" which was
-not updated to "2.11.0" before the RC was cut. This is a version-string mismatch.
+```bash
+uv run --project .apache-magpie/tools/release-verify release-verify version \
+ --tree apache-airflow-2.11.0-source-release --rc-tag 2.11.0-rc1 --manifest setup.cfg --manifest airflow/__init__.py
+```
+
+Tool output:
+
+```json
+{
+ "step": "version-consistency",
+ "status": "FAIL",
+ "expected_version": "2.11.0",
+ "results": [
+ {
+ "file": "setup.cfg",
+ "extracted": "2.11.0",
+ "match": true
+ },
+ {
+ "file": "airflow/__init__.py",
+ "extracted": "2.11.0.dev0",
+ "match": false
+ }
+ ]
+}
+```
diff --git a/uv.lock b/uv.lock
index 606ff601..94f16848 100644
--- a/uv.lock
+++ b/uv.lock
@@ -44,6 +44,7 @@ members = [
"preflight-audit",
"redactor",
"release-config",
+ "release-verify",
"reproducible-archive",
"sandbox-lint",
"security-tracker-stats-dashboard",
@@ -1723,6 +1724,21 @@ dev = [
[package.metadata.requires-dev]
dev = [{ name = "magpie-dev", editable = "tools/dev" }]
+[[package]]
+name = "release-verify"
+version = "0.1.0"
+source = { editable = "tools/release-verify" }
+
+[package.dev-dependencies]
+dev = [
+ { name = "magpie-dev" },
+]
+
+[package.metadata]
+
+[package.metadata.requires-dev]
+dev = [{ name = "magpie-dev", editable = "tools/dev" }]
+
[[package]]
name = "reproducible-archive"
version = "0.1.0"