You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@magpie.apache.org by po...@apache.org on 2026/09/16 10:20:36 UTC
(magpie) branch main updated: feat(cve-json)!: require a purl on every affected entry (#1247)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/magpie.git
The following commit(s) were added to refs/heads/main by this push:
new aa078470 feat(cve-json)!: require a purl on every affected entry (#1247)
aa078470 is described below
commit aa0784704f99fe3497906d488e308dbb105f844b
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Wed Sep 16 12:20:30 2026 +0200
feat(cve-json)!: require a purl on every affected entry (#1247)
* feat(cve-json)!: require a purl on every affected entry
#1197 added `affected[].packageURL` as opt-in, off by default, so that an
adopter who set neither new key got byte-identical records. In practice a
record without a purl cannot be promoted by the CNA, so "off by default"
means the default configuration silently produces records that are
rejected later -- after allocation, after the advisory is drafted, at the
point where the cost of going back is highest.
The purl is now produced for every entry, by three means in order:
1. `product.purl_type`, when configured, as before.
2. Otherwise derived from `product.default_collection_url`. An adopter
publishing to PyPI has already said so; needing to repeat it in a
second key is what left the field unset. Only hosts whose purl type is
unambiguous are mapped (PyPI, npm, crates.io, RubyGems, NuGet) -- a
guessed type yields a valid-looking identifier pointing at the wrong
ecosystem.
3. Otherwise a ValueError naming the key to set, instead of an emitted
record missing the field.
`purl_type = "none"` opts out deliberately, for a product with no package
host -- a Helm chart served from the project's own site, say. The point is
that a record without a purl is now a decision someone wrote down rather
than a default nobody noticed.
This is the same posture the version parser already takes: it refuses to
emit a record whose `version` is a range expression rather than shipping
JSON that Vulnogram will reject.
BREAKING CHANGE: an adopter whose `purl_type` is unset and whose
`default_collection_url` is not a recognised package host now gets an
error instead of a record without a purl. Set `product.purl_type`, or set
it to "none" to opt out explicitly.
Generated-by: Claude Opus 5
* feat(cve-json)!: emit the purl as the only package identifier
Review feedback on #1247: apache/security-vulnogram#253 promoted the
Package URL to the *recommended* identifier and made the CVE tool derive
`collectionURL` / `packageName` from it when a record is serialised for
publication -- while reporting a stored pair that disagrees with the
purl. Writing both leaves two identifiers free to drift apart, so an
entry now carries one or the other:
* a `packageURL` alone, for every product that has a purl;
* the legacy `collectionURL` / `packageName` pair only for a product
configured `purl_type = "none"`, where there is no purl to derive them
from.
Two consumers read that identifier and had to stop assuming
`packageName`:
* the attachment table's package and project-link cells, which now read
the name back out of the purl (`split_purl`) and map its type to a
project page -- PyPI, npm, crates.io, RubyGems, NuGet. The column is
no longer PyPI-specific, so it is headed "Project URL".
* the oauth-api push guard, which compares `(package, product)`
signatures against the already-published record. An older record
stores the legacy pair, so without normalising both shapes to the same
package the guard would refuse every push over a rename that never
happened.
BREAKING CHANGE: `affected[]` entries no longer carry `collectionURL` /
`packageName` unless the product is configured `purl_type = "none"`. The
CVE tool derives the pair from the purl on publication.
Generated-by: Claude Opus 5
---
.../cve-tool-vulnogram/generate-cve-json/SKILL.md | 58 ++++--
.../src/generate_cve_json/__init__.py | 6 +
.../src/generate_cve_json/cve_json.py | 182 ++++++++++++++---
.../tests/fixtures/cve-json-config-providers.toml | 3 +-
.../tests/test_generate_cve_json.py | 218 ++++++++++++++++++---
.../oauth-api/src/vulnogram_api/merge_mode.py | 40 +++-
.../oauth-api/src/vulnogram_api/record_update.py | 4 +-
.../oauth-api/tests/test_merge_mode.py | 31 +++
8 files changed, 466 insertions(+), 76 deletions(-)
diff --git a/tools/cve-tool-vulnogram/generate-cve-json/SKILL.md b/tools/cve-tool-vulnogram/generate-cve-json/SKILL.md
index 95f80cb1..f92a4d8b 100644
--- a/tools/cve-tool-vulnogram/generate-cve-json/SKILL.md
+++ b/tools/cve-tool-vulnogram/generate-cve-json/SKILL.md
@@ -106,13 +106,40 @@ diff the two to see what the tool has added / what you changed by hand.
`product.default_package_name`, `product.default_collection_url`).
They are used as the identity for *Affected versions* lines that
don't start with a recognisable per-package directory name
- (see the multi-product note below).
- - `product.purl_type` *(optional, config only)* — the Package URL type
- for the project's packages (`pypi`, `npm`, `cargo`, …). When set,
- every `affected[]` entry also carries a `packageURL`
- (`pkg:<type>/<packageName>`); when unset the field is omitted
- entirely. Per the CVE Record Format the purl never includes a
- version — the entry's `versions[]` carries the range.
+ (see the multi-product note below). `--collection-url` reaches the
+ record only through the purl it is used to derive (see
+ `product.purl_type` below) — except for a product that opts out of
+ purls, the one case a record still carries a `collectionURL`.
+ - `product.purl_type` *(config only)* — the Package URL type
+ for the project's packages (`pypi`, `npm`, `cargo`, …). Every
+ `affected[]` entry carries a `packageURL` (`pkg:<type>/<packageName>`).
+ Per the CVE Record Format the purl never includes a version — the
+ entry's `versions[]` carries the range.
+
+ **The purl is the only package identifier the record carries.** The
+ ASF CVE tool treats the Package URL as the recommended identifier and
+ derives the legacy `collectionURL` / `packageName` pair from it when
+ the record is serialised for publication, so the generator emits the
+ purl alone. Writing the pair as well would hand the tool two
+ identifiers that can disagree — which it reports rather than
+ silently reconciles.
+
+ **A purl is required, not optional.** A record needs one to be
+ promoted, so the generator refuses to emit a record it cannot build
+ a purl for rather than producing one the CNA will reject later.
+ Leaving the key unset is fine when the type can be **derived from
+ `product.default_collection_url`** — PyPI, npm, crates.io,
+ RubyGems and NuGet hosts are recognised — and only hosts whose type
+ is unambiguous are mapped, because a guessed type yields a
+ valid-looking identifier pointing at the wrong ecosystem.
+
+ Set `purl_type = "none"` to state deliberately that a product has no
+ package host — a Helm chart served from the project's own site, for
+ instance. That is an explicit decision rather than an omission, which
+ is the distinction the previous optional behaviour lost. Those
+ entries — and only those — carry `collectionURL` / `packageName`
+ instead, because there is no purl for the CVE tool to derive them
+ from.
Name normalisation follows the package-url spec per type, and is
implemented for the types whose rules have been read from it:
`pypi` (lowercased, `_` becomes `-`) and `npm` (scope becomes the
@@ -121,14 +148,15 @@ diff the two to see what the tool has added / what you changed by hand.
mixed-case npm packages were grandfathered in). Any other type has
its name passed through unchanged, and a name needing namespace
semantics is covered by `product.purl_namespace` below, and without
- one such a type emits no purl rather than a guessed one.
+ one such a type is an error rather than a guessed purl.
- `product.purl_namespace` *(optional, config only)* — the namespace
for purl types that require one: a Maven `groupId`, a Composer
vendor, a Go module prefix. Types the spec gives no namespace
(`pypi`, `cargo`, `gem`, `nuget`) ignore it, and an npm scope
carried in the package name itself wins over it. A type that
- requires a namespace and has none emits no purl — a guessed Maven
- groupId would point at another organisation's artifact.
+ requires a namespace and has none is an error naming this key — a
+ guessed Maven groupId would point at another organisation's artifact,
+ and silently omitting the purl produces an unpromotable record.
- `--product-for PACKAGE=PRODUCT` — override the CVE product display
name for a specific `packageName`. Repeat to override multiple
packages. Useful when a package is not in the project's
@@ -182,7 +210,7 @@ filled in through a prior `security-issue-sync` run. In particular:
**Multi-product CVEs are supported** — put one package per line,
prefixing each with the package directory name as it appears in
the adopter's repo, and the script emits one `affected[]` entry
- per line with the right `product` / `packageName`. Example
+ per line with the right `product` and package identity. Example
(illustrative — using a hypothetical `apache-foo` project's
sub-project layout):
@@ -381,9 +409,11 @@ same tracking-issue body produces the same JSON bytes.
The generated record matches what Vulnogram exports after a save,
minus editor cruft. Notable fields:
-- `containers.cna.affected[]` — `vendor`, `product`, `collectionURL`,
- `packageName` and a `versions[]` entry with `version`, `lessThan`,
- `status: "affected"`, `versionType: "semver"`.
+- `containers.cna.affected[]` — `vendor`, `product`, `packageURL` (the
+ version-less purl) and a `versions[]` entry with `version`, `lessThan`,
+ `status: "affected"`, `versionType: "semver"`. An entry for a product
+ configured with `purl_type = "none"` carries `collectionURL` /
+ `packageName` in place of the purl.
- `containers.cna.descriptions[]` — both a plain `value` and an HTML
`supportingMedia` alternative (Vulnogram's WYSIWYG mode needs both).
- `containers.cna.problemTypes[].descriptions[]` — `cweId`,
diff --git a/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/__init__.py b/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/__init__.py
index 49b70a8d..64d9cb91 100644
--- a/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/__init__.py
+++ b/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/__init__.py
@@ -28,6 +28,7 @@ from generate_cve_json.cve_json import (
COLLECTION_URL_TO_PROJECT_URL_TEMPLATE,
NEXT_VERSION_TOKEN_RE,
PROJECT_DISPLAY_MAP,
+ PURL_TYPE_TO_PROJECT_URL_TEMPLATE,
_build_attachment_body,
_has_vendor_advisory_reference,
_is_cna_ready_for_review,
@@ -56,7 +57,9 @@ from generate_cve_json.cve_json import (
parse_cve_id,
parse_cwe,
parse_url_list,
+ resolve_purl_type,
resolve_title,
+ split_purl,
wrap_cve_record,
)
@@ -64,6 +67,7 @@ __all__ = [
"COLLECTION_URL_TO_PROJECT_URL_TEMPLATE",
"NEXT_VERSION_TOKEN_RE",
"PROJECT_DISPLAY_MAP",
+ "PURL_TYPE_TO_PROJECT_URL_TEMPLATE",
"_build_attachment_body",
"_has_vendor_advisory_reference",
"_is_cna_ready_for_review",
@@ -92,6 +96,8 @@ __all__ = [
"parse_cve_id",
"parse_cwe",
"parse_url_list",
+ "resolve_purl_type",
"resolve_title",
+ "split_purl",
"wrap_cve_record",
]
diff --git a/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/cve_json.py b/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/cve_json.py
index cf59bf0b..b961e942 100644
--- a/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/cve_json.py
+++ b/tools/cve-tool-vulnogram/generate-cve-json/src/generate_cve_json/cve_json.py
@@ -27,10 +27,13 @@ The output shape is modelled on a real CVE record exported from
Vulnogram so that a paste-and-save round-trip does not mangle the
record. In particular, the script produces:
-* ``affected[]`` entries with ``vendor``, ``product``, ``collectionURL``
- (PyPI), ``packageName`` (``apache-airflow``), an optional
- ``packageURL`` (a version-less purl, emitted only when the project
- configures ``product.purl_type``) and a semver range;
+* ``affected[]`` entries with ``vendor``, ``product``, a version-less
+ ``packageURL`` (``pkg:pypi/apache-airflow``) and a semver range. The
+ purl is the CNA's recommended package identifier: Vulnogram derives
+ the legacy ``collectionURL`` / ``packageName`` pair from it when the
+ record is serialised for publication, so the generator emits that
+ pair only for a product with no purl at all
+ (``product.purl_type = "none"``);
* a ``descriptions[]`` entry with both a plain ``value`` and an HTML
``supportingMedia`` alternative (Vulnogram's WYSIWYG mode expects
both);
@@ -301,8 +304,10 @@ DEFAULT_VENDOR: str = ""
DEFAULT_PRODUCT: str = ""
DEFAULT_PACKAGE_NAME: str = ""
DEFAULT_COLLECTION_URL: str = ""
-# Optional: empty unless the project sets `product.purl_type`, in which case
-# every affected entry also carries a version-less `packageURL`.
+# The purl type for the project's packages. Empty means "derive it from
+# `product.default_collection_url`"; `"none"` opts the product out of purls
+# and falls its entries back to the legacy `collectionURL` / `packageName`
+# pair.
DEFAULT_PURL_TYPE: str = ""
# Optional: the namespace for types that require one — a Maven groupId, a
# Composer vendor, a Go module prefix. Ignored for types that have none.
@@ -452,6 +457,9 @@ NEXT_VERSION_TOKEN_RE = re.compile(r"<\s*NEXT\s+VERSION\b", re.IGNORECASE)
# Only collection URLs Airflow actually emits today need entries here —
# unknown URLs fall through to ``None`` and the attachment table
# renders ``—`` for them.
+# Entries normally carry a purl rather than the legacy pair, and the table
+# reads their link out of :data:`PURL_TYPE_TO_PROJECT_URL_TEMPLATE`; this map
+# serves the ``purl_type = "none"`` entries that still carry a collection URL.
COLLECTION_URL_TO_PROJECT_URL_TEMPLATE: dict[str, str] = {
"https://pypi.python.org": "https://pypi.org/project/{package}/",
"https://pypi.org": "https://pypi.org/project/{package}/",
@@ -1084,10 +1092,10 @@ def build_affected(
The *Affected versions* body field is parsed line by line. Each line
that starts with an Airflow package directory name
(``apache-airflow`` or ``apache-airflow-project-<dir>``) creates
- an entry with that package's derived product / packageName. Lines
- without a recognisable prefix — or an empty field — fall back to
- the explicit ``product`` / ``package_name`` arguments, which is the
- historical single-entry shape.
+ an entry with that package's derived product / package identity.
+ Lines without a recognisable prefix — or an empty field — fall back
+ to the explicit ``product`` / ``package_name`` arguments, which is
+ the historical single-entry shape.
``product_overrides`` is consulted *before*
:data:`PROJECT_DISPLAY_MAP` and lets callers shadow the
@@ -1107,15 +1115,38 @@ def build_affected(
entry_package = package_name
entry_product = product
version_source = version_expr or affected_versions_value
- # Keys are inserted in alphabetical order; `packageURL` is optional
- # and omitted entirely when no purl can be built for the package.
- entry: dict = {
- "collectionURL": collection_url,
- "defaultStatus": "unaffected",
- "packageName": entry_package,
- }
- purl = compute_purl(purl_type or "", entry_package, namespace=purl_namespace or "")
- if purl:
+ # The purl is the CNA's recommended package identifier, and the CVE
+ # tool derives `collectionURL` / `packageName` from it when both are
+ # absent — while reporting a stored pair that disagrees with the purl.
+ # So an entry carries one or the other and never both: the purl when
+ # there is one, the legacy pair only for a product that opted out of
+ # purls with `purl_type = "none"` because its ecosystem has none.
+ resolved_type = resolve_purl_type(purl_type or "", collection_url)
+ purl: str | None = None
+ if resolved_type != PURL_TYPE_NONE:
+ purl = compute_purl(resolved_type, entry_package, namespace=purl_namespace or "")
+ if not purl:
+ raise ValueError(
+ f"Could not build a Package URL (purl) for {entry_package!r}. "
+ f"A CVE record needs `affected[].packageURL` to be promoted, "
+ f"so the record is not emitted without one.\n"
+ f"Fix one of:\n"
+ f' - set `product.purl_type` (e.g. "pypi", "npm", "maven") '
+ f"when it cannot be derived from "
+ f"`product.default_collection_url` ({collection_url!r});\n"
+ f" - set `product.purl_namespace` when the type requires one "
+ f"(maven groupId, composer vendor, go module prefix);\n"
+ f' - set `product.purl_type = "{PURL_TYPE_NONE}"` to state '
+ f"deliberately that this product has no package host."
+ )
+ # Keys are inserted in alphabetical order.
+ entry: dict = {}
+ if purl is None:
+ entry["collectionURL"] = collection_url
+ entry["defaultStatus"] = "unaffected"
+ if purl is None:
+ entry["packageName"] = entry_package
+ else:
entry["packageURL"] = purl
entry["product"] = entry_product
entry["vendor"] = vendor
@@ -1402,6 +1433,58 @@ def _encode_purl_segments(namespace: str) -> list[str]:
return [urllib.parse.quote(segment, safe="") for segment in namespace.split("/") if segment]
+# Package host -> purl type. Lets `purl_type` be derived from the
+# `collection_url` an adopter already configures, so the common case needs no
+# new config at all. Only hosts whose purl type is unambiguous are listed; a
+# guessed type produces a valid-looking identifier pointing at the wrong
+# ecosystem, which is worse than asking the adopter.
+_PURL_TYPE_BY_COLLECTION_HOST = {
+ "pypi.org": "pypi",
+ "pypi.python.org": "pypi",
+ "files.pythonhosted.org": "pypi",
+ "registry.npmjs.org": "npm",
+ "www.npmjs.com": "npm",
+ "npmjs.com": "npm",
+ "crates.io": "cargo",
+ "rubygems.org": "gem",
+ "www.nuget.org": "nuget",
+ "nuget.org": "nuget",
+}
+
+# Purl type -> canonical project-page URL template. The attachment table
+# renders a clickable link per package, and an entry now carries the purl
+# rather than the `collectionURL` / `packageName` pair that link used to be
+# built from — so the link is read back out of the purl instead. Only the
+# hosts `_PURL_TYPE_BY_COLLECTION_HOST` recognises need an entry; an
+# unmapped type renders `—`.
+PURL_TYPE_TO_PROJECT_URL_TEMPLATE: dict[str, str] = {
+ "pypi": "https://pypi.org/project/{package}/",
+ "npm": "https://www.npmjs.com/package/{package}",
+ "cargo": "https://crates.io/crates/{package}",
+ "gem": "https://rubygems.org/gems/{package}",
+ "nuget": "https://www.nuget.org/packages/{package}",
+}
+
+# Explicit opt-out for an adopter distributing outside any package host the
+# purl spec models. Spelled out rather than implied by an empty value, so a
+# record without a purl is always a decision someone wrote down.
+PURL_TYPE_NONE = "none"
+
+
+def resolve_purl_type(configured: str, collection_url: str) -> str:
+ """Return the purl type to use: the configured one, else derived from the
+ collection URL, else empty.
+
+ Derivation exists so that `purl_type` is not a second thing to configure
+ for the common case — an adopter publishing to PyPI has already said so in
+ `default_collection_url`.
+ """
+ if configured:
+ return configured.lower()
+ host = urllib.parse.urlsplit(collection_url or "").netloc.lower()
+ return _PURL_TYPE_BY_COLLECTION_HOST.get(host, "")
+
+
def compute_purl(purl_type: str, package_name: str, namespace: str = "") -> str | None:
"""Return the Package URL (purl) for a package, or ``None``.
@@ -1459,6 +1542,32 @@ def compute_purl(purl_type: str, package_name: str, namespace: str = "") -> str
return f"pkg:{normalised_type}/" + "/".join(segments)
+def split_purl(purl: str) -> tuple[str, str] | None:
+ """Return ``(type, package name)`` read back out of a version-less purl.
+
+ The attachment table shows a package name and a project-page link, and
+ an entry now carries the purl instead of the ``collectionURL`` /
+ ``packageName`` pair those cells used to read. Percent-encoding is
+ undone and the namespace is rejoined onto the name, so an npm scope
+ reads the way the registry spells it (``@angular/animation``).
+
+ Display only — the JSON is never rebuilt from this. Returns ``None``
+ for anything that is not a ``pkg:<type>/<path>`` string.
+ """
+ if not purl.startswith("pkg:"):
+ return None
+ # `compute_purl` emits neither qualifiers nor a subpath; strip them
+ # anyway so a hand-edited record still renders a table.
+ remainder = purl[len("pkg:") :].split("#", 1)[0].split("?", 1)[0]
+ purl_type, separator, path = remainder.partition("/")
+ if not separator or not purl_type:
+ return None
+ segments = [urllib.parse.unquote(segment) for segment in path.split("/") if segment]
+ if not segments:
+ return None
+ return purl_type.lower(), "/".join(segments)
+
+
def compute_package_url(collection_url: str, package_name: str) -> str | None:
"""Return the canonical project-page URL for a package.
@@ -1631,6 +1740,27 @@ def _escape_table_cell(value: str) -> str:
return value.replace("|", "\\|").replace("\n", " ").strip()
+def _package_identity(entry: dict) -> tuple[str, str | None]:
+ """Return ``(package name, project page URL)`` for one ``affected[]``
+ entry, for the attachment table.
+
+ Reads the legacy ``collectionURL`` / ``packageName`` pair when the entry
+ carries it — the ``purl_type = "none"`` products, and any record
+ hand-edited in Vulnogram — and otherwise reads both back out of
+ ``packageURL``. Either cell falls back to ``—`` when it cannot be
+ resolved.
+ """
+ package = (entry.get("packageName") or "").strip()
+ if package:
+ return package, compute_package_url(entry.get("collectionURL") or "", package)
+ parsed = split_purl((entry.get("packageURL") or "").strip())
+ if parsed is None:
+ return "", None
+ purl_type, package = parsed
+ template = PURL_TYPE_TO_PROJECT_URL_TEMPLATE.get(purl_type)
+ return package, template.format(package=package) if template else None
+
+
def _build_affected_table(affected: list[dict]) -> str:
"""Return the markdown table summarising ``affected[]`` for the
attachment block.
@@ -1638,19 +1768,19 @@ def _build_affected_table(affected: list[dict]) -> str:
One row per package (so multi-product CVEs surface every entry the
JSON will land in Vulnogram). Columns: package name, product,
versions (rendered back to ``>= X, < Y`` shape via
- :func:`format_version_range`), and a clickable PyPI project-page
- URL derived from ``collectionURL`` + ``packageName`` so a reader
- can jump straight to the package without reconstructing the URL.
+ :func:`format_version_range`), and a clickable project-page URL so a
+ reader can jump straight to the package without reconstructing it.
+ Both the name and the URL come from :func:`_package_identity`, which
+ reads whichever package identifier the entry carries.
"""
if not affected:
return ""
- header = "| # | Package | Product | Versions | PyPI URL |\n| :-- | :-- | :-- | :-- | :-- |\n"
+ header = "| # | Package | Product | Versions | Project URL |\n| :-- | :-- | :-- | :-- | :-- |\n"
rows: list[str] = []
for index, entry in enumerate(affected, start=1):
- package = (entry.get("packageName") or "").strip()
+ package, package_url = _package_identity(entry)
product = (entry.get("product") or "").strip()
versions = format_version_range(entry.get("versions") or [])
- package_url = compute_package_url(entry.get("collectionURL") or "", package)
package_cell = f"`{_escape_table_cell(package)}`" if package else "—"
product_cell = _escape_table_cell(product) or "—"
versions_cell = f"`{_escape_table_cell(versions)}`" if versions else "—"
@@ -1697,7 +1827,7 @@ def _build_attachment_body(
reference_count = len(cna.get("references") or [])
title_cell = _escape_table_cell(title) or "—"
state_cell = f"`{cna_private_state}`" if cna_private_state else "— (`--no-envelope`)"
- package_names = [_escape_table_cell((entry.get("packageName") or "").strip()) for entry in affected]
+ package_names = [_escape_table_cell(_package_identity(entry)[0]) for entry in affected]
package_names = [name for name in package_names if name]
packages_cell = ", ".join(f"`{name}`" for name in package_names) if package_names else "—"
metric_table = (
diff --git a/tools/cve-tool-vulnogram/generate-cve-json/tests/fixtures/cve-json-config-providers.toml b/tools/cve-tool-vulnogram/generate-cve-json/tests/fixtures/cve-json-config-providers.toml
index d6e60615..f0d17f0c 100644
--- a/tools/cve-tool-vulnogram/generate-cve-json/tests/fixtures/cve-json-config-providers.toml
+++ b/tools/cve-tool-vulnogram/generate-cve-json/tests/fixtures/cve-json-config-providers.toml
@@ -32,7 +32,8 @@ default_product = "Apache Example"
default_package_name = "apache-example"
default_collection_url = "https://pypi.python.org"
-# Optional. Package URL type for `affected[].packageURL`; omit to emit no purl.
+# Package URL type for `affected[].packageURL`. Derived from
+# `default_collection_url` when unset; `"none"` opts out explicitly.
purl_type = "pypi"
[cna]
diff --git a/tools/cve-tool-vulnogram/generate-cve-json/tests/test_generate_cve_json.py b/tools/cve-tool-vulnogram/generate-cve-json/tests/test_generate_cve_json.py
index ded18438..4b3fb3ed 100644
--- a/tools/cve-tool-vulnogram/generate-cve-json/tests/test_generate_cve_json.py
+++ b/tools/cve-tool-vulnogram/generate-cve-json/tests/test_generate_cve_json.py
@@ -56,7 +56,9 @@ from generate_cve_json import (
parse_cve_id,
parse_cwe,
parse_url_list,
+ resolve_purl_type,
resolve_title,
+ split_purl,
wrap_cve_record,
)
from generate_cve_json.cve_json import is_bot_credit, normalise_severity, to_html
@@ -609,13 +611,13 @@ class TestBuildAffectedSingleProduct:
def test_empty_field_emits_one_placeholder_entry(self):
entries = build_affected("", **DEFAULT_AFFECTED_ARGS)
assert len(entries) == 1
- assert entries[0]["packageName"] == "apache-example"
+ assert entries[0]["packageURL"] == "pkg:pypi/apache-example"
assert entries[0]["product"] == "Apache Example"
def test_bare_version_range_uses_defaults(self):
entries = build_affected("<3.2.2", **DEFAULT_AFFECTED_ARGS)
assert len(entries) == 1
- assert entries[0]["packageName"] == "apache-example"
+ assert entries[0]["packageURL"] == "pkg:pypi/apache-example"
assert entries[0]["product"] == "Apache Example"
assert entries[0]["versions"][0]["lessThan"] == "3.2.2"
@@ -625,7 +627,7 @@ class TestBuildAffectedSingleProduct:
**DEFAULT_AFFECTED_ARGS,
)
assert len(entries) == 1
- assert entries[0]["packageName"] == "apache-example-project-foo"
+ assert entries[0]["packageURL"] == "pkg:pypi/apache-example-project-foo"
assert entries[0]["product"] == "Apache Example Project Foo"
assert entries[0]["versions"][0]["lessThanOrEqual"] == "6.5.0"
@@ -641,7 +643,7 @@ class TestBuildAffectedSingleProduct:
assert len(entries) == 1
# Package prefix wins over the default argument because the body
# has said the package name explicitly.
- assert entries[0]["packageName"] == "apache-example"
+ assert entries[0]["packageURL"] == "pkg:pypi/apache-example"
assert entries[0]["product"] == "Apache Example"
@@ -653,10 +655,10 @@ class TestBuildAffectedMultiProduct:
)
assert len(entries) == 2
es, os_ = entries
- assert es["packageName"] == "apache-example-project-foo"
+ assert es["packageURL"] == "pkg:pypi/apache-example-project-foo"
assert es["product"] == "Apache Example Project Foo"
assert es["versions"][0]["lessThanOrEqual"] == "6.5.0"
- assert os_["packageName"] == "apache-example-project-kerfluffle"
+ assert os_["packageURL"] == "pkg:pypi/apache-example-project-kerfluffle"
assert os_["product"] == "Apache Example Project Kerfluffle"
assert os_["versions"][0]["lessThanOrEqual"] == "1.9.0"
@@ -665,9 +667,9 @@ class TestBuildAffectedMultiProduct:
"- apache-example-project-foo <=6.5.0\n* apache-example-project-kerfluffle <=1.9.0",
**DEFAULT_AFFECTED_ARGS,
)
- assert [e["packageName"] for e in entries] == [
- "apache-example-project-foo",
- "apache-example-project-kerfluffle",
+ assert [e["packageURL"] for e in entries] == [
+ "pkg:pypi/apache-example-project-foo",
+ "pkg:pypi/apache-example-project-kerfluffle",
]
def test_blank_lines_between_entries_are_ignored(self):
@@ -709,10 +711,15 @@ class TestBuildAffectedMultiProduct:
package_name="apache-example-helm-chart",
collection_url="https://airflow.apache.org/",
version_start=None,
+ # A Helm chart served from the project's own site is not on a
+ # package host the purl spec models, so it opts out explicitly.
+ purl_type="none",
)
assert len(entries) == 1
assert entries[0]["packageName"] == "apache-example-helm-chart"
+ assert entries[0]["collectionURL"] == "https://airflow.apache.org/"
assert entries[0]["product"] == "Apache Example Helm Chart"
+ assert "packageURL" not in entries[0]
def test_multi_line_is_deterministic(self):
# Re-generating must produce byte-identical output, which is
@@ -736,11 +743,11 @@ class TestBuildAffectedMultiProduct:
)
assert len(entries) == 2
es, os_ = entries
- assert es["packageName"] == "apache-example-project-foo"
+ assert es["packageURL"] == "pkg:pypi/apache-example-project-foo"
assert es["versions"] == [
{"status": "affected", "version": "0", "versionType": "semver"},
]
- assert os_["packageName"] == "apache-example-project-kerfluffle"
+ assert os_["packageURL"] == "pkg:pypi/apache-example-project-kerfluffle"
assert os_["versions"] == [
{"status": "affected", "version": "0", "versionType": "semver"},
]
@@ -754,7 +761,7 @@ class TestBuildAffectedMultiProduct:
**DEFAULT_AFFECTED_ARGS,
)
assert len(entries) == 1
- assert entries[0]["packageName"] == "apache-example-project-xyz"
+ assert entries[0]["packageURL"] == "pkg:pypi/apache-example-project-xyz"
assert entries[0]["versions"] == [
{"status": "affected", "version": "2.0.0", "versionType": "semver"},
]
@@ -1253,9 +1260,9 @@ class TestFormatVersionRange:
def _ready_cna_for_attachment() -> dict:
- """Same shape as _ready_cna() but with explicit packageName /
- product / collectionURL on every affected[] entry — what
- build_affected() actually emits in practice."""
+ """Same shape as _ready_cna(): affected[] entries as build_affected()
+ actually emits them, i.e. carrying a packageURL rather than the legacy
+ collectionURL / packageName pair."""
return _ready_cna()
@@ -1283,7 +1290,7 @@ class TestBuildAttachmentBody:
)
assert "| Vulnogram state | — (`--no-envelope`) |" in body
- def test_per_package_table_includes_pypi_url_and_versions(self):
+ def test_per_package_table_includes_project_url_and_versions(self):
body = _build_attachment_body(
cve_id="CVE-2026-00001",
json_text="{}",
@@ -1291,7 +1298,7 @@ class TestBuildAttachmentBody:
cna_private_state="REVIEW",
)
assert "**Packages this JSON covers:**" in body
- assert "| # | Package | Product | Versions | PyPI URL |" in body
+ assert "| # | Package | Product | Versions | Project URL |" in body
assert (
"| 1 | `apache-example` | Apache Example | `>= 3.0.0, < 3.2.0` | <https://pypi.org/project/apache-example/> |"
in body
@@ -1558,7 +1565,7 @@ class TestBuildCnaContainer:
def test_affected_entry_uses_correct_package(self):
cna = build_cna_container(**_CNA_DEFAULTS)
- assert any(a["packageName"] == "apache-example" for a in cna["affected"])
+ assert any(a["packageURL"] == "pkg:pypi/apache-example" for a in cna["affected"])
def test_description_value_appears_in_descriptions(self):
cna = build_cna_container(**_CNA_DEFAULTS)
@@ -1732,10 +1739,41 @@ class TestBuildAffectedPackageURL:
"pkg:pypi/apache-example-project-foo",
]
- def test_omits_the_field_entirely_without_a_purl_type(self):
+ def test_derives_the_type_from_the_collection_url_when_unset(self):
+ # `default_collection_url` already says where the package lives, so the
+ # common case needs no second config key.
entries = build_affected("<3.2.2", **DEFAULT_AFFECTED_ARGS)
+ assert entries[0]["packageURL"] == "pkg:pypi/apache-example"
+
+ @pytest.mark.parametrize(
+ "collection_url, expected",
+ [
+ ("https://pypi.org/", "pypi"),
+ ("https://pypi.python.org", "pypi"),
+ ("https://registry.npmjs.org", "npm"),
+ ("https://crates.io", "cargo"),
+ ("https://rubygems.org", "gem"),
+ ("https://example.invalid/", ""),
+ ("", ""),
+ ],
+ )
+ def test_resolve_purl_type_derives_from_the_collection_url(self, collection_url, expected):
+ assert resolve_purl_type("", collection_url) == expected
+
+ def test_a_configured_type_beats_the_derived_one(self):
+ assert resolve_purl_type("npm", "https://pypi.org/") == "npm"
+
+ def test_an_explicit_none_type_opts_out(self):
+ entries = build_affected("<3.2.2", **DEFAULT_AFFECTED_ARGS, purl_type="none")
assert "packageURL" not in entries[0]
+ def test_raises_when_no_purl_can_be_built(self):
+ # A record without a purl cannot be promoted, so refuse to emit one
+ # rather than ship it and have the CNA reject it later.
+ args = {**DEFAULT_AFFECTED_ARGS, "collection_url": "https://example.invalid/"}
+ with pytest.raises(ValueError, match="Package URL"):
+ build_affected("<3.2.2", **args)
+
def test_the_purl_never_carries_a_version(self):
entries = build_affected(
"apache-example <3.2.2",
@@ -1752,9 +1790,9 @@ class TestCnaContainerPackageURL:
cna = build_cna_container(**{**_CNA_DEFAULTS, "purl_type": "pypi"})
assert cna["affected"][0]["packageURL"] == "pkg:pypi/apache-example"
- def test_omits_package_url_when_no_purl_type_is_configured(self):
+ def test_derives_the_purl_type_when_none_is_configured(self):
cna = build_cna_container(**_CNA_DEFAULTS)
- assert "packageURL" not in cna["affected"][0]
+ assert cna["affected"][0]["packageURL"] == "pkg:pypi/apache-example"
class TestComputePurlNamespaceRequiredTypes:
@@ -1824,12 +1862,144 @@ class TestBuildAffectedPurlNamespace:
)
assert entries[0]["packageURL"] == "pkg:maven/org.apache.example/apache-example"
- def test_a_namespace_requiring_type_emits_nothing_without_one(self):
- entries = build_affected("<3.2.2", **DEFAULT_AFFECTED_ARGS, purl_type="maven")
- assert "packageURL" not in entries[0]
+ def test_a_namespace_requiring_type_raises_without_one(self):
+ # Guessing a groupId points at another organisation's artifact, so the
+ # generator stops and names the key to set.
+ with pytest.raises(ValueError, match="purl_namespace"):
+ build_affected("<3.2.2", **DEFAULT_AFFECTED_ARGS, purl_type="maven")
def test_the_cna_container_passes_the_namespace_through(self):
cna = build_cna_container(
**{**_CNA_DEFAULTS, "purl_type": "maven", "purl_namespace": "org.apache.example"}
)
assert cna["affected"][0]["packageURL"] == "pkg:maven/org.apache.example/apache-example"
+
+
+# ---------------------------------------------------------------------------
+# The purl is the only package identifier an entry carries
+# ---------------------------------------------------------------------------
+
+
+class TestPackageIdentifierShape:
+ def test_an_entry_with_a_purl_omits_the_legacy_pair(self):
+ # The CVE tool derives `collectionURL` / `packageName` from the purl
+ # when both are absent, and reports a stored pair that disagrees with
+ # it. Emitting both hands it two sources of truth to disagree about.
+ entry = build_affected("<3.2.2", **DEFAULT_AFFECTED_ARGS)[0]
+ assert entry["packageURL"] == "pkg:pypi/apache-example"
+ assert "collectionURL" not in entry
+ assert "packageName" not in entry
+
+ def test_the_none_opt_out_keeps_the_legacy_pair(self):
+ # The one case the legacy identifiers are still wanted: an ecosystem
+ # the purl spec does not model, so there is nothing to derive from.
+ entry = build_affected("<3.2.2", **DEFAULT_AFFECTED_ARGS, purl_type="none")[0]
+ assert entry["collectionURL"] == "https://pypi.python.org"
+ assert entry["packageName"] == "apache-example"
+ assert "packageURL" not in entry
+
+ def test_keys_stay_in_alphabetical_order(self):
+ # Vulnogram round-trips the record, and a stable key order is what
+ # keeps `--attach` byte-identical across re-runs.
+ assert list(build_affected("<3.2.2", **DEFAULT_AFFECTED_ARGS)[0]) == [
+ "defaultStatus",
+ "packageURL",
+ "product",
+ "vendor",
+ "versions",
+ ]
+
+ def test_the_opted_out_keys_stay_in_alphabetical_order(self):
+ entry = build_affected("<3.2.2", **DEFAULT_AFFECTED_ARGS, purl_type="none")[0]
+ assert list(entry) == [
+ "collectionURL",
+ "defaultStatus",
+ "packageName",
+ "product",
+ "vendor",
+ "versions",
+ ]
+
+
+# ---------------------------------------------------------------------------
+# `split_purl` — reading the identifier back for the attachment table
+# ---------------------------------------------------------------------------
+
+
+class TestSplitPurl:
+ @pytest.mark.parametrize(
+ "purl, expected",
+ [
+ ("pkg:pypi/apache-example", ("pypi", "apache-example")),
+ ("pkg:npm/%40angular/animation", ("npm", "@angular/animation")),
+ ("pkg:maven/org.apache.example/thing", ("maven", "org.apache.example/thing")),
+ # A hand-edited record may spell the type in any case, and may
+ # carry the qualifiers / subpath `compute_purl` never emits.
+ ("pkg:PyPI/apache-example", ("pypi", "apache-example")),
+ ("pkg:pypi/apache-example?repository_url=x#sub", ("pypi", "apache-example")),
+ ],
+ )
+ def test_reads_the_type_and_name_back_out(self, purl, expected):
+ assert split_purl(purl) == expected
+
+ @pytest.mark.parametrize(
+ "value",
+ [
+ "",
+ "apache-example",
+ "https://pypi.org/project/apache-example/",
+ "pkg:pypi",
+ "pkg:pypi/",
+ "pkg:/name",
+ ],
+ )
+ def test_returns_none_for_anything_that_is_not_a_purl(self, value):
+ assert split_purl(value) is None
+
+ def test_round_trips_an_npm_scope(self):
+ purl = compute_purl("npm", "@angular/animation")
+ assert purl is not None
+ assert split_purl(purl) == ("npm", "@angular/animation")
+
+
+class TestAttachmentTableFromPurl:
+ def test_derives_the_package_and_link_from_the_purl(self):
+ # Entries no longer carry `packageName` / `collectionURL`, so the
+ # table's name and link cells are read back out of the purl — for
+ # whichever registry the project publishes to, not just PyPI.
+ cna = _ready_cna_for_attachment()
+ cna["affected"] = build_affected(
+ "<3.2.2",
+ vendor="Apache Software Foundation",
+ product="Apache Example",
+ package_name="@apache/example",
+ collection_url="https://registry.npmjs.org",
+ version_start=None,
+ )
+ body = _build_attachment_body(
+ cve_id="CVE-2026-00001",
+ json_text="{}",
+ cna=cna,
+ cna_private_state="REVIEW",
+ )
+ assert (
+ "| 1 | `@apache/example` | Apache Example | `< 3.2.2` "
+ "| <https://www.npmjs.com/package/@apache/example> |"
+ ) in body
+ assert "| Affected packages | `@apache/example` |" in body
+
+ def test_a_purl_type_with_no_project_page_still_names_the_package(self):
+ cna = _ready_cna_for_attachment()
+ cna["affected"] = build_affected(
+ "<3.2.2",
+ **DEFAULT_AFFECTED_ARGS,
+ purl_type="maven",
+ purl_namespace="org.apache.example",
+ )
+ body = _build_attachment_body(
+ cve_id="CVE-2026-00001",
+ json_text="{}",
+ cna=cna,
+ cna_private_state="REVIEW",
+ )
+ assert "| 1 | `org.apache.example/apache-example` | Apache Example | `< 3.2.2` | — |" in body
diff --git a/tools/cve-tool-vulnogram/oauth-api/src/vulnogram_api/merge_mode.py b/tools/cve-tool-vulnogram/oauth-api/src/vulnogram_api/merge_mode.py
index 54b82543..2fe6d187 100644
--- a/tools/cve-tool-vulnogram/oauth-api/src/vulnogram_api/merge_mode.py
+++ b/tools/cve-tool-vulnogram/oauth-api/src/vulnogram_api/merge_mode.py
@@ -50,6 +50,7 @@ the API was built that way long before this module existed.
from __future__ import annotations
import copy
+import urllib.parse
from typing import Any
@@ -148,15 +149,34 @@ def _merge_references_by_url(
return merged
+def _package_identity(entry: dict[str, Any]) -> str:
+ """Return the package an ``affected[]`` entry names, whichever
+ identifier it carries.
+
+ Records authored before the CVE tool promoted the Package URL to the
+ recommended identifier carry ``packageName``; records generated today
+ carry ``packageURL`` alone and the tool derives the legacy pair from it
+ on publication. The two spellings have to compare equal, or the
+ product-change guard refuses every push against an older record over a
+ package that never changed.
+ """
+ purl = str(entry.get("packageURL") or "").strip()
+ if purl.startswith("pkg:"):
+ remainder = purl[len("pkg:") :].split("#", 1)[0].split("?", 1)[0]
+ _, separator, path = remainder.partition("/")
+ segments = [urllib.parse.unquote(segment) for segment in path.split("/") if segment]
+ if separator and segments:
+ return "/".join(segments)
+ return str(entry.get("packageName") or "")
+
+
def _product_signature(entry: dict[str, Any]) -> tuple[str, str]:
- """Return ``(packageName, product)`` for an ``affected[]`` entry.
+ """Return ``(package, product)`` for an ``affected[]`` entry.
Either field may be missing; the signature uses ``""`` to keep
set semantics consistent.
"""
- package = str(entry.get("packageName") or "")
- product = str(entry.get("product") or "")
- return (package, product)
+ return (_package_identity(entry), str(entry.get("product") or ""))
def _diff_affected_products(
@@ -167,9 +187,11 @@ def _diff_affected_products(
changes between the current and new affected[] arrays.
Returns an empty list when both arrays carry the same
- ``(packageName, product)`` signatures (order ignored). Otherwise
- returns one line per dropped, added, or renamed signature so the
- caller can decide whether to refuse or warn.
+ ``(package, product)`` signatures (order ignored) — where the package
+ is read from ``packageURL`` or ``packageName``, whichever the entry
+ carries, so the two identifier shapes do not read as a rename.
+ Otherwise returns one line per dropped, added, or renamed signature so
+ the caller can decide whether to refuse or warn.
"""
current_sigs = {_product_signature(entry) for entry in current if isinstance(entry, dict)}
new_sigs = {_product_signature(entry) for entry in new if isinstance(entry, dict)}
@@ -184,10 +206,10 @@ def _diff_affected_products(
diffs: list[str] = []
for sig in sorted(current_sigs - new_sigs):
package, product = sig
- diffs.append(f" - removed: packageName={package!r}, product={product!r}")
+ diffs.append(f" - removed: package={package!r}, product={product!r}")
for sig in sorted(new_sigs - current_sigs):
package, product = sig
- diffs.append(f" + added: packageName={package!r}, product={product!r}")
+ diffs.append(f" + added: package={package!r}, product={product!r}")
return diffs
diff --git a/tools/cve-tool-vulnogram/oauth-api/src/vulnogram_api/record_update.py b/tools/cve-tool-vulnogram/oauth-api/src/vulnogram_api/record_update.py
index 9f96a100..3d3d96fe 100644
--- a/tools/cve-tool-vulnogram/oauth-api/src/vulnogram_api/record_update.py
+++ b/tools/cve-tool-vulnogram/oauth-api/src/vulnogram_api/record_update.py
@@ -34,7 +34,7 @@ script intentionally does not automate.
**Merge mode (default on)** — before the POST, the script fetches
the record's current state and applies three safety nets against
the document about to be pushed: state-downgrade refusal,
-references-by-URL merge, product/packageName-change refusal. See
+references-by-URL merge, product/package-change refusal. See
:mod:`vulnogram_api.merge_mode` for the full rules, and the
``--allow-state-downgrade`` / ``--replace-references`` /
``--allow-product-change`` / ``--full-replace`` flags below for the
@@ -122,7 +122,7 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
"--allow-product-change",
action="store_true",
help=(
- "Allow `affected[].product` / `packageName` changes vs "
+ "Allow `affected[].product` / package changes vs "
"the current record. Required when the change is "
"intentional (broadening scope to add a new package, or "
"correcting the originally-published name); refused by "
diff --git a/tools/cve-tool-vulnogram/oauth-api/tests/test_merge_mode.py b/tools/cve-tool-vulnogram/oauth-api/tests/test_merge_mode.py
index 6923e524..668be872 100644
--- a/tools/cve-tool-vulnogram/oauth-api/tests/test_merge_mode.py
+++ b/tools/cve-tool-vulnogram/oauth-api/tests/test_merge_mode.py
@@ -383,6 +383,37 @@ class TestProductChangeGuard:
assert "apache-foo-bar" in joined
assert "apache-foo" in joined
+ def test_a_purl_and_the_package_name_it_derives_are_the_same_package(self):
+ # The generator now emits `packageURL` alone; a record authored
+ # before that carries the legacy pair. Same package either way, so
+ # the guard must not read the switch as a rename and refuse every
+ # push against an older record.
+ diffs = _diff_affected_products(
+ current=[
+ {
+ "collectionURL": "https://pypi.python.org",
+ "packageName": "apache-foo",
+ "product": "Apache Foo",
+ }
+ ],
+ new=[{"packageURL": "pkg:pypi/apache-foo", "product": "Apache Foo"}],
+ )
+ assert diffs == []
+
+ def test_a_purl_naming_a_different_package_is_still_refused(self):
+ diffs = _diff_affected_products(
+ current=[{"packageName": "apache-foo", "product": "Apache Foo"}],
+ new=[{"packageURL": "pkg:pypi/apache-bar", "product": "Apache Foo"}],
+ )
+ assert diffs != []
+
+ def test_an_npm_scope_survives_the_purl_encoding(self):
+ diffs = _diff_affected_products(
+ current=[{"packageName": "@apache/foo", "product": "Apache Foo"}],
+ new=[{"packageURL": "pkg:npm/%40apache/foo", "product": "Apache Foo"}],
+ )
+ assert diffs == []
+
def test_diff_empty_when_unchanged(self):
diffs = _diff_affected_products(
current=[{"packageName": "apache-foo", "product": "Apache Foo"}],