You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by xu...@apache.org on 2026/07/27 06:12:07 UTC
(opendal) 01/01: ci: use trusted publishing for Rust releases
This is an automated email from the ASF dual-hosted git repository.
Xuanwo pushed a commit to branch xuanwo/rust-trusted-publishing
in repository https://gitbox.apache.org/repos/asf/opendal.git
commit 3e6eb914b4e32f99a42404744c2925fff9fd2d7b
Author: Xuanwo <gi...@xuanwo.io>
AuthorDate: Mon Jul 27 14:11:50 2026 +0800
ci: use trusted publishing for Rust releases
---
.agents/skills/opendal-release/SKILL.md | 76 ++-
.../scripts/bootstrap-rust-crates.sh | 152 +++++
.github/ISSUE_TEMPLATE/3-new-release.md | 1 +
.github/scripts/release_rust/README.md | 33 +-
.github/scripts/release_rust/bootstrap.py | 759 +++++++++++++++++++++
.github/scripts/release_rust/publish.py | 34 +-
.github/scripts/release_rust/test_bootstrap.py | 494 ++++++++++++++
.github/scripts/release_rust/test_publish.py | 100 +++
.../release_rust/test_trusted_publishing.py | 116 ++++
.github/scripts/release_rust/trusted_publishing.py | 159 +++++
.github/workflows/bootstrap_rust_crates.yml | 76 +++
.github/workflows/release_rust.yml | 8 +-
website/community/release/release.md | 58 +-
13 files changed, 2034 insertions(+), 32 deletions(-)
diff --git a/.agents/skills/opendal-release/SKILL.md b/.agents/skills/opendal-release/SKILL.md
index 039b768074..f8c6a3de7f 100644
--- a/.agents/skills/opendal-release/SKILL.md
+++ b/.agents/skills/opendal-release/SKILL.md
@@ -46,14 +46,15 @@ Use these states explicitly when reporting status:
1. `planning`: release discussion/tracking issue/version bump not done.
2. `bump-pr`: version/changelog/upgrade/dependency updates are in PR.
-3. `rc-tagged`: signed RC tag exists and was pushed.
-4. `rc-ci`: tag-triggered release workflows are still running or failed.
-5. `artifacts-built`: `just release` generated local ASF source artifacts.
-6. `dist-dev-uploaded`: artifacts are committed to ASF SVN `dist/dev`.
-7. `nexus-closed`: Java staging repo is closed and publicly accessible.
-8. `vote-open`: GitHub Discussion vote is open.
-9. `vote-passed`: at least 72 hours elapsed and binding vote requirements are met.
-10. `official-release`: final tag, `dist/release`, package repositories, GitHub release, and announcement are complete.
+3. `crates-bootstrapped`: the bootstrap workflow has authenticated every Rust crate in the scanned `main` publish plan, and every name exists with the exact Trusted Publisher and `trustpub_only` enabled.
+4. `rc-tagged`: signed RC tag exists and was pushed.
+5. `rc-ci`: tag-triggered release workflows are still running or failed.
+6. `artifacts-built`: `just release` generated local ASF source artifacts.
+7. `dist-dev-uploaded`: artifacts are committed to ASF SVN `dist/dev`.
+8. `nexus-closed`: Java staging repo is closed and publicly accessible.
+9. `vote-open`: GitHub Discussion vote is open.
+10. `vote-passed`: at least 72 hours elapsed and binding vote requirements are met.
+11. `official-release`: final tag, `dist/release`, package repositories, GitHub release, and announcement are complete.
If a release fails before `official-release`, abandon that RC, clean up wrong staged artifacts where needed, drop the Maven staging repo, and create the next RC.
@@ -69,6 +70,8 @@ If a release fails before `official-release`, abandon that RC, clean up wrong st
- `website/community/release/release.md`
- `dev/src/release/package.rs`
- `.github/workflows/release_*.yml`
+ - `.github/workflows/bootstrap_rust_crates.yml`
+ - `.github/scripts/release_rust/bootstrap.py`
- `.github/scripts/release_rust/plan.py`
- `.github/scripts/release_rust/publish.py`
@@ -95,6 +98,59 @@ When preparing a bump PR:
Before opening the PR, check whether PR templates exist and use them. Keep the PR body self-contained and reviewer-facing.
+## Bootstrap Rust Crates
+
+During release preparation, wait until the intended Rust crate-name set is
+present on `apache/opendal` `main`. The release manager chooses the exact
+reservation time, normally about three days before the planned release. When
+the release manager decides to reserve the names, dispatch the Rust crate
+bootstrap workflow:
+
+```bash
+.agents/skills/opendal-release/scripts/bootstrap-rust-crates.sh
+```
+
+This command performs the transition to `crates-bootstrapped`. Run it without
+asking for a second confirmation after a PMC release manager explicitly chooses
+the reservation time. Do not run it for release status checks or dry-run
+planning. A `0.0.0` package is an externally visible and irreversible namespace
+reservation, not an ASF software release or release artifact.
+
+The helper:
+
+- Requires a clean checkout at the current `apache/opendal` `main`.
+- Verifies that the `rust-bootstrap` environment has a required-reviewer protection rule.
+- Dispatches `bootstrap_rust_crates.yml` without inputs.
+- Resolves the exact run, checks its `headSha`, and waits for completion.
+- Blocks while the `rust-bootstrap` environment awaits PMC approval on every run.
+- Verifies publicly that every crate in the checked publish plan exists and has `trustpub_only` enabled after the authenticated workflow audit succeeds.
+
+The workflow always scans the publish plan from its `main` commit. Before any
+write, the protected job uses the bootstrap token to authenticate ownership and
+audit the exact Trusted Publisher configuration of every existing planned
+crate. It never modifies an established crate. For missing names, it publishes
+a dependency-free `0.0.0` namespace reservation, creates the single allowed
+Trusted Publisher for `apache/opendal`, `release_rust.yml`, and the `release`
+environment, and enables `trustpub_only`. Reruns reconcile every placeholder.
+The protected authenticated audit runs even when discovery finds no bootstrap
+candidates.
+
+The one-time migration of existing, established crates to Trusted Publishing is
+an independent administrative prerequisite. Configure the same
+`apache/opendal`, `release_rust.yml`, and `release` publisher identity and
+enable `trustpub_only` for every existing crate. This workflow audits but never
+changes established crates, and it fails if that migration is incomplete.
+Complete the migration before using the OIDC-only release workflow.
+
+Repository administrators must configure the `rust-bootstrap` GitHub
+environment with PMC required reviewers and the
+`CARGO_REGISTRY_BOOTSTRAP_TOKEN` secret. The token must have only `publish-new`
+and `trusted-publishing` endpoint scopes, with crate scopes restricted to
+OpenDAL package names. Never expose this token to the normal release workflow.
+
+If the Rust publish plan gains another crate after a successful reservation
+run, run the helper again at the release manager's chosen time.
+
## RC Tagging
Before creating an RC tag:
@@ -102,6 +158,7 @@ Before creating an RC tag:
- Resolve the remote that points to `apache/opendal`; do not assume it is named `origin`.
- Confirm the bump PR or required fix PR is merged.
- Confirm the tag target commit exactly.
+- Confirm the Rust crate bootstrap workflow succeeded and no later change added a crate to the publish plan.
- Confirm no existing tag uses the intended RC version.
Tag and push:
@@ -232,7 +289,8 @@ Release safety requirements from the 0.56.0 cycle:
- `core/testkit` / `opendal-testkit` must be in the Rust publish plan when top-level `opendal` references it through the `tests` feature.
- Publish helpers must use `cargo publish --package <name>` rather than relying on workspace defaults.
- Repo-local `dev-dependencies` can break packaging even with `cargo publish --no-verify`; use `.github/scripts/release_rust/publish.py` and keep its tests green.
-- Trusted publishing tokens cannot create new crates. If a new crate name is introduced, verify creation permissions or pre-create/publish manually.
+- The bootstrap workflow must succeed before RC tagging, and it must be rerun if the publish plan later gains a crate.
+- The normal release workflow must use Trusted Publishing only. It fetches and revokes a new OIDC-derived crates.io token for every publish attempt, including retries.
If release helper CI is noisy, rebase onto latest `origin/main`, run targeted helper tests locally, then update the PR.
diff --git a/.agents/skills/opendal-release/scripts/bootstrap-rust-crates.sh b/.agents/skills/opendal-release/scripts/bootstrap-rust-crates.sh
new file mode 100755
index 0000000000..60ad4050ad
--- /dev/null
+++ b/.agents/skills/opendal-release/scripts/bootstrap-rust-crates.sh
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -euo pipefail
+
+skill_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+repo_dir="$(git -C "${skill_dir}" rev-parse --show-toplevel)"
+cd "${repo_dir}"
+
+for command in gh git jq python3; do
+ command -v "${command}" >/dev/null || {
+ echo "required command is unavailable: ${command}" >&2
+ exit 1
+ }
+done
+
+if [[ "$#" -ne 0 ]]; then
+ echo "usage: $0" >&2
+ exit 1
+fi
+if [[ -n "$(git status --porcelain)" ]]; then
+ echo "the release checkout must be clean" >&2
+ exit 1
+fi
+
+apache_remote="$(
+ git remote -v |
+ awk '$2 ~ /github.com[:\/]apache\/opendal(\.git)?$/ && $3 == "(fetch)" { print $1; exit }'
+)"
+if [[ -z "${apache_remote}" ]]; then
+ echo "cannot find a git remote for apache/opendal" >&2
+ exit 1
+fi
+
+git fetch "${apache_remote}" main
+source_commit="$(git rev-parse FETCH_HEAD)"
+if [[ "$(git rev-parse HEAD)" != "${source_commit}" ]]; then
+ echo "the release checkout must be at the current apache/opendal main: ${source_commit}" >&2
+ exit 1
+fi
+
+git cat-file -e \
+ "${source_commit}:.github/workflows/bootstrap_rust_crates.yml"
+git cat-file -e \
+ "${source_commit}:.github/scripts/release_rust/bootstrap.py"
+
+repo="apache/opendal"
+workflow="bootstrap_rust_crates.yml"
+environment="rust-bootstrap"
+if ! environment_json="$(gh api "repos/${repo}/environments/${environment}")"; then
+ echo "GitHub environment ${environment} is not configured" >&2
+ exit 1
+fi
+if ! jq -e \
+ '[.protection_rules[]? | select(.type == "required_reviewers")] | length > 0' \
+ >/dev/null <<<"${environment_json}"; then
+ echo "GitHub environment ${environment} must require reviewers" >&2
+ exit 1
+fi
+
+run_title="Bootstrap Rust crates at ${source_commit}"
+started_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
+
+if ! dispatch_output="$(
+ gh workflow run "${workflow}" \
+ --repo "${repo}" \
+ --ref main 2>&1
+)"; then
+ echo "${dispatch_output}" >&2
+ exit 1
+fi
+echo "${dispatch_output}"
+
+run_id="$(
+ sed -nE \
+ 's#.*github\.com/apache/opendal/actions/runs/([0-9]+).*#\1#p' \
+ <<<"${dispatch_output}" |
+ tail -n 1
+)"
+deadline=$((SECONDS + 120))
+while [[ -z "${run_id}" && ${SECONDS} -lt ${deadline} ]]; do
+ runs="$(
+ gh run list \
+ --repo "${repo}" \
+ --workflow "${workflow}" \
+ --event workflow_dispatch \
+ --limit 50 \
+ --json databaseId,displayTitle,headSha,createdAt
+ )"
+ run_id="$(
+ jq -r \
+ --arg title "${run_title}" \
+ --arg head_sha "${source_commit}" \
+ --arg started_at "${started_at}" \
+ '[.[]
+ | select(
+ .displayTitle == $title
+ and .headSha == $head_sha
+ and .createdAt >= $started_at
+ )
+ ]
+ | sort_by(.createdAt)
+ | last
+ | .databaseId // empty' \
+ <<<"${runs}"
+ )"
+ if [[ -z "${run_id}" ]]; then
+ sleep 2
+ fi
+done
+
+if [[ -z "${run_id}" ]]; then
+ echo "could not resolve the dispatched ${workflow} run" >&2
+ exit 1
+fi
+
+run_head_sha="$(
+ gh run view "${run_id}" --repo "${repo}" --json headSha --jq '.headSha'
+)"
+if [[ "${run_head_sha}" != "${source_commit}" ]]; then
+ echo "workflow run ${run_id} uses ${run_head_sha}, expected ${source_commit}" >&2
+ exit 1
+fi
+
+echo "Waiting for crates.io bootstrap run ${run_id}"
+if ! gh run watch "${run_id}" --repo "${repo}" --exit-status; then
+ gh run view "${run_id}" --repo "${repo}" --log-failed
+ exit 1
+fi
+
+gh run view "${run_id}" \
+ --repo "${repo}" \
+ --json displayTitle,headSha,status,conclusion,url \
+ --jq '{title: .displayTitle, head_sha: .headSha, status, conclusion, url}'
+
+python3 .github/scripts/release_rust/bootstrap.py verify
diff --git a/.github/ISSUE_TEMPLATE/3-new-release.md b/.github/ISSUE_TEMPLATE/3-new-release.md
index 637e858017..0fdc282c47 100644
--- a/.github/ISSUE_TEMPLATE/3-new-release.md
+++ b/.github/ISSUE_TEMPLATE/3-new-release.md
@@ -25,6 +25,7 @@ This issue is used to track tasks of the opendal ${opendal_version} release.
- [ ] Update docs
- [ ] Generate dependencies list
- [ ] Check Rust crates.io publish plan
+- [ ] Reserve and audit Rust crate names at the release manager's chosen time
- [ ] Push release candidate tag to GitHub
#### ASF Side
diff --git a/.github/scripts/release_rust/README.md b/.github/scripts/release_rust/README.md
index 882e42901a..19208df46b 100644
--- a/.github/scripts/release_rust/README.md
+++ b/.github/scripts/release_rust/README.md
@@ -49,18 +49,47 @@ PACKAGES="$(python3 .github/scripts/release_rust/plan.py)" \
python3 .github/scripts/release_rust/publish.py
```
+Live publishing requires a GitHub Actions job with `id-token: write`. Dry runs
+do not request a Trusted Publishing token.
+
`publish.py` wraps `cargo publish` with the release-specific behavior we need:
- retries crates.io rate limits by using the server-provided retry time
+- fetches and revokes a fresh GitHub OIDC token for every publish attempt
- uses `cargo publish --package <name>` so workspace packages publish the intended crate
- temporarily removes repo-local `dev-dependencies` from the package manifest before publishing
- restores every touched manifest and lockfile after each package
+## Bootstrap new crate names
+
+`bootstrap.py` reserves new crate names and audits the complete Rust publish
+plan. Its `discover` command reports missing names and existing `0.0.0`
+placeholders without using credentials. Its `apply` command uses the protected
+bootstrap token to:
+
+- authenticate ownership and audit the exact Trusted Publisher of every
+ existing planned crate before any write
+- publish a dependency-free `0.0.0` namespace reservation for every missing name
+- configure `apache/opendal`, `release_rust.yml`, and the `release` environment
+ as a placeholder's only Trusted Publisher
+- enable `trustpub_only` and reconcile partially completed placeholders
+- perform a final authenticated audit of every planned crate
+
+The `verify` command reads public crates.io metadata and confirms that all
+planned crates exist and require Trusted Publishing. The authenticated `apply`
+audit is authoritative for ownership and the exact publisher configuration.
+
+Migration of existing crates is independent. `bootstrap.py apply` refuses to
+modify an established crate. The `rust-bootstrap` protected job always runs,
+including when the plan contains no missing names or placeholders.
+
+Version `0.0.0` only reserves a crates.io namespace. It is not an ASF software
+release or release artifact.
+
## Tests
Run the unit tests with:
```bash
-python3 .github/scripts/release_rust/test_plan.py
-python3 .github/scripts/release_rust/test_publish.py
+python3 -m unittest discover -s .github/scripts/release_rust -p "test_*.py"
```
diff --git a/.github/scripts/release_rust/bootstrap.py b/.github/scripts/release_rust/bootstrap.py
new file mode 100755
index 0000000000..4285665224
--- /dev/null
+++ b/.github/scripts/release_rust/bootstrap.py
@@ -0,0 +1,759 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import argparse
+import json
+import os
+import shutil
+import subprocess
+import tempfile
+import time
+import tomllib
+import urllib.error
+import urllib.parse
+import urllib.request
+from dataclasses import dataclass
+from pathlib import Path
+
+from plan import plan
+from publish import parse_retry_after
+from publish import should_retry
+
+
+SCRIPT_PATH = Path(__file__).resolve()
+PROJECT_DIR = SCRIPT_PATH.parents[3]
+REGISTRY_URL = "https://crates.io"
+REPOSITORY = "https://github.com/apache/opendal"
+PLACEHOLDER_VERSION = "0.0.0"
+PLACEHOLDER_DESCRIPTION = "Namespace reservation for a crate planned by Apache OpenDAL."
+PUBLISHER = {
+ "repository_owner": "apache",
+ "repository_name": "opendal",
+ "workflow_filename": "release_rust.yml",
+ "environment": "release",
+}
+USER_AGENT = "apache-opendal-release-bootstrap/1.0 (https://github.com/apache/opendal)"
+
+
+class ApiError(RuntimeError):
+ def __init__(self, operation: str, status: int, detail: str):
+ super().__init__(f"{operation} failed with HTTP {status}: {detail}")
+ self.status = status
+
+
+@dataclass(frozen=True)
+class PlannedCrate:
+ name: str
+ path: str
+
+
+@dataclass(frozen=True)
+class ReconcileResult:
+ name: str
+ actions: tuple[str, ...]
+
+
+def planned_crates(project_dir: Path = PROJECT_DIR) -> list[PlannedCrate]:
+ project_dir = project_dir.resolve()
+ crates: list[PlannedCrate] = []
+ names: set[str] = set()
+
+ for package_path in plan(project_dir):
+ manifest_path = project_dir / package_path / "Cargo.toml"
+ with manifest_path.open("rb") as fp:
+ package = tomllib.load(fp)["package"]
+
+ name = package["name"]
+ if name in names:
+ raise RuntimeError(
+ f"duplicate crates.io package name in publish plan: {name}"
+ )
+ names.add(name)
+ crates.append(PlannedCrate(name=name, path=package_path))
+
+ return crates
+
+
+def _api_error_detail(error: urllib.error.HTTPError) -> str:
+ try:
+ response = error.read().decode("utf-8", errors="replace")
+ finally:
+ error.close()
+ try:
+ body = json.loads(response)
+ errors = body.get("errors", [])
+ return "; ".join(item["detail"] for item in errors)
+ except (json.JSONDecodeError, AttributeError, KeyError, TypeError):
+ return response
+
+
+class CratesIoClient:
+ def __init__(self, registry_url: str = REGISTRY_URL, token: str | None = None):
+ self.registry_url = registry_url.rstrip("/")
+ self.token = token
+
+ def _request(
+ self,
+ method: str,
+ path: str,
+ operation: str,
+ body: dict[str, object] | None = None,
+ authenticated: bool = False,
+ ) -> dict[str, object]:
+ headers = {
+ "Accept": "application/json",
+ "User-Agent": USER_AGENT,
+ }
+ if authenticated:
+ if not self.token:
+ raise RuntimeError(f"{operation} requires a crates.io API token")
+ headers["Authorization"] = self.token
+
+ data = None
+ if body is not None:
+ data = json.dumps(body).encode()
+ headers["Content-Type"] = "application/json"
+
+ request = urllib.request.Request(
+ f"{self.registry_url}{path}",
+ data=data,
+ headers=headers,
+ method=method,
+ )
+ retryable = method in {"GET", "PATCH"}
+ for attempt in range(5):
+ try:
+ with urllib.request.urlopen(request, timeout=30) as response:
+ return json.load(response)
+ except urllib.error.HTTPError as error:
+ should_retry = retryable and error.code in {429, 502, 503, 504}
+ if should_retry and attempt < 4:
+ retry_after = (
+ error.headers.get("Retry-After") if error.headers else None
+ )
+ delay = (
+ int(retry_after)
+ if retry_after and retry_after.isdigit()
+ else 2**attempt
+ )
+ error.read()
+ error.close()
+ print(
+ f"{operation} returned HTTP {error.code}; retrying in {delay}s",
+ flush=True,
+ )
+ time.sleep(delay)
+ continue
+ raise ApiError(
+ operation, error.code, _api_error_detail(error)
+ ) from None
+ except urllib.error.URLError as error:
+ if retryable and attempt < 4:
+ delay = 2**attempt
+ print(
+ f"{operation} failed: {error.reason}; retrying in {delay}s",
+ flush=True,
+ )
+ time.sleep(delay)
+ continue
+ raise RuntimeError(f"{operation} failed: {error.reason}") from None
+
+ raise AssertionError("unreachable")
+
+ def get_crate(self, name: str) -> dict[str, object] | None:
+ encoded_name = urllib.parse.quote(name, safe="")
+ try:
+ response = self._request(
+ "GET",
+ f"/api/v1/crates/{encoded_name}",
+ f"reading crate {name}",
+ )
+ except ApiError as error:
+ if error.status == 404:
+ return None
+ raise
+ krate = response.get("crate")
+ if not isinstance(krate, dict):
+ raise RuntimeError(f"crates.io returned invalid metadata for {name}")
+ return krate
+
+ def get_version(self, name: str, version: str) -> dict[str, object]:
+ encoded_name = urllib.parse.quote(name, safe="")
+ encoded_version = urllib.parse.quote(version, safe="")
+ response = self._request(
+ "GET",
+ f"/api/v1/crates/{encoded_name}/{encoded_version}",
+ f"reading {name} {version}",
+ )
+ version_data = response.get("version")
+ if not isinstance(version_data, dict):
+ raise RuntimeError(
+ f"crates.io returned invalid version metadata for {name} {version}"
+ )
+ return version_data
+
+ def list_github_configs(self, name: str) -> list[dict[str, object]]:
+ query = urllib.parse.urlencode({"crate": name, "per_page": 100})
+ response = self._request(
+ "GET",
+ f"/api/v1/trusted_publishing/github_configs?{query}",
+ f"listing Trusted Publishers for {name}",
+ authenticated=True,
+ )
+ configs = response.get("github_configs")
+ if not isinstance(configs, list) or not all(
+ isinstance(config, dict) for config in configs
+ ):
+ raise RuntimeError(
+ f"crates.io returned invalid Trusted Publisher metadata for {name}"
+ )
+ return configs
+
+ def create_github_config(self, name: str) -> dict[str, object]:
+ config = {"crate": name, **PUBLISHER}
+ response = self._request(
+ "POST",
+ "/api/v1/trusted_publishing/github_configs",
+ f"creating the Trusted Publisher for {name}",
+ body={"github_config": config},
+ authenticated=True,
+ )
+ created = response.get("github_config")
+ if not isinstance(created, dict):
+ raise RuntimeError(
+ f"crates.io returned invalid Trusted Publisher metadata for {name}"
+ )
+ return created
+
+ def set_trustpub_only(self, name: str) -> dict[str, object]:
+ encoded_name = urllib.parse.quote(name, safe="")
+ response = self._request(
+ "PATCH",
+ f"/api/v1/crates/{encoded_name}",
+ f"enabling Trusted Publishing only for {name}",
+ body={"crate": {"trustpub_only": True}},
+ authenticated=True,
+ )
+ krate = response.get("crate")
+ if not isinstance(krate, dict):
+ raise RuntimeError(f"crates.io returned invalid metadata for {name}")
+ return krate
+
+
+def _normalized_repository(value: object) -> str:
+ if not isinstance(value, str):
+ return ""
+ normalized = value.rstrip("/")
+ if normalized.endswith(".git"):
+ normalized = normalized[:-4]
+ return normalized.lower()
+
+
+def validate_crate_metadata(
+ planned: PlannedCrate, metadata: dict[str, object], client: CratesIoClient
+) -> None:
+ if metadata.get("id") != planned.name:
+ raise RuntimeError(
+ f"crate name mismatch for {planned.name}: got {metadata.get('id')!r}"
+ )
+ if _normalized_repository(metadata.get("repository")) != _normalized_repository(
+ REPOSITORY
+ ):
+ raise RuntimeError(
+ f"{planned.name} already exists with an unexpected repository: "
+ f"{metadata.get('repository')!r}"
+ )
+
+ if metadata.get("max_version") == PLACEHOLDER_VERSION:
+ if metadata.get("description") != PLACEHOLDER_DESCRIPTION:
+ raise RuntimeError(
+ f"{planned.name} has an unexpected {PLACEHOLDER_VERSION} placeholder"
+ )
+ version = client.get_version(planned.name, PLACEHOLDER_VERSION)
+ if version.get("num") != PLACEHOLDER_VERSION:
+ raise RuntimeError(
+ f"{planned.name} placeholder version could not be verified"
+ )
+
+
+def _is_expected_config(name: str, config: dict[str, object]) -> bool:
+ repository_owner = config.get("repository_owner")
+ repository_name = config.get("repository_name")
+ return all(
+ (
+ config.get("crate") == name,
+ isinstance(repository_owner, str)
+ and repository_owner.lower() == PUBLISHER["repository_owner"],
+ isinstance(repository_name, str)
+ and repository_name.lower() == PUBLISHER["repository_name"],
+ config.get("workflow_filename") == PUBLISHER["workflow_filename"],
+ config.get("environment") == PUBLISHER["environment"],
+ )
+ )
+
+
+def validate_github_configs(name: str, configs: list[dict[str, object]]) -> None:
+ if len(configs) != 1 or not _is_expected_config(name, configs[0]):
+ compact = [
+ {
+ key: config.get(key)
+ for key in (
+ "repository_owner",
+ "repository_name",
+ "workflow_filename",
+ "environment",
+ )
+ }
+ for config in configs
+ ]
+ raise RuntimeError(
+ f"{name} has unexpected Trusted Publisher configurations: "
+ f"{json.dumps(compact, sort_keys=True)}"
+ )
+
+
+def preflight_authenticated(
+ packages: list[PlannedCrate],
+ candidate_names: set[str],
+ client: CratesIoClient,
+) -> list[str]:
+ verified: list[str] = []
+ for planned in packages:
+ metadata = client.get_crate(planned.name)
+ is_candidate = planned.name in candidate_names
+ if metadata is None:
+ if not is_candidate:
+ raise RuntimeError(
+ f"{planned.name} is missing but was not selected for bootstrap"
+ )
+ verified.append(planned.name)
+ continue
+
+ validate_crate_metadata(planned, metadata, client)
+ is_placeholder = metadata.get("max_version") == PLACEHOLDER_VERSION
+ if is_placeholder != is_candidate:
+ state = "a placeholder" if is_placeholder else "an established crate"
+ raise RuntimeError(
+ f"{planned.name} is now {state}, which does not match discovery"
+ )
+
+ configs = client.list_github_configs(planned.name)
+ if is_placeholder:
+ if configs:
+ validate_github_configs(planned.name, configs)
+ else:
+ validate_github_configs(planned.name, configs)
+ if metadata.get("trustpub_only") is not True:
+ raise RuntimeError(
+ f"{planned.name} does not require Trusted Publishing"
+ )
+ verified.append(planned.name)
+ return verified
+
+
+def verify_authenticated(
+ packages: list[PlannedCrate], client: CratesIoClient
+) -> list[str]:
+ verified: list[str] = []
+ for planned in packages:
+ metadata = client.get_crate(planned.name)
+ if metadata is None:
+ raise RuntimeError(f"{planned.name} does not exist on crates.io")
+ validate_crate_metadata(planned, metadata, client)
+ validate_github_configs(planned.name, client.list_github_configs(planned.name))
+ if metadata.get("trustpub_only") is not True:
+ raise RuntimeError(f"{planned.name} does not require Trusted Publishing")
+ verified.append(planned.name)
+ return verified
+
+
+def _placeholder_manifest(name: str) -> str:
+ return f"""# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[package]
+name = {json.dumps(name)}
+version = {json.dumps(PLACEHOLDER_VERSION)}
+edition = "2024"
+rust-version = "1.91"
+description = {json.dumps(PLACEHOLDER_DESCRIPTION)}
+homepage = "https://opendal.apache.org/"
+repository = {json.dumps(REPOSITORY)}
+license = "Apache-2.0"
+readme = "README.md"
+include = ["src/lib.rs", "README.md", "LICENSE", "NOTICE"]
+"""
+
+
+def _placeholder_readme(name: str) -> str:
+ return f"""# {name}
+
+This crate belongs to [Apache OpenDAL]({REPOSITORY}).
+
+Version {PLACEHOLDER_VERSION} reserves the crates.io package name for Apache
+OpenDAL. It is not an ASF software release, contains no implementation, and must
+not be used as a dependency.
+"""
+
+
+PLACEHOLDER_LIB = """// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#![doc = include_str!("../README.md")]
+"""
+
+
+def write_placeholder_package(
+ project_dir: Path, planned: PlannedCrate, package_dir: Path
+) -> None:
+ source_dir = package_dir / "src"
+ source_dir.mkdir()
+ (package_dir / "Cargo.toml").write_text(
+ _placeholder_manifest(planned.name), encoding="utf-8"
+ )
+ (package_dir / "README.md").write_text(
+ _placeholder_readme(planned.name), encoding="utf-8"
+ )
+ (source_dir / "lib.rs").write_text(PLACEHOLDER_LIB, encoding="utf-8")
+ shutil.copyfile(project_dir / "LICENSE", package_dir / "LICENSE")
+ shutil.copyfile(project_dir / "NOTICE", package_dir / "NOTICE")
+
+
+def publish_placeholder(project_dir: Path, planned: PlannedCrate, token: str) -> None:
+ with tempfile.TemporaryDirectory(prefix=f"{planned.name}-bootstrap-") as tmpdir:
+ package_dir = Path(tmpdir)
+ write_placeholder_package(project_dir, planned, package_dir)
+
+ env = os.environ.copy()
+ env["CARGO_REGISTRY_TOKEN"] = token
+ command = ["cargo", "publish", "--manifest-path", "Cargo.toml"]
+ while True:
+ process = subprocess.run(
+ command,
+ cwd=package_dir,
+ env=env,
+ check=False,
+ text=True,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ )
+ output = process.stdout or ""
+ print(output, end="", flush=True)
+ if process.returncode == 0:
+ return
+ if should_retry(output):
+ delay = parse_retry_after(output)
+ print(
+ f"crates.io rate limited {planned.name}; sleeping {delay}s",
+ flush=True,
+ )
+ time.sleep(delay)
+ continue
+ raise subprocess.CalledProcessError(
+ process.returncode, command, output=output
+ )
+
+
+def wait_for_crate(
+ client: CratesIoClient, name: str, timeout: int = 120
+) -> dict[str, object]:
+ deadline = time.monotonic() + timeout
+ while True:
+ metadata = client.get_crate(name)
+ if metadata is not None:
+ return metadata
+ if time.monotonic() >= deadline:
+ raise RuntimeError(
+ f"timed out waiting for {name} to become visible on crates.io"
+ )
+ time.sleep(2)
+
+
+def wait_for_trustpub_only(
+ client: CratesIoClient, name: str, timeout: int = 120
+) -> dict[str, object]:
+ deadline = time.monotonic() + timeout
+ while True:
+ metadata = client.get_crate(name)
+ if metadata is not None and metadata.get("trustpub_only") is True:
+ return metadata
+ if time.monotonic() >= deadline:
+ raise RuntimeError(
+ f"timed out waiting for {name} to require Trusted Publishing"
+ )
+ time.sleep(2)
+
+
+def wait_for_expected_config(
+ client: CratesIoClient, name: str, timeout: int = 120
+) -> None:
+ deadline = time.monotonic() + timeout
+ while True:
+ configs = client.list_github_configs(name)
+ if configs:
+ validate_github_configs(name, configs)
+ return
+ if time.monotonic() >= deadline:
+ raise RuntimeError(
+ f"timed out waiting for the Trusted Publisher for {name}"
+ )
+ time.sleep(2)
+
+
+def reconcile_crate(
+ project_dir: Path,
+ planned: PlannedCrate,
+ client: CratesIoClient,
+ token: str,
+) -> ReconcileResult:
+ actions: list[str] = []
+ metadata = client.get_crate(planned.name)
+ if metadata is None:
+ publish_placeholder(project_dir, planned, token)
+ metadata = wait_for_crate(client, planned.name)
+ actions.append("created placeholder")
+ elif metadata.get("max_version") != PLACEHOLDER_VERSION:
+ raise RuntimeError(
+ f"{planned.name} became an established crate after discovery; "
+ "refusing to modify it in the bootstrap workflow"
+ )
+
+ validate_crate_metadata(planned, metadata, client)
+
+ configs = client.list_github_configs(planned.name)
+ if not configs:
+ created = client.create_github_config(planned.name)
+ if not _is_expected_config(planned.name, created):
+ raise RuntimeError(
+ f"crates.io created an unexpected Trusted Publisher for {planned.name}"
+ )
+ actions.append("configured Trusted Publishing")
+ else:
+ validate_github_configs(planned.name, configs)
+
+ if metadata.get("trustpub_only") is not True:
+ updated = client.set_trustpub_only(planned.name)
+ if updated.get("trustpub_only") is not True:
+ raise RuntimeError(
+ f"crates.io did not enable Trusted Publishing only for {planned.name}"
+ )
+ actions.append("enabled Trusted Publishing only")
+
+ verified_metadata = wait_for_trustpub_only(client, planned.name)
+ validate_crate_metadata(planned, verified_metadata, client)
+ wait_for_expected_config(client, planned.name)
+
+ if not actions:
+ actions.append("verified")
+ return ReconcileResult(planned.name, tuple(actions))
+
+
+def discover(
+ project_dir: Path, client: CratesIoClient
+) -> tuple[list[PlannedCrate], list[str], list[str]]:
+ packages = planned_crates(project_dir)
+ missing: list[str] = []
+ placeholders: list[str] = []
+ for planned in packages:
+ metadata = client.get_crate(planned.name)
+ if metadata is None:
+ missing.append(planned.name)
+ continue
+ validate_crate_metadata(planned, metadata, client)
+ if metadata.get("max_version") == PLACEHOLDER_VERSION:
+ placeholders.append(planned.name)
+ return packages, missing, placeholders
+
+
+def verify_public(project_dir: Path, client: CratesIoClient) -> list[str]:
+ verified: list[str] = []
+ for planned in planned_crates(project_dir):
+ metadata = client.get_crate(planned.name)
+ if metadata is None:
+ raise RuntimeError(f"{planned.name} does not exist on crates.io")
+ validate_crate_metadata(planned, metadata, client)
+ if metadata.get("trustpub_only") is not True:
+ raise RuntimeError(f"{planned.name} does not require Trusted Publishing")
+ verified.append(planned.name)
+ return verified
+
+
+def _write_summary(title: str, lines: list[str]) -> None:
+ summary_path = os.environ.get("GITHUB_STEP_SUMMARY")
+ if not summary_path:
+ return
+ source_commit = os.environ.get("GITHUB_SHA", "unknown")
+ with Path(summary_path).open("a", encoding="utf-8") as summary:
+ summary.write(f"## {title}\n\n")
+ summary.write(f"Source commit: `{source_commit}`\n\n")
+ for line in lines:
+ summary.write(f"- {line}\n")
+ summary.write("\n")
+
+
+def run_discover(args: argparse.Namespace) -> int:
+ client = CratesIoClient(args.registry_url)
+ packages, missing, placeholders = discover(args.project_dir, client)
+ candidates = [*missing, *placeholders]
+ result = {
+ "packages": len(packages),
+ "missing": missing,
+ "placeholders": placeholders,
+ "bootstrap_candidates": candidates,
+ }
+ print(json.dumps(result, indent=2))
+ _write_summary(
+ "Rust crate bootstrap discovery",
+ [
+ f"Publishable crates: {len(packages)}",
+ f"Missing crates: {', '.join(missing) if missing else 'none'}",
+ f"Placeholder crates: {', '.join(placeholders) if placeholders else 'none'}",
+ ],
+ )
+ return 0
+
+
+def run_apply(args: argparse.Namespace) -> int:
+ token = os.environ.get("CARGO_REGISTRY_BOOTSTRAP_TOKEN")
+ if not token:
+ raise RuntimeError("CARGO_REGISTRY_BOOTSTRAP_TOKEN is not set")
+
+ client = CratesIoClient(args.registry_url, token=token)
+ packages, missing, placeholders = discover(args.project_dir, client)
+ candidate_set = {*missing, *placeholders}
+ candidates = [package for package in packages if package.name in candidate_set]
+
+ results: list[ReconcileResult] = []
+ authenticated: list[str] = []
+ try:
+ preflight_authenticated(packages, candidate_set, client)
+ print(
+ f"authenticated preflight passed for {len(packages)} planned crates",
+ flush=True,
+ )
+ print(
+ f"bootstrap candidates: {len(candidates)}",
+ flush=True,
+ )
+ for planned in candidates:
+ result = reconcile_crate(args.project_dir, planned, client, token)
+ results.append(result)
+ print(f"{result.name}: {', '.join(result.actions)}", flush=True)
+ authenticated = verify_authenticated(packages, client)
+ print(
+ f"authenticated final audit passed for {len(authenticated)} planned crates",
+ flush=True,
+ )
+ except Exception as error:
+ _write_summary(
+ "Rust crate bootstrap",
+ [
+ *(
+ f"`{result.name}`: {', '.join(result.actions)}"
+ for result in results
+ ),
+ f"Failed: {error}",
+ ],
+ )
+ raise
+
+ _write_summary(
+ "Rust crate bootstrap",
+ [
+ f"Authenticated preflight: {len(packages)} planned crates",
+ *(f"`{result.name}`: {', '.join(result.actions)}" for result in results),
+ f"Authenticated final audit: {len(authenticated)} planned crates",
+ ],
+ )
+ return 0
+
+
+def run_verify(args: argparse.Namespace) -> int:
+ verified = verify_public(args.project_dir, CratesIoClient(args.registry_url))
+ print(json.dumps({"verified": verified}, indent=2))
+ return 0
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(
+ description="Create and secure crates.io names in the OpenDAL Rust publish plan."
+ )
+ parser.add_argument(
+ "--project-dir",
+ type=Path,
+ default=PROJECT_DIR,
+ help="Path to the repository root.",
+ )
+ parser.add_argument(
+ "--registry-url",
+ default=REGISTRY_URL,
+ help="crates.io-compatible registry API URL.",
+ )
+ subparsers = parser.add_subparsers(dest="command", required=True)
+
+ discover_parser = subparsers.add_parser(
+ "discover", help="Report missing and placeholder crate names."
+ )
+ discover_parser.set_defaults(run=run_discover)
+
+ apply_parser = subparsers.add_parser(
+ "apply",
+ help="Audit all planned crates and reconcile missing names and placeholders.",
+ )
+ apply_parser.set_defaults(run=run_apply)
+
+ verify_parser = subparsers.add_parser(
+ "verify", help="Verify public crate existence and Trusted Publishing only."
+ )
+ verify_parser.set_defaults(run=run_verify)
+
+ args = parser.parse_args()
+ args.project_dir = args.project_dir.resolve()
+ return args.run(args)
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/.github/scripts/release_rust/publish.py b/.github/scripts/release_rust/publish.py
index 9472457ebe..8643da235f 100644
--- a/.github/scripts/release_rust/publish.py
+++ b/.github/scripts/release_rust/publish.py
@@ -24,12 +24,15 @@ import subprocess
import time
import tomllib
from collections.abc import Iterable
+from contextlib import nullcontext
from dataclasses import dataclass
from datetime import datetime
from datetime import timezone
from email.utils import parsedate_to_datetime
from pathlib import Path
+from trusted_publishing import temporary_trusted_publishing_token
+
SCRIPT_PATH = Path(__file__).resolve()
PROJECT_DIR = SCRIPT_PATH.parents[3]
@@ -215,20 +218,31 @@ def publish_package(project_dir: Path, package: str, dry_run: bool) -> None:
if patch is not None:
cmd.append("--allow-dirty")
- proc = subprocess.run(
- cmd,
- cwd=package_dir,
- check=False,
- env=os.environ,
- text=True,
- stdout=subprocess.PIPE,
- stderr=subprocess.STDOUT,
+ token_context = (
+ nullcontext(None) if dry_run else temporary_trusted_publishing_token()
)
+ with token_context as token:
+ env = os.environ.copy()
+ if token is None:
+ env.pop("CARGO_REGISTRY_TOKEN", None)
+ else:
+ env["CARGO_REGISTRY_TOKEN"] = token
+
+ proc = subprocess.run(
+ cmd,
+ cwd=package_dir,
+ check=False,
+ env=env,
+ text=True,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ )
+
+ output = proc.stdout or ""
+ print(output, end="", flush=True)
finally:
restore(patch)
- output = proc.stdout or ""
- print(output, end="", flush=True)
if proc.returncode == 0:
return
if already_published(output):
diff --git a/.github/scripts/release_rust/test_bootstrap.py b/.github/scripts/release_rust/test_bootstrap.py
new file mode 100644
index 0000000000..43a5b4f5f8
--- /dev/null
+++ b/.github/scripts/release_rust/test_bootstrap.py
@@ -0,0 +1,494 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import argparse
+import io
+import subprocess
+import tempfile
+import tomllib
+import unittest
+import urllib.error
+from pathlib import Path
+from unittest import mock
+
+from bootstrap import CratesIoClient
+from bootstrap import PLACEHOLDER_DESCRIPTION
+from bootstrap import PLACEHOLDER_VERSION
+from bootstrap import PUBLISHER
+from bootstrap import PROJECT_DIR
+from bootstrap import REPOSITORY
+from bootstrap import PlannedCrate
+from bootstrap import ReconcileResult
+from bootstrap import _placeholder_manifest
+from bootstrap import discover
+from bootstrap import preflight_authenticated
+from bootstrap import reconcile_crate
+from bootstrap import run_apply
+from bootstrap import verify_authenticated
+from bootstrap import write_placeholder_package
+
+
+def metadata(name: str, *, version: str = "1.0.0", trustpub_only: bool = False):
+ return {
+ "id": name,
+ "max_version": version,
+ "repository": REPOSITORY,
+ "description": (
+ PLACEHOLDER_DESCRIPTION
+ if version == PLACEHOLDER_VERSION
+ else "An Apache OpenDAL crate"
+ ),
+ "trustpub_only": trustpub_only,
+ }
+
+
+def expected_config(name: str):
+ return {"crate": name, **PUBLISHER}
+
+
+class FakeClient:
+ def __init__(self, name: str, krate=None, configs=None):
+ self.name = name
+ self.krate = krate
+ self.configs = list(configs or [])
+ self.created_configs = 0
+ self.restricted = 0
+
+ def get_crate(self, name: str):
+ self._assert_name(name)
+ return None if self.krate is None else dict(self.krate)
+
+ def get_version(self, name: str, version: str):
+ self._assert_name(name)
+ return {"num": version}
+
+ def list_github_configs(self, name: str):
+ self._assert_name(name)
+ return [dict(config) for config in self.configs]
+
+ def create_github_config(self, name: str):
+ self._assert_name(name)
+ config = expected_config(name)
+ self.configs.append(config)
+ self.created_configs += 1
+ return dict(config)
+
+ def set_trustpub_only(self, name: str):
+ self._assert_name(name)
+ self.krate["trustpub_only"] = True
+ self.restricted += 1
+ return dict(self.krate)
+
+ def _assert_name(self, name: str):
+ if name != self.name:
+ raise AssertionError(f"expected {self.name}, got {name}")
+
+
+class JsonResponse(io.BytesIO):
+ def __enter__(self):
+ return self
+
+ def __exit__(self, exc_type, exc_value, traceback):
+ self.close()
+
+
+class BootstrapTest(unittest.TestCase):
+ def test_bootstrap_workflow_is_input_free_and_always_protected(self):
+ workflow = (
+ PROJECT_DIR / ".github/workflows/bootstrap_rust_crates.yml"
+ ).read_text(encoding="utf-8")
+ dispatch = workflow.split("\non:\n", 1)[1].split("\npermissions:\n", 1)[0]
+
+ self.assertEqual(dispatch, " workflow_dispatch:\n")
+ self.assertIn(" environment: rust-bootstrap\n", workflow)
+ self.assertNotIn("candidate_count", workflow)
+
+ def test_publisher_matches_the_release_workflow(self):
+ workflow = (PROJECT_DIR / ".github/workflows/release_rust.yml").read_text(
+ encoding="utf-8"
+ )
+ publish_job = workflow.split("\n publish:\n", 1)[1]
+
+ self.assertEqual(PUBLISHER["workflow_filename"], "release_rust.yml")
+ self.assertEqual(PUBLISHER["environment"], "release")
+ self.assertIn(" environment: release\n", publish_job)
+ self.assertIn(" id-token: write\n", publish_job)
+
+ def test_placeholder_manifest_is_dependency_free(self):
+ manifest = tomllib.loads(_placeholder_manifest("opendal-service-new"))
+
+ self.assertEqual(manifest["package"]["name"], "opendal-service-new")
+ self.assertEqual(manifest["package"]["version"], PLACEHOLDER_VERSION)
+ self.assertNotIn("dependencies", manifest)
+ self.assertNotIn("dev-dependencies", manifest)
+ self.assertNotIn("build-dependencies", manifest)
+
+ def test_placeholder_can_be_packaged_offline(self):
+ planned = PlannedCrate("opendal-service-new", "core/services/new")
+ with tempfile.TemporaryDirectory() as tmpdir:
+ package_dir = Path(tmpdir)
+ write_placeholder_package(PROJECT_DIR, planned, package_dir)
+
+ process = subprocess.run(
+ [
+ "cargo",
+ "package",
+ "--manifest-path",
+ str(package_dir / "Cargo.toml"),
+ "--no-verify",
+ "--offline",
+ ],
+ check=False,
+ text=True,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ )
+
+ self.assertEqual(process.returncode, 0, process.stdout)
+ self.assertTrue(
+ (
+ package_dir
+ / "target"
+ / "package"
+ / f"{planned.name}-{PLACEHOLDER_VERSION}.crate"
+ ).is_file()
+ )
+
+ def test_public_read_retries_a_transient_registry_failure(self):
+ error = urllib.error.HTTPError(
+ "https://crates.io/api/v1/crates/opendal",
+ 503,
+ "unavailable",
+ {},
+ io.BytesIO(b""),
+ )
+ response = JsonResponse(
+ b'{"crate":{"id":"opendal","repository":'
+ b'"https://github.com/apache/opendal"}}'
+ )
+
+ with (
+ mock.patch(
+ "bootstrap.urllib.request.urlopen",
+ side_effect=(error, response),
+ ),
+ mock.patch("bootstrap.time.sleep") as sleep,
+ ):
+ krate = CratesIoClient().get_crate("opendal")
+
+ self.assertEqual(krate["id"], "opendal")
+ sleep.assert_called_once_with(1)
+
+ def test_discovery_does_not_migrate_established_crates(self):
+ planned = PlannedCrate("opendal-core", "core/core")
+ client = FakeClient(planned.name, metadata(planned.name))
+
+ with mock.patch("bootstrap.planned_crates", return_value=[planned]):
+ packages, missing, incomplete = discover(Path(), client)
+
+ self.assertEqual(packages, [planned])
+ self.assertEqual(missing, [])
+ self.assertEqual(incomplete, [])
+
+ def test_discovery_selects_an_incomplete_placeholder(self):
+ planned = PlannedCrate("opendal-service-new", "core/services/new")
+ client = FakeClient(
+ planned.name,
+ metadata(planned.name, version=PLACEHOLDER_VERSION),
+ )
+
+ with mock.patch("bootstrap.planned_crates", return_value=[planned]):
+ _, missing, incomplete = discover(Path(), client)
+
+ self.assertEqual(missing, [])
+ self.assertEqual(incomplete, [planned.name])
+
+ def test_discovery_selects_a_ready_placeholder(self):
+ planned = PlannedCrate("opendal-service-new", "core/services/new")
+ client = FakeClient(
+ planned.name,
+ metadata(
+ planned.name,
+ version=PLACEHOLDER_VERSION,
+ trustpub_only=True,
+ ),
+ [expected_config(planned.name)],
+ )
+
+ with mock.patch("bootstrap.planned_crates", return_value=[planned]):
+ _, missing, placeholders = discover(Path(), client)
+
+ self.assertEqual(missing, [])
+ self.assertEqual(placeholders, [planned.name])
+
+ def test_authenticated_preflight_audits_an_established_crate(self):
+ planned = PlannedCrate("opendal-core", "core/core")
+ client = FakeClient(
+ planned.name,
+ metadata(planned.name, trustpub_only=True),
+ [expected_config(planned.name)],
+ )
+
+ verified = preflight_authenticated([planned], set(), client)
+
+ self.assertEqual(verified, [planned.name])
+ self.assertEqual(client.created_configs, 0)
+ self.assertEqual(client.restricted, 0)
+
+ def test_authenticated_preflight_rejects_an_unexpected_publisher(self):
+ planned = PlannedCrate("opendal-core", "core/core")
+ unexpected = {
+ **expected_config(planned.name),
+ "workflow_filename": "other.yml",
+ }
+ client = FakeClient(
+ planned.name,
+ metadata(planned.name, trustpub_only=True),
+ [unexpected],
+ )
+
+ with self.assertRaisesRegex(RuntimeError, "unexpected Trusted Publisher"):
+ preflight_authenticated([planned], set(), client)
+
+ self.assertEqual(client.created_configs, 0)
+ self.assertEqual(client.restricted, 0)
+
+ def test_authenticated_preflight_rejects_a_ready_placeholder_with_wrong_publisher(
+ self,
+ ):
+ planned = PlannedCrate("opendal-service-new", "core/services/new")
+ unexpected = {
+ **expected_config(planned.name),
+ "workflow_filename": "other.yml",
+ }
+ client = FakeClient(
+ planned.name,
+ metadata(
+ planned.name,
+ version=PLACEHOLDER_VERSION,
+ trustpub_only=True,
+ ),
+ [unexpected],
+ )
+
+ with self.assertRaisesRegex(RuntimeError, "unexpected Trusted Publisher"):
+ preflight_authenticated([planned], {planned.name}, client)
+
+ self.assertEqual(client.created_configs, 0)
+ self.assertEqual(client.restricted, 0)
+
+ def test_authenticated_preflight_rejects_unmigrated_established_crate(self):
+ planned = PlannedCrate("opendal-core", "core/core")
+ client = FakeClient(
+ planned.name,
+ metadata(planned.name),
+ [expected_config(planned.name)],
+ )
+
+ with self.assertRaisesRegex(RuntimeError, "does not require"):
+ preflight_authenticated([planned], set(), client)
+
+ def test_authenticated_verification_requires_the_exact_publisher(self):
+ planned = PlannedCrate("opendal-core", "core/core")
+ unexpected = {
+ **expected_config(planned.name),
+ "environment": "other",
+ }
+ client = FakeClient(
+ planned.name,
+ metadata(planned.name, trustpub_only=True),
+ [unexpected],
+ )
+
+ with self.assertRaisesRegex(RuntimeError, "unexpected Trusted Publisher"):
+ verify_authenticated([planned], client)
+
+ def test_apply_preflights_the_complete_plan_before_reconciling(self):
+ candidate = PlannedCrate("opendal-service-new", "core/services/new")
+ established = PlannedCrate("opendal-core", "core/core")
+ packages = [candidate, established]
+ client = object()
+ calls = []
+
+ def preflight(actual_packages, candidate_names, actual_client):
+ calls.append(("preflight", actual_packages, candidate_names, actual_client))
+
+ def reconcile(project_dir, planned, actual_client, token):
+ calls.append(("reconcile", planned, actual_client, token))
+ return ReconcileResult(planned.name, ("verified",))
+
+ def verify(actual_packages, actual_client):
+ calls.append(("verify", actual_packages, actual_client))
+ return [package.name for package in actual_packages]
+
+ args = argparse.Namespace(project_dir=Path(), registry_url="registry")
+ with (
+ mock.patch.dict(
+ "bootstrap.os.environ",
+ {"CARGO_REGISTRY_BOOTSTRAP_TOKEN": "bootstrap-token"},
+ clear=True,
+ ),
+ mock.patch("bootstrap.CratesIoClient", return_value=client),
+ mock.patch(
+ "bootstrap.discover",
+ return_value=(packages, [candidate.name], []),
+ ),
+ mock.patch("bootstrap.preflight_authenticated", preflight),
+ mock.patch("bootstrap.reconcile_crate", reconcile),
+ mock.patch("bootstrap.verify_authenticated", verify),
+ ):
+ self.assertEqual(run_apply(args), 0)
+
+ self.assertEqual(
+ [call[0] for call in calls],
+ ["preflight", "reconcile", "verify"],
+ )
+ self.assertEqual(calls[0][1], packages)
+ self.assertEqual(calls[0][2], {candidate.name})
+ self.assertEqual(calls[2][1], packages)
+
+ def test_apply_does_not_mutate_when_authenticated_preflight_fails(self):
+ candidate = PlannedCrate("opendal-service-new", "core/services/new")
+ args = argparse.Namespace(project_dir=Path(), registry_url="registry")
+
+ with (
+ mock.patch.dict(
+ "bootstrap.os.environ",
+ {"CARGO_REGISTRY_BOOTSTRAP_TOKEN": "bootstrap-token"},
+ clear=True,
+ ),
+ mock.patch("bootstrap.CratesIoClient"),
+ mock.patch(
+ "bootstrap.discover",
+ return_value=([candidate], [candidate.name], []),
+ ),
+ mock.patch(
+ "bootstrap.preflight_authenticated",
+ side_effect=RuntimeError("audit failed"),
+ ),
+ mock.patch("bootstrap.reconcile_crate") as reconcile,
+ mock.patch("bootstrap.verify_authenticated") as verify,
+ self.assertRaisesRegex(RuntimeError, "audit failed"),
+ ):
+ run_apply(args)
+
+ reconcile.assert_not_called()
+ verify.assert_not_called()
+
+ def test_missing_crate_is_created_configured_and_restricted(self):
+ planned = PlannedCrate("opendal-service-new", "core/services/new")
+ client = FakeClient(planned.name)
+
+ def publish(project_dir, package, token):
+ self.assertEqual(package, planned)
+ self.assertEqual(token, "bootstrap-token")
+ client.krate = metadata(planned.name, version=PLACEHOLDER_VERSION)
+
+ with (
+ tempfile.TemporaryDirectory() as tmpdir,
+ mock.patch("bootstrap.publish_placeholder", publish),
+ ):
+ result = reconcile_crate(Path(tmpdir), planned, client, "bootstrap-token")
+
+ self.assertEqual(
+ result.actions,
+ (
+ "created placeholder",
+ "configured Trusted Publishing",
+ "enabled Trusted Publishing only",
+ ),
+ )
+ self.assertEqual(client.created_configs, 1)
+ self.assertEqual(client.restricted, 1)
+
+ def test_partial_placeholder_resumes_without_republishing(self):
+ planned = PlannedCrate("opendal-service-new", "core/services/new")
+ client = FakeClient(
+ planned.name,
+ metadata(planned.name, version=PLACEHOLDER_VERSION),
+ )
+
+ with (
+ tempfile.TemporaryDirectory() as tmpdir,
+ mock.patch("bootstrap.publish_placeholder") as publish,
+ ):
+ result = reconcile_crate(Path(tmpdir), planned, client, "bootstrap-token")
+
+ publish.assert_not_called()
+ self.assertEqual(
+ result.actions,
+ (
+ "configured Trusted Publishing",
+ "enabled Trusted Publishing only",
+ ),
+ )
+
+ def test_ready_placeholder_is_a_noop(self):
+ planned = PlannedCrate("opendal-service-new", "core/services/new")
+ client = FakeClient(
+ planned.name,
+ metadata(
+ planned.name,
+ version=PLACEHOLDER_VERSION,
+ trustpub_only=True,
+ ),
+ [expected_config(planned.name)],
+ )
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ result = reconcile_crate(Path(tmpdir), planned, client, "bootstrap-token")
+
+ self.assertEqual(result.actions, ("verified",))
+ self.assertEqual(client.created_configs, 0)
+ self.assertEqual(client.restricted, 0)
+
+ def test_established_crate_is_never_migrated(self):
+ planned = PlannedCrate("opendal-core", "core/core")
+ client = FakeClient(planned.name, metadata(planned.name))
+
+ with (
+ tempfile.TemporaryDirectory() as tmpdir,
+ self.assertRaisesRegex(RuntimeError, "established crate"),
+ ):
+ reconcile_crate(Path(tmpdir), planned, client, "bootstrap-token")
+
+ self.assertEqual(client.created_configs, 0)
+ self.assertEqual(client.restricted, 0)
+
+ def test_unexpected_publisher_fails_closed(self):
+ planned = PlannedCrate("opendal-core", "core/core")
+ unexpected = {
+ **expected_config(planned.name),
+ "workflow_filename": "other.yml",
+ }
+ client = FakeClient(
+ planned.name,
+ metadata(planned.name, version=PLACEHOLDER_VERSION),
+ [unexpected],
+ )
+
+ with (
+ tempfile.TemporaryDirectory() as tmpdir,
+ self.assertRaisesRegex(RuntimeError, "unexpected Trusted Publisher"),
+ ):
+ reconcile_crate(Path(tmpdir), planned, client, "bootstrap-token")
+
+ self.assertEqual(client.restricted, 0)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/.github/scripts/release_rust/test_publish.py b/.github/scripts/release_rust/test_publish.py
index 2da69b3817..bb6a929440 100644
--- a/.github/scripts/release_rust/test_publish.py
+++ b/.github/scripts/release_rust/test_publish.py
@@ -18,10 +18,13 @@
import tempfile
import textwrap
import unittest
+from contextlib import contextmanager
from pathlib import Path
+from unittest import mock
from publish import local_dev_dependency_names
from publish import load_manifest
+from publish import publish_package
from publish import strip_local_dev_dependencies
@@ -81,6 +84,103 @@ class ReleaseRustPublishTest(unittest.TestCase):
self.assertNotIn("local-dev-multiline", stripped)
self.assertNotIn("local-target-dev", stripped)
+ def test_live_publish_fetches_a_new_token_for_every_attempt(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ root = Path(tmpdir)
+ package = root / "crate"
+ write(
+ package / "Cargo.toml",
+ """
+ [package]
+ name = "crate"
+ version = "0.1.0"
+ """,
+ )
+
+ tokens = iter(("first-token", "second-token"))
+ released: list[str] = []
+ cargo_tokens: list[str | None] = []
+
+ @contextmanager
+ def token_provider():
+ token = next(tokens)
+ try:
+ yield token
+ finally:
+ released.append(token)
+
+ results = iter(
+ (
+ subprocess_result(
+ 1, "Too many requests. Please try again after invalid."
+ ),
+ subprocess_result(0, "published"),
+ )
+ )
+
+ def run(*args, **kwargs):
+ cargo_tokens.append(kwargs["env"].get("CARGO_REGISTRY_TOKEN"))
+ return next(results)
+
+ with (
+ mock.patch.dict(
+ "publish.os.environ",
+ {"CARGO_REGISTRY_TOKEN": "legacy-token"},
+ clear=True,
+ ),
+ mock.patch(
+ "publish.temporary_trusted_publishing_token", token_provider
+ ),
+ mock.patch("publish.subprocess.run", run),
+ mock.patch("publish.time.sleep") as sleep,
+ ):
+ publish_package(
+ root,
+ "crate",
+ dry_run=False,
+ )
+
+ self.assertEqual(cargo_tokens, ["first-token", "second-token"])
+ self.assertEqual(released, ["first-token", "second-token"])
+ sleep.assert_called_once_with(610)
+
+ def test_dry_run_does_not_request_a_trusted_publishing_token(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ root = Path(tmpdir)
+ package = root / "crate"
+ write(
+ package / "Cargo.toml",
+ """
+ [package]
+ name = "crate"
+ version = "0.1.0"
+ """,
+ )
+
+ with (
+ mock.patch.dict(
+ "publish.os.environ",
+ {"CARGO_REGISTRY_TOKEN": "legacy-token"},
+ clear=True,
+ ),
+ mock.patch(
+ "publish.temporary_trusted_publishing_token"
+ ) as token_provider,
+ mock.patch(
+ "publish.subprocess.run",
+ return_value=subprocess_result(0, "checked"),
+ ) as run,
+ ):
+ publish_package(root, "crate", dry_run=True)
+
+ token_provider.assert_not_called()
+ self.assertIn("--dry-run", run.call_args.args[0])
+ self.assertNotIn("CARGO_REGISTRY_TOKEN", run.call_args.kwargs["env"])
+
+
+def subprocess_result(returncode: int, output: str):
+ return mock.Mock(returncode=returncode, stdout=output)
+
if __name__ == "__main__":
unittest.main()
diff --git a/.github/scripts/release_rust/test_trusted_publishing.py b/.github/scripts/release_rust/test_trusted_publishing.py
new file mode 100644
index 0000000000..4b982d3b3f
--- /dev/null
+++ b/.github/scripts/release_rust/test_trusted_publishing.py
@@ -0,0 +1,116 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import io
+import json
+import unittest
+from unittest import mock
+from urllib.parse import parse_qs
+from urllib.parse import urlsplit
+
+from trusted_publishing import _oidc_request_url
+from trusted_publishing import request_trusted_publishing_token
+from trusted_publishing import revoke_trusted_publishing_token
+from trusted_publishing import temporary_trusted_publishing_token
+
+
+class JsonResponse(io.BytesIO):
+ def __enter__(self):
+ return self
+
+ def __exit__(self, exc_type, exc_value, traceback):
+ self.close()
+
+
+class TrustedPublishingTest(unittest.TestCase):
+ def test_oidc_request_url_preserves_existing_query(self):
+ result = _oidc_request_url(
+ "https://example.test/token?api-version=1", "crates.io"
+ )
+
+ query = parse_qs(urlsplit(result).query)
+ self.assertEqual(query["api-version"], ["1"])
+ self.assertEqual(query["audience"], ["crates.io"])
+
+ def test_token_exchange_uses_github_oidc_and_crates_io(self):
+ requests = []
+ responses = iter(
+ (
+ JsonResponse(b'{"value":"github-jwt"}'),
+ JsonResponse(b'{"token":"crates-token"}'),
+ )
+ )
+
+ def urlopen(request, timeout):
+ requests.append(request)
+ self.assertEqual(timeout, 30)
+ return next(responses)
+
+ environment = {
+ "ACTIONS_ID_TOKEN_REQUEST_URL": "https://github.test/oidc?api-version=1",
+ "ACTIONS_ID_TOKEN_REQUEST_TOKEN": "request-token",
+ }
+ with (
+ mock.patch.dict("trusted_publishing.os.environ", environment, clear=True),
+ mock.patch("trusted_publishing.urllib.request.urlopen", urlopen),
+ ):
+ token = request_trusted_publishing_token()
+
+ self.assertEqual(token, "crates-token")
+ self.assertEqual(requests[0].get_method(), "GET")
+ self.assertEqual(
+ parse_qs(urlsplit(requests[0].full_url).query)["audience"],
+ ["crates.io"],
+ )
+ self.assertEqual(
+ requests[0].get_header("Authorization"), "Bearer request-token"
+ )
+ self.assertEqual(requests[1].get_method(), "POST")
+ self.assertEqual(json.loads(requests[1].data), {"jwt": "github-jwt"})
+
+ def test_revoke_uses_the_temporary_token(self):
+ requests = []
+
+ def urlopen(request, timeout):
+ requests.append(request)
+ self.assertEqual(timeout, 30)
+ return JsonResponse(b"")
+
+ with mock.patch("trusted_publishing.urllib.request.urlopen", urlopen):
+ revoke_trusted_publishing_token("crates-token")
+
+ self.assertEqual(requests[0].get_method(), "DELETE")
+ self.assertEqual(requests[0].get_header("Authorization"), "Bearer crates-token")
+
+ def test_temporary_token_is_always_revoked(self):
+ with (
+ mock.patch(
+ "trusted_publishing.request_trusted_publishing_token",
+ return_value="temporary-token",
+ ),
+ mock.patch("trusted_publishing.revoke_trusted_publishing_token") as revoke,
+ ):
+ with self.assertRaisesRegex(RuntimeError, "publish failed"):
+ with temporary_trusted_publishing_token() as token:
+ self.assertEqual(token, "temporary-token")
+ raise RuntimeError("publish failed")
+
+ revoke.assert_called_once_with("temporary-token", "https://crates.io")
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/.github/scripts/release_rust/trusted_publishing.py b/.github/scripts/release_rust/trusted_publishing.py
new file mode 100755
index 0000000000..fc58242f98
--- /dev/null
+++ b/.github/scripts/release_rust/trusted_publishing.py
@@ -0,0 +1,159 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import json
+import os
+import urllib.error
+import urllib.parse
+import urllib.request
+from contextlib import contextmanager
+from collections.abc import Iterator
+
+
+DEFAULT_REGISTRY_URL = "https://crates.io"
+USER_AGENT = "apache-opendal-release/1.0 (https://github.com/apache/opendal)"
+
+
+def _response_error(operation: str, error: urllib.error.HTTPError) -> RuntimeError:
+ try:
+ response = error.read().decode("utf-8", errors="replace")
+ finally:
+ error.close()
+ try:
+ details = json.loads(response)
+ errors = details.get("errors", [])
+ response = "; ".join(error["detail"] for error in errors)
+ except (json.JSONDecodeError, AttributeError, KeyError, TypeError):
+ pass
+ return RuntimeError(f"{operation} failed with HTTP {error.code}: {response}")
+
+
+def _request_json(request: urllib.request.Request, operation: str) -> dict[str, object]:
+ try:
+ with urllib.request.urlopen(request, timeout=30) as response:
+ return json.load(response)
+ except urllib.error.HTTPError as error:
+ raise _response_error(operation, error) from None
+ except urllib.error.URLError as error:
+ raise RuntimeError(f"{operation} failed: {error.reason}") from None
+
+
+def _oidc_request_url(request_url: str, audience: str) -> str:
+ parsed = urllib.parse.urlsplit(request_url)
+ query = urllib.parse.parse_qsl(parsed.query, keep_blank_values=True)
+ query.append(("audience", audience))
+ return urllib.parse.urlunsplit(
+ (
+ parsed.scheme,
+ parsed.netloc,
+ parsed.path,
+ urllib.parse.urlencode(query),
+ parsed.fragment,
+ )
+ )
+
+
+def _mask_secret(value: str) -> None:
+ if os.environ.get("GITHUB_ACTIONS") == "true":
+ print(f"::add-mask::{value}", flush=True)
+
+
+# The GitHub Action runs once per step. This helper repeats the same exchange
+# and revoke protocol around every cargo publish attempt.
+def request_trusted_publishing_token(
+ registry_url: str = DEFAULT_REGISTRY_URL,
+) -> str:
+ request_url = os.environ.get("ACTIONS_ID_TOKEN_REQUEST_URL")
+ request_token = os.environ.get("ACTIONS_ID_TOKEN_REQUEST_TOKEN")
+ if not request_url or not request_token:
+ raise RuntimeError(
+ "GitHub OIDC is unavailable; grant this job `id-token: write`"
+ )
+
+ audience = registry_url.rstrip("/")
+ for prefix in ("https://", "http://"):
+ if audience.startswith(prefix):
+ audience = audience.removeprefix(prefix)
+ break
+ oidc_request = urllib.request.Request(
+ _oidc_request_url(request_url, audience),
+ headers={
+ "Authorization": f"Bearer {request_token}",
+ "User-Agent": USER_AGENT,
+ },
+ )
+ oidc_response = _request_json(oidc_request, "requesting a GitHub OIDC token")
+ jwt = oidc_response.get("value")
+ if not isinstance(jwt, str) or not jwt:
+ raise RuntimeError("GitHub OIDC response did not contain a token")
+ _mask_secret(jwt)
+
+ token_request = urllib.request.Request(
+ f"{registry_url.rstrip('/')}/api/v1/trusted_publishing/tokens",
+ data=json.dumps({"jwt": jwt}).encode(),
+ headers={
+ "Content-Type": "application/json",
+ "User-Agent": USER_AGENT,
+ },
+ method="POST",
+ )
+ token_response = _request_json(
+ token_request, "exchanging a crates.io Trusted Publishing token"
+ )
+ token = token_response.get("token")
+ if not isinstance(token, str) or not token:
+ raise RuntimeError(
+ "crates.io Trusted Publishing response did not contain a token"
+ )
+ _mask_secret(token)
+ return token
+
+
+def revoke_trusted_publishing_token(
+ token: str, registry_url: str = DEFAULT_REGISTRY_URL
+) -> None:
+ request = urllib.request.Request(
+ f"{registry_url.rstrip('/')}/api/v1/trusted_publishing/tokens",
+ headers={
+ "Authorization": f"Bearer {token}",
+ "User-Agent": USER_AGENT,
+ },
+ method="DELETE",
+ )
+ try:
+ with urllib.request.urlopen(request, timeout=30):
+ return
+ except urllib.error.HTTPError as error:
+ raise _response_error(
+ "revoking a crates.io Trusted Publishing token", error
+ ) from None
+ except urllib.error.URLError as error:
+ raise RuntimeError(
+ f"revoking a crates.io Trusted Publishing token failed: {error.reason}"
+ ) from None
+
+
+@contextmanager
+def temporary_trusted_publishing_token(
+ registry_url: str = DEFAULT_REGISTRY_URL,
+) -> Iterator[str]:
+ token = request_trusted_publishing_token(registry_url)
+ try:
+ yield token
+ finally:
+ revoke_trusted_publishing_token(token, registry_url)
diff --git a/.github/workflows/bootstrap_rust_crates.yml b/.github/workflows/bootstrap_rust_crates.yml
new file mode 100644
index 0000000000..0bf1cb5c1a
--- /dev/null
+++ b/.github/workflows/bootstrap_rust_crates.yml
@@ -0,0 +1,76 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Bootstrap Rust Crates
+run-name: Bootstrap Rust crates at ${{ github.sha }}
+
+on:
+ workflow_dispatch:
+
+permissions:
+ contents: read
+
+concurrency:
+ group: bootstrap-rust-crates
+ cancel-in-progress: false
+
+jobs:
+ discover:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v7
+ with:
+ persist-credentials: false
+ ref: ${{ github.sha }}
+ - name: Validate main dispatch
+ run: |
+ set -euo pipefail
+ if [[ "${GITHUB_REF}" != "refs/heads/main" ]]; then
+ echo "Bootstrap Rust Crates must be dispatched from main" >&2
+ exit 1
+ fi
+ test "$(git rev-parse HEAD)" = "${GITHUB_SHA}"
+ - uses: actions/setup-python@v6
+ with:
+ python-version: "3.11"
+ - name: Discover Rust crates
+ run: python3 .github/scripts/release_rust/bootstrap.py discover
+
+ bootstrap:
+ needs: discover
+ runs-on: ubuntu-latest
+ environment: rust-bootstrap
+ steps:
+ - uses: actions/checkout@v7
+ with:
+ persist-credentials: false
+ ref: ${{ github.sha }}
+ - name: Validate main dispatch
+ run: |
+ set -euo pipefail
+ if [[ "${GITHUB_REF}" != "refs/heads/main" ]]; then
+ echo "Bootstrap Rust Crates must be dispatched from main" >&2
+ exit 1
+ fi
+ test "$(git rev-parse HEAD)" = "${GITHUB_SHA}"
+ - uses: actions/setup-python@v6
+ with:
+ python-version: "3.11"
+ - name: Audit and bootstrap crates.io packages
+ env:
+ CARGO_REGISTRY_BOOTSTRAP_TOKEN: ${{ secrets.CARGO_REGISTRY_BOOTSTRAP_TOKEN }}
+ run: python3 .github/scripts/release_rust/bootstrap.py apply
diff --git a/.github/workflows/release_rust.yml b/.github/workflows/release_rust.yml
index 83a15641b1..34013babb4 100644
--- a/.github/workflows/release_rust.yml
+++ b/.github/workflows/release_rust.yml
@@ -27,6 +27,7 @@ on:
branches:
- main
paths:
+ - ".github/workflows/bootstrap_rust_crates.yml"
- ".github/workflows/release_rust.yml"
- ".github/scripts/release_rust/**"
- "core/testkit/Cargo.toml"
@@ -43,6 +44,8 @@ jobs:
packages: ${{ steps.plan.outputs.packages }}
steps:
- uses: actions/checkout@v7
+ - name: Test release helpers
+ run: python3 -m unittest discover -s .github/scripts/release_rust -p "test_*.py"
- name: Plan publish matrix
id: plan
run: python3 .github/scripts/release_rust/plan.py --github-output
@@ -51,7 +54,9 @@ jobs:
if: ${{ startsWith(github.ref, 'refs/tags/') && !contains(github.ref, '-') }}
needs: plan
runs-on: ubuntu-latest
+ environment: release
permissions:
+ contents: read
id-token: write
steps:
- uses: actions/checkout@v7
@@ -69,11 +74,8 @@ jobs:
with:
need-rocksdb: true
need-protoc: true
- - uses: rust-lang/crates-io-auth-action@c6f97d42243bad5fab37ca0427f495c86d5b1a18
- id: auth
- name: Publish Rust crates
env:
PACKAGES: ${{ needs.plan.outputs.packages }}
LD_LIBRARY_PATH: ${{ env.JAVA_HOME }}/lib/server:${{ env.LD_LIBRARY_PATH }}
- CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN || steps.auth.outputs.token }}
run: python3 .github/scripts/release_rust/publish.py
diff --git a/website/community/release/release.md b/website/community/release/release.md
index cfabffe78a..32b615946c 100644
--- a/website/community/release/release.md
+++ b/website/community/release/release.md
@@ -104,6 +104,52 @@ Download and setup `cargo-deny`. You can refer to [cargo-deny](https://embarkstu
Running `python3 ./scripts/dependencies.py generate` to update the dependency list of every package.
+### Bootstrap Rust crate names
+
+During release preparation, wait until the intended Rust crate-name set is on
+`apache/opendal` `main`. The release manager chooses the exact reservation time,
+normally about three days before the planned release. At that time, check out
+the current `main` commit and run:
+
+```shell
+.agents/skills/opendal-release/scripts/bootstrap-rust-crates.sh
+```
+
+The helper dispatches the manual `Bootstrap Rust Crates` workflow without
+inputs. The workflow scans the Rust publish plan from the `main` commit that
+GitHub records as the run's `headSha`. The helper resolves that exact run,
+checks the `headSha`, and waits for it to finish. A PMC member must approve the
+protected `rust-bootstrap` environment on every run.
+
+The protected job authenticates ownership and audits every existing planned
+crate before it writes anything. Each established crate must already have
+`apache/opendal`, `release_rust.yml`, and the `release` environment as its only
+Trusted Publisher, with `trustpub_only` enabled. The workflow never modifies an
+established crate. For each missing name, it publishes a dependency-free
+`0.0.0` package, configures that Trusted Publisher, and enables
+`trustpub_only`. Reruns reconcile every placeholder. The protected authenticated
+audit runs even when there are no bootstrap candidates.
+
+The one-time migration of existing crates is a separate administrative task.
+Configure the same `apache/opendal`, `release_rust.yml`, and `release` publisher
+identity and enable `trustpub_only` for every existing crate. This workflow
+audits but never modifies established crates, and it fails if the migration is
+incomplete. Complete that migration before using the OIDC-only Rust release
+workflow.
+
+The `0.0.0` package is a namespace reservation, not an ASF software release or
+release artifact. crates.io exposes the reservation publicly, and publishing it
+is irreversible. The release manager decides when the names are stable enough
+to reserve. Run the helper again if a later `main` commit adds another planned
+crate. Do not create the RC tag until every name in the current publish plan has
+passed the workflow.
+
+Repository administrators must configure the `rust-bootstrap` GitHub
+environment with PMC required reviewers and a
+`CARGO_REGISTRY_BOOTSTRAP_TOKEN` secret. The crates.io token must have only the
+`publish-new` and `trusted-publishing` endpoint scopes and OpenDAL-specific crate
+scopes. The normal Rust release workflow must not have access to this token.
+
### Push release candidate tag
After bump version PR gets merged, push the release candidate tag:
@@ -149,14 +195,10 @@ of calling `cargo publish` directly. The helper temporarily removes repo-local
step, same-version dev dependencies and dev-only cycles can block the release
even though they are not needed by downstream users.
-:::caution
-
-crates.io trusted publishing tokens cannot create new crates. If this release
-introduces a new crate name, make sure a crates.io token that is allowed to
-create crates is available as `CARGO_REGISTRY_TOKEN`, or publish the new crate
-manually before relying on trusted publishing.
-
-:::
+The normal workflow publishes only through Trusted Publishing. The helper
+fetches and revokes a fresh OIDC-derived crates.io token for every publish
+attempt, including attempts retried after a rate limit. New crate names must
+already exist from the reservation workflow.
## ASF Side