(iceberg-rust) branch main updated: ci: check new crates are reserved on crates.io before merge (#3185)

[sc...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

CTTY pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg-rust.git


The following commit(s) were added to refs/heads/main by this push:
     new cb912c910 ci: check new crates are reserved on crates.io before merge (#3185)
cb912c910 is described below

commit cb912c91066f4fc9bca8b5cdebc40e79aef38d9f
Author: Kevin Liu <ke...@users.noreply.github.com>
AuthorDate: Fri Sep 11 15:30:44 2026 -0700

    ci: check new crates are reserved on crates.io before merge (#3185)
    
    * ci: check new crates are reserved on crates.io before merge
    
    Trusted publishing can't create a crate, and the release workflow publishes
    the whole workspace in one `cargo publish --workspace` run, which cargo
    aborts if any crate already exists at the release version. So the existing
    "publish the new crate by hand after the workflow fails" instructions no
    longer work.
    
    Add a lint step that fails if a publishable crate is missing from
    crates.io, and rewrite the release guide appendix around reserving the
    crate with a placeholder version when it is added.
    
    Co-Authored-By: Claude Fable 5.1 <no...@anthropic.com>
    
    * docs: reserve the crate name with a standalone placeholder crate
    
    Overriding the version inside the workspace fails because iceberg depends
    on the new crate at the workspace version. Verified with cargo publish
    --dry-run.
    
    Co-Authored-By: Claude Fable 5.1 <no...@anthropic.com>
    
    * ci: match the existing publishable-crate filter; document partial publish recovery
    
    Co-Authored-By: Claude Fable 5.1 <no...@anthropic.com>
    
    * docs: clarify that reserving a crate needs a token with the publish-new scope
    
    Owning the existing crates does not imply the token in ~/.cargo/credentials.toml
    has publish-new, and crates.io reports a missing scope as a generic
    authentication failure. Say so, and point at where to create the token.
    
    Co-Authored-By: Claude Fable 5.1 <no...@anthropic.com>
    
    * docs: fix stale first-publish note, empty placeholder, and owner invitations
    
    - The release steps still said the publish workflow is expected to fail on
      new crates and linked to a heading that no longer exists.
    - `cargo new --lib` generates a non-empty lib.rs; use mkdir and touch.
    - `cargo owner --add <user>` sends an invitation that must be accepted.
    
    Co-Authored-By: Claude Fable 5.1 <no...@anthropic.com>
    
    * ci: fail the crates.io check if the crate list is empty
    
    A failed `cargo metadata` inside `for pkg in $(...)` never trips
    `bash -e`, so the loop ran zero times and the step passed without
    checking anything. Capture the list first and fail if it is empty.
    
    Co-Authored-By: Claude Fable 5.1 <no...@anthropic.com>
    
    ---------
    
    Co-authored-by: Claude Fable 5.1 <no...@anthropic.com>
---
 .github/workflows/ci.yml |  16 ++++++++
 website/src/release.md   | 101 +++++++++++++++++++++++++++++++++++------------
 2 files changed, 92 insertions(+), 25 deletions(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 856026bd1..5bb2cc178 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -78,6 +78,22 @@ jobs:
       - name: Check crates package LICENSE and NOTICE
         run: make check-license-notice
 
+      # Trusted publishing cannot create a crate (https://crates.io/docs/trusted-publishing),
+      # so a new publishable crate must be reserved on crates.io before it is merged.
+      # See "Adding a new crate" in website/src/release.md.
+      - name: Check publishable crates exist on crates.io
+        run: |
+          pkgs=$(cargo metadata --no-deps --format-version=1 | jq -r '.packages[] | select(.publish != []) | .name')
+          # A failed `cargo metadata` would otherwise leave the loop empty and the step green.
+          [ -n "$pkgs" ] || { echo "::error::No publishable crates found in the workspace, expected at least one"; exit 1; }
+          for pkg in $pkgs; do
+            # --registry is required, otherwise cargo resolves the local workspace member.
+            if ! cargo info --registry crates-io "$pkg" >/dev/null; then
+              echo "::error::$pkg is not on crates.io. Reserve it before merging; see 'Adding a new crate' in website/src/release.md"
+              exit 1
+            fi
+          done
+
       - name: Cargo Machete
         run: cargo machete
 
diff --git a/website/src/release.md b/website/src/release.md
index 5e207bc41..4b97fc6f7 100644
--- a/website/src/release.md
+++ b/website/src/release.md
@@ -531,9 +531,8 @@ The creation of the final release tag triggers the publish workflow for crates.
 Python packages are manually triggered later.
 Please verify that the triggered workflows for the crates succeeded.
 
-Note, this workflow for crates is expected to fail if new crates are being published for the first time.
-In this instance, a committer must manually publish the crate in order to continue.
-See [publishing a crate for the first time](#publishing-a-crate-for-the-first-time) for what steps to take.
+New crates are reserved on crates.io when they are added, so the workflow should not fail on a new crate.
+If it does, see [if the Publish workflow fails on a missing crate](#if-the-publish-workflow-fails-on-a-missing-crate).
 Once all the crates are published, Python publishing should start.
 
 Python publishing is performed by a GitHub workflow, however the trigger is manual.
@@ -614,45 +613,97 @@ however it is in the community's best interest to publish it soon after.
 
 ## Appendix
 
-### Publishing a crate for the first time
+### Adding a new crate
 
-Publishing the Iceberg crates is automated using GitHub Actions,
-which authenticates with crates.io using trusted publishing.
+Every publishable crate must exist on crates.io before the pull request that adds it is merged.
+CI enforces this with a check that fails for any publishable crate missing from crates.io.
 
-Trusted publishing must first be configured for each crate.
-If the crate has never been published, crates.io rejects attempts to publish.
+The release workflow publishes crates with [trusted publishing](https://crates.io/docs/trusted-publishing), which cannot create a crate.
+crates.io requires the first version of a crate to be published manually with an API token ([announcement](https://blog.rust-lang.org/2025/07/11/crates-io-development-update-2025-07/)).
+Publishing the crate by hand after the release workflow fails does not work either: the workflow runs `cargo publish --workspace`, which refuses to run if any crate already exists at the release version.
 
-When a release workflow fails due to the presence of a new crate, a committer must perform steps enumerated below.
-Once complete, future versions can be published automatically using GitHub Actions.
+A committer therefore reserves the crate when it is added, in three steps: publish a placeholder version, configure owners, and configure trusted publishing.
 
-#### Initial crate publish
+#### Prerequisites
 
-Manually publish the crate using the following command.
-You **must** have the source code checked out matching the pushed Git tag for the release.
+Any crates.io account can publish an unclaimed name, and the first publisher becomes the crate's only owner.
+Reserve the crate promptly once the pull request is open, and have a committer do it, since the reserver must hand ownership to the project.
+
+The committer needs:
+
+- A crates.io account linked to their GitHub account.
+- Membership in the `apache/iceberg-private` GitHub team, with the `read:org` permission granted to crates.io, in order to add that team as an owner.
+- A crates.io API token with the `publish-new` and `change-owners` scopes, used with `cargo login`.
+  Existing tokens usually lack `publish-new`; create one at [crates.io/settings/tokens](https://crates.io/settings/tokens).
+
+Committers who already own the existing Iceberg crates meet the first two.
+List them with `cargo owner --list iceberg`.
+
+#### Step 1: Reserve the crate name
+
+Publish an empty placeholder crate as version `0.0.0`, so it never collides with a real release.
+The placeholder is created outside the repository; overriding the version inside the workspace does not work because other workspace crates depend on the new crate at the workspace version.
 
 ```shell
-cargo publish --package <package-name>
+mkdir -p /tmp/<package-name>/src
+cd /tmp/<package-name>
+touch src/lib.rs
+cat > Cargo.toml <<'EOF'
+[package]
+name = "<package-name>"
+version = "0.0.0"
+edition = "2024"
+license = "Apache-2.0"
+description = "Placeholder that reserves the crate name for Apache Iceberg Rust. See the repository for the real crate."
+repository = "https://github.com/apache/iceberg-rust"
+EOF
+cargo publish --dry-run
+cargo publish
 ```
 
-#### Configure crate permissions
+A `403 Forbidden: authentication failed` error means the token lacks the `publish-new` scope.
 
-After publishing succeeds, the crate must be configured to allow other committers to publish.
+#### Step 2: Configure crate owners
 
-Add the GitHub team that Apache Iceberg committers are a member of.
+Add the GitHub team that owns the existing Iceberg crates.
 
 ```shell
-cargo owner --add github:<github-team-org>:<github-team-name>
+cargo owner --add github:apache:iceberg-private <package-name>
 ```
 
-Additionally, add two PMC members (excluding yourself) as owners of the crate.
-A GitHub team cannot manage permissions for the crate, so it is important that individuals have ownership to continue being able to manage the crate should a PMC member become inactive.
-See the [Cargo documentation for `cargo owner`](https://doc.rust-lang.org/cargo/reference/publishing.html#cargo-owner) for reference.
+Then add two PMC members other than yourself as individual owners.
 
 ```shell
-cargo owner --add <github-handle>
+cargo owner --add <github-handle> <package-name>
 ```
 
-#### Configure trusted publishing
+This sends an invitation; ask each invitee to accept it on crates.io.
+
+Individual owners are required because a team owner can publish and yank versions but cannot add or remove owners.
+Without them, the crate could become unmanageable if a PMC member becomes inactive.
+See the [Cargo documentation for `cargo owner`](https://doc.rust-lang.org/cargo/reference/publishing.html#cargo-owner).
+
+#### Step 3: Configure trusted publishing
+
+As a crate owner, open the crate's settings on the crates.io website and add a trusted publisher, following the [crates.io trusted publishing documentation](https://crates.io/docs/trusted-publishing).
+Use an existing Iceberg crate as the reference:
 
-Review the [crates.io trusted publishing documentation](https://crates.io/docs/trusted-publishing) for the latest instructions on how to configure it.
-You should also review another already-existing crate for reference.
+- Repository: `apache/iceberg-rust`
+- Workflow: `publish.yml`
+- Environment: `publish`
+
+Then re-run CI on the pull request.
+The next release publishes the crate's first real version along with the rest of the workspace.
+
+#### If the Publish workflow fails on a missing crate
+
+Reserve the crate as above, using the `0.0.0` placeholder and never the release version.
+
+Then check the failed run's log for `Uploaded` lines to see which crates were already published, since cargo uploads crates one at a time in dependency order.
+
+- If nothing was uploaded, re-run the failed workflow run from the GitHub Actions UI. Re-running keeps the original tag as the workflow ref.
+- If some crates were uploaded, publish the rest from a clean checkout of the release tag, excluding each crate that was already published:
+
+```shell
+cargo publish --workspace --all-features --exclude <published-crate> --exclude <published-crate>
+```