You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2026/08/29 08:16:25 UTC
(spark-connect-rust) branch branch-4.2 updated: [SPARK-59101][SPARK-59100] Release from a chosen apache upstream branch on dispatch, and publish release candidates as GitHub pre-releases
This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 979ccd7 [SPARK-59101][SPARK-59100] Release from a chosen apache upstream branch on dispatch, and publish release candidates as GitHub pre-releases
979ccd7 is described below
commit 979ccd767a38533c93410bf5b367e604aa3afdd8
Author: Hyukjin Kwon <hy...@databricks.com>
AuthorDate: Sat Aug 29 17:16:05 2026 +0900
[SPARK-59101][SPARK-59100] Release from a chosen apache upstream branch on dispatch, and publish release candidates as GitHub pre-releases
### What changes were proposed in this pull request?
Two related Release-workflow changes, squash-merged together:
**[SPARK-59101] Release from a chosen apache upstream branch on manual dispatch**
Add a typed `branch` input to the Release workflow's manual dispatch, and route every source checkout through a new `resolve-source` job:
- **Manual dispatch** now fetches the typed branch (e.g. `master` or `branch-4.2`) from **apache/spark-connect-rust**, instead of using whatever the committer's fork checkout happens to contain. The workflow still *runs* from the fork, so it keeps using the fork's `CARGO_REGISTRY_TOKEN` / `PYPI_API_TOKEN`.
- **Tag push** is unchanged: it releases the tagged commit on the fork the tag was pushed to.
- The repository/ref decision is made in exactly one place (`resolve-source`) and consumed by `validate-version`, `publish-crates`, `build-wheels`, and `build-sdist`.
**[SPARK-59100] Publish release candidates as GitHub pre-releases**
- Push a semver pre-release tag (e.g. `v4.2.0-rc1`) with `Cargo.toml = 4.2.0-rc1` / `pyproject.toml = 4.2.0rc1`. `validate-version` normalizes the Cargo hyphen spelling against PEP 440 and emits an `is_prerelease` output.
- A new `publish-github-release` job attaches all wheels (Linux x86_64/aarch64, macOS universal2, Windows x86_64) + the sdist to a GitHub Release, marked `--prerelease` for RC tags, via the `gh` CLI (ASF Actions policy allowlists `actions/*` only) with job-level `contents: write`. Idempotent (create-or-`--clobber`-upload).
- For a **pre-release** tag the `publish-crates` (crates.io) and `publish-wheels` (PyPI) jobs are **skipped**, so an RC never lands on an immutable index and stays deletable. A **final** tag (no `-rcN`) is unchanged (crates.io + PyPI) and additionally gets a regular GitHub Release.
### Why are the changes needed?
- A fork-dispatched release could silently ship the fork's stale copy rather than the canonical apache code; the source should come from the upstream repo (SPARK-59101).
- PyPI and crates.io are immutable, so they are a poor fit for a throwaway RC; GitHub Releases can be marked pre-release and freely deleted, so users can `pip install <asset-url>` to try an RC and we can withdraw it by deleting the release + tag (SPARK-59100).
### Does this PR introduce _any_ user-facing change?
No change to released artifacts. It changes how a release is cut and adds an opt-in pre-release publishing path (via an `-rcN` tag).
### How was this patch tested?
`release.yml` parses as valid YAML with the expected jobs; the `resolve-source` outputs feed the four source-checkout jobs, and the `if:`/`needs` wiring gates crates.io/PyPI off for pre-release tags while running the GitHub-Release job on tag push. The existing dry-run path is unaffected.
This pull request and its description were written by Isaac.
---
**Piggybacked doc change:** also documents the release umbrella JIRA convention in `AGENTS.md` — each `connect-rust-x.y.z` release is tracked by an umbrella Epic (SPARK-59035 for 4.2.0, SPARK-59102 for 4.3.0) whose members attach via an `is part of` (Incorporates) link rather than the Epic Link field.
Closes #84 from HyukjinKwon/release-from-upstream-branch.
Authored-by: Hyukjin Kwon <hy...@databricks.com>
Signed-off-by: Hyukjin Kwon <hy...@databricks.com>
(cherry picked from commit aff372ddeb932b16788dd68d6cdd413698bed3cd)
Signed-off-by: Hyukjin Kwon <hy...@databricks.com>
---
.github/workflows/release.yml | 543 ++++++++++++++++++++++++++++++++----------
AGENTS.md | 8 +
2 files changed, 431 insertions(+), 120 deletions(-)
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index fede7a5..527461f 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -17,36 +17,74 @@
# under the License.
#
-# Release workflow for the Rust crates (crates.io) and the Python wheel (PyPI).
+# Release workflow for pyspark-client-rust (the Rust crates on crates.io + the
+# Python wheel on PyPI). Modeled on Apache Spark's release workflow: it is
+# DISPATCHED FROM A COMMITTER'S FORK (which holds the publish tokens), takes the
+# same inputs, and follows the same release-candidate-then-finalize flow.
#
-# Following the Apache Spark model, releases are cut by a committer from THEIR
-# OWN FORK, not from the canonical apache/ repository:
+# Inputs (identical to apache/spark's release workflow):
+# branch apache/spark-connect-rust branch the release source is fetched
+# from (always from apache, never the fork's own copy). Default
+# master. Unused by finalize, which uses the RC tag as its source.
+# release-version Final version, e.g. 4.2.0. Leave empty (with rc-count) for a dry
+# run. Must equal the Cargo.toml/pyproject.toml version on `branch`
+# (asserted) -- the RC number never appears in the artifact version.
+# rc-count RC number, e.g. 1. Leave empty (with release-version) for a dry run.
+# finalize true => convert the named RC into the OFFICIAL release
+# (IRREVERSIBLE). Default false.
#
-# 1. Enable GitHub Actions on your fork.
-# 2. Add the release secrets to the fork:
-# - CARGO_REGISTRY_TOKEN (crates.io API token)
-# - PYPI_API_TOKEN (PyPI API token)
-# 3. Bump the version in Cargo.toml and pyproject.toml (they must match).
-# 4. Dispatch this workflow with dry_run=true to rehearse (default), then push
-# a matching `v<version>` tag (or dispatch with dry_run=false) to publish.
+# The three modes are derived from the inputs (see the `resolve` job):
#
-# A dry run fully rehearses BOTH sides without publishing: it packages and
-# verify-builds every crate in dependency order (`cargo publish --dry-run`) and
-# builds every wheel and the sdist. A tag push (or an explicit dry_run=false
-# dispatch) additionally publishes to crates.io and PyPI. Both publish steps are
-# idempotent, so a re-run after a partial failure skips what already landed.
+# * Dry run (release-version AND rc-count empty): build every wheel + sdist and
+# `cargo publish --dry-run`. Creates no tag, publishes nothing. This is the
+# testable path and also runs on a schedule to keep the release path healthy.
+#
+# * Cut an RC (release-version + rc-count, finalize=false): build the wheels +
+# sdist -- versioned as the FINAL release-version, since the RC number lives
+# only in the tag/release name, never in the artifact version -- then
+# auto-create and push the tag `v<version>-rc<rc>` to this fork and attach the
+# artifacts to a GitHub *pre-release* of that tag. crates.io/PyPI are NOT
+# touched, so an RC stays fully deletable.
+#
+# * Finalize (finalize=true, naming an existing RC): check out the RC tag,
+# `twine upload` the EXACT wheels+sdist from that RC's GitHub pre-release to
+# PyPI, `cargo publish` the identical source to crates.io, push a permanent
+# `v<version>` git tag marking the released commit, then delete every
+# `v<version>-rc*` GitHub pre-release (and its tag) from this fork. GitHub
+# Releases are only RC staging; the official release lives on PyPI + crates.io,
+# marked in git by the immutable `v<version>` tag.
+#
+# Required fork secrets for a real run: CARGO_REGISTRY_TOKEN (crates.io) and
+# PYPI_API_TOKEN (PyPI). Without them, only dry runs work. The default GITHUB_TOKEN
+# creates/deletes the RC tags and GitHub Releases on the fork.
+#
+# To drop an RC without finalizing: delete its GitHub pre-release and the
+# `v<version>-rc<n>` tag from the fork (nothing was published to PyPI/crates.io).
name: Release
on:
+ schedule:
+ # Periodic dry run (runs only in the repo hosting the workflow; GitHub does not
+ # run scheduled workflows in forks), to catch release-path breakage early.
+ - cron: '0 7 */2 * *'
workflow_dispatch:
inputs:
- dry_run:
- description: "Build and validate artifacts but do not publish"
+ branch:
+ description: 'Branch to release. Leave release-version/rc-count empty to launch a dryrun. Dispatch this workflow only in the forked repository.'
+ required: true
+ default: master
+ release-version:
+ description: 'Release version (e.g. 4.2.0). Leave it empty to launch a dryrun.'
+ required: false
+ rc-count:
+ description: 'RC number (e.g. 1). Leave it empty to launch a dryrun.'
+ required: false
+ finalize:
+ description: 'Whether to convert RC to the official release (IRREVERSIBLE)'
+ required: true
type: boolean
- default: true
- push:
- tags: ["v*"]
+ default: false
permissions:
contents: read
@@ -57,118 +95,165 @@ concurrency:
group: release
cancel-in-progress: false
+env:
+ # The release source is always fetched from this canonical apache repository.
+ UPSTREAM_REPO: apache/spark-connect-rust
+
jobs:
- validate-version:
- name: Validate version
+ # A real (RC/finalize) run must come from a committer's fork, which holds the publish
+ # tokens; the canonical apache repo only does dry runs. `resolve`'s `if:` already skips
+ # real runs there, but a bare skip is a green, all-skipped no-op that explains nothing.
+ # This job runs ONLY in that mistaken case and fails loudly with instructions.
+ reject-real-release-from-apache:
+ name: Reject real releases dispatched from the apache repo
runs-on: ubuntu-latest
+ if: >-
+ github.repository == 'apache/spark-connect-rust' &&
+ (inputs.release-version != '' || inputs.rc-count != '')
steps:
- - uses: actions/checkout@v4
- - name: Cargo and pyproject versions agree (and match the tag, if any)
+ - run: |
+ echo "::error::Real releases (RC or finalize) must be dispatched from a committer's FORK of spark-connect-rust (which holds CARGO_REGISTRY_TOKEN / PYPI_API_TOKEN) — not from apache/spark-connect-rust. This canonical repository only runs dry runs: leave release-version and rc-count empty. See the workflow header for the fork-based release process."
+ exit 1
+
+ resolve:
+ name: Resolve release mode
+ runs-on: ubuntu-latest
+ # In the canonical apache repo, only dry runs are allowed (release-version and
+ # rc-count empty). A fork may perform real runs. This mirrors apache/spark.
+ # NOTE: the `env` context is not available in a job-level `if:`, so the canonical
+ # repo is spelled out literally here; it must match `env.UPSTREAM_REPO` below.
+ if: >-
+ github.repository != 'apache/spark-connect-rust' ||
+ (inputs.release-version == '' && inputs.rc-count == '')
+ outputs:
+ mode: ${{ steps.r.outputs.mode }}
+ version: ${{ steps.r.outputs.version }}
+ rc_tag: ${{ steps.r.outputs.rc_tag }}
+ branch: ${{ steps.r.outputs.branch }}
+ # The branch resolved to a single immutable commit SHA, checked out everywhere
+ # downstream so the wheels, sdist, and RC tag can never come from different commits.
+ sha: ${{ steps.r.outputs.sha }}
+ steps:
+ - name: Determine mode and validate inputs
+ id: r
+ env:
+ IN_BRANCH: ${{ inputs.branch }}
+ IN_VERSION: ${{ inputs.release-version }}
+ IN_RC: ${{ inputs.rc-count }}
+ IN_FINALIZE: ${{ inputs.finalize }}
run: |
set -euo pipefail
- CARGO_VER=$(grep -m1 '^version' Cargo.toml | awk -F'"' '{print $2}')
- PYPROJECT_VER=$(grep -m1 '^version' pyproject.toml | awk -F'"' '{print $2}')
- echo "Cargo.toml=${CARGO_VER} pyproject.toml=${PYPROJECT_VER}"
- if [[ "${CARGO_VER}" != "${PYPROJECT_VER}" ]]; then
- echo "::error::Cargo.toml (${CARGO_VER}) and pyproject.toml (${PYPROJECT_VER}) versions differ"
+ BRANCH="${IN_BRANCH:-master}"
+ RV="${IN_VERSION:-}"
+ RC="${IN_RC:-}"
+ FIN="${IN_FINALIZE:-false}"
+
+ # release-version and rc-count must be provided together (or neither).
+ if { [ -n "$RV" ] && [ -z "$RC" ]; } || { [ -z "$RV" ] && [ -n "$RC" ]; }; then
+ echo "::error::Provide BOTH 'release-version' and 'rc-count', or leave BOTH empty for a dry run."
exit 1
fi
- # Internal path-dependency version pins (e.g. spark-connect-core -> proto)
- # must track the workspace version, or `cargo publish` embeds a stale
- # requirement into the published crate.
- BAD=$(grep -rEn 'path = "\.\./spark-connect[^"]*"' crates/*/Cargo.toml \
- | grep -E 'version = "' \
- | grep -v "version = \"${CARGO_VER}\"" || true)
- if [[ -n "${BAD}" ]]; then
- echo "::error::internal path-dep version pin(s) do not match ${CARGO_VER}:"
- echo "${BAD}"
- exit 1
+
+ if [ "$FIN" = "true" ]; then
+ if [ -z "$RV" ] || [ -z "$RC" ]; then
+ echo "::error::finalize=true requires 'release-version' and 'rc-count' naming the RC to promote."
+ exit 1
+ fi
+ MODE=finalize
+ elif [ -n "$RV" ]; then
+ MODE=rc
+ else
+ MODE=dryrun
fi
- if [[ "${GITHUB_REF}" == refs/tags/* ]]; then
- TAG="${GITHUB_REF##*/}"
- if [[ "${TAG}" != "v${CARGO_VER}" ]]; then
- echo "::error::Pushed tag ${TAG} does not match version v${CARGO_VER}"
+
+ RC_TAG=""
+ [ -n "$RV" ] && RC_TAG="v${RV}-rc${RC}"
+
+ # Pin the branch to one immutable commit SHA so every downstream checkout
+ # (wheels, sdist, RC tag) uses the SAME source even if the branch moves
+ # mid-run. finalize does not need it (it checks out the RC tag instead).
+ SHA=""
+ if [ "$MODE" = "rc" ] || [ "$MODE" = "dryrun" ]; then
+ SHA=$(git ls-remote "https://github.com/${UPSTREAM_REPO}.git" "refs/heads/${BRANCH}" | awk 'NR==1{print $1}')
+ if [ -z "$SHA" ]; then
+ echo "::error::Branch '${BRANCH}' not found on ${UPSTREAM_REPO}."
exit 1
fi
fi
- publish-crates:
- name: Publish Rust crates to crates.io
- needs: validate-version
- runs-on: ubuntu-latest
- steps:
- - uses: actions/checkout@v4
- - name: Install Rust toolchain
- run: |
- rustup toolchain install stable --profile minimal
- rustup default stable
- - name: Install protoc
- run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
- - name: cargo publish (dependency-ordered; dry-run unless releasing)
+ {
+ echo "mode=$MODE"
+ echo "version=$RV"
+ echo "rc_tag=$RC_TAG"
+ echo "branch=$BRANCH"
+ echo "sha=$SHA"
+ } >> "$GITHUB_OUTPUT"
+ echo "Resolved: mode=$MODE version='${RV}' rc='${RC}' tag='${RC_TAG}' branch='${BRANCH}' sha='${SHA}'"
+
+ - name: Check out release source (for version validation)
+ if: steps.r.outputs.mode == 'dryrun' || steps.r.outputs.mode == 'rc'
+ uses: actions/checkout@v4
+ with:
+ repository: ${{ env.UPSTREAM_REPO }}
+ ref: ${{ steps.r.outputs.sha }}
+
+ - name: Validate versions
+ # Runs for dry run AND rc. The Cargo<->pyproject agreement and the path-dep pin
+ # checks always run (so the scheduled dry run actually exercises them); the
+ # release-version equality check is additional and rc-only.
+ if: steps.r.outputs.mode == 'dryrun' || steps.r.outputs.mode == 'rc'
env:
- CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }}
- # true only for an explicit dry-run dispatch; tag pushes and an explicit
- # dry_run=false dispatch both evaluate to "false" and publish for real.
- DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run }}
+ MODE: ${{ steps.r.outputs.mode }}
+ V: ${{ steps.r.outputs.version }}
run: |
set -euo pipefail
- # Leaf-first; cargo (>=1.90) orders and verify-builds these together,
- # resolving siblings from the local workspace.
- CRATES=(apache-spark-connect-proto apache-spark-connect-core apache-spark-connect)
- VER=$(grep -m1 '^version' Cargo.toml | awk -F'"' '{print $2}')
+ CARGO_VER=$(grep -m1 '^version' Cargo.toml | awk -F'"' '{print $2}')
+ PYPROJECT_VER=$(grep -m1 '^version' pyproject.toml | awk -F'"' '{print $2}')
+ echo "mode=${MODE} Cargo=${CARGO_VER} pyproject=${PYPROJECT_VER} release-version=${V:-<none>}"
- if [[ "${DRY_RUN}" == "true" ]]; then
- # Rehearse the real publish. Because siblings resolve from the local
- # workspace, this works even before anything exists on crates.io.
- pkgs=(); for c in "${CRATES[@]}"; do pkgs+=(-p "$c"); done
- echo "::group::cargo publish --dry-run ${pkgs[*]}"
- cargo publish --dry-run "${pkgs[@]}"
- echo "::endgroup::"
- echo "Dry run only — nothing published."
- exit 0
+ # 1. Cargo.toml and pyproject.toml must declare the same version.
+ if [ "$CARGO_VER" != "$PYPROJECT_VER" ]; then
+ echo "::error::Cargo.toml (${CARGO_VER}) and pyproject.toml (${PYPROJECT_VER}) versions disagree."
+ exit 1
fi
- # Real publish. Skip any crate whose current version is already on
- # crates.io so a re-run after a partial failure is idempotent.
- pkgs=()
- for c in "${CRATES[@]}"; do
- if curl -sf -H "User-Agent: spark-connect-rust-release (dev@spark.apache.org)" \
- "https://crates.io/api/v1/crates/${c}/${VER}" >/dev/null; then
- echo "${c} ${VER} is already on crates.io — skipping"
- else
- pkgs+=(-p "$c")
- fi
- done
- if [[ ${#pkgs[@]} -eq 0 ]]; then
- echo "All crates already published at ${VER}; nothing to do."
- exit 0
+ # 2. Internal path-dep pins (e.g. spark-connect-core -> proto) must track the
+ # workspace version, or `cargo publish` embeds a stale hard dep into the
+ # published crate and breaks downstream consumers.
+ BAD=$(grep -rEn 'path = "\.\./spark-connect[^"]*"' crates/*/Cargo.toml \
+ | grep -E 'version = "' \
+ | grep -v "version = \"${CARGO_VER}\"" || true)
+ if [ -n "$BAD" ]; then
+ echo "::error::internal path-dep version pin(s) do not match the workspace version ${CARGO_VER}:"
+ echo "$BAD"
+ exit 1
+ fi
+
+ # 3. For an RC, the declared version must equal the requested release-version.
+ if [ "$MODE" = "rc" ] && [ "$CARGO_VER" != "$V" ]; then
+ echo "::error::Branch '${{ steps.r.outputs.branch }}' declares ${CARGO_VER}, but release-version=${V}. Bump the branch to ${V} (Cargo.toml, pyproject.toml, and the internal path-dep pins) before cutting the RC."
+ exit 1
fi
- echo "::group::cargo publish ${pkgs[*]}"
- # cargo publishes in dependency order and waits for the index between
- # crates, so no manual sleep is needed.
- cargo publish "${pkgs[@]}"
- echo "::endgroup::"
+ # ---- Build (dry run + RC only; finalize reuses the RC's already-built artifacts) ----
build-wheels:
name: Build wheel (${{ matrix.target }})
- needs: validate-version
+ needs: resolve
+ if: needs.resolve.outputs.mode == 'dryrun' || needs.resolve.outputs.mode == 'rc'
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
include:
- - os: ubuntu-latest
- target: x86_64-unknown-linux-gnu
- zig: true
- - os: ubuntu-latest
- target: aarch64-unknown-linux-gnu
- zig: true
- - os: macos-latest
- target: universal2-apple-darwin
- - os: windows-latest
- target: x86_64-pc-windows-msvc
+ - { os: ubuntu-latest, target: x86_64-unknown-linux-gnu, zig: true }
+ - { os: ubuntu-latest, target: aarch64-unknown-linux-gnu, zig: true }
+ - { os: macos-latest, target: universal2-apple-darwin }
+ - { os: windows-latest, target: x86_64-pc-windows-msvc }
steps:
- uses: actions/checkout@v4
+ with:
+ repository: ${{ env.UPSTREAM_REPO }}
+ ref: ${{ needs.resolve.outputs.sha }}
- uses: actions/setup-python@v5
with:
python-version: "3.11"
@@ -195,10 +280,8 @@ jobs:
Windows) choco install protoc -y ;;
esac
- name: Build abi3 wheel with maturin
- # abi3-py39 produces a single wheel per (platform, arch) valid for
- # CPython 3.9+. `--zig` cross-links the Linux wheels against an old glibc
- # (manylinux2014), so they install on essentially any modern Linux rather
- # than only on the runner's glibc.
+ # abi3-py39 produces one wheel per (platform, arch) valid for CPython 3.9+.
+ # `--zig` cross-links the Linux wheels against an old glibc (manylinux2014).
shell: bash
run: |
python -m pip install --upgrade maturin
@@ -215,16 +298,18 @@ jobs:
build-sdist:
name: Build source distribution
- needs: validate-version
+ needs: resolve
+ if: needs.resolve.outputs.mode == 'dryrun' || needs.resolve.outputs.mode == 'rc'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
+ with:
+ repository: ${{ env.UPSTREAM_REPO }}
+ ref: ${{ needs.resolve.outputs.sha }}
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Build sdist with maturin
- # sdist packages source only (no compile), so protoc is not needed here.
- # It is the install fallback for platforms without a prebuilt wheel.
run: |
python -m pip install --upgrade maturin
maturin sdist --out dist
@@ -233,28 +318,246 @@ jobs:
name: wheels-sdist
path: dist/*.tar.gz
- publish-wheels:
- name: Publish wheel + sdist to PyPI
- needs: [build-wheels, build-sdist]
- if: github.event_name == 'push' || (github.event_name == 'workflow_dispatch' && inputs.dry_run == false)
+ # ---- Dry run: rehearse the crates.io packaging without publishing ----
+ dryrun-crates:
+ name: Dry-run crates.io packaging
+ needs: resolve
+ if: needs.resolve.outputs.mode == 'dryrun'
runs-on: ubuntu-latest
steps:
- - uses: actions/setup-python@v5
+ - uses: actions/checkout@v4
with:
- python-version: "3.11"
+ repository: ${{ env.UPSTREAM_REPO }}
+ ref: ${{ needs.resolve.outputs.sha }}
+ - name: Install Rust toolchain
+ run: |
+ rustup toolchain install stable --profile minimal
+ rustup default stable
+ - name: Install protoc
+ run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
+ - name: cargo publish --dry-run (dependency-ordered)
+ run: |
+ set -euo pipefail
+ cargo publish --dry-run \
+ -p apache-spark-connect-proto \
+ -p apache-spark-connect-core \
+ -p apache-spark-connect
+
+ # ---- Cut an RC: tag + GitHub pre-release with the artifacts (no crates.io/PyPI) ----
+ publish-rc:
+ name: Cut RC GitHub pre-release
+ needs: [resolve, build-wheels, build-sdist]
+ if: needs.resolve.outputs.mode == 'rc'
+ runs-on: ubuntu-latest
+ permissions:
+ contents: write # push the RC tag + create the GitHub Release on the fork
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ repository: ${{ env.UPSTREAM_REPO }}
+ ref: ${{ needs.resolve.outputs.sha }}
+ fetch-depth: 0 # full history so the tag push carries all reachable objects
- uses: actions/download-artifact@v4
with:
- # Matches every wheels-<target> artifact plus wheels-sdist.
pattern: wheels-*
path: dist
merge-multiple: true
- - name: Publish to PyPI
- # Fork-based release: authenticate with the committer's PyPI token via twine
- # (avoids third-party publish actions; ASF Actions policy allowlists actions/*).
+ - name: Create and push the RC tag to this fork
+ env:
+ GH_TOKEN: ${{ github.token }}
+ TAG: ${{ needs.resolve.outputs.rc_tag }}
+ VERSION: ${{ needs.resolve.outputs.version }}
+ run: |
+ set -euo pipefail
+ git config user.name "${{ github.actor }}"
+ git config user.email "${{ github.actor }}@users.noreply.github.com"
+ if git rev-parse -q --verify "refs/tags/${TAG}" >/dev/null; then
+ echo "Local tag ${TAG} already exists; reusing."
+ else
+ git tag -a "${TAG}" -m "Release candidate ${TAG} of pyspark-client-rust ${VERSION}"
+ fi
+ # Push the RC tag (and the objects it reaches) to THIS fork, so the GitHub
+ # pre-release below can hang off it. Force so re-cutting the same RC is idempotent.
+ git push --force "https://x-access-token:${GH_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" "refs/tags/${TAG}"
+ - name: Create the GitHub pre-release with the artifacts
+ env:
+ GH_TOKEN: ${{ github.token }}
+ TAG: ${{ needs.resolve.outputs.rc_tag }}
+ VERSION: ${{ needs.resolve.outputs.version }}
+ run: |
+ set -euo pipefail
+ ls -l dist
+ NOTES="Release candidate ${TAG} of pyspark-client-rust ${VERSION}."$'\n\n'"> Not published to PyPI or crates.io. Install directly from a wheel/sdist asset below, e.g. \`pip install <asset-url>\`. This pre-release is deleted when the RC is finalized (or dropped)."
+ # Idempotent: a re-run updates the existing release and clobbers its assets.
+ if gh release view "${TAG}" --repo "${GITHUB_REPOSITORY}" >/dev/null 2>&1; then
+ gh release upload "${TAG}" dist/* --repo "${GITHUB_REPOSITORY}" --clobber
+ else
+ gh release create "${TAG}" dist/* --repo "${GITHUB_REPOSITORY}" \
+ --prerelease --title "${TAG}" --notes "${NOTES}"
+ fi
+
+ # ---- Finalize: promote the named RC into the official release (IRREVERSIBLE) ----
+ finalize:
+ name: Finalize release to PyPI + crates.io (IRREVERSIBLE)
+ needs: resolve
+ if: needs.resolve.outputs.mode == 'finalize'
+ runs-on: ubuntu-latest
+ permissions:
+ contents: write # delete the RC pre-releases + tags on the fork
+ steps:
+ - name: Abort window
+ run: |
+ echo "=============================================================================="
+ echo " CONVERTING RC ${{ needs.resolve.outputs.rc_tag }} INTO THE OFFICIAL RELEASE"
+ echo " ${{ needs.resolve.outputs.version }} -> PyPI + crates.io. THIS IS IRREVERSIBLE."
+ echo " Cancel this workflow now if you did not intend to finalize."
+ echo " Continuing in 60 seconds..."
+ echo "=============================================================================="
+ sleep 60
+ - name: Verify the RC pre-release exists
+ env:
+ GH_TOKEN: ${{ github.token }}
+ TAG: ${{ needs.resolve.outputs.rc_tag }}
+ run: |
+ set -euo pipefail
+ if ! gh release view "${TAG}" --repo "${GITHUB_REPOSITORY}" >/dev/null 2>&1; then
+ echo "::error::RC GitHub pre-release ${TAG} not found on ${GITHUB_REPOSITORY}. Cut the RC first."
+ exit 1
+ fi
+ - name: Check out the RC tag (the exact released source)
+ uses: actions/checkout@v4
+ with:
+ # The RC tag lives on the fork where the RC was cut (this repository).
+ repository: ${{ github.repository }}
+ ref: ${{ needs.resolve.outputs.rc_tag }}
+ fetch-depth: 0
+ - name: Validate the RC source versions
+ run: |
+ set -euo pipefail
+ V="${{ needs.resolve.outputs.version }}"
+ CARGO_VER=$(grep -m1 '^version' Cargo.toml | awk -F'"' '{print $2}')
+ PYPROJECT_VER=$(grep -m1 '^version' pyproject.toml | awk -F'"' '{print $2}')
+ echo "RC tag source: Cargo=${CARGO_VER} pyproject=${PYPROJECT_VER}; release-version=${V}"
+ # Cargo and pyproject must both equal the release-version (agreement + expected value).
+ if [ "$CARGO_VER" != "$V" ] || [ "$PYPROJECT_VER" != "$V" ]; then
+ echo "::error::RC tag ${{ needs.resolve.outputs.rc_tag }} declares Cargo=${CARGO_VER} / pyproject=${PYPROJECT_VER}, not ${V}."
+ exit 1
+ fi
+ # Path-dep pins must match, or `cargo publish` ships a stale hard dep (same check as resolve).
+ BAD=$(grep -rEn 'path = "\.\./spark-connect[^"]*"' crates/*/Cargo.toml \
+ | grep -E 'version = "' \
+ | grep -v "version = \"${CARGO_VER}\"" || true)
+ if [ -n "$BAD" ]; then
+ echo "::error::internal path-dep version pin(s) in the RC source do not match ${CARGO_VER}:"
+ echo "$BAD"
+ exit 1
+ fi
+ - uses: actions/setup-python@v5
+ with:
+ python-version: "3.11"
+ - name: Download the RC artifacts (the exact tested wheels + sdist)
+ env:
+ GH_TOKEN: ${{ github.token }}
+ TAG: ${{ needs.resolve.outputs.rc_tag }}
+ run: |
+ set -euo pipefail
+ mkdir -p dist
+ gh release download "${TAG}" --repo "${GITHUB_REPOSITORY}" --dir dist --pattern '*'
+ ls -l dist
+ - name: Publish wheels + sdist to PyPI
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
run: |
+ set -euo pipefail
python -m pip install --upgrade twine
# --skip-existing makes a re-run after a partial upload safe.
twine upload --skip-existing dist/*
+ - name: Install Rust toolchain
+ run: |
+ rustup toolchain install stable --profile minimal
+ rustup default stable
+ - name: Install protoc
+ run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
+ - name: Publish crates to crates.io (from the identical RC source)
+ env:
+ CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }}
+ run: |
+ set -euo pipefail
+ CRATES=(apache-spark-connect-proto apache-spark-connect-core apache-spark-connect)
+ VER="${{ needs.resolve.outputs.version }}"
+ # Skip any crate already on crates.io at this version so a re-run is idempotent.
+ pkgs=()
+ for c in "${CRATES[@]}"; do
+ if curl -sf -H "User-Agent: spark-connect-rust-release (dev@spark.apache.org)" \
+ "https://crates.io/api/v1/crates/${c}/${VER}" >/dev/null; then
+ echo "${c} ${VER} is already on crates.io — skipping"
+ else
+ pkgs+=(-p "$c")
+ fi
+ done
+ if [ ${#pkgs[@]} -eq 0 ]; then
+ echo "All crates already published at ${VER}; nothing to do."
+ else
+ # cargo publishes in dependency order and waits for the index between crates.
+ cargo publish "${pkgs[@]}"
+ fi
+ - name: Tag the released commit with the permanent v<version> tag
+ env:
+ GH_TOKEN: ${{ github.token }}
+ TAG: v${{ needs.resolve.outputs.version }}
+ VERSION: ${{ needs.resolve.outputs.version }}
+ run: |
+ set -euo pipefail
+ git config user.name "${{ github.actor }}"
+ git config user.email "${{ github.actor }}@users.noreply.github.com"
+ URL="https://x-access-token:${GH_TOKEN}@github.com/${GITHUB_REPOSITORY}.git"
+ # HEAD is the RC tag's commit (the exact released source), checked out above.
+ LOCAL_SHA=$(git rev-parse HEAD)
+ if ! git rev-parse -q --verify "refs/tags/${TAG}" >/dev/null; then
+ git tag -a "${TAG}" -m "pyspark-client-rust ${VERSION}"
+ fi
+ # A release tag is immutable. Look the remote tag up EXPLICITLY: a network/auth
+ # failure then aborts under set -e (instead of being mistaken for "already
+ # exists"). Only an exact same-commit tag is a no-op; a tag already pointing at
+ # a DIFFERENT commit is a hard error (we must never silently move a release tag,
+ # nor report success while it marks the wrong source).
+ REMOTE_LS=$(git ls-remote --tags "${URL}" "refs/tags/${TAG}")
+ if [ -z "${REMOTE_LS}" ]; then
+ git push "${URL}" "refs/tags/${TAG}" # a push failure is fatal under set -e
+ echo "Pushed release tag ${TAG} -> ${LOCAL_SHA}."
+ else
+ # Annotated tags list a peeled '^{}' line with the commit; prefer it, else the ref line.
+ REMOTE_SHA=$(printf '%s\n' "${REMOTE_LS}" | awk '/\^\{\}$/{c=$1} END{if(c!="")print c}')
+ [ -z "${REMOTE_SHA}" ] && REMOTE_SHA=$(printf '%s\n' "${REMOTE_LS}" | awk 'NR==1{print $1}')
+ if [ "${REMOTE_SHA}" = "${LOCAL_SHA}" ]; then
+ echo "Release tag ${TAG} already points at ${LOCAL_SHA}; nothing to do."
+ else
+ echo "::error::Release tag ${TAG} already exists on ${GITHUB_REPOSITORY} at ${REMOTE_SHA}, but the release commit is ${LOCAL_SHA}. Refusing to move an immutable release tag."
+ exit 1
+ fi
+ fi
+
+ - name: Remove all RC GitHub pre-releases + tags for this version
+ env:
+ GH_TOKEN: ${{ github.token }}
+ VERSION: ${{ needs.resolve.outputs.version }}
+ run: |
+ set -euo pipefail
+ # Every v<version>-rcN pre-release is RC staging; the official release now lives
+ # on PyPI + crates.io, so delete them all (and their tags) from this fork.
+ # Escape dots: grep -E treats '.' as any char, so an unescaped 4.2.0 would also
+ # match tags like v4x2x0-rcN.
+ ESC_V="${VERSION//./\\.}"
+ # Fetch the tag list in its own command so a `gh` failure (rate limit/network)
+ # aborts loudly under `set -e`, instead of being swallowed by the `|| true` that
+ # is only meant to absorb grep's "no match" exit code.
+ ALL_TAGS=$(gh release list --repo "${GITHUB_REPOSITORY}" --limit 200 --json tagName --jq '.[].tagName')
+ mapfile -t RCS < <(printf '%s\n' "${ALL_TAGS}" | grep -E "^v${ESC_V}-rc[0-9]+$" || true)
+ if [ "${#RCS[@]}" -eq 0 ]; then
+ echo "No v${VERSION}-rc* GitHub releases to remove."
+ fi
+ for t in "${RCS[@]}"; do
+ echo "Deleting RC release + tag ${t}"
+ gh release delete "${t}" --repo "${GITHUB_REPOSITORY}" --cleanup-tag --yes
+ done
diff --git a/AGENTS.md b/AGENTS.md
index 5414592..0515a38 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -76,6 +76,14 @@ the Rust and Python sides, no stubs, green CI.
This keeps connect-rust work in its own release notes instead of leaking into
core Spark's. `dev/create_spark_jira.py` and `dev/merge_connect_rust_pr.py`
already follow this; keep it consistent when editing tickets by hand.
+- **Release umbrella JIRA: one Epic per `connect-rust-x.y.z` release.** Each
+ release is tracked by an umbrella Epic titled `spark-connect-rust
+ connect-rust-x.y.z release` (e.g. `SPARK-59035` for `connect-rust-4.2.0`,
+ `SPARK-59102` for `connect-rust-4.3.0`), with *Affects Version/s* = the prior
+ release and *Fix Version/s* = this release. Attach every JIRA that ships in the
+ release to it with an **`is part of` (Incorporates) issue link** — the umbrella
+ `incorporates` the child, the child `is part of` the umbrella — **not** the
+ Epic Link field. Use only that one mechanism so the two never drift apart.
## 1. Environment setup
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org