You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Hyukjin Kwon <gu...@apache.org> on 2026/09/01 22:07:28 UTC
[VOTE] Release Apache Spark Connect Rust Client (+ pyspark-client in Rust) 4.2.0 (RC2)
Please vote on releasing the following candidate as Apache Spark Connect
Rust
Client 4.2.0 RC2.
This is the *first* release of the ground-up Rust rewrite of the Spark
Connect
Python client (published to PyPI as pyspark-client-rust): a drop-in
replacement
for pyspark-client <https://pypi.org/project/pyspark-client/>, backed by a
native Rust core instead of py4j/grpcio.
The vote is open until Thursday, September 7, 2026, 5:00 PM PDT and passes
if a majority of +1 PMC votes are cast, with a minimum of
3 +1 votes.
[ ] +1 Release this package as Apache Spark Connect Rust Client 4.2.0
[ ] -1 Do not release this package because …
TAG:
https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2
(commit: ce1fa08)
RELEASE FILES (wheels + sdist, built and attached by GitHub Actions):
https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2
DOCUMENTATION:
https://apache.github.io/spark-connect-rust/
LIST OF ISSUES:
https://issues.apache.org/jira/projects/SPARK/versions/12357484
HOW TO TRY IT — PYTHON (pip)
The release attaches abi3 wheels (Linux x86_64/aarch64, macOS universal2,
Windows x86_64; CPython 3.9+) plus a source distribution. Grab the wheel for
your platform from the release page and install it directly, e.g.:
pip install https://github.com/apache/spark-connect-rust/releases/download/v4.2.0-rc2/pyspark_client_rust-4.2.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Then use it exactly like PySpark. See if your existing workloads work with
this.
HOW TO TEST IT — RUST (cargo)
The Rust crates are intentionally NOT on crates.io for a release candidate
(crates.io is immutable; the crates are published there only when the
release is
finalized). Test against the tagged source instead:
# A) Pin a git dependency to the RC tag in your project's Cargo.toml:
[dependencies]
apache-spark-connect = { git =
"https://github.com/apache/spark-connect-rust", tag = "v4.2.0-rc2" }
# B) Or build and run the crates' own tests from the tag:
git clone --branch v4.2.0-rc2 https://github.com/apache/spark-connect-rust
cd spark-connect-rust
cargo build --release
cargo test
HIGHLIGHTS
- *Drop-in PySpark Connect Python API*, verified against real PySpark
4.2.0 two
ways: the official PySpark test suite runs against this client, and a
behavioral API-parity sweep compares the Python API surface (functions
and
DataFrame / Column / SparkSession / GroupedData / Window methods)
call-by-call
to PySpark.
- Spark Connect Rust API *full* coverage, including *Rust UDF*s
(docs: https://apache.github.io/spark-connect-rust/).
- *Native Rust transport* via tonic; cloudpickle vendored to match
upstream, so
no grpcio/py4j at runtime. and *native type conversion.*
WHY DOES THE VERSION START AT 4.2.0?
Even though this is the first release of the rewrite, the version starts at
4.2.0 to track the Apache Spark / PySpark version it targets. This (1)
removes
any ambiguity about which Spark version a given client is compatible with,
and
(2) lets PySpark users pip install pyspark-client-rust== exactly
as they would pyspark-client, keeping the same versioning across both.
Thanks,
Hyukjin Kwon
Re: [VOTE] Release Apache Spark Connect Rust Client (+ pyspark-client in Rust) 4.2.0 (RC2)
Posted by Gengliang Wang <lt...@gmail.com>.
+1
On Thu, Sep 3, 2026 at 4:10 AM Yang Jie <ya...@apache.org> wrote:
> +1
>
> On 2026/09/03 01:12:00 bo yang wrote:
> > +1 (non binding)
> >
> >
> > On Wed, Sep 2, 2026 at 5:36 PM Ruifeng Zheng <zr...@gmail.com> wrote:
> >
> > > +1
> > >
> > > On Thu, Sep 3, 2026 at 6:07 AM L. C. Hsieh <vi...@gmail.com> wrote:
> > >
> > >> +1, on the technical merits.
> > >>
> > >> I verified RC2 end-to-end and the software is in good shape — build,
> > >> tests, and
> > >> functional results are all correct (details below). Two
> > >> packaging/release-policy
> > >> items stood out that I'd ask the RM to look at before this is
> finalized;
> > >> I don't
> > >> think either is a code problem, but the first in particular may matter
> > >> for a
> > >> formal ASF release, so I'm noting them rather than blocking.
> > >>
> > >> ## What I verified (all passed)
> > >>
> > >> Identity & integrity:
> > >> - git tag v4.2.0-rc2 resolves to commit ce1fa08 (matches the vote
> email).
> > >> - Downloaded macOS universal2 wheel and the sdist; both SHA-256 match
> the
> > >> digests GitHub recorded for the assets.
> > >> - proto pinning in the tree is PROTO_VERSION=v4.2.0 /
> SPARK_SHA=32f7299,
> > >> consistent with targeting Spark 4.2.0.
> > >>
> > >> Python wheel (installed into a throwaway venv):
> > >> - The macOS universal2 wheel installs and imports as a drop-in
> `pyspark`
> > >> (pyspark-client-rust 4.2.0). The abi3 native extension is a genuine
> > >> universal2 binary (x86_64 + arm64 slices) and loads on Apple
> Silicon.
> > >> - Ran real workloads against a local Spark 4.2.0 Connect server (the
> > >> bundled
> > >> spark-connect_2.13-4.2.0.jar from the apache/spark:4.2.0 image):
> > >> range/count,
> > >> groupBy aggregation, SQL, createDataFrame + filter — all correct;
> > >> server-reported spark.version = 4.2.0.
> > >> - collect() and toPandas() type conversion is correct across
> > >> int/string/double/
> > >> boolean and NULL->NaN.
> > >>
> > >> Rust crates (from the tag):
> > >> - cargo build --release and cargo test --release both pass, no
> failures
> > >> (server-dependent e2e tests are ignored by design without a live
> > >> server).
> > >> - The Rust `sql` example runs end-to-end against the live server (SQL
> ->
> > >> write
> > >> parquet -> read back), confirming the native tonic transport works
> > >> against a
> > >> real Spark 4.2.0 server.
> > >>
> > >> ## Notes for the RM
> > >>
> > >> 1. The candidate is not published to the ASF distribution area and is
> not
> > >> signed. The release files live only on GitHub Releases. There
> > >> is no .asc PGP signature, no .sha512, and no reference to the Spark
> > >> KEYS
> > >> file. GitHub's SHA-256 digests establish transfer integrity only,
> not
> > >> that
> > >> a release manager signed the artifacts. ASF policy is that the
> thing
> > >> being
> > >> voted on is a signed source release in the dist area; convenience
> > >> binaries
> > >> are not the subject of the vote. Notably the sibling Spark Connect
> Go
> > >> client
> > >> RC does publish to dist.apache.org, so there is an established
> > >> pattern here
> > >> for this candidate to follow.
> > >>
> > >> 2. The source distribution does not ship a LICENSE. The sdist
> > >> (pyspark_client_rust-4.2.0.tar.gz) has 390 files and none of them
> is a
> > >> LICENSE file, so downstream repackagers get no license text — this
> one
> > >> should be fixed regardless. Separately, the tree bundles some
> vendored
> > >> third-party code, so a NOTICE also looks like it's needed here (and
> > >> there
> > >> isn't one); worth the RM double-checking the LICENSE/NOTICE against
> > >> what's
> > >> actually bundled.
> > >>
> > >> ## Minor (non-blocking) observation
> > >>
> > >> DataFrame.show() renders rows in a compact form (e.g. `[apple, 123]`)
> > >> rather
> > >> than PySpark's bordered ASCII table. Values are correct; flagging only
> > >> because
> > >> the client is presented as a strict drop-in and some users scrape
> show()
> > >> output.
> > >>
> > >>
> > >> Thanks for driving this — the technical work looks solid.
> > >>
> > >>
> > >> On Tue, Sep 1, 2026 at 3:08 PM Hyukjin Kwon <gu...@apache.org>
> wrote:
> > >>
> > >>> Please vote on releasing the following candidate as Apache Spark
> Connect
> > >>> Rust
> > >>> Client 4.2.0 RC2.
> > >>>
> > >>> This is the *first* release of the ground-up Rust rewrite of the
> Spark
> > >>> Connect
> > >>> Python client (published to PyPI as pyspark-client-rust): a drop-in
> > >>> replacement
> > >>> for pyspark-client <https://pypi.org/project/pyspark-client/>,
> backed
> > >>> by a native Rust core instead of py4j/grpcio.
> > >>>
> > >>> The vote is open until Thursday, September 7, 2026, 5:00 PM PDT and
> > >>> passes if a majority of +1 PMC votes are cast, with a minimum of
> > >>> 3 +1 votes.
> > >>>
> > >>> [ ] +1 Release this package as Apache Spark Connect Rust Client 4.2.0
> > >>> [ ] -1 Do not release this package because …
> > >>>
> > >>> TAG:
> > >>> https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2
> > >>> (commit: ce1fa08)
> > >>>
> > >>> RELEASE FILES (wheels + sdist, built and attached by GitHub Actions):
> > >>> https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2
> > >>>
> > >>> DOCUMENTATION:
> > >>> https://apache.github.io/spark-connect-rust/
> > >>>
> > >>> LIST OF ISSUES:
> > >>> https://issues.apache.org/jira/projects/SPARK/versions/12357484
> > >>> HOW TO TRY IT — PYTHON (pip)
> > >>>
> > >>> The release attaches abi3 wheels (Linux x86_64/aarch64, macOS
> universal2,
> > >>> Windows x86_64; CPython 3.9+) plus a source distribution. Grab the
> wheel
> > >>> for
> > >>> your platform from the release page and install it directly, e.g.:
> > >>>
> > >>> pip install
> https://github.com/apache/spark-connect-rust/releases/download/v4.2.0-rc2/pyspark_client_rust-4.2.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
> > >>>
> > >>> Then use it exactly like PySpark. See if your existing workloads work
> > >>> with this.
> > >>> HOW TO TEST IT — RUST (cargo)
> > >>>
> > >>> The Rust crates are intentionally NOT on crates.io for a release
> > >>> candidate
> > >>> (crates.io is immutable; the crates are published there only when
> the
> > >>> release is
> > >>> finalized). Test against the tagged source instead:
> > >>>
> > >>> # A) Pin a git dependency to the RC tag in your project's
> Cargo.toml:
> > >>> [dependencies]
> > >>> apache-spark-connect = { git = "
> https://github.com/apache/spark-connect-rust", tag = "v4.2.0-rc2" }
> > >>>
> > >>> # B) Or build and run the crates' own tests from the tag:
> > >>> git clone --branch v4.2.0-rc2
> https://github.com/apache/spark-connect-rust
> > >>> cd spark-connect-rust
> > >>> cargo build --release
> > >>> cargo test
> > >>>
> > >>> HIGHLIGHTS
> > >>>
> > >>> - *Drop-in PySpark Connect Python API*, verified against real
> > >>> PySpark 4.2.0 two
> > >>> ways: the official PySpark test suite runs against this client,
> and a
> > >>> behavioral API-parity sweep compares the Python API surface
> > >>> (functions and
> > >>> DataFrame / Column / SparkSession / GroupedData / Window methods)
> > >>> call-by-call
> > >>> to PySpark.
> > >>> - Spark Connect Rust API *full* coverage, including *Rust UDF*s
> > >>> (docs: https://apache.github.io/spark-connect-rust/).
> > >>> - *Native Rust transport* via tonic; cloudpickle vendored to match
> > >>> upstream, so
> > >>> no grpcio/py4j at runtime. and *native type conversion.*
> > >>>
> > >>> WHY DOES THE VERSION START AT 4.2.0?
> > >>>
> > >>> Even though this is the first release of the rewrite, the version
> starts
> > >>> at
> > >>> 4.2.0 to track the Apache Spark / PySpark version it targets. This
> (1)
> > >>> removes
> > >>> any ambiguity about which Spark version a given client is compatible
> > >>> with, and
> > >>> (2) lets PySpark users pip install pyspark-client-rust== exactly
> > >>> as they would pyspark-client, keeping the same versioning across
> both.
> > >>>
> > >>> Thanks,
> > >>> Hyukjin Kwon
> > >>>
> > >>
> > >
> > > --
> > > Ruifeng Zheng
> > > E-mail: zrflibra@gmail.com
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: [VOTE] Release Apache Spark Connect Rust Client (+ pyspark-client in Rust) 4.2.0 (RC2)
Posted by Yang Jie <ya...@apache.org>.
+1
On 2026/09/03 01:12:00 bo yang wrote:
> +1 (non binding)
>
>
> On Wed, Sep 2, 2026 at 5:36 PM Ruifeng Zheng <zr...@gmail.com> wrote:
>
> > +1
> >
> > On Thu, Sep 3, 2026 at 6:07 AM L. C. Hsieh <vi...@gmail.com> wrote:
> >
> >> +1, on the technical merits.
> >>
> >> I verified RC2 end-to-end and the software is in good shape — build,
> >> tests, and
> >> functional results are all correct (details below). Two
> >> packaging/release-policy
> >> items stood out that I'd ask the RM to look at before this is finalized;
> >> I don't
> >> think either is a code problem, but the first in particular may matter
> >> for a
> >> formal ASF release, so I'm noting them rather than blocking.
> >>
> >> ## What I verified (all passed)
> >>
> >> Identity & integrity:
> >> - git tag v4.2.0-rc2 resolves to commit ce1fa08 (matches the vote email).
> >> - Downloaded macOS universal2 wheel and the sdist; both SHA-256 match the
> >> digests GitHub recorded for the assets.
> >> - proto pinning in the tree is PROTO_VERSION=v4.2.0 / SPARK_SHA=32f7299,
> >> consistent with targeting Spark 4.2.0.
> >>
> >> Python wheel (installed into a throwaway venv):
> >> - The macOS universal2 wheel installs and imports as a drop-in `pyspark`
> >> (pyspark-client-rust 4.2.0). The abi3 native extension is a genuine
> >> universal2 binary (x86_64 + arm64 slices) and loads on Apple Silicon.
> >> - Ran real workloads against a local Spark 4.2.0 Connect server (the
> >> bundled
> >> spark-connect_2.13-4.2.0.jar from the apache/spark:4.2.0 image):
> >> range/count,
> >> groupBy aggregation, SQL, createDataFrame + filter — all correct;
> >> server-reported spark.version = 4.2.0.
> >> - collect() and toPandas() type conversion is correct across
> >> int/string/double/
> >> boolean and NULL->NaN.
> >>
> >> Rust crates (from the tag):
> >> - cargo build --release and cargo test --release both pass, no failures
> >> (server-dependent e2e tests are ignored by design without a live
> >> server).
> >> - The Rust `sql` example runs end-to-end against the live server (SQL ->
> >> write
> >> parquet -> read back), confirming the native tonic transport works
> >> against a
> >> real Spark 4.2.0 server.
> >>
> >> ## Notes for the RM
> >>
> >> 1. The candidate is not published to the ASF distribution area and is not
> >> signed. The release files live only on GitHub Releases. There
> >> is no .asc PGP signature, no .sha512, and no reference to the Spark
> >> KEYS
> >> file. GitHub's SHA-256 digests establish transfer integrity only, not
> >> that
> >> a release manager signed the artifacts. ASF policy is that the thing
> >> being
> >> voted on is a signed source release in the dist area; convenience
> >> binaries
> >> are not the subject of the vote. Notably the sibling Spark Connect Go
> >> client
> >> RC does publish to dist.apache.org, so there is an established
> >> pattern here
> >> for this candidate to follow.
> >>
> >> 2. The source distribution does not ship a LICENSE. The sdist
> >> (pyspark_client_rust-4.2.0.tar.gz) has 390 files and none of them is a
> >> LICENSE file, so downstream repackagers get no license text — this one
> >> should be fixed regardless. Separately, the tree bundles some vendored
> >> third-party code, so a NOTICE also looks like it's needed here (and
> >> there
> >> isn't one); worth the RM double-checking the LICENSE/NOTICE against
> >> what's
> >> actually bundled.
> >>
> >> ## Minor (non-blocking) observation
> >>
> >> DataFrame.show() renders rows in a compact form (e.g. `[apple, 123]`)
> >> rather
> >> than PySpark's bordered ASCII table. Values are correct; flagging only
> >> because
> >> the client is presented as a strict drop-in and some users scrape show()
> >> output.
> >>
> >>
> >> Thanks for driving this — the technical work looks solid.
> >>
> >>
> >> On Tue, Sep 1, 2026 at 3:08 PM Hyukjin Kwon <gu...@apache.org> wrote:
> >>
> >>> Please vote on releasing the following candidate as Apache Spark Connect
> >>> Rust
> >>> Client 4.2.0 RC2.
> >>>
> >>> This is the *first* release of the ground-up Rust rewrite of the Spark
> >>> Connect
> >>> Python client (published to PyPI as pyspark-client-rust): a drop-in
> >>> replacement
> >>> for pyspark-client <https://pypi.org/project/pyspark-client/>, backed
> >>> by a native Rust core instead of py4j/grpcio.
> >>>
> >>> The vote is open until Thursday, September 7, 2026, 5:00 PM PDT and
> >>> passes if a majority of +1 PMC votes are cast, with a minimum of
> >>> 3 +1 votes.
> >>>
> >>> [ ] +1 Release this package as Apache Spark Connect Rust Client 4.2.0
> >>> [ ] -1 Do not release this package because …
> >>>
> >>> TAG:
> >>> https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2
> >>> (commit: ce1fa08)
> >>>
> >>> RELEASE FILES (wheels + sdist, built and attached by GitHub Actions):
> >>> https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2
> >>>
> >>> DOCUMENTATION:
> >>> https://apache.github.io/spark-connect-rust/
> >>>
> >>> LIST OF ISSUES:
> >>> https://issues.apache.org/jira/projects/SPARK/versions/12357484
> >>> HOW TO TRY IT — PYTHON (pip)
> >>>
> >>> The release attaches abi3 wheels (Linux x86_64/aarch64, macOS universal2,
> >>> Windows x86_64; CPython 3.9+) plus a source distribution. Grab the wheel
> >>> for
> >>> your platform from the release page and install it directly, e.g.:
> >>>
> >>> pip install https://github.com/apache/spark-connect-rust/releases/download/v4.2.0-rc2/pyspark_client_rust-4.2.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
> >>>
> >>> Then use it exactly like PySpark. See if your existing workloads work
> >>> with this.
> >>> HOW TO TEST IT — RUST (cargo)
> >>>
> >>> The Rust crates are intentionally NOT on crates.io for a release
> >>> candidate
> >>> (crates.io is immutable; the crates are published there only when the
> >>> release is
> >>> finalized). Test against the tagged source instead:
> >>>
> >>> # A) Pin a git dependency to the RC tag in your project's Cargo.toml:
> >>> [dependencies]
> >>> apache-spark-connect = { git = "https://github.com/apache/spark-connect-rust", tag = "v4.2.0-rc2" }
> >>>
> >>> # B) Or build and run the crates' own tests from the tag:
> >>> git clone --branch v4.2.0-rc2 https://github.com/apache/spark-connect-rust
> >>> cd spark-connect-rust
> >>> cargo build --release
> >>> cargo test
> >>>
> >>> HIGHLIGHTS
> >>>
> >>> - *Drop-in PySpark Connect Python API*, verified against real
> >>> PySpark 4.2.0 two
> >>> ways: the official PySpark test suite runs against this client, and a
> >>> behavioral API-parity sweep compares the Python API surface
> >>> (functions and
> >>> DataFrame / Column / SparkSession / GroupedData / Window methods)
> >>> call-by-call
> >>> to PySpark.
> >>> - Spark Connect Rust API *full* coverage, including *Rust UDF*s
> >>> (docs: https://apache.github.io/spark-connect-rust/).
> >>> - *Native Rust transport* via tonic; cloudpickle vendored to match
> >>> upstream, so
> >>> no grpcio/py4j at runtime. and *native type conversion.*
> >>>
> >>> WHY DOES THE VERSION START AT 4.2.0?
> >>>
> >>> Even though this is the first release of the rewrite, the version starts
> >>> at
> >>> 4.2.0 to track the Apache Spark / PySpark version it targets. This (1)
> >>> removes
> >>> any ambiguity about which Spark version a given client is compatible
> >>> with, and
> >>> (2) lets PySpark users pip install pyspark-client-rust== exactly
> >>> as they would pyspark-client, keeping the same versioning across both.
> >>>
> >>> Thanks,
> >>> Hyukjin Kwon
> >>>
> >>
> >
> > --
> > Ruifeng Zheng
> > E-mail: zrflibra@gmail.com
> >
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [VOTE] Release Apache Spark Connect Rust Client (+ pyspark-client in Rust) 4.2.0 (RC2)
Posted by bo yang <bo...@gmail.com>.
+1 (non binding)
On Wed, Sep 2, 2026 at 5:36 PM Ruifeng Zheng <zr...@gmail.com> wrote:
> +1
>
> On Thu, Sep 3, 2026 at 6:07 AM L. C. Hsieh <vi...@gmail.com> wrote:
>
>> +1, on the technical merits.
>>
>> I verified RC2 end-to-end and the software is in good shape — build,
>> tests, and
>> functional results are all correct (details below). Two
>> packaging/release-policy
>> items stood out that I'd ask the RM to look at before this is finalized;
>> I don't
>> think either is a code problem, but the first in particular may matter
>> for a
>> formal ASF release, so I'm noting them rather than blocking.
>>
>> ## What I verified (all passed)
>>
>> Identity & integrity:
>> - git tag v4.2.0-rc2 resolves to commit ce1fa08 (matches the vote email).
>> - Downloaded macOS universal2 wheel and the sdist; both SHA-256 match the
>> digests GitHub recorded for the assets.
>> - proto pinning in the tree is PROTO_VERSION=v4.2.0 / SPARK_SHA=32f7299,
>> consistent with targeting Spark 4.2.0.
>>
>> Python wheel (installed into a throwaway venv):
>> - The macOS universal2 wheel installs and imports as a drop-in `pyspark`
>> (pyspark-client-rust 4.2.0). The abi3 native extension is a genuine
>> universal2 binary (x86_64 + arm64 slices) and loads on Apple Silicon.
>> - Ran real workloads against a local Spark 4.2.0 Connect server (the
>> bundled
>> spark-connect_2.13-4.2.0.jar from the apache/spark:4.2.0 image):
>> range/count,
>> groupBy aggregation, SQL, createDataFrame + filter — all correct;
>> server-reported spark.version = 4.2.0.
>> - collect() and toPandas() type conversion is correct across
>> int/string/double/
>> boolean and NULL->NaN.
>>
>> Rust crates (from the tag):
>> - cargo build --release and cargo test --release both pass, no failures
>> (server-dependent e2e tests are ignored by design without a live
>> server).
>> - The Rust `sql` example runs end-to-end against the live server (SQL ->
>> write
>> parquet -> read back), confirming the native tonic transport works
>> against a
>> real Spark 4.2.0 server.
>>
>> ## Notes for the RM
>>
>> 1. The candidate is not published to the ASF distribution area and is not
>> signed. The release files live only on GitHub Releases. There
>> is no .asc PGP signature, no .sha512, and no reference to the Spark
>> KEYS
>> file. GitHub's SHA-256 digests establish transfer integrity only, not
>> that
>> a release manager signed the artifacts. ASF policy is that the thing
>> being
>> voted on is a signed source release in the dist area; convenience
>> binaries
>> are not the subject of the vote. Notably the sibling Spark Connect Go
>> client
>> RC does publish to dist.apache.org, so there is an established
>> pattern here
>> for this candidate to follow.
>>
>> 2. The source distribution does not ship a LICENSE. The sdist
>> (pyspark_client_rust-4.2.0.tar.gz) has 390 files and none of them is a
>> LICENSE file, so downstream repackagers get no license text — this one
>> should be fixed regardless. Separately, the tree bundles some vendored
>> third-party code, so a NOTICE also looks like it's needed here (and
>> there
>> isn't one); worth the RM double-checking the LICENSE/NOTICE against
>> what's
>> actually bundled.
>>
>> ## Minor (non-blocking) observation
>>
>> DataFrame.show() renders rows in a compact form (e.g. `[apple, 123]`)
>> rather
>> than PySpark's bordered ASCII table. Values are correct; flagging only
>> because
>> the client is presented as a strict drop-in and some users scrape show()
>> output.
>>
>>
>> Thanks for driving this — the technical work looks solid.
>>
>>
>> On Tue, Sep 1, 2026 at 3:08 PM Hyukjin Kwon <gu...@apache.org> wrote:
>>
>>> Please vote on releasing the following candidate as Apache Spark Connect
>>> Rust
>>> Client 4.2.0 RC2.
>>>
>>> This is the *first* release of the ground-up Rust rewrite of the Spark
>>> Connect
>>> Python client (published to PyPI as pyspark-client-rust): a drop-in
>>> replacement
>>> for pyspark-client <https://pypi.org/project/pyspark-client/>, backed
>>> by a native Rust core instead of py4j/grpcio.
>>>
>>> The vote is open until Thursday, September 7, 2026, 5:00 PM PDT and
>>> passes if a majority of +1 PMC votes are cast, with a minimum of
>>> 3 +1 votes.
>>>
>>> [ ] +1 Release this package as Apache Spark Connect Rust Client 4.2.0
>>> [ ] -1 Do not release this package because …
>>>
>>> TAG:
>>> https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2
>>> (commit: ce1fa08)
>>>
>>> RELEASE FILES (wheels + sdist, built and attached by GitHub Actions):
>>> https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2
>>>
>>> DOCUMENTATION:
>>> https://apache.github.io/spark-connect-rust/
>>>
>>> LIST OF ISSUES:
>>> https://issues.apache.org/jira/projects/SPARK/versions/12357484
>>> HOW TO TRY IT — PYTHON (pip)
>>>
>>> The release attaches abi3 wheels (Linux x86_64/aarch64, macOS universal2,
>>> Windows x86_64; CPython 3.9+) plus a source distribution. Grab the wheel
>>> for
>>> your platform from the release page and install it directly, e.g.:
>>>
>>> pip install https://github.com/apache/spark-connect-rust/releases/download/v4.2.0-rc2/pyspark_client_rust-4.2.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
>>>
>>> Then use it exactly like PySpark. See if your existing workloads work
>>> with this.
>>> HOW TO TEST IT — RUST (cargo)
>>>
>>> The Rust crates are intentionally NOT on crates.io for a release
>>> candidate
>>> (crates.io is immutable; the crates are published there only when the
>>> release is
>>> finalized). Test against the tagged source instead:
>>>
>>> # A) Pin a git dependency to the RC tag in your project's Cargo.toml:
>>> [dependencies]
>>> apache-spark-connect = { git = "https://github.com/apache/spark-connect-rust", tag = "v4.2.0-rc2" }
>>>
>>> # B) Or build and run the crates' own tests from the tag:
>>> git clone --branch v4.2.0-rc2 https://github.com/apache/spark-connect-rust
>>> cd spark-connect-rust
>>> cargo build --release
>>> cargo test
>>>
>>> HIGHLIGHTS
>>>
>>> - *Drop-in PySpark Connect Python API*, verified against real
>>> PySpark 4.2.0 two
>>> ways: the official PySpark test suite runs against this client, and a
>>> behavioral API-parity sweep compares the Python API surface
>>> (functions and
>>> DataFrame / Column / SparkSession / GroupedData / Window methods)
>>> call-by-call
>>> to PySpark.
>>> - Spark Connect Rust API *full* coverage, including *Rust UDF*s
>>> (docs: https://apache.github.io/spark-connect-rust/).
>>> - *Native Rust transport* via tonic; cloudpickle vendored to match
>>> upstream, so
>>> no grpcio/py4j at runtime. and *native type conversion.*
>>>
>>> WHY DOES THE VERSION START AT 4.2.0?
>>>
>>> Even though this is the first release of the rewrite, the version starts
>>> at
>>> 4.2.0 to track the Apache Spark / PySpark version it targets. This (1)
>>> removes
>>> any ambiguity about which Spark version a given client is compatible
>>> with, and
>>> (2) lets PySpark users pip install pyspark-client-rust== exactly
>>> as they would pyspark-client, keeping the same versioning across both.
>>>
>>> Thanks,
>>> Hyukjin Kwon
>>>
>>
>
> --
> Ruifeng Zheng
> E-mail: zrflibra@gmail.com
>
Re: [VOTE] Release Apache Spark Connect Rust Client (+ pyspark-client in Rust) 4.2.0 (RC2)
Posted by Ruifeng Zheng <zr...@gmail.com>.
+1
On Thu, Sep 3, 2026 at 6:07 AM L. C. Hsieh <vi...@gmail.com> wrote:
> +1, on the technical merits.
>
> I verified RC2 end-to-end and the software is in good shape — build,
> tests, and
> functional results are all correct (details below). Two
> packaging/release-policy
> items stood out that I'd ask the RM to look at before this is finalized; I
> don't
> think either is a code problem, but the first in particular may matter for
> a
> formal ASF release, so I'm noting them rather than blocking.
>
> ## What I verified (all passed)
>
> Identity & integrity:
> - git tag v4.2.0-rc2 resolves to commit ce1fa08 (matches the vote email).
> - Downloaded macOS universal2 wheel and the sdist; both SHA-256 match the
> digests GitHub recorded for the assets.
> - proto pinning in the tree is PROTO_VERSION=v4.2.0 / SPARK_SHA=32f7299,
> consistent with targeting Spark 4.2.0.
>
> Python wheel (installed into a throwaway venv):
> - The macOS universal2 wheel installs and imports as a drop-in `pyspark`
> (pyspark-client-rust 4.2.0). The abi3 native extension is a genuine
> universal2 binary (x86_64 + arm64 slices) and loads on Apple Silicon.
> - Ran real workloads against a local Spark 4.2.0 Connect server (the
> bundled
> spark-connect_2.13-4.2.0.jar from the apache/spark:4.2.0 image):
> range/count,
> groupBy aggregation, SQL, createDataFrame + filter — all correct;
> server-reported spark.version = 4.2.0.
> - collect() and toPandas() type conversion is correct across
> int/string/double/
> boolean and NULL->NaN.
>
> Rust crates (from the tag):
> - cargo build --release and cargo test --release both pass, no failures
> (server-dependent e2e tests are ignored by design without a live server).
> - The Rust `sql` example runs end-to-end against the live server (SQL ->
> write
> parquet -> read back), confirming the native tonic transport works
> against a
> real Spark 4.2.0 server.
>
> ## Notes for the RM
>
> 1. The candidate is not published to the ASF distribution area and is not
> signed. The release files live only on GitHub Releases. There
> is no .asc PGP signature, no .sha512, and no reference to the Spark KEYS
> file. GitHub's SHA-256 digests establish transfer integrity only, not
> that
> a release manager signed the artifacts. ASF policy is that the thing
> being
> voted on is a signed source release in the dist area; convenience
> binaries
> are not the subject of the vote. Notably the sibling Spark Connect Go
> client
> RC does publish to dist.apache.org, so there is an established pattern
> here
> for this candidate to follow.
>
> 2. The source distribution does not ship a LICENSE. The sdist
> (pyspark_client_rust-4.2.0.tar.gz) has 390 files and none of them is a
> LICENSE file, so downstream repackagers get no license text — this one
> should be fixed regardless. Separately, the tree bundles some vendored
> third-party code, so a NOTICE also looks like it's needed here (and
> there
> isn't one); worth the RM double-checking the LICENSE/NOTICE against
> what's
> actually bundled.
>
> ## Minor (non-blocking) observation
>
> DataFrame.show() renders rows in a compact form (e.g. `[apple, 123]`)
> rather
> than PySpark's bordered ASCII table. Values are correct; flagging only
> because
> the client is presented as a strict drop-in and some users scrape show()
> output.
>
>
> Thanks for driving this — the technical work looks solid.
>
>
> On Tue, Sep 1, 2026 at 3:08 PM Hyukjin Kwon <gu...@apache.org> wrote:
>
>> Please vote on releasing the following candidate as Apache Spark Connect
>> Rust
>> Client 4.2.0 RC2.
>>
>> This is the *first* release of the ground-up Rust rewrite of the Spark
>> Connect
>> Python client (published to PyPI as pyspark-client-rust): a drop-in
>> replacement
>> for pyspark-client <https://pypi.org/project/pyspark-client/>, backed by
>> a native Rust core instead of py4j/grpcio.
>>
>> The vote is open until Thursday, September 7, 2026, 5:00 PM PDT and
>> passes if a majority of +1 PMC votes are cast, with a minimum of
>> 3 +1 votes.
>>
>> [ ] +1 Release this package as Apache Spark Connect Rust Client 4.2.0
>> [ ] -1 Do not release this package because …
>>
>> TAG:
>> https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2
>> (commit: ce1fa08)
>>
>> RELEASE FILES (wheels + sdist, built and attached by GitHub Actions):
>> https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2
>>
>> DOCUMENTATION:
>> https://apache.github.io/spark-connect-rust/
>>
>> LIST OF ISSUES:
>> https://issues.apache.org/jira/projects/SPARK/versions/12357484
>> HOW TO TRY IT — PYTHON (pip)
>>
>> The release attaches abi3 wheels (Linux x86_64/aarch64, macOS universal2,
>> Windows x86_64; CPython 3.9+) plus a source distribution. Grab the wheel
>> for
>> your platform from the release page and install it directly, e.g.:
>>
>> pip install https://github.com/apache/spark-connect-rust/releases/download/v4.2.0-rc2/pyspark_client_rust-4.2.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
>>
>> Then use it exactly like PySpark. See if your existing workloads work
>> with this.
>> HOW TO TEST IT — RUST (cargo)
>>
>> The Rust crates are intentionally NOT on crates.io for a release
>> candidate
>> (crates.io is immutable; the crates are published there only when the
>> release is
>> finalized). Test against the tagged source instead:
>>
>> # A) Pin a git dependency to the RC tag in your project's Cargo.toml:
>> [dependencies]
>> apache-spark-connect = { git = "https://github.com/apache/spark-connect-rust", tag = "v4.2.0-rc2" }
>>
>> # B) Or build and run the crates' own tests from the tag:
>> git clone --branch v4.2.0-rc2 https://github.com/apache/spark-connect-rust
>> cd spark-connect-rust
>> cargo build --release
>> cargo test
>>
>> HIGHLIGHTS
>>
>> - *Drop-in PySpark Connect Python API*, verified against real PySpark
>> 4.2.0 two
>> ways: the official PySpark test suite runs against this client, and a
>> behavioral API-parity sweep compares the Python API surface
>> (functions and
>> DataFrame / Column / SparkSession / GroupedData / Window methods)
>> call-by-call
>> to PySpark.
>> - Spark Connect Rust API *full* coverage, including *Rust UDF*s
>> (docs: https://apache.github.io/spark-connect-rust/).
>> - *Native Rust transport* via tonic; cloudpickle vendored to match
>> upstream, so
>> no grpcio/py4j at runtime. and *native type conversion.*
>>
>> WHY DOES THE VERSION START AT 4.2.0?
>>
>> Even though this is the first release of the rewrite, the version starts
>> at
>> 4.2.0 to track the Apache Spark / PySpark version it targets. This (1)
>> removes
>> any ambiguity about which Spark version a given client is compatible
>> with, and
>> (2) lets PySpark users pip install pyspark-client-rust== exactly
>> as they would pyspark-client, keeping the same versioning across both.
>>
>> Thanks,
>> Hyukjin Kwon
>>
>
--
Ruifeng Zheng
E-mail: zrflibra@gmail.com
Re: [VOTE] Release Apache Spark Connect Rust Client (+ pyspark-client in Rust) 4.2.0 (RC2)
Posted by "L. C. Hsieh" <vi...@gmail.com>.
+1, on the technical merits.
I verified RC2 end-to-end and the software is in good shape — build, tests,
and
functional results are all correct (details below). Two
packaging/release-policy
items stood out that I'd ask the RM to look at before this is finalized; I
don't
think either is a code problem, but the first in particular may matter for a
formal ASF release, so I'm noting them rather than blocking.
## What I verified (all passed)
Identity & integrity:
- git tag v4.2.0-rc2 resolves to commit ce1fa08 (matches the vote email).
- Downloaded macOS universal2 wheel and the sdist; both SHA-256 match the
digests GitHub recorded for the assets.
- proto pinning in the tree is PROTO_VERSION=v4.2.0 / SPARK_SHA=32f7299,
consistent with targeting Spark 4.2.0.
Python wheel (installed into a throwaway venv):
- The macOS universal2 wheel installs and imports as a drop-in `pyspark`
(pyspark-client-rust 4.2.0). The abi3 native extension is a genuine
universal2 binary (x86_64 + arm64 slices) and loads on Apple Silicon.
- Ran real workloads against a local Spark 4.2.0 Connect server (the bundled
spark-connect_2.13-4.2.0.jar from the apache/spark:4.2.0 image):
range/count,
groupBy aggregation, SQL, createDataFrame + filter — all correct;
server-reported spark.version = 4.2.0.
- collect() and toPandas() type conversion is correct across
int/string/double/
boolean and NULL->NaN.
Rust crates (from the tag):
- cargo build --release and cargo test --release both pass, no failures
(server-dependent e2e tests are ignored by design without a live server).
- The Rust `sql` example runs end-to-end against the live server (SQL ->
write
parquet -> read back), confirming the native tonic transport works
against a
real Spark 4.2.0 server.
## Notes for the RM
1. The candidate is not published to the ASF distribution area and is not
signed. The release files live only on GitHub Releases. There
is no .asc PGP signature, no .sha512, and no reference to the Spark KEYS
file. GitHub's SHA-256 digests establish transfer integrity only, not
that
a release manager signed the artifacts. ASF policy is that the thing
being
voted on is a signed source release in the dist area; convenience
binaries
are not the subject of the vote. Notably the sibling Spark Connect Go
client
RC does publish to dist.apache.org, so there is an established pattern
here
for this candidate to follow.
2. The source distribution does not ship a LICENSE. The sdist
(pyspark_client_rust-4.2.0.tar.gz) has 390 files and none of them is a
LICENSE file, so downstream repackagers get no license text — this one
should be fixed regardless. Separately, the tree bundles some vendored
third-party code, so a NOTICE also looks like it's needed here (and there
isn't one); worth the RM double-checking the LICENSE/NOTICE against
what's
actually bundled.
## Minor (non-blocking) observation
DataFrame.show() renders rows in a compact form (e.g. `[apple, 123]`) rather
than PySpark's bordered ASCII table. Values are correct; flagging only
because
the client is presented as a strict drop-in and some users scrape show()
output.
Thanks for driving this — the technical work looks solid.
On Tue, Sep 1, 2026 at 3:08 PM Hyukjin Kwon <gu...@apache.org> wrote:
> Please vote on releasing the following candidate as Apache Spark Connect
> Rust
> Client 4.2.0 RC2.
>
> This is the *first* release of the ground-up Rust rewrite of the Spark
> Connect
> Python client (published to PyPI as pyspark-client-rust): a drop-in
> replacement
> for pyspark-client <https://pypi.org/project/pyspark-client/>, backed by
> a native Rust core instead of py4j/grpcio.
>
> The vote is open until Thursday, September 7, 2026, 5:00 PM PDT and passes
> if a majority of +1 PMC votes are cast, with a minimum of
> 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark Connect Rust Client 4.2.0
> [ ] -1 Do not release this package because …
>
> TAG:
> https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2
> (commit: ce1fa08)
>
> RELEASE FILES (wheels + sdist, built and attached by GitHub Actions):
> https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2
>
> DOCUMENTATION:
> https://apache.github.io/spark-connect-rust/
>
> LIST OF ISSUES:
> https://issues.apache.org/jira/projects/SPARK/versions/12357484
> HOW TO TRY IT — PYTHON (pip)
>
> The release attaches abi3 wheels (Linux x86_64/aarch64, macOS universal2,
> Windows x86_64; CPython 3.9+) plus a source distribution. Grab the wheel
> for
> your platform from the release page and install it directly, e.g.:
>
> pip install https://github.com/apache/spark-connect-rust/releases/download/v4.2.0-rc2/pyspark_client_rust-4.2.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
>
> Then use it exactly like PySpark. See if your existing workloads work with
> this.
> HOW TO TEST IT — RUST (cargo)
>
> The Rust crates are intentionally NOT on crates.io for a release candidate
> (crates.io is immutable; the crates are published there only when the
> release is
> finalized). Test against the tagged source instead:
>
> # A) Pin a git dependency to the RC tag in your project's Cargo.toml:
> [dependencies]
> apache-spark-connect = { git = "https://github.com/apache/spark-connect-rust", tag = "v4.2.0-rc2" }
>
> # B) Or build and run the crates' own tests from the tag:
> git clone --branch v4.2.0-rc2 https://github.com/apache/spark-connect-rust
> cd spark-connect-rust
> cargo build --release
> cargo test
>
> HIGHLIGHTS
>
> - *Drop-in PySpark Connect Python API*, verified against real PySpark
> 4.2.0 two
> ways: the official PySpark test suite runs against this client, and a
> behavioral API-parity sweep compares the Python API surface (functions
> and
> DataFrame / Column / SparkSession / GroupedData / Window methods)
> call-by-call
> to PySpark.
> - Spark Connect Rust API *full* coverage, including *Rust UDF*s
> (docs: https://apache.github.io/spark-connect-rust/).
> - *Native Rust transport* via tonic; cloudpickle vendored to match
> upstream, so
> no grpcio/py4j at runtime. and *native type conversion.*
>
> WHY DOES THE VERSION START AT 4.2.0?
>
> Even though this is the first release of the rewrite, the version starts at
> 4.2.0 to track the Apache Spark / PySpark version it targets. This (1)
> removes
> any ambiguity about which Spark version a given client is compatible with,
> and
> (2) lets PySpark users pip install pyspark-client-rust== exactly
> as they would pyspark-client, keeping the same versioning across both.
>
> Thanks,
> Hyukjin Kwon
>
Re: [VOTE] Release Apache Spark Connect Rust Client (+ pyspark-client in Rust) 4.2.0 (RC2)
Posted by Kousuke Saruta <sa...@apache.org>.
+1
2026年9月2日(水) 16:43 Holden Karau <ho...@gmail.com>:
> +1
>
>
> Twitter: https://twitter.com/holdenkarau
> Fight Health Insurance: https://www.fighthealthinsurance.com/
> <https://www.fighthealthinsurance.com/?q=hk_email>
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> Pronouns: she/her
>
> On Tue, Sep 1, 2026 at 9:15 PM Dongjoon Hyun <do...@apache.org> wrote:
>
>> +1
>>
>> Dongjoon.
>>
>> On 2026/09/01 22:08:01 Hyukjin Kwon wrote:
>> > Starting with my own +1.
>> >
>> > PS: we have a discussion in the dev mailing list, and I am sync'ed with
>> the
>> > original authors.
>> >
>> > On Wed, 2 Sept 2026 at 07:07, Hyukjin Kwon <gu...@apache.org>
>> wrote:
>> >
>> > > Please vote on releasing the following candidate as Apache Spark
>> Connect
>> > > Rust
>> > > Client 4.2.0 RC2.
>> > >
>> > > This is the *first* release of the ground-up Rust rewrite of the Spark
>> > > Connect
>> > > Python client (published to PyPI as pyspark-client-rust): a drop-in
>> > > replacement
>> > > for pyspark-client <https://pypi.org/project/pyspark-client/>,
>> backed by
>> > > a native Rust core instead of py4j/grpcio.
>> > >
>> > > The vote is open until Thursday, September 7, 2026, 5:00 PM PDT and
>> passes
>> > > if a majority of +1 PMC votes are cast, with a minimum of
>> > > 3 +1 votes.
>> > >
>> > > [ ] +1 Release this package as Apache Spark Connect Rust Client 4.2.0
>> > > [ ] -1 Do not release this package because …
>> > >
>> > > TAG:
>> > > https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2
>> > > (commit: ce1fa08)
>> > >
>> > > RELEASE FILES (wheels + sdist, built and attached by GitHub Actions):
>> > > https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2
>> > >
>> > > DOCUMENTATION:
>> > > https://apache.github.io/spark-connect-rust/
>> > >
>> > > LIST OF ISSUES:
>> > > https://issues.apache.org/jira/projects/SPARK/versions/12357484
>> > > HOW TO TRY IT — PYTHON (pip)
>> > >
>> > > The release attaches abi3 wheels (Linux x86_64/aarch64, macOS
>> universal2,
>> > > Windows x86_64; CPython 3.9+) plus a source distribution. Grab the
>> wheel
>> > > for
>> > > your platform from the release page and install it directly, e.g.:
>> > >
>> > > pip install
>> https://github.com/apache/spark-connect-rust/releases/download/v4.2.0-rc2/pyspark_client_rust-4.2.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
>> > >
>> > > Then use it exactly like PySpark. See if your existing workloads work
>> with
>> > > this.
>> > > HOW TO TEST IT — RUST (cargo)
>> > >
>> > > The Rust crates are intentionally NOT on crates.io for a release
>> candidate
>> > > (crates.io is immutable; the crates are published there only when the
>> > > release is
>> > > finalized). Test against the tagged source instead:
>> > >
>> > > # A) Pin a git dependency to the RC tag in your project's
>> Cargo.toml:
>> > > [dependencies]
>> > > apache-spark-connect = { git = "
>> https://github.com/apache/spark-connect-rust", tag = "v4.2.0-rc2" }
>> > >
>> > > # B) Or build and run the crates' own tests from the tag:
>> > > git clone --branch v4.2.0-rc2
>> https://github.com/apache/spark-connect-rust
>> > > cd spark-connect-rust
>> > > cargo build --release
>> > > cargo test
>> > >
>> > > HIGHLIGHTS
>> > >
>> > > - *Drop-in PySpark Connect Python API*, verified against real
>> PySpark
>> > > 4.2.0 two
>> > > ways: the official PySpark test suite runs against this client,
>> and a
>> > > behavioral API-parity sweep compares the Python API surface
>> (functions
>> > > and
>> > > DataFrame / Column / SparkSession / GroupedData / Window methods)
>> > > call-by-call
>> > > to PySpark.
>> > > - Spark Connect Rust API *full* coverage, including *Rust UDF*s
>> > > (docs: https://apache.github.io/spark-connect-rust/).
>> > > - *Native Rust transport* via tonic; cloudpickle vendored to match
>> > > upstream, so
>> > > no grpcio/py4j at runtime. and *native type conversion.*
>> > >
>> > > WHY DOES THE VERSION START AT 4.2.0?
>> > >
>> > > Even though this is the first release of the rewrite, the version
>> starts at
>> > > 4.2.0 to track the Apache Spark / PySpark version it targets. This (1)
>> > > removes
>> > > any ambiguity about which Spark version a given client is compatible
>> with,
>> > > and
>> > > (2) lets PySpark users pip install pyspark-client-rust== exactly
>> > > as they would pyspark-client, keeping the same versioning across both.
>> > >
>> > > Thanks,
>> > > Hyukjin Kwon
>> > >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
Re: [VOTE] Release Apache Spark Connect Rust Client (+ pyspark-client in Rust) 4.2.0 (RC2)
Posted by Holden Karau <ho...@gmail.com>.
+1
Twitter: https://twitter.com/holdenkarau
Fight Health Insurance: https://www.fighthealthinsurance.com/
<https://www.fighthealthinsurance.com/?q=hk_email>
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Pronouns: she/her
On Tue, Sep 1, 2026 at 9:15 PM Dongjoon Hyun <do...@apache.org> wrote:
> +1
>
> Dongjoon.
>
> On 2026/09/01 22:08:01 Hyukjin Kwon wrote:
> > Starting with my own +1.
> >
> > PS: we have a discussion in the dev mailing list, and I am sync'ed with
> the
> > original authors.
> >
> > On Wed, 2 Sept 2026 at 07:07, Hyukjin Kwon <gu...@apache.org> wrote:
> >
> > > Please vote on releasing the following candidate as Apache Spark
> Connect
> > > Rust
> > > Client 4.2.0 RC2.
> > >
> > > This is the *first* release of the ground-up Rust rewrite of the Spark
> > > Connect
> > > Python client (published to PyPI as pyspark-client-rust): a drop-in
> > > replacement
> > > for pyspark-client <https://pypi.org/project/pyspark-client/>, backed
> by
> > > a native Rust core instead of py4j/grpcio.
> > >
> > > The vote is open until Thursday, September 7, 2026, 5:00 PM PDT and
> passes
> > > if a majority of +1 PMC votes are cast, with a minimum of
> > > 3 +1 votes.
> > >
> > > [ ] +1 Release this package as Apache Spark Connect Rust Client 4.2.0
> > > [ ] -1 Do not release this package because …
> > >
> > > TAG:
> > > https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2
> > > (commit: ce1fa08)
> > >
> > > RELEASE FILES (wheels + sdist, built and attached by GitHub Actions):
> > > https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2
> > >
> > > DOCUMENTATION:
> > > https://apache.github.io/spark-connect-rust/
> > >
> > > LIST OF ISSUES:
> > > https://issues.apache.org/jira/projects/SPARK/versions/12357484
> > > HOW TO TRY IT — PYTHON (pip)
> > >
> > > The release attaches abi3 wheels (Linux x86_64/aarch64, macOS
> universal2,
> > > Windows x86_64; CPython 3.9+) plus a source distribution. Grab the
> wheel
> > > for
> > > your platform from the release page and install it directly, e.g.:
> > >
> > > pip install
> https://github.com/apache/spark-connect-rust/releases/download/v4.2.0-rc2/pyspark_client_rust-4.2.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
> > >
> > > Then use it exactly like PySpark. See if your existing workloads work
> with
> > > this.
> > > HOW TO TEST IT — RUST (cargo)
> > >
> > > The Rust crates are intentionally NOT on crates.io for a release
> candidate
> > > (crates.io is immutable; the crates are published there only when the
> > > release is
> > > finalized). Test against the tagged source instead:
> > >
> > > # A) Pin a git dependency to the RC tag in your project's Cargo.toml:
> > > [dependencies]
> > > apache-spark-connect = { git = "
> https://github.com/apache/spark-connect-rust", tag = "v4.2.0-rc2" }
> > >
> > > # B) Or build and run the crates' own tests from the tag:
> > > git clone --branch v4.2.0-rc2
> https://github.com/apache/spark-connect-rust
> > > cd spark-connect-rust
> > > cargo build --release
> > > cargo test
> > >
> > > HIGHLIGHTS
> > >
> > > - *Drop-in PySpark Connect Python API*, verified against real
> PySpark
> > > 4.2.0 two
> > > ways: the official PySpark test suite runs against this client, and
> a
> > > behavioral API-parity sweep compares the Python API surface
> (functions
> > > and
> > > DataFrame / Column / SparkSession / GroupedData / Window methods)
> > > call-by-call
> > > to PySpark.
> > > - Spark Connect Rust API *full* coverage, including *Rust UDF*s
> > > (docs: https://apache.github.io/spark-connect-rust/).
> > > - *Native Rust transport* via tonic; cloudpickle vendored to match
> > > upstream, so
> > > no grpcio/py4j at runtime. and *native type conversion.*
> > >
> > > WHY DOES THE VERSION START AT 4.2.0?
> > >
> > > Even though this is the first release of the rewrite, the version
> starts at
> > > 4.2.0 to track the Apache Spark / PySpark version it targets. This (1)
> > > removes
> > > any ambiguity about which Spark version a given client is compatible
> with,
> > > and
> > > (2) lets PySpark users pip install pyspark-client-rust== exactly
> > > as they would pyspark-client, keeping the same versioning across both.
> > >
> > > Thanks,
> > > Hyukjin Kwon
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: [VOTE] Release Apache Spark Connect Rust Client (+ pyspark-client in Rust) 4.2.0 (RC2)
Posted by Dongjoon Hyun <do...@apache.org>.
+1
Dongjoon.
On 2026/09/01 22:08:01 Hyukjin Kwon wrote:
> Starting with my own +1.
>
> PS: we have a discussion in the dev mailing list, and I am sync'ed with the
> original authors.
>
> On Wed, 2 Sept 2026 at 07:07, Hyukjin Kwon <gu...@apache.org> wrote:
>
> > Please vote on releasing the following candidate as Apache Spark Connect
> > Rust
> > Client 4.2.0 RC2.
> >
> > This is the *first* release of the ground-up Rust rewrite of the Spark
> > Connect
> > Python client (published to PyPI as pyspark-client-rust): a drop-in
> > replacement
> > for pyspark-client <https://pypi.org/project/pyspark-client/>, backed by
> > a native Rust core instead of py4j/grpcio.
> >
> > The vote is open until Thursday, September 7, 2026, 5:00 PM PDT and passes
> > if a majority of +1 PMC votes are cast, with a minimum of
> > 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark Connect Rust Client 4.2.0
> > [ ] -1 Do not release this package because …
> >
> > TAG:
> > https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2
> > (commit: ce1fa08)
> >
> > RELEASE FILES (wheels + sdist, built and attached by GitHub Actions):
> > https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2
> >
> > DOCUMENTATION:
> > https://apache.github.io/spark-connect-rust/
> >
> > LIST OF ISSUES:
> > https://issues.apache.org/jira/projects/SPARK/versions/12357484
> > HOW TO TRY IT — PYTHON (pip)
> >
> > The release attaches abi3 wheels (Linux x86_64/aarch64, macOS universal2,
> > Windows x86_64; CPython 3.9+) plus a source distribution. Grab the wheel
> > for
> > your platform from the release page and install it directly, e.g.:
> >
> > pip install https://github.com/apache/spark-connect-rust/releases/download/v4.2.0-rc2/pyspark_client_rust-4.2.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
> >
> > Then use it exactly like PySpark. See if your existing workloads work with
> > this.
> > HOW TO TEST IT — RUST (cargo)
> >
> > The Rust crates are intentionally NOT on crates.io for a release candidate
> > (crates.io is immutable; the crates are published there only when the
> > release is
> > finalized). Test against the tagged source instead:
> >
> > # A) Pin a git dependency to the RC tag in your project's Cargo.toml:
> > [dependencies]
> > apache-spark-connect = { git = "https://github.com/apache/spark-connect-rust", tag = "v4.2.0-rc2" }
> >
> > # B) Or build and run the crates' own tests from the tag:
> > git clone --branch v4.2.0-rc2 https://github.com/apache/spark-connect-rust
> > cd spark-connect-rust
> > cargo build --release
> > cargo test
> >
> > HIGHLIGHTS
> >
> > - *Drop-in PySpark Connect Python API*, verified against real PySpark
> > 4.2.0 two
> > ways: the official PySpark test suite runs against this client, and a
> > behavioral API-parity sweep compares the Python API surface (functions
> > and
> > DataFrame / Column / SparkSession / GroupedData / Window methods)
> > call-by-call
> > to PySpark.
> > - Spark Connect Rust API *full* coverage, including *Rust UDF*s
> > (docs: https://apache.github.io/spark-connect-rust/).
> > - *Native Rust transport* via tonic; cloudpickle vendored to match
> > upstream, so
> > no grpcio/py4j at runtime. and *native type conversion.*
> >
> > WHY DOES THE VERSION START AT 4.2.0?
> >
> > Even though this is the first release of the rewrite, the version starts at
> > 4.2.0 to track the Apache Spark / PySpark version it targets. This (1)
> > removes
> > any ambiguity about which Spark version a given client is compatible with,
> > and
> > (2) lets PySpark users pip install pyspark-client-rust== exactly
> > as they would pyspark-client, keeping the same versioning across both.
> >
> > Thanks,
> > Hyukjin Kwon
> >
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [VOTE] Release Apache Spark Connect Rust Client (+ pyspark-client in Rust) 4.2.0 (RC2)
Posted by Hyukjin Kwon <gu...@apache.org>.
Starting with my own +1.
PS: we have a discussion in the dev mailing list, and I am sync'ed with the
original authors.
On Wed, 2 Sept 2026 at 07:07, Hyukjin Kwon <gu...@apache.org> wrote:
> Please vote on releasing the following candidate as Apache Spark Connect
> Rust
> Client 4.2.0 RC2.
>
> This is the *first* release of the ground-up Rust rewrite of the Spark
> Connect
> Python client (published to PyPI as pyspark-client-rust): a drop-in
> replacement
> for pyspark-client <https://pypi.org/project/pyspark-client/>, backed by
> a native Rust core instead of py4j/grpcio.
>
> The vote is open until Thursday, September 7, 2026, 5:00 PM PDT and passes
> if a majority of +1 PMC votes are cast, with a minimum of
> 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark Connect Rust Client 4.2.0
> [ ] -1 Do not release this package because …
>
> TAG:
> https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2
> (commit: ce1fa08)
>
> RELEASE FILES (wheels + sdist, built and attached by GitHub Actions):
> https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2
>
> DOCUMENTATION:
> https://apache.github.io/spark-connect-rust/
>
> LIST OF ISSUES:
> https://issues.apache.org/jira/projects/SPARK/versions/12357484
> HOW TO TRY IT — PYTHON (pip)
>
> The release attaches abi3 wheels (Linux x86_64/aarch64, macOS universal2,
> Windows x86_64; CPython 3.9+) plus a source distribution. Grab the wheel
> for
> your platform from the release page and install it directly, e.g.:
>
> pip install https://github.com/apache/spark-connect-rust/releases/download/v4.2.0-rc2/pyspark_client_rust-4.2.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
>
> Then use it exactly like PySpark. See if your existing workloads work with
> this.
> HOW TO TEST IT — RUST (cargo)
>
> The Rust crates are intentionally NOT on crates.io for a release candidate
> (crates.io is immutable; the crates are published there only when the
> release is
> finalized). Test against the tagged source instead:
>
> # A) Pin a git dependency to the RC tag in your project's Cargo.toml:
> [dependencies]
> apache-spark-connect = { git = "https://github.com/apache/spark-connect-rust", tag = "v4.2.0-rc2" }
>
> # B) Or build and run the crates' own tests from the tag:
> git clone --branch v4.2.0-rc2 https://github.com/apache/spark-connect-rust
> cd spark-connect-rust
> cargo build --release
> cargo test
>
> HIGHLIGHTS
>
> - *Drop-in PySpark Connect Python API*, verified against real PySpark
> 4.2.0 two
> ways: the official PySpark test suite runs against this client, and a
> behavioral API-parity sweep compares the Python API surface (functions
> and
> DataFrame / Column / SparkSession / GroupedData / Window methods)
> call-by-call
> to PySpark.
> - Spark Connect Rust API *full* coverage, including *Rust UDF*s
> (docs: https://apache.github.io/spark-connect-rust/).
> - *Native Rust transport* via tonic; cloudpickle vendored to match
> upstream, so
> no grpcio/py4j at runtime. and *native type conversion.*
>
> WHY DOES THE VERSION START AT 4.2.0?
>
> Even though this is the first release of the rewrite, the version starts at
> 4.2.0 to track the Apache Spark / PySpark version it targets. This (1)
> removes
> any ambiguity about which Spark version a given client is compatible with,
> and
> (2) lets PySpark users pip install pyspark-client-rust== exactly
> as they would pyspark-client, keeping the same versioning across both.
>
> Thanks,
> Hyukjin Kwon
>