You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2026/08/24 04:03:08 UTC
[PR] [SPARK-58960] Drop-in Python (pyspark) wrapper over the Rust core [spark-connect-rust]
HyukjinKwon opened a new pull request, #51:
URL: https://github.com/apache/spark-connect-rust/pull/51
## Summary
Layers a thin **drop-in Python (`pyspark`) wrapper** over the Rust Spark Connect client so existing Spark Connect code runs unchanged. Published to PyPI as `pyspark-client-rust`; the import package stays `pyspark`.
> **Stacked on #50** (the Rust core). This PR is based on the `split-rust-core` branch, so its diff shows only the Python additions. It will be retargeted to `master` once #50 merges.
## What's in this PR
- **`crates/pyspark-rs`** — PyO3 bindings that build the `_pyspark` extension module, wrapping the pure-Rust `spark-connect` API (`publish = false`; shipped as a wheel).
- **`python/pyspark/**`** — the drop-in `pyspark` package (thin wrapper + vendored `pyspark.pandas` / `cloudpickle` / `pyspark.testing` for compatibility).
- **`pyproject.toml`** — maturin build; distribution name `pyspark-client-rust`.
- **`.cargo/config.toml`** — macOS link flags for the CPython extension.
- **CI** — `build_python_connect.yml` runs the reference-vs-ours parity gate and an end-to-end benchmark against a real Spark 4.2.0 Connect server; `release.yml` extended to build abi3 wheels and publish to PyPI (committer-fork model).
- Docs, the parity ledger, and the official-test harness scripts.
## Verification
- `cargo build -p pyspark-rs` compiles the extension; core `cargo test` still passes (21 suites).
- The wrapper imports under Python 3.11 with 444 functions exposed; it is a complete drop-in for the reference `pyspark` Spark Connect client.
JIRA: https://issues.apache.org/jira/browse/SPARK-58960
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58960] Drop-in Python (pyspark) wrapper over the Rust core [spark-connect-rust]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #51:
URL: https://github.com/apache/spark-connect-rust/pull/51#issuecomment-5394462146
To ground the plan-building coverage against a concrete Spark version: the golden-proto tests that validate `plan.rs` are captured from the **Apache Spark 4.2.0** reference client, so every plan / expression / function they cover is asserted **byte-for-byte** against what pyspark 4.2.0 emits on the wire.
So the two paths are complementary rather than one substituting for the other:
- **golden-proto tests** — validate our plan building against Spark 4.2.0's exact protobufs (this is where `plan.rs` is covered), and
- **official connect suite** (via `rust_transport_plugin.py`) — validates the transport + Arrow round-trip against a live 4.2.0 server.
You're right that the official-suite path routes plan-building through upstream pyspark, so it isn't what covers `plan.rs` — the golden tests are. The honest remaining gap is that the 4.2.0 golden capture doesn't yet cover the newly-added items (the +100 functions and the closure/UDTF/streaming methods added in this stack); extending the capture to those is the way to close it, and that's tracked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58960] Drop-in Python (pyspark) wrapper over the Rust core [spark-connect-rust]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #51: [SPARK-58960] Drop-in Python (pyspark) wrapper over the Rust core
URL: https://github.com/apache/spark-connect-rust/pull/51
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58960] Drop-in Python (pyspark) wrapper over the Rust core [spark-connect-rust]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #51:
URL: https://github.com/apache/spark-connect-rust/pull/51#issuecomment-5403248305
Addressed the assisted-review findings on this PR. Commit on `HyukjinKwon:split-python-wrapper`.
**1. [High] All server errors collapsed to `RuntimeError`** — fixed. `crates/pyspark-rs/src/errors.rs` now maps `SparkError.kind` to the corresponding `pyspark.errors` class — `Analysis → AnalysisException`, `Parse → ParseException`, `IllegalArgument → IllegalArgumentException`, `QueryExecution → QueryExecutionException`, `Python → PythonException`, `StreamingQuery → StreamingQueryException`, and the builtin-style kinds to their `PySpark*` subclasses (which also subclass the Python builtins), etc. It imports `pyspark.errors` under the GIL and constructs the typed exception, falling back to `PyRuntimeError` only if the import fails. So `except AnalysisException:` / `ParseException` / `IllegalArgumentException` catch server errors again. All 29 mapped class names verified present in the vendored `pyspark.errors`.
**2. [High] Streaming listeners didn't deliver live events** — fixed (root cause on #50). `python/pyspark/sql/streaming/query.py`'s `StreamingQueryListenerBus` daemon thread now loops the **incremental** `streamListenerEvents()` iterator and dispatches `onQueryStarted`/`onQueryProgress`/`onQueryIdle`/`onQueryTerminated` as each event arrives; it stops when the last listener is removed. Critically, `PyListenerEventStream.__next__` releases the GIL (`py.detach`) around the blocking wait — otherwise the daemon thread would freeze the interpreter between events (the same GIL concern from the earlier round, applied to the listener bus).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58960] Drop-in Python (pyspark) wrapper over the Rust core [spark-connect-rust]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #51:
URL: https://github.com/apache/spark-connect-rust/pull/51#issuecomment-5393949844
Thanks — all the wrapper-specific findings are addressed (pushed here); the core findings were fixed on #50 and this branch is now genuinely stacked on it.
**Stack is real now.** Re-synced this branch's base to #50's tip, so the diff is just the Python wrapper surface (no duplicated Rust core).
**Plan-builder bugs reaching the wheel.** Fixed at the source on #50 (`dropna`/`hint`/`replace`/`pivot`/`fillna`, plus `createTempView`, the caching family, `LocalRelation` schema, etc.), so the wheel inherits the fixes.
**Harness overclaiming parity.** You're exactly right that `rust_transport_plugin.py` routes plan-building through upstream pyspark, so the official suite exercises **transport + Arrow**, not our `plan.rs`. Updated the README CI section and added a "coverage boundary" note in `docs/design/OFFICIAL_TESTS_HARNESS.md`: parity is golden-proto tests (plan building) **plus** the official suite (transport/Arrow); neither alone is sufficient. No longer presented as end-to-end parity for the plan layer.
**GIL across blocking RPCs — fixed.** Wrapped the `block_on` sites in `transport.rs` (connect/execute_plan/reattach/release/analyze/config/interrupt/fetch_error_details) and the blocking `PyDataFrame` actions (`collect`/`count`/`show`/`first`/`head`/`take`/`toArrow`) in `py.detach`, matching what `ResponseStream::__next__` already did — so Python threads no longer serialize on network round-trips. Also `value_to_py` now returns `PyResult` and propagates with `?` instead of `.unwrap()` panicking across FFI.
**pandas surface.** `python/pyspark/pandas/**` is vendored verbatim from Apache Spark (license headers intact); what this PR adds is only the entry-point bridges (`DataFrame.pandas_api()` / `plot`) into it, not a reimplementation — so it carries the usual per-release sync obligation, and the "golden tests don't cover it" caveat is noted.
Appreciate the review — the transport-seam feedback in particular made the harness's real (and its non-)coverage explicit in the docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58960] Drop-in Python (pyspark) wrapper over the Rust core [spark-connect-rust]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #51:
URL: https://github.com/apache/spark-connect-rust/pull/51#issuecomment-5401469609
Thanks — the GIL item and the provenance ask are both addressed. Commit on `HyukjinKwon:split-python-wrapper`.
## `session.rs` GIL
You were right that it read as a missed file. Every blocking RPC in `session.rs` now releases the GIL via `py.detach(...)`, matching `dataframe.rs`/`transport.rs` (9 detach sites):
- `get_or_create` (the connect handshake — the one you singled out),
- `interrupt_all` / `interrupt_tag` / `interrupt_operation` — so the cancellation path is available while a worker thread blocks, which was the sharpest concern,
- `stop`, `version`, `add_artifact` / `add_artifacts`, `build_resource_profile`.
Two you listed that I deliberately did **not** change, because in this client they don't do I/O:
- `sql` — returns a lazy `DataFrame` wrapping `LogicalPlan::Sql`; it's infallible and issues no AnalyzePlan. The round-trip happens on the action (`collect`/`show`/…), which already detaches in `dataframe.rs`. (This differs from reference pyspark, where `spark.sql()` can execute eagerly for commands — here `sql` is a pure transformation.)
- `table` — likewise lazy (`read().table()`), no RPC.
If you'd rather `sql`/`table` detach defensively anyway I can add it, but as written there's no blocking call to wrap.
On the two-threads regression test (one slow query, assert the other progresses): agreed it's the right test and that the parity suite won't exercise it since it's effectively single-threaded. It needs a live server, so I can't run it in the offline/parity harness here; tracking it to add once the fork CI has a server, and it would also guard the `dataframe.rs` detaches from regressing.
## Provenance of the vendored trees
Added a "Provenance" section to the PR description. Short version: `pyspark.pandas`, `pyspark.testing`, `cloudpickle`, `pyspark.resource/{__init__,information,requests}.py`, and `pyspark.sql.datasource.py` are **verbatim** from Apache Spark with headers intact (re-synced from the matching release tag). The **adapted** files — where the interesting diff lives — are `pyspark/resource/profile.py` (routes `ResourceProfile.id` through the `_pyspark` extension instead of a JVM builder), the `pyspark/sql/streaming/{query,readwriter}.py` re-exports + client-side listener bus, and the minimal `pyspark/sql/streaming/{listener,datasource}.py` subsets.
## WASM packer / own PR
Same as on #50: keeping WASM in the stack is the project owner's call for now, and the encode/decode round-trip test per `AbiType` (incl. null/empty-array) belongs in the `wasm-udf` CI job that has `wasmtime` + a compiled module — adding it there as a follow-up rather than a test that can't actually round-trip.
The newer surface you flagged as un-reviewed (streaming, `toLocalIterator`, `ResourceProfile`, `dataSource.register`, avro/protobuf/sha2/window dispatch, datetime/Decimal literals) — no expectation that you've signed off on it; happy to walk through any of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58960] Drop-in Python (pyspark) wrapper over the Rust core [spark-connect-rust]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #51:
URL: https://github.com/apache/spark-connect-rust/pull/51#issuecomment-5398144026
Third pass. One item from my last comment is still open, and it's the case I'd previously singled out as the worst of the set.
## Still open: `session.rs` holds the GIL
`crates/pyspark-rs/src/session.rs` contains **zero** `detach` calls, while `dataframe.rs` and `transport.rs` were both fixed thoroughly. Given how consistent those two are, this reads as a missed file rather than a decision.
The methods that block while holding the GIL:
- **`get_or_create` (line 50)** — the one I flagged last time. It runs the connect handshake with the GIL held, so a server that isn't answering freezes the whole interpreter, signal handling included; Ctrl-C may not land until the timeout expires.
- **`sql` (line 115)** — issues an AnalyzePlan round-trip.
- `stop`, `version`, `table`, `interrupt_all` / `interrupt_tag` / `interrupt_operation`, `add_artifact` / `add_artifacts`, `build_resource_profile`. I confirmed `interrupt_all` and `add_artifacts` reach `block_on(self.client...)` in the core, so those are unambiguous.
`interrupt_*` is the sharpest in practice, and worth calling out as more than a performance nit. Interrupt exists to cancel a query running on another thread — but the caller can't acquire the GIL to *invoke* it while the blocked worker holds it. So the cancellation path is liable to be unavailable exactly when a user needs it. (`transport.rs`'s `interrupt` does detach; it's the session-level wrapper that doesn't.)
Same fix as `dataframe.rs`: thread a `py: Python<'_>` parameter through and wrap the call in `py.detach(...)`.
This won't surface in the official suite, which is effectively single-threaded, so it needs a deliberate test — two threads, one slow query, assert the other makes progress — rather than relying on the parity gate. That test would also protect the `dataframe.rs` fixes from regressing.
## Confirmed fixed since last pass
- **The stack is real.** `git merge-base --is-ancestor` now succeeds against #50's branch, so fixes there flow here.
- **GIL released across the transport RPCs** — all of `connect`, `execute_plan`, `reattach_execute`, `release_execute`, `analyze_plan`, `config`, `interrupt`, `fetch_error_details` go through `py.detach(...)`.
- **`PyDataFrame`** — `collect`, `count`, `show`, `first`, `head`, `take`, `to_arrow`, `to_local_iterator`, and the iterator's `next` all detach.
- **`row.rs`** is down to one `unwrap`, so the FFI-unwind concern is essentially closed.
- The plan-builder bugs are fixed on #50, so the published wheel no longer inherits them.
## On the WASM UDF packer
The Python side of the WASM UDF feature lives here (`python/pyspark_wasm_udf/`), with the Rust half on #50. I've put my substantive feedback on that PR since the crates are there, but two points touch this one:
The runner in `__init__.py` does the actual work — encoding each row's arguments into wasm linear memory, calling the entrypoint, decoding the result — and it's hand-rolled pointer arithmetic (`packed >> 32`, `& 0xFFFFFFFF`, explicit `alloc`/`dealloc`). It has no tests. A round-trip test per `AbiType`, including the null and empty-array cases, would be cheap relative to the risk; a decode bug here produces silently wrong column values rather than an error.
I'd also suggest the whole feature move to its own PR stacked on this one. It's a distinct capability from "drop-in pyspark wrapper," it spans both PRs, and it carries a deployment prerequisite (`wasmtime` on every executor) that a wrapper PR shouldn't be quietly introducing. Reviewing the ABI and the by-value cloudpickling on their own merits would be easier there, and it'd be revertible without touching the wrapper.
## On the newer surface
Structured Streaming exposure, `toLocalIterator`, `ResourceProfile`, `dataSource.register`, avro/protobuf/sha2/window dispatch, datetime/Decimal literals, and the vendored `pyspark.resource` / `pyspark.sql.datasource` / `pyspark.pandas` trees all arrived across the last two passes. I haven't reviewed that in depth and don't want to imply otherwise.
The vendored trees still raise the provenance question from my first comment: whether they're byte-identical to upstream (so license headers and a per-release re-sync story are the concern) or modified (in which case the diffs are the interesting part). A line in the PR description stating which, and how they get re-synced when Spark cuts a release, would save every future reviewer from re-deriving it — and it's the kind of thing an ASF release audit will ask about anyway.
Finally, on the test-harness boundary from my first comment: now that the plan-builder bugs are fixed on #50, this is no longer urgent, but the structural point stands — the official suite builds plans with upstream pyspark and never executes `plan.rs`, so it can't defend the layer that ships. I've left that discussion with the golden-test item on #50 rather than duplicating it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58960] Drop-in Python (pyspark) wrapper over the Rust core [spark-connect-rust]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #51:
URL: https://github.com/apache/spark-connect-rust/pull/51#issuecomment-5390804504
This implements ALL API in PySpark connect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58960] Drop-in Python (pyspark) wrapper over the Rust core [spark-connect-rust]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #51:
URL: https://github.com/apache/spark-connect-rust/pull/51#issuecomment-5392429440
Re-reviewed this as PART 2 of the split. The Rust-core findings from #49 are on #50 where they belong; this comment covers what's specific to the Python wrapper and the test harness. I verified each against this branch — all still apply.
## This branch isn't actually stacked on #50
It declares `split-rust-core` as its base, but the two branches share no lineage: `git merge-base --is-ancestor pr-50 pr-51` fails, and both fork independently from `4f43dfa` on master. This branch carries its own complete copy of the Rust core — all 21 files under `crates/spark-connect/src`, plus `release.yml` and `.asf.yaml`.
GitHub's diff view makes it look like a clean follow-up, which hides two problems: merging #50 and then this would conflict or silently revert core changes depending on order, and any fix pushed to #50 won't appear here. A rebase onto #50's branch would make the stack real and shrink this diff to just the Python surface.
Also worth flagging that the split wasn't purely mechanical: the final state differs from #49 by roughly +3,800/-900 across 27 files, including a new `crates/spark-connect/src/merge.rs` and ~800 new lines in `functions.rs`. That new surface hasn't been reviewed before, so #49's review doesn't transfer wholesale.
## The plan-builder bugs reach the published wheel
`python/pyspark/sql/dataframe.py` is 28 lines whose substance is `from pyspark._pyspark import DataFrame` — it re-exports the Rust `PyDataFrame` directly. So a user who `pip install`s `pyspark-client-rust` and calls `df.dropna(how="all")` goes through `pyspark-rs` → `spark-connect` → the plan builder, and inherits every silent-argument-drop bug I filed on #50 (`dropna`'s `how`, `hint`'s `parameters`, `replace` on non-numerics, explicit-value `pivot`).
For something positioned as a drop-in `pyspark` replacement, "same API call, quietly different answer" is the worst available failure mode — worse than an error, because nothing tells the user to look. The fixes belong on #50; I'm noting it here so the wheel's exposure is visible on the PR that ships it.
## The official test harness structurally cannot catch those bugs
The parity harness is the main quality argument for this PR, but it exercises a different plan builder than the one that ships.
`scripts/rust_transport_plugin.py` monkeypatches the upstream `SparkConnectClient` so its gRPC stub becomes `RustConnectStub`. In that path **upstream pyspark builds the plan** — `_RustStub.ExecutePlan(request)` receives an upstream protobuf object and calls `request.SerializeToString()`, and the bytes go out through `execute_plan_raw` + `BytesCodec` untouched. Rust's job there is transport and Arrow decoding only; `crates/spark-connect/src/plan.rs` never executes.
So the paths diverge exactly where it matters:
| Path | Who builds the plan | Exercises `plan.rs`? |
|---|---|---|
| Rust-native (`examples`) | Rust `spark-connect` | yes |
| Published wheel (`python/pyspark/`) | Rust, via `_pyspark` | yes |
| Official suite (`rust_transport_plugin.py`) | upstream pyspark | **no** |
The code paths users run and the ones the official suite covers don't intersect on plan building. A green parity gate therefore says nothing about the `dropna`/`hint`/`replace`/`pivot` bugs — which is precisely why they reached review unnoticed.
I want to be clear this isn't an argument against the harness: routing the official suite through the Rust transport is a genuinely good idea and it does validate the transport and Arrow paths well. The problem is only that it's being presented as parity evidence for a layer it bypasses. Making that boundary explicit in the docs — and covering plan building with golden tests instead — would keep the harness's real value without overclaiming.
## The GIL is held across blocking RPCs
The streaming path gets this right: `ResponseStream::__next__` wraps the blocking read in `py.detach()` and races it against cancellation (`transport.rs:114`), and the comment there explains exactly why. The other call sites don't follow it — `transport.rs` lines 167, 193, 207, 220, 227, 235, 242, 251 (`connect`, `execute_plan`, `reattach_execute`, `release_execute`, `analyze_plan`, `config`, `interrupt`, `fetch_error_details`) all call `block_on(...)` while holding the GIL, as do the `spark-connect` crate's `collect`/`count`/`first`/`show` reached through `pyspark-rs`.
Each of those is a synchronous network round-trip, so while one is in flight every other Python thread is blocked. Concretely:
- Four threads each running `spark.sql(q).collect()` serialize completely instead of overlapping — threading buys nothing.
- A background thread running a slow query stalls the main thread, so even a `print` progress loop freezes and the program looks hung.
- `SessionBuilder.get_or_create` blocks the whole interpreter through the connect handshake, including signal handling, so a server that isn't answering can make Ctrl-C unresponsive until timeout.
This matters more than usual for a drop-in replacement: real pyspark uses `grpcio`, which releases the GIL properly, so the same user code that scales across threads on upstream pyspark will serialize here. It also won't show up in the official suite, which is largely single-threaded. The fix is mechanical — wrap each `block_on` the way `__next__` already does; the `spark-connect` methods need a `py` handle threaded through first.
Smaller item nearby: `value_to_py` (`crates/pyspark-rs/src/row.rs`) `.unwrap()`s every `into_pyobject`. These are near-infallible, but a panic there unwinds across the FFI boundary instead of raising a Python exception — `?` would be safer.
## On the `pandas` surface
This branch vendors a large `python/pyspark/pandas/**` tree (~80 files). I didn't audit it in depth, but it's worth stating what the intent is: whether it's vendored verbatim from upstream (in which case its provenance and license headers need to be exact, and it will need a sync story per Spark release) or modified. If any of it is stubbed against the Rust core, the same "golden tests don't cover it" gap applies there too.
Overall: the transport seam here is the strongest part of the change, and the strangler-fig approach is a smart way to get the official suite involved. My concerns are the unreal stack (this should be rebased onto #50), the wheel inheriting the plan-builder bugs, and the GIL behaviour — the last one being the item most likely to be reported as "the Rust client is slower than pyspark" once people run it under threads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58960] Drop-in Python (pyspark) wrapper over the Rust core [spark-connect-rust]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #51:
URL: https://github.com/apache/spark-connect-rust/pull/51#issuecomment-5397718149
Re-reviewed after the updates. Two of the three things I raised here are fixed, and I verified them in the code rather than trusting the commit messages. One is fixed only partially, in a way that leaves the case I'd called the worst still broken.
## Confirmed fixed
- **The stack is real now.** `git merge-base --is-ancestor` succeeds against #50's branch, so this genuinely builds on PART 1 and fixes pushed there will flow here. Thanks for rebasing.
- **The GIL is released across the transport RPCs.** Every `block_on` in `transport.rs` is now wrapped — `connect`, `execute_plan`, `reattach_execute`, `release_execute`, `analyze_plan`, `config`, `interrupt`, `fetch_error_details` all go through `py.detach(...)`, matching what `__next__` already did.
- **`PyDataFrame` too.** `collect`, `count`, `show`, `first`, `head`, `take`, `to_arrow`, `to_local_iterator`, and the iterator's `next` all detach. That's the hot path covered properly.
- **`row.rs`** is down to a single `unwrap`, so the FFI-unwind concern is largely gone.
## Still open: `session.rs` holds the GIL, including the case I flagged as worst
`crates/pyspark-rs/src/session.rs` contains **zero** `detach` calls, and several of its methods do network I/O:
- `get_or_create` (line 50) — calls `builder.get_or_create()` while holding the GIL. This is the specific case I singled out last time: it blocks the entire interpreter through the connect handshake, so a server that isn't answering makes the process unresponsive — signal handling included, so Ctrl-C may not land until timeout.
- `sql` (line 115) — `self.session.sql(query)` issues an AnalyzePlan round-trip.
- Plus `stop`, `version`, `table`, `interrupt_all` / `interrupt_tag` / `interrupt_operation`, `add_artifact` / `add_artifacts`, `build_resource_profile`. `interrupt_all` and `add_artifacts` reach `block_on(self.client...)` in the core, so they're unambiguously blocking.
`interrupt_*` is the sharpest of these in practice. Interrupt exists to cancel a query running on another thread, but the caller cannot acquire the GIL to *invoke* it while the blocked worker holds it — so on the current code the cancellation path is liable to be unavailable exactly when it's needed. (`transport.rs`'s `interrupt` does detach; it's the session-level wrapper that doesn't.)
The fix is the same shape as the one already applied to `dataframe.rs` — these methods need a `py: Python<'_>` parameter so they can wrap the call in `py.detach(...)`. Given how consistently `dataframe.rs` was fixed, I suspect `session.rs` was simply missed rather than deliberately skipped.
Worth noting neither the old nor the new behaviour here is observable in the official suite, which is effectively single-threaded — so this needs a deliberate test (two threads, one slow query, assert the other makes progress) rather than relying on the parity gate.
## On the new surface
A lot arrived since the last pass — Structured Streaming exposure, `toLocalIterator`, `ResourceProfile`, `dataSource.register`, avro/protobuf/sha2/window dispatch, vendored `pyspark.resource` and `pyspark.sql.datasource`, datetime/Decimal literals, and a WASM UDF packer (`python/pyspark_wasm_udf/`). I haven't reviewed that in depth and I don't want to imply I have.
Two structural observations rather than defects:
The vendored trees (`pyspark.resource`, `pyspark.sql.datasource`, and the earlier `pyspark.pandas`) still raise the provenance question from my last comment: whether they're byte-identical to upstream (so license headers and a per-release sync story are what matter) or modified (in which case the diffs are the interesting part). A note in the PR description saying which, and how they get re-synced when Spark cuts a release, would save every future reviewer the same investigation.
And the WASM UDF work spans both PRs — `wasm_udf.rs` and the macro/build crates on #50, the Python packer here. It's a genuinely separate capability from "drop-in pyspark wrapper," with its own runtime prerequisite (`wasmtime` on the executors). It would be easier to evaluate, and easier to revert if it doesn't pan out, as its own PR on top of this stack. I've filed the workspace-membership problem it causes (`cargo test --workspace` fails without the wasm32 target) on #50, since the crates live there.
Nothing else from my earlier comment has changed status: the plan-builder bugs are fixed on #50, so the wheel no longer inherits them, and the test-harness boundary (the official suite builds plans with upstream pyspark and never executes `plan.rs`) still holds — I've left the follow-up on that with the golden-test discussion on #50.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58960] Drop-in Python (pyspark) wrapper over the Rust core [spark-connect-rust]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #51:
URL: https://github.com/apache/spark-connect-rust/pull/51#issuecomment-5403048203
**Assisted code review** (vendored Spark trees — `pandas`/`testing`/`cloudpickle` — skipped; focused on the PyO3 bindings and adapted wrappers).
**1. [High] All server errors collapse to `RuntimeError`** — `crates/pyspark-rs/src/errors.rs:8`
`spark_error_to_py_exception` maps **every** `SparkError` to `PyRuntimeError`, discarding the `error_class`/kind the Rust core carefully preserves, and nothing in the Python connect layer re-maps it. So `except AnalysisException:` / `ParseException` / `IllegalArgumentException` etc. won't catch server errors — a regression against the "complete drop-in for the reference pyspark client" claim. Map by `error_class`/`SparkErrorKind` to the corresponding `pyspark.errors` exception types.
**2. [High] Streaming query listeners don't deliver live events** (root cause in #50)
`python/pyspark/sql/streaming/query.py:_query_event_handler` calls `streamListenerEvents()` once; the underlying Rust `stream_listener_events()` (#50, `streaming.rs:869`) returns a batch only when the stream closes, so live progress/idle/terminated events never reach registered listeners. See #50 finding 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org