You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2026/07/06 03:15:30 UTC

[PR] [SPARK-57948][PYTHON][CORE] Fix pipelined Python UDF data transfer over Unix domain sockets [spark]

HyukjinKwon opened a new pull request, #57023:
URL: https://github.com/apache/spark/pull/57023

   ### What changes were proposed in this pull request?
   
   [SPARK-56642](https://issues.apache.org/jira/browse/SPARK-56642) added pipelined
   JVM->Python UDF data transfer via `BasePythonRunner.createPipelinedDataIn`
   (`core/.../api/python/PythonRunner.scala`). That code reads the worker through the
   legacy blocking-socket API:
   
   ```scala
   worker.channel.socket().setSoTimeout(...)          // idle-timeout detection
   worker.channel.socket().getInputStream             // read side of the pipeline
   ```
   
   For a **Unix domain** `SocketChannel`, `SocketChannel.socket()` throws
   `java.lang.UnsupportedOperationException` — a UDS channel has no `java.net.Socket`
   adapter, and `SO_TIMEOUT` does not apply to it. So whenever
   `spark.python.unix.domain.socket.enabled=true` is combined with pipelined execution
   (`spark.python.udf.pipelined.enabled=true`), every pipelined Arrow/Python-UDF
   evaluation fails:
   
   ```
   java.lang.UnsupportedOperationException: Not supported
       at java.base/sun.nio.ch.SocketChannelImpl.socket(SocketChannelImpl.java:228)
       at org.apache.spark.api.python.BasePythonRunner.createPipelinedDataIn(PythonRunner.scala:447)
       at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:390)
       at ...ArrowEvalPythonEvaluatorFactory.evaluate(ArrowEvalPythonExec.scala:209)
   ```
   
   This PR makes the pipelined read path UDS-aware, mirroring the existing
   `if (!isUnixDomainSock)` handling already used elsewhere in `PythonRunner`
   (the sync-mode `ReaderInputStream` reads the channel directly and does idle-timeout
   via selector waits, not `SO_TIMEOUT`):
   
   - In UDS mode, read straight from the channel with
     `Channels.newInputStream(worker.channel)` and skip the `SO_TIMEOUT`-based
     idle-timeout wrapper (a blocking channel stream never raises
     `SocketTimeoutException`, so the idle-timeout branch is simply never taken —
     matching sync mode's behavior for UDS).
   - In TCP mode the behavior is unchanged.
   
   ### Why are the changes needed?
   
   The scheduled `Build / Unix Domain Socket (JDK 17, Scala 2.13)` (`build_uds`) master
   lane is red: `PYSPARK_UDS_MODE=true` turns on Unix domain sockets, and the pipelined
   UDF path then throws `UnsupportedOperationException` for every Arrow/Python UDF,
   failing the `pyspark-sql` module (12 errors + 1 downstream assertion failure) and the
   docs generation. Unix domain sockets are a supported transport, so this is a real
   regression, not a flake.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No — internal JVM<->Python transport only. It restores correct behavior when Unix
   domain sockets and pipelined UDF execution are both enabled.
   
   ### How was this patch tested?
   
   `pyspark.sql.tests.pandas.test_pipelined_udf` (forces
   `spark.python.udf.pipelined.enabled=true`) and
   `pyspark.sql.tests.arrow.test_arrow_python_udf` were run with `PYSPARK_UDS_MODE=true`
   — i.e. exactly the pipelined + Unix-domain-socket combination that was broken — on
   GitHub Actions:
   
   - **Before (apache/spark `master`, `build_uds`):** `pyspark-sql` FAILED with
     `UnsupportedOperationException: Not supported` at
     `createPipelinedDataIn(PythonRunner.scala:447)` —
     https://github.com/apache/spark/actions/runs/28488010762
   - **After (this change, fork validation):** PASS — https://github.com/HyukjinKwon/spark/actions/runs/28762213817 — `test_pipelined_udf` (pipelined mode asserted active) + `test_arrow_python_udf` both PASS under `PYSPARK_UDS_MODE=true`
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57948][PYTHON][CORE] Fix pipelined Python UDF data transfer over Unix domain sockets [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #57023:
URL: https://github.com/apache/spark/pull/57023#issuecomment-4888791449

   Closing in favor of #57024, which fixes the identical SPARK-56642 UDS + pipelined-UDF regression with a cleaner diff (opened ~2 seconds apart by another agent in the same multi-agent session). Convergence to avoid a duplicate PR for one bug.
   
   The validation I ran here still stands and I've added it to #57024: `test_pipelined_udf` + `test_arrow_python_udf` both pass under `PYSPARK_UDS_MODE=true` (https://github.com/HyukjinKwon/spark/actions/runs/28762213817). JIRA SPARK-57948 will be marked a duplicate of the SPARK-56642 follow-up.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57948][PYTHON][CORE] Fix pipelined Python UDF data transfer over Unix domain sockets [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #57023: [SPARK-57948][PYTHON][CORE] Fix pipelined Python UDF data transfer over Unix domain sockets
URL: https://github.com/apache/spark/pull/57023


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org