You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2026/10/21 16:44:49 UTC
[PR] [SPARK-59718][SQL][4.x] Add opt-in in-process Python UDF execution via Arrow C Data Interface [spark]
viirya opened a new pull request, #59676:
URL: https://github.com/apache/spark/pull/59676
### What changes were proposed in this pull request?
Backport of #58978 to branch-4.x.
The cherry-pick conflicted only in the root `pom.xml`, where the new `jep.version` property sits next to `ammonite.version`, which differs on this branch (3.0.8 instead of 3.0.9). The branch's version is kept; otherwise the change is the same as on master. SPARK-59275 and SPARK-60071, which this change relies on, are already on this branch.
### Why are the changes needed?
Same as #58978. Worker-based Arrow UDFs exchange data between executor JVMs and separate Python processes, which can dominate execution time for large inputs or outputs with relatively inexpensive computation. In-process UDFs run CPython inside the executor JVM through JEP and exchange Arrow buffers through the C Data Interface.
### Does this PR introduce _any_ user-facing change?
Yes, as described in #58978: a new opt-in `pyspark.inprocess.inprocess_udf` API, the `spark.python.inProcess.sitePackages` and `spark.sql.execution.pythonUDF.inProcess.fullValidation.enabled` configurations, and the `InProcessPythonPlugin` executor plugin. JEP and `arrow-c-data` are `provided` dependencies that users install separately. Existing Python UDF execution is unchanged.
### How was this patch tested?
The tests added in #58978 pass on this branch with JDK 17: `InProcessPythonRuntimeSuite`, `InProcessPythonUDFSuite`, `InProcessArrowBridgeSuite`, `ConvertToCatalystSuite`, `CollapseProjectSuite` and `InvalidInputErrorsSuite`, as do `PythonUDFSuite`, `ArrowEvalPythonExecSuite`, `BatchEvalPythonExecSuite`, `ExtractPythonUDFsSuite`, `RowQueueSuite`, `CachedTableSuite` and `DatasetCacheSuite`, except `SPARK-27052` in `PythonUDFSuite`, which needs PyArrow in the system Python used by workers. The PySpark suites `test_inprocess_runtime` (50 tests) and `test_inprocess_udf` (83 tests) pass on macOS/Python 3.13 (PyArrow 23, JEP 4.3.1).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (review, implementation fixes, tests, CI integration, benchmarks, and documentation; model/version not recorded)
Generated-by: Claude Code (Claude Opus 5.5; performance fixes, review fixes, tests, benchmarks and this backport)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59718][SQL][4.x] Add opt-in in-process Python UDF execution via Arrow C Data Interface [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #59676:
URL: https://github.com/apache/spark/pull/59676#issuecomment-6343655386
Thank you @dongjoon-hyun !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59718][SQL][4.x] Add opt-in in-process Python UDF execution via Arrow C Data Interface [spark]
Posted by "spark-test (via GitHub)" <gi...@apache.org>.
spark-test commented on PR #59676:
URL: https://github.com/apache/spark/pull/59676#issuecomment-6340942338
The AI review requested by @HyukjinKwon did not finish. A committer can add the `ai-review-requested` label again to retry.
<!-- SPARK_OSS_REVIEW_REQUEST 33943535568 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59718][SQL][4.x] Add opt-in in-process Python UDF execution via Arrow C Data Interface [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #59676: [SPARK-59718][SQL][4.x] Add opt-in in-process Python UDF execution via Arrow C Data Interface
URL: https://github.com/apache/spark/pull/59676
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59718][SQL][4.x] Add opt-in in-process Python UDF execution via Arrow C Data Interface [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #59676:
URL: https://github.com/apache/spark/pull/59676#issuecomment-6340816759
**Merge Summary:**
- merged into branch-4.x https://github.com/apache/spark/commit/2bb7dd1f11e568fc79461330f1d7a746e6d20fa0
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org