You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/07/07 23:23:19 UTC
[PR] [SPARK-58016][PYTHON] Revert Arrow-optimized Python UDFs and Arrow-based PySpark IPC by default [spark]
Yicong-Huang opened a new pull request, #57094:
URL: https://github.com/apache/spark/pull/57094
### What changes were proposed in this pull request?
This PR reverts [SPARK-54555](https://issues.apache.org/jira/browse/SPARK-54555) (commit ea0a35e065d, "Enable Arrow-optimized Python UDFs and Arrow-based PySpark IPC by default"), restoring the following config defaults to `false`:
- `spark.sql.execution.pythonUDF.arrow.enabled`
- `spark.sql.execution.pythonUDTF.arrow.enabled`
- `spark.sql.execution.arrow.pyspark.enabled` (via its fallback `spark.sql.execution.arrow.enabled`)
The associated user-facing documentation and migration-guide entries added by SPARK-54555 are reverted as well, and the `test_unified_udf` test is restored to no longer force the conf off.
The revert was applied with `git revert`; the only manual conflict resolution was in the migration guide and the Arrow/pandas tutorial, where subsequent PRs had touched adjacent lines. Only the entries introduced by SPARK-54555 were removed; unrelated 4.1-to-4.2 migration notes are preserved.
### Why are the changes needed?
Enabling these Arrow optimizations by default in 4.2.0 is being reverted so the change does not ship on by default in the 4.2.0 release. Users who want the Arrow-optimized behavior can still opt in explicitly via the same configs.
### Does this PR introduce _any_ user-facing change?
Yes. It restores the previous default configuration: regular Python UDFs, Python UDTFs, and PySpark Arrow-based columnar data exchange (`DataFrame.toPandas` / `SparkSession.createDataFrame`) are once again non-Arrow by default. This is a revert of an unreleased change (SPARK-54555 targeted 4.2.0), so there is no behavior change relative to released Spark versions.
### How was this patch tested?
Reverting existing changes; relies on existing PySpark test coverage that runs with the Arrow conf both enabled and disabled. `sql/catalyst` compiles.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58016][PYTHON] Revert Arrow-optimized Python UDFs and Arrow-based PySpark IPC by default [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #57094:
URL: https://github.com/apache/spark/pull/57094#issuecomment-4927676267
Closing this PR. Instead of reverting, we will fix forward to reduce the regression via:
- https://github.com/apache/spark/pull/57099
- https://github.com/apache/spark/pull/57104
- https://github.com/apache/spark/pull/57105
Note: the fix spans two parts. In addition to the PySpark-side changes above, a complementary fix is also needed in the Apache Arrow (PyArrow) community itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58016][PYTHON] Revert Arrow-optimized Python UDFs and Arrow-based PySpark IPC by default [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang closed pull request #57094: [SPARK-58016][PYTHON] Revert Arrow-optimized Python UDFs and Arrow-based PySpark IPC by default
URL: https://github.com/apache/spark/pull/57094
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org