You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "xinrong-meng (via GitHub)" <gi...@apache.org> on 2025/01/14 08:55:44 UTC
[PR] [SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
xinrong-meng opened a new pull request, #49482:
URL: https://github.com/apache/spark/pull/49482
### What changes were proposed in this pull request?
Turn on Arrow optimization for Python UDFs by default
### Why are the changes needed?
Arrow optimization was introduced in 3.4.0. See [SPARK-40307](https://issues.apache.org/jira/browse/SPARK-40307) for more context.
Arrow-optimized Python UDF is approximately 1.6 times faster than the original pickled Python UDF. More details can be found in [this blog post](https://www.databricks.com/blog/arrow-optimized-python-udfs-apache-sparktm-35).
In version 4.0.0, we propose enabling the optimization by default. If PyArrow is not installed, it will fall back to the original pickled Python UDF.
### Does this PR introduce _any_ user-facing change?
Yes
### How was this patch tested?
Existing tests
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #49482:
URL: https://github.com/apache/spark/pull/49482#discussion_r1942153333
##########
python/pyspark/sql/utils.py:
##########
@@ -71,6 +71,15 @@
pass
+has_arrow: bool = False
Review Comment:
Good point! Resolved, thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #49482:
URL: https://github.com/apache/spark/pull/49482#discussion_r1942147785
##########
python/pyspark/sql/utils.py:
##########
@@ -71,6 +71,15 @@
pass
+has_arrow: bool = False
Review Comment:
Can we address this comment @xinrong-meng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #49482: [SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default
URL: https://github.com/apache/spark/pull/49482
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on PR #49482:
URL: https://github.com/apache/spark/pull/49482#issuecomment-2644331586
Failed tests seem irrelevant:
```
[info] MySQLNamespaceSuite:
[info] org.apache.spark.sql.jdbc.v2.MySQLNamespaceSuite *** ABORTED *** (10 seconds, 369 milliseconds)
[info] com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"driver failed programming external connectivity on endpoint condescending_lumiere (7a051f139d30436d8a1e231e3f4aeb991784c60ddc1f391ee8c1fc587c8ce2ca): Error starting userland proxy: listen tcp4 0.0.0.0:39901: bind: address already in use"}
```
Retriggering tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on PR #49482:
URL: https://github.com/apache/spark/pull/49482#issuecomment-2611368289
A quick update: the PR is blocked by UDT support in Arrow Python UDF, which I’m currently working on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #49482:
URL: https://github.com/apache/spark/pull/49482#issuecomment-2699566083
There are some subtle diff found. I will revert this for now, and enable it back later after improving this more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #49482:
URL: https://github.com/apache/spark/pull/49482#issuecomment-2611621015
Thank you for the updated context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on PR #49482:
URL: https://github.com/apache/spark/pull/49482#issuecomment-2652683313
Thank you @dongjoon-hyun !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #49482:
URL: https://github.com/apache/spark/pull/49482#discussion_r1947396399
##########
python/pyspark/sql/utils.py:
##########
@@ -71,6 +71,15 @@
pass
+has_arrow: bool = False
Review Comment:
probably we can import it inside the function/class body
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #49482:
URL: https://github.com/apache/spark/pull/49482#discussion_r1943588933
##########
python/pyspark/sql/utils.py:
##########
@@ -71,6 +71,15 @@
pass
+has_arrow: bool = False
Review Comment:
There will be a circular import if we do that. Let me follow up with a separate PR instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #49482:
URL: https://github.com/apache/spark/pull/49482#issuecomment-2633606849
Let's make this ready ASAP for 4.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on PR #49482:
URL: https://github.com/apache/spark/pull/49482#issuecomment-2610844943
Thanks @dongjoon-hyun I just got some free cycles for that and will resolve it ASAP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on PR #49482:
URL: https://github.com/apache/spark/pull/49482#issuecomment-2632534151
Thanks @dongjoon-hyun for attention! The current proposal is to fall back to the existing (non-Arrow-optimized) Python UDF when UDT is involved. My understanding is that no further testing is needed and the code change is minimal (just an if-else), but I respect the community’s decision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on PR #49482:
URL: https://github.com/apache/spark/pull/49482#issuecomment-2632535147
I marked it as WIP because I wanted to file a separate PR for the fallback mechanism with tests. Once that PR is in, this PR will be unblocked immediately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #49482:
URL: https://github.com/apache/spark/pull/49482#issuecomment-2608097024
Gentle ping, @xinrong-meng .
If this is targeting Apache Spark 4.0, we had better have this before February 1st.
- https://spark.apache.org/versioning-policy.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #49482:
URL: https://github.com/apache/spark/pull/49482#discussion_r1942148189
##########
python/pyspark/sql/utils.py:
##########
@@ -71,6 +71,15 @@
pass
+has_arrow: bool = False
Review Comment:
Good point! I think it’s more intuitive in the other direction, having the tests reference sql/utils instead. Let me file a follow-up for easier review, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #49482:
URL: https://github.com/apache/spark/pull/49482#issuecomment-2638349882
Can you take a look at the test failure to make sure? I think those failures look related.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #49482:
URL: https://github.com/apache/spark/pull/49482#discussion_r1914465635
##########
python/pyspark/sql/utils.py:
##########
@@ -71,6 +71,15 @@
pass
+has_arrow: bool = False
Review Comment:
I think we can use `from pyspark.testing.utils import have_pyarrow`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #49482:
URL: https://github.com/apache/spark/pull/49482#discussion_r1942148189
##########
python/pyspark/sql/utils.py:
##########
@@ -71,6 +71,15 @@
pass
+has_arrow: bool = False
Review Comment:
Good point! I think it’s more intuitive in the other direction, having the tests reference sql/utils instead. Let me file a follow-up for easier review, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on PR #49482:
URL: https://github.com/apache/spark/pull/49482#issuecomment-2635536415
The Arrow fallback PR is in so this PR should be unblocked. I’ll keep an eye on testing and make it ready ASAP!
Thank you @HyukjinKwon @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #49482:
URL: https://github.com/apache/spark/pull/49482#issuecomment-2635538080
can we address https://github.com/apache/spark/pull/49482#discussion_r1914465635 @xinrong-meng ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org