You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2025/03/05 02:21:20 UTC
[PR] [SPARK-51393][PYTHON] Fallback to regular Python UDF when Arrow is not found but Arrow-optimized Python UDFs enabled [spark]
HyukjinKwon opened a new pull request, #50160:
URL: https://github.com/apache/spark/pull/50160
### What changes were proposed in this pull request?
This PR extracts legitimate improvement in https://github.com/apache/spark/pull/49482. Falls back regular Python UDF when Arrow is not found but Arrow-optimized Python UDFs enabled.
### Why are the changes needed?
To minimize end user impact.
### Does this PR introduce _any_ user-facing change?
Yes, it falls back to regular Python UDF when Arrow is not found but Arrow-optimized Python UDFs enabled.
### How was this patch tested?
Manually tested.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51393][PYTHON] Fallback to regular Python UDF when Arrow is not found but Arrow-optimized Python UDFs enabled [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #50160:
URL: https://github.com/apache/spark/pull/50160#issuecomment-2702303740
Just for the record, PyPy 3.10 CIs have been broken in both `master` and `branch-4.0` after the reverting commit, https://github.com/apache/spark/commit/eb4855315049d2e8ab0105efbbd1a00f8c01aadb .
- https://github.com/apache/spark/actions/runs/13679068978
I raised SPARK-51393's priority to `Blocker` level.
cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51393][PYTHON] Fallback to regular Python UDF when Arrow is not found but Arrow-optimized Python UDFs enabled [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #50160:
URL: https://github.com/apache/spark/pull/50160#issuecomment-2702462302
Merged to master and branch-4.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51393][PYTHON] Fallback to regular Python UDF when Arrow is not found but Arrow-optimized Python UDFs enabled [spark]
Posted by "allisonwang-db (via GitHub)" <gi...@apache.org>.
allisonwang-db commented on code in PR #50160:
URL: https://github.com/apache/spark/pull/50160#discussion_r1982284472
##########
python/pyspark/sql/udf.py:
##########
@@ -128,6 +130,18 @@ def _create_py_udf(
else:
is_arrow_enabled = useArrow
+ if is_arrow_enabled:
+ try:
+ require_minimum_pandas_version()
+ require_minimum_pyarrow_version()
+ except ImportError:
+ is_arrow_enabled = False
+ warnings.warn(
+ "Arrow optimization failed to enable because PyArrow/pandas is not installed. "
Review Comment:
This looks good! But it would be great to make it consistent with above.
##########
python/pyspark/sql/connect/udf.py:
##########
@@ -77,6 +79,17 @@ def _create_py_udf(
eval_type: int = PythonEvalType.SQL_BATCHED_UDF
if is_arrow_enabled:
+ try:
+ require_minimum_pandas_version()
+ require_minimum_pyarrow_version()
+ except ImportError:
+ is_arrow_enabled = False
+ warnings.warn(
+ "Arrow optimization failed to enable because PyArrow is not installed. "
Review Comment:
nit: Looks like we also need Pandas here
```suggestion
"Arrow optimization failed to enable because PyArrow or Pandas is not installed. "
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51393][PYTHON] Fallback to regular Python UDF when Arrow is not found but Arrow-optimized Python UDFs enabled [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #50160: [SPARK-51393][PYTHON] Fallback to regular Python UDF when Arrow is not found but Arrow-optimized Python UDFs enabled
URL: https://github.com/apache/spark/pull/50160
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51393][PYTHON] Fallback to regular Python UDF when Arrow is not found but Arrow-optimized Python UDFs enabled [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #50160:
URL: https://github.com/apache/spark/pull/50160#discussion_r1981879585
##########
python/pyspark/sql/connect/udf.py:
##########
@@ -77,6 +79,17 @@ def _create_py_udf(
eval_type: int = PythonEvalType.SQL_BATCHED_UDF
if is_arrow_enabled:
+ try:
+ require_minimum_pandas_version()
+ require_minimum_pyarrow_version()
+ except ImportError:
+ is_arrow_enabled = False
+ warnings.warn(
Review Comment:
This seems to fail at linter. Could you check this, @HyukjinKwon ?
```
./python/pyspark/sql/connect/udf.py:87:13: F821 undefined name 'warnings'
warnings.warn(
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51393][PYTHON] Fallback to regular Python UDF when Arrow is not found but Arrow-optimized Python UDFs enabled [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #50160:
URL: https://github.com/apache/spark/pull/50160#discussion_r1980598264
##########
python/pyspark/sql/connect/udf.py:
##########
@@ -78,6 +80,15 @@ def _create_py_udf(
eval_type: int = PythonEvalType.SQL_BATCHED_UDF
+ if is_arrow_enabled:
+ require_minimum_pandas_version()
+ require_minimum_pyarrow_version()
+ warnings.warn(
+ "Arrow optimization failed to enable because PyArrow is not installed. "
+ "Falling back to a non-Arrow-optimized UDF.",
+ RuntimeWarning,
+ )
Review Comment:
This needs try-catch and set `is_arrow_enabled` to `False`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org