You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "xinrong-meng (via GitHub)" <gi...@apache.org> on 2023/05/26 00:35:11 UTC
[GitHub] [spark] xinrong-meng opened a new pull request, #41321: [WIP] Support StructType input in Arrow-optimized Python UDFs
xinrong-meng opened a new pull request, #41321:
URL: https://github.com/apache/spark/pull/41321
### What changes were proposed in this pull request?
Support StructType input in Arrow-optimized Python UDFs.
### Why are the changes needed?
Parity with pickled Python UDFs.
### Does this PR introduce _any_ user-facing change?
StructType input is supported in Arrow-optimized Python UDFs now.
### How was this patch tested?
TODO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-meng commented on pull request #41321: [SPARK-43893][PYTHON][CONNECT] Non-atomic data type support in Arrow-optimized Python UDF
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on PR #41321:
URL: https://github.com/apache/spark/pull/41321#issuecomment-1572492288
@ueshin @HyukjinKwon @zhengruifeng may I ask for your review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on a diff in pull request #41321: [SPARK-43893][PYTHON][CONNECT] Non-atomic data type support in Arrow-optimized Python UDF
Posted by "allisonwang-db (via GitHub)" <gi...@apache.org>.
allisonwang-db commented on code in PR #41321:
URL: https://github.com/apache/spark/pull/41321#discussion_r1218626596
##########
python/pyspark/sql/udf.py:
##########
@@ -129,18 +127,12 @@ def _create_py_udf(
else useArrow
)
regular_udf = _create_udf(f, returnType, PythonEvalType.SQL_BATCHED_UDF)
- return_type = regular_udf.returnType
try:
is_func_with_args = len(getfullargspec(f).args) > 0
except TypeError:
is_func_with_args = False
- is_output_atomic_type = (
- not isinstance(return_type, StructType)
- and not isinstance(return_type, MapType)
- and not isinstance(return_type, ArrayType)
- )
if is_arrow_enabled:
- if is_output_atomic_type and is_func_with_args:
+ if is_func_with_args:
Review Comment:
Just curious, why can't we enable arrow optimization for UDFs that do not have input arguments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-meng commented on a diff in pull request #41321: [SPARK-43893][PYTHON][CONNECT] Non-atomic data type support in Arrow-optimized Python UDF
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #41321:
URL: https://github.com/apache/spark/pull/41321#discussion_r1220083440
##########
python/pyspark/sql/udf.py:
##########
@@ -129,18 +127,12 @@ def _create_py_udf(
else useArrow
)
regular_udf = _create_udf(f, returnType, PythonEvalType.SQL_BATCHED_UDF)
- return_type = regular_udf.returnType
try:
is_func_with_args = len(getfullargspec(f).args) > 0
except TypeError:
is_func_with_args = False
- is_output_atomic_type = (
- not isinstance(return_type, StructType)
- and not isinstance(return_type, MapType)
- and not isinstance(return_type, ArrayType)
- )
if is_arrow_enabled:
- if is_output_atomic_type and is_func_with_args:
+ if is_func_with_args:
Review Comment:
Arrow optimization only makes sense when UDFs have inputs because it's an improvement on input (de)serialization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-meng commented on a diff in pull request #41321: [SPARK-43893][PYTHON][CONNECT] Non-atomic data type support in Arrow-optimized Python UDF
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #41321:
URL: https://github.com/apache/spark/pull/41321#discussion_r1222243033
##########
python/pyspark/sql/udf.py:
##########
@@ -129,18 +127,12 @@ def _create_py_udf(
else useArrow
)
regular_udf = _create_udf(f, returnType, PythonEvalType.SQL_BATCHED_UDF)
- return_type = regular_udf.returnType
try:
is_func_with_args = len(getfullargspec(f).args) > 0
except TypeError:
is_func_with_args = False
- is_output_atomic_type = (
- not isinstance(return_type, StructType)
- and not isinstance(return_type, MapType)
- and not isinstance(return_type, ArrayType)
- )
if is_arrow_enabled:
- if is_output_atomic_type and is_func_with_args:
+ if is_func_with_args:
Review Comment:
Good point. The warning will be removed when we enable Arrow by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-meng commented on pull request #41321: [SPARK-43893][PYTHON][CONNECT] Non-atomic data type support in Arrow-optimized Python UDF
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on PR #41321:
URL: https://github.com/apache/spark/pull/41321#issuecomment-1579560755
Merged to master, thanks all!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a diff in pull request #41321: [SPARK-43893][PYTHON][CONNECT] Non-atomic data type support in Arrow-optimized Python UDF
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #41321:
URL: https://github.com/apache/spark/pull/41321#discussion_r1213529660
##########
python/pyspark/sql/pandas/serializers.py:
##########
@@ -298,26 +299,39 @@ class ArrowStreamPandasUDFSerializer(ArrowStreamPandasSerializer):
Serializer used by Python worker to evaluate Pandas UDFs
"""
- def __init__(self, timezone, safecheck, assign_cols_by_name, df_for_struct=False):
+ def __init__(
+ self,
+ timezone,
+ safecheck,
+ assign_cols_by_name,
+ df_for_struct=False,
+ eval_type=PythonEvalType.SQL_SCALAR_PANDAS_UDF,
Review Comment:
It shouldn't take `eval_type` here as the same as `df_for_struct` that is calculated from the `eval_type` in `worker.py`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-meng commented on a diff in pull request #41321: [SPARK-43893][PYTHON][CONNECT] Non-atomic data type support in Arrow-optimized Python UDF
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #41321:
URL: https://github.com/apache/spark/pull/41321#discussion_r1213770380
##########
python/pyspark/sql/pandas/serializers.py:
##########
@@ -298,26 +299,39 @@ class ArrowStreamPandasUDFSerializer(ArrowStreamPandasSerializer):
Serializer used by Python worker to evaluate Pandas UDFs
"""
- def __init__(self, timezone, safecheck, assign_cols_by_name, df_for_struct=False):
+ def __init__(
+ self,
+ timezone,
+ safecheck,
+ assign_cols_by_name,
+ df_for_struct=False,
+ eval_type=PythonEvalType.SQL_SCALAR_PANDAS_UDF,
Review Comment:
Docstrings on how `df_for_struct` and `struct_in_pandas` is set according to eval_types, as well as user-facing documentation on struct type behaviors would be helpful. I may work on it as a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-meng commented on a diff in pull request #41321: [SPARK-43893][PYTHON][CONNECT] Non-atomic data type support in Arrow-optimized Python UDF
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #41321:
URL: https://github.com/apache/spark/pull/41321#discussion_r1213766572
##########
python/pyspark/sql/pandas/serializers.py:
##########
@@ -298,26 +299,39 @@ class ArrowStreamPandasUDFSerializer(ArrowStreamPandasSerializer):
Serializer used by Python worker to evaluate Pandas UDFs
"""
- def __init__(self, timezone, safecheck, assign_cols_by_name, df_for_struct=False):
+ def __init__(
+ self,
+ timezone,
+ safecheck,
+ assign_cols_by_name,
+ df_for_struct=False,
+ eval_type=PythonEvalType.SQL_SCALAR_PANDAS_UDF,
Review Comment:
Good point, adjusted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-meng closed pull request #41321: [SPARK-43893][PYTHON][CONNECT] Non-atomic data type support in Arrow-optimized Python UDF
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng closed pull request #41321: [SPARK-43893][PYTHON][CONNECT] Non-atomic data type support in Arrow-optimized Python UDF
URL: https://github.com/apache/spark/pull/41321
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on a diff in pull request #41321: [SPARK-43893][PYTHON][CONNECT] Non-atomic data type support in Arrow-optimized Python UDF
Posted by "allisonwang-db (via GitHub)" <gi...@apache.org>.
allisonwang-db commented on code in PR #41321:
URL: https://github.com/apache/spark/pull/41321#discussion_r1220538154
##########
python/pyspark/sql/udf.py:
##########
@@ -129,18 +127,12 @@ def _create_py_udf(
else useArrow
)
regular_udf = _create_udf(f, returnType, PythonEvalType.SQL_BATCHED_UDF)
- return_type = regular_udf.returnType
try:
is_func_with_args = len(getfullargspec(f).args) > 0
except TypeError:
is_func_with_args = False
- is_output_atomic_type = (
- not isinstance(return_type, StructType)
- and not isinstance(return_type, MapType)
- and not isinstance(return_type, ArrayType)
- )
if is_arrow_enabled:
- if is_output_atomic_type and is_func_with_args:
+ if is_func_with_args:
Review Comment:
But does it hurt if we enable Arrow for functions without input arguments? Then we can eliminate this restriction and enable Arrow for all kinds of Python UDFs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org