You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "xinrong-meng (via GitHub)" <gi...@apache.org> on 2023/06/14 21:52:27 UTC
[GitHub] [spark] xinrong-meng opened a new pull request, #41603: [SPARK-43903][PYTHON][CONNECT] Improve ArrayType input support in Arrow Python UDF
xinrong-meng opened a new pull request, #41603:
URL: https://github.com/apache/spark/pull/41603
### What changes were proposed in this pull request?
Improve ArrayType input support in Arrow-optimized Python UDF.
### Why are the changes needed?
Reach parity with Pickled Python UDF.
### Does this PR introduce _any_ user-facing change?
Yes. TODO
### How was this patch tested?
TODO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #41603: [SPARK-43903][PYTHON][CONNECT] Improve ArrayType input support in Arrow Python UDF
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #41603: [SPARK-43903][PYTHON][CONNECT] Improve ArrayType input support in Arrow Python UDF
URL: https://github.com/apache/spark/pull/41603
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-meng commented on a diff in pull request #41603: [WIP][SPARK-43903][PYTHON][CONNECT] Improve ArrayType input support in Arrow Python UDF
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #41603:
URL: https://github.com/apache/spark/pull/41603#discussion_r1231526720
##########
python/pyspark/sql/pandas/types.py:
##########
@@ -699,7 +710,9 @@ def convert_timestamp_ntz(value: Any) -> Any:
elif isinstance(dt, UserDefinedType):
udt: UserDefinedType = dt
- conv = _converter(udt.sqlType(), _struct_in_pandas="row") or (lambda x: x)
+ conv = _converter(udt.sqlType(), _struct_in_pandas="row", _ndarray_as_list=False) or (
Review Comment:
Modified, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a diff in pull request #41603: [WIP][SPARK-43903][PYTHON][CONNECT] Improve ArrayType input support in Arrow Python UDF
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #41603:
URL: https://github.com/apache/spark/pull/41603#discussion_r1231514639
##########
python/pyspark/sql/pandas/types.py:
##########
@@ -582,7 +585,12 @@ def convert_array(value: Any) -> Any:
return None
elif isinstance(value, np.ndarray):
# `pyarrow.Table.to_pandas` uses `np.ndarray`.
- return np.array([_element_conv(v) for v in value]) # type: ignore[misc]
+ if ndarray_as_list:
Review Comment:
`_ndarray_as_list` instead of `ndarray_as_list`?
##########
python/pyspark/sql/pandas/types.py:
##########
@@ -582,7 +585,12 @@ def convert_array(value: Any) -> Any:
return None
elif isinstance(value, np.ndarray):
# `pyarrow.Table.to_pandas` uses `np.ndarray`.
- return np.array([_element_conv(v) for v in value]) # type: ignore[misc]
+ if ndarray_as_list:
Review Comment:
Btw, shall we define a new function for it's `True` or `False` to reduce any overhead, similar to ones for `StructType`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a diff in pull request #41603: [WIP][SPARK-43903][PYTHON][CONNECT] Improve ArrayType input support in Arrow Python UDF
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #41603:
URL: https://github.com/apache/spark/pull/41603#discussion_r1231520341
##########
python/pyspark/sql/pandas/types.py:
##########
@@ -699,7 +710,9 @@ def convert_timestamp_ntz(value: Any) -> Any:
elif isinstance(dt, UserDefinedType):
udt: UserDefinedType = dt
- conv = _converter(udt.sqlType(), _struct_in_pandas="row") or (lambda x: x)
+ conv = _converter(udt.sqlType(), _struct_in_pandas="row", _ndarray_as_list=False) or (
Review Comment:
`_ndarray_as_list` sounds to be `True` for `UserDefinedType`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-meng commented on a diff in pull request #41603: [WIP][SPARK-43903][PYTHON][CONNECT] Improve ArrayType input support in Arrow Python UDF
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #41603:
URL: https://github.com/apache/spark/pull/41603#discussion_r1231552983
##########
python/pyspark/sql/pandas/types.py:
##########
@@ -578,29 +578,44 @@ def _converter(
if isinstance(dt, ArrayType):
_element_conv = _converter(dt.elementType, _struct_in_pandas, _ndarray_as_list)
- def convert_array(value: Any) -> Any:
- if value is None:
- return None
- elif isinstance(value, np.ndarray):
- element_list = (
- [_element_conv(v) for v in value]
- if _element_conv is not None
- else [v for v in value]
- )
-
- if _ndarray_as_list:
+ if _ndarray_as_list:
+
+ def convert_array_ndarray_as_list(value: Any) -> Any:
+ if value is None:
+ return None
+ elif isinstance(value, np.ndarray):
# In Arrow Python UDF, ArrayType is converted to `np.ndarray`
# whereas a list is expected.
- return element_list
+ return (
+ [_element_conv(v) for v in value]
+ if _element_conv is not None
+ else [v for v in value]
+ )
else:
+ assert isinstance(value, list)
+ # otherwise, `list` should be used.
+ return [_element_conv(v) for v in value] # type: ignore[misc]
+
+ return convert_array_ndarray_as_list
+ else:
+
+ def convert_array_ndarray_as_ndarray(value: Any) -> Any:
Review Comment:
Good point!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-meng commented on a diff in pull request #41603: [WIP][SPARK-43903][PYTHON][CONNECT] Improve ArrayType input support in Arrow Python UDF
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #41603:
URL: https://github.com/apache/spark/pull/41603#discussion_r1231536052
##########
python/pyspark/sql/pandas/types.py:
##########
@@ -582,7 +585,12 @@ def convert_array(value: Any) -> Any:
return None
elif isinstance(value, np.ndarray):
# `pyarrow.Table.to_pandas` uses `np.ndarray`.
- return np.array([_element_conv(v) for v in value]) # type: ignore[misc]
+ if ndarray_as_list:
Review Comment:
Adjusted, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-meng commented on a diff in pull request #41603: [WIP][SPARK-43903][PYTHON][CONNECT] Improve ArrayType input support in Arrow Python UDF
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #41603:
URL: https://github.com/apache/spark/pull/41603#discussion_r1231553251
##########
python/pyspark/sql/pandas/types.py:
##########
@@ -578,29 +578,44 @@ def _converter(
if isinstance(dt, ArrayType):
_element_conv = _converter(dt.elementType, _struct_in_pandas, _ndarray_as_list)
- def convert_array(value: Any) -> Any:
- if value is None:
- return None
- elif isinstance(value, np.ndarray):
- element_list = (
- [_element_conv(v) for v in value]
- if _element_conv is not None
- else [v for v in value]
- )
-
- if _ndarray_as_list:
+ if _ndarray_as_list:
+
+ def convert_array_ndarray_as_list(value: Any) -> Any:
+ if value is None:
+ return None
+ elif isinstance(value, np.ndarray):
# In Arrow Python UDF, ArrayType is converted to `np.ndarray`
# whereas a list is expected.
- return element_list
+ return (
+ [_element_conv(v) for v in value]
+ if _element_conv is not None
Review Comment:
Adjusted.
##########
python/pyspark/sql/pandas/types.py:
##########
@@ -569,30 +572,56 @@ def correct_dtype(pser: pd.Series) -> pd.Series:
return correct_dtype
def _converter(
- dt: DataType, _struct_in_pandas: Optional[str]
+ dt: DataType, _struct_in_pandas: Optional[str], _ndarray_as_list: bool
) -> Optional[Callable[[Any], Any]]:
if isinstance(dt, ArrayType):
- _element_conv = _converter(dt.elementType, _struct_in_pandas)
- if _element_conv is None:
- return None
+ _element_conv = _converter(dt.elementType, _struct_in_pandas, _ndarray_as_list)
- def convert_array(value: Any) -> Any:
- if value is None:
- return None
- elif isinstance(value, np.ndarray):
- # `pyarrow.Table.to_pandas` uses `np.ndarray`.
- return np.array([_element_conv(v) for v in value]) # type: ignore[misc]
- else:
- assert isinstance(value, list)
- # otherwise, `list` should be used.
- return [_element_conv(v) for v in value] # type: ignore[misc]
+ if _ndarray_as_list:
- return convert_array
+ def convert_array_ndarray_as_list(value: Any) -> Any:
+ if value is None:
+ return None
+ elif isinstance(value, np.ndarray):
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a diff in pull request #41603: [WIP][SPARK-43903][PYTHON][CONNECT] Improve ArrayType input support in Arrow Python UDF
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #41603:
URL: https://github.com/apache/spark/pull/41603#discussion_r1231539888
##########
python/pyspark/sql/pandas/types.py:
##########
@@ -578,29 +578,44 @@ def _converter(
if isinstance(dt, ArrayType):
_element_conv = _converter(dt.elementType, _struct_in_pandas, _ndarray_as_list)
- def convert_array(value: Any) -> Any:
- if value is None:
- return None
- elif isinstance(value, np.ndarray):
- element_list = (
- [_element_conv(v) for v in value]
- if _element_conv is not None
- else [v for v in value]
- )
-
- if _ndarray_as_list:
+ if _ndarray_as_list:
+
+ def convert_array_ndarray_as_list(value: Any) -> Any:
+ if value is None:
+ return None
+ elif isinstance(value, np.ndarray):
# In Arrow Python UDF, ArrayType is converted to `np.ndarray`
# whereas a list is expected.
- return element_list
+ return (
+ [_element_conv(v) for v in value]
+ if _element_conv is not None
+ else [v for v in value]
+ )
else:
+ assert isinstance(value, list)
+ # otherwise, `list` should be used.
+ return [_element_conv(v) for v in value] # type: ignore[misc]
+
+ return convert_array_ndarray_as_list
+ else:
+
+ def convert_array_ndarray_as_ndarray(value: Any) -> Any:
Review Comment:
In this case, we can return `None` if `_element_conv is None`?
##########
python/pyspark/sql/pandas/types.py:
##########
@@ -578,29 +578,44 @@ def _converter(
if isinstance(dt, ArrayType):
_element_conv = _converter(dt.elementType, _struct_in_pandas, _ndarray_as_list)
- def convert_array(value: Any) -> Any:
- if value is None:
- return None
- elif isinstance(value, np.ndarray):
- element_list = (
- [_element_conv(v) for v in value]
- if _element_conv is not None
- else [v for v in value]
- )
-
- if _ndarray_as_list:
+ if _ndarray_as_list:
+
+ def convert_array_ndarray_as_list(value: Any) -> Any:
+ if value is None:
+ return None
+ elif isinstance(value, np.ndarray):
# In Arrow Python UDF, ArrayType is converted to `np.ndarray`
# whereas a list is expected.
- return element_list
+ return (
+ [_element_conv(v) for v in value]
+ if _element_conv is not None
Review Comment:
In this case, `_element_conv` can be `_element_conv or (lambda x: x)` outside of function?
##########
python/pyspark/sql/pandas/types.py:
##########
@@ -569,30 +572,56 @@ def correct_dtype(pser: pd.Series) -> pd.Series:
return correct_dtype
def _converter(
- dt: DataType, _struct_in_pandas: Optional[str]
+ dt: DataType, _struct_in_pandas: Optional[str], _ndarray_as_list: bool
) -> Optional[Callable[[Any], Any]]:
if isinstance(dt, ArrayType):
- _element_conv = _converter(dt.elementType, _struct_in_pandas)
- if _element_conv is None:
- return None
+ _element_conv = _converter(dt.elementType, _struct_in_pandas, _ndarray_as_list)
- def convert_array(value: Any) -> Any:
- if value is None:
- return None
- elif isinstance(value, np.ndarray):
- # `pyarrow.Table.to_pandas` uses `np.ndarray`.
- return np.array([_element_conv(v) for v in value]) # type: ignore[misc]
- else:
- assert isinstance(value, list)
- # otherwise, `list` should be used.
- return [_element_conv(v) for v in value] # type: ignore[misc]
+ if _ndarray_as_list:
- return convert_array
+ def convert_array_ndarray_as_list(value: Any) -> Any:
+ if value is None:
+ return None
+ elif isinstance(value, np.ndarray):
Review Comment:
Do we still need to check this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #41603: [SPARK-43903][PYTHON][CONNECT] Improve ArrayType input support in Arrow Python UDF
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #41603:
URL: https://github.com/apache/spark/pull/41603#issuecomment-1600495998
merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org