You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ueshin (via GitHub)" <gi...@apache.org> on 2023/08/16 21:11:14 UTC
[GitHub] [spark] ueshin opened a new pull request, #42520: [SPARK-44836][PYTHON] Refactor Arrow Python UDTF
ueshin opened a new pull request, #42520:
URL: https://github.com/apache/spark/pull/42520
### What changes were proposed in this pull request?
Refactors Arrow Python UDTF.
### Why are the changes needed?
Arrow Python UDTF is not need to be redefined when creating it. It can be handled in `worker.py`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
The existing tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #42520: [SPARK-44836][PYTHON] Refactor Arrow Python UDTF
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #42520:
URL: https://github.com/apache/spark/pull/42520#issuecomment-1681315872
For 3.5: #42522 as this changes Spark Connect client implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin closed pull request #42520: [SPARK-44836][PYTHON] Refactor Arrow Python UDTF
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin closed pull request #42520: [SPARK-44836][PYTHON] Refactor Arrow Python UDTF
URL: https://github.com/apache/spark/pull/42520
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #42520: [SPARK-44836][PYTHON] Refactor Arrow Python UDTF
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #42520:
URL: https://github.com/apache/spark/pull/42520#issuecomment-1681537506
Thanks! merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #42520: [SPARK-44836][PYTHON] Refactor Arrow Python UDTF
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #42520:
URL: https://github.com/apache/spark/pull/42520#issuecomment-1681276866
cc @dtenedor @allisonwang-db
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on a diff in pull request #42520: [SPARK-44836][PYTHON] Refactor Arrow Python UDTF
Posted by "allisonwang-db (via GitHub)" <gi...@apache.org>.
allisonwang-db commented on code in PR #42520:
URL: https://github.com/apache/spark/pull/42520#discussion_r1296461894
##########
python/pyspark/worker.py:
##########
@@ -628,9 +628,36 @@ def verify_result(result):
)
return result
- return lambda *a, **kw: map(
- lambda res: (res, arrow_return_type), map(verify_result, f(*a, **kw))
- )
+ def evaluate(*args: pd.Series, **kwargs: pd.Series):
+ try:
+ if len(args) == 0 and len(kwargs) == 0:
+ yield verify_result(pd.DataFrame(f())), arrow_return_type
Review Comment:
I see we have a try catch statement for the entire evaluate function. Can we actually only wrap the invocation of `f()`? This is to separate the user's code error from the error thrown by the worker code.
##########
python/pyspark/worker.py:
##########
@@ -628,9 +628,36 @@ def verify_result(result):
)
return result
- return lambda *a, **kw: map(
- lambda res: (res, arrow_return_type), map(verify_result, f(*a, **kw))
- )
+ def evaluate(*args: pd.Series, **kwargs: pd.Series):
+ try:
+ if len(args) == 0 and len(kwargs) == 0:
+ yield verify_result(pd.DataFrame(f())), arrow_return_type
+ else:
+ # Create tuples from the input pandas Series, each tuple
+ # represents a row across all Series.
+ keys = list(kwargs.keys())
+ len_args = len(args)
+ row_tuples = zip(*args, *[kwargs[key] for key in keys])
+ for row in row_tuples:
+ res = f(
+ *row[:len_args],
+ **{key: row[len_args + i] for i, key in enumerate(keys)},
+ )
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org