You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ueshin (via GitHub)" <gi...@apache.org> on 2023/09/13 21:47:02 UTC
[GitHub] [spark] ueshin opened a new pull request, #42915: [SPARK-45159][PYTHON] Handle named arguments only when necessary
ueshin opened a new pull request, #42915:
URL: https://github.com/apache/spark/pull/42915
### What changes were proposed in this pull request?
Handles named arguments only when necessary.
### Why are the changes needed?
Constructing `kwargs` as `dict` could be expensive. It should be done only when necessary.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #42915: [SPARK-45159][PYTHON] Handle named arguments only when necessary
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #42915:
URL: https://github.com/apache/spark/pull/42915#issuecomment-1720022792
cc @HyukjinKwon @zhengruifeng @allisonwang-db
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #42915: [SPARK-45159][PYTHON] Handle named arguments only when necessary
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #42915:
URL: https://github.com/apache/spark/pull/42915#issuecomment-1720388881
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #42915: [SPARK-45159][PYTHON] Handle named arguments only when necessary
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #42915: [SPARK-45159][PYTHON] Handle named arguments only when necessary
URL: https://github.com/apache/spark/pull/42915
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on a diff in pull request #42915: [SPARK-45159][PYTHON] Handle named arguments only when necessary
Posted by "allisonwang-db (via GitHub)" <gi...@apache.org>.
allisonwang-db commented on code in PR #42915:
URL: https://github.com/apache/spark/pull/42915#discussion_r1327519954
##########
python/pyspark/worker.py:
##########
@@ -810,28 +847,26 @@ def check_return_value(res):
},
)
- def evaluate(*args: pd.Series, **kwargs: pd.Series):
- if len(args) == 0 and len(kwargs) == 0:
+ def evaluate(*args: pd.Series):
+ if len(args) == 0:
res = func()
check_return_value(res)
yield verify_result(pd.DataFrame(res)), arrow_return_type
else:
# Create tuples from the input pandas Series, each tuple
# represents a row across all Series.
- keys = list(kwargs.keys())
- len_args = len(args)
- row_tuples = zip(*args, *[kwargs[key] for key in keys])
+ row_tuples = zip(*args)
for row in row_tuples:
- res = func(
- *row[:len_args],
- **{key: row[len_args + i] for i, key in enumerate(keys)},
- )
+ res = func(*row)
check_return_value(res)
yield verify_result(pd.DataFrame(res)), arrow_return_type
return evaluate
- eval = wrap_arrow_udtf(getattr(udtf, "eval"), return_type)
+ eval_func_kwargs_support, args_kwargs_offsets = wrap_kwargs_support(
Review Comment:
It would be really good to comment here on why we need to wrap the kwargs separately. This can provide valuable context for those who work on this code in the future. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org