You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2023/09/26 04:53:55 UTC
[GitHub] [spark] zhengruifeng opened a new pull request, #43117: [SPARK-45329][PYTHON][CONNECT] DataFrame methods avoid pandas conversion if possible
zhengruifeng opened a new pull request, #43117:
URL: https://github.com/apache/spark/pull/43117
### What changes were proposed in this pull request?
DataFrame methods avoid pandas conversion if possible
### Why are the changes needed?
minor optimization:
before: `arrow table` -> `pandas dataframe` -> `scalar result`
after: `arrow table` -> `scalar result`
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
ci
### Was this patch authored or co-authored using generative AI tooling?
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #43117: [SPARK-45329][PYTHON][CONNECT] DataFrame methods skip pandas conversion
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #43117:
URL: https://github.com/apache/spark/pull/43117#issuecomment-1736752430
> Failed tests seem to be irrelevant to the PR, please retrigger them. Otherwise looks great! It would be nice to extract `table[0][0].to_py()` as a method or add a comment to help understand what it does. I don't have a strong opinion on that though.
hmm, `table[0][0]` is to exact the first column of the first row, I guess it maybe straightforward
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #43117: [SPARK-45329][PYTHON][CONNECT] DataFrame methods skip pandas conversion
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #43117:
URL: https://github.com/apache/spark/pull/43117#issuecomment-1736754967
thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #43117: [SPARK-45329][PYTHON][CONNECT] DataFrame methods skip pandas conversion
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #43117: [SPARK-45329][PYTHON][CONNECT] DataFrame methods skip pandas conversion
URL: https://github.com/apache/spark/pull/43117
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #43117: [SPARK-45329][PYTHON][CONNECT] DataFrame methods skip pandas conversion
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43117:
URL: https://github.com/apache/spark/pull/43117#issuecomment-1734999526
cc @xinrong-meng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-meng commented on pull request #43117: [SPARK-45329][PYTHON][CONNECT] DataFrame methods skip pandas conversion
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on PR #43117:
URL: https://github.com/apache/spark/pull/43117#issuecomment-1736383945
Failed tests seem to be irrelevant to the PR, please retrigger them.
Otherwise looks great!
It would be nice to extract `table[0][0].to_py()` as a method or add a comment to help understand what it does. I don't have a strong opinion on that though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org