You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "xinrong-meng (via GitHub)" <gi...@apache.org> on 2023/01/26 07:32:49 UTC

[GitHub] [spark] xinrong-meng opened a new pull request, #39753: [SPARK-42125][CONNECT][PYTHON] Pandas UDF in Spark Connect

xinrong-meng opened a new pull request, #39753:
URL: https://github.com/apache/spark/pull/39753

   ### What changes were proposed in this pull request?
   Support Pandas UDF in Spark Connect.
   
   
   ### Why are the changes needed?
   To reach parity with the vanilla PySpark.
   
   
   ### Does this PR introduce _any_ user-facing change?
   Yes. Pandas UDF is supported in Spark Connect.
   
   
   ### How was this patch tested?
   Existing tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-meng closed pull request #39753: [SPARK-42125][CONNECT][PYTHON] Pandas UDF in Spark Connect

Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng closed pull request #39753: [SPARK-42125][CONNECT][PYTHON] Pandas UDF in Spark Connect
URL: https://github.com/apache/spark/pull/39753


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-meng commented on a diff in pull request #39753: [SPARK-42125][CONNECT][PYTHON] Pandas UDF in Spark Connect

Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #39753:
URL: https://github.com/apache/spark/pull/39753#discussion_r1090101065


##########
connector/connect/common/src/main/protobuf/spark/connect/expressions.proto:
##########
@@ -297,7 +297,7 @@ message Expression {
   }
 }
 
-message ScalarInlineUserDefinedFunction {
+message CommonUserDefinedFunction {

Review Comment:
   1. Series-to-Scalar Pandas UDFs `groupby(..).agg(..)` should be able to share the same message, so I suggested removing the `Scalar` piece.
   
   2. As for the `Inline` piece, I was thinking we were about to make both registered UDF and unregistered(inline) UDF share the same message so I proposed to remove it.
   
   `CommonInlineUserDefinedFunction` sounds fine. I am just wondering if my understanding of point 2. is correct. @grundprinzip @hvanhovell 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] vicennial commented on a diff in pull request #39753: [SPARK-42125][CONNECT][PYTHON] Pandas UDF in Spark Connect

Posted by "vicennial (via GitHub)" <gi...@apache.org>.
vicennial commented on code in PR #39753:
URL: https://github.com/apache/spark/pull/39753#discussion_r1088952970


##########
connector/connect/common/src/main/protobuf/spark/connect/expressions.proto:
##########
@@ -297,7 +297,7 @@ message Expression {
   }
 }
 
-message ScalarInlineUserDefinedFunction {
+message CommonUserDefinedFunction {

Review Comment:
   Quite valid points, +1 to `CommonInlineUserDefinedFunction`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-meng commented on pull request #39753: [SPARK-42125][CONNECT][PYTHON] Pandas UDF in Spark Connect

Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on PR #39753:
URL: https://github.com/apache/spark/pull/39753#issuecomment-1409601795

   Merged to master and branch-3.4. Thanks all!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-meng commented on pull request #39753: [SPARK-42125][CONNECT][PYTHON] Pandas UDF in Spark Connect

Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on PR #39753:
URL: https://github.com/apache/spark/pull/39753#issuecomment-1409604165

   Merged it to unblock follow-up PRs. Feel free to leave comments, I will address them in the follow-ups.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-meng commented on a diff in pull request #39753: [SPARK-42125][CONNECT][PYTHON] Pandas UDF in Spark Connect

Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #39753:
URL: https://github.com/apache/spark/pull/39753#discussion_r1090630628


##########
connector/connect/common/src/main/protobuf/spark/connect/expressions.proto:
##########
@@ -297,7 +297,7 @@ message Expression {
   }
 }
 
-message ScalarInlineUserDefinedFunction {
+message CommonUserDefinedFunction {

Review Comment:
   Renamed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-meng commented on a diff in pull request #39753: [SPARK-42125][CONNECT][PYTHON] Pandas UDF in Spark Connect

Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #39753:
URL: https://github.com/apache/spark/pull/39753#discussion_r1090101065


##########
connector/connect/common/src/main/protobuf/spark/connect/expressions.proto:
##########
@@ -297,7 +297,7 @@ message Expression {
   }
 }
 
-message ScalarInlineUserDefinedFunction {
+message CommonUserDefinedFunction {

Review Comment:
   1. Series-to-Scalar Pandas UDFs `groupby(..).agg(..)` should be able to share the same message, so I suggested removing the `Scalar` piece.
   
   2. As for the `Inline` piece, I was thinking we were about to make both registered UDF and unregistered(inline) UDF share the same message so I proposed to remove it.
   
   `CommonInlineUserDefinedFunction` sounds fine. I am just wondering if my understanding of point 2. is correct. @grundprinzip



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-meng commented on pull request #39753: [SPARK-42125][CONNECT][PYTHON] Pandas UDF in Spark Connect

Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on PR #39753:
URL: https://github.com/apache/spark/pull/39753#issuecomment-1406160212

   @HyukjinKwon @zhengruifeng @grundprinzip May I get a review, please?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-meng commented on pull request #39753: [SPARK-42125][CONNECT][PYTHON] Pandas UDF in Spark Connect

Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on PR #39753:
URL: https://github.com/apache/spark/pull/39753#issuecomment-1406133181

   I would prefer to file a follow-up PR to reuse `pyspark.sql.tests.test_pandas_udf` test cases.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] grundprinzip commented on a diff in pull request #39753: [SPARK-42125][CONNECT][PYTHON] Pandas UDF in Spark Connect

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #39753:
URL: https://github.com/apache/spark/pull/39753#discussion_r1088698727


##########
connector/connect/common/src/main/protobuf/spark/connect/expressions.proto:
##########
@@ -297,7 +297,7 @@ message Expression {
   }
 }
 
-message ScalarInlineUserDefinedFunction {
+message CommonUserDefinedFunction {

Review Comment:
   So I'm not sure I 100% agree with the renaming, and would like to invite a bit more feedback from @HyukjinKwon @hvanhovell and others.
   
   The reason why we picked initially `ScalarInlineUserDefinedFunction` was because it represents a scalar function with an inlined defined function body.
   
   If we rename this to `CommonUserDefinedFunction` it's not 100% clear if the message is a scalar function or an aggregate function, does it contain the function body or not.
   
   Personally, if we really have to get rid of the scalar piece, I would prefer `CommonInlineUserDefinedFunction`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org