You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/09/26 04:02:21 UTC

[GitHub] [spark] grundprinzip opened a new pull request, #37993: [Cleanup] Update generated proto files for Spark Connect

grundprinzip opened a new pull request, #37993:
URL: https://github.com/apache/spark/pull/37993

   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
     8. If you want to add or modify an error type or message, please read the guideline first in
        'core/src/main/resources/error/README.md'.
   -->
   
   ### What changes were proposed in this pull request?
   This patch cleans up the generated proto files from the initial Spark Connect import. The previous files had a Databricks specific go module path embedded in the generated Python descriptor. This is now removed. No new functionality added.
   
   ### Why are the changes needed?
   
   Cleanup.
   
   ### Does this PR introduce _any_ user-facing change?
   No.
   
   ### How was this patch tested?
   The generated files are used during the regular testing for Spark Connect.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #37993: [SPARK-40557][CONNECT] Update generated proto files for Spark Connect

Posted by GitBox <gi...@apache.org>.
grundprinzip commented on code in PR #37993:
URL: https://github.com/apache/spark/pull/37993#discussion_r988554994


##########
python/pyspark/sql/connect/proto/base_pb2.py:
##########
@@ -28,26 +28,20 @@
 _sym_db = _symbol_database.Default()
 
 
-from pyspark.sql.connect.proto import (
-    commands_pb2 as spark_dot_connect_dot_commands__pb2,
-)
-from pyspark.sql.connect.proto import (
-    relations_pb2 as spark_dot_connect_dot_relations__pb2,
-)
+from pyspark.sql.connect.proto import commands_pb2 as spark_dot_connect_dot_commands__pb2

Review Comment:
   In contrast to Java in Python the generated code is the final artifact. The only way to achieve this would be to use a proper build system for Python but that's probably going to be a much bigger task.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon closed pull request #37993: [SPARK-40557][CONNECT] Update generated proto files for Spark Connect

Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #37993: [SPARK-40557][CONNECT] Update generated proto files for Spark Connect
URL: https://github.com/apache/spark/pull/37993


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] amaliujia commented on a diff in pull request #37993: [SPARK-40557][CONNECT] Update generated proto files for Spark Connect

Posted by GitBox <gi...@apache.org>.
amaliujia commented on code in PR #37993:
URL: https://github.com/apache/spark/pull/37993#discussion_r988542154


##########
python/pyspark/sql/connect/proto/base_pb2.py:
##########
@@ -28,26 +28,20 @@
 _sym_db = _symbol_database.Default()
 
 
-from pyspark.sql.connect.proto import (
-    commands_pb2 as spark_dot_connect_dot_commands__pb2,
-)
-from pyspark.sql.connect.proto import (
-    relations_pb2 as spark_dot_connect_dot_relations__pb2,
-)
+from pyspark.sql.connect.proto import commands_pb2 as spark_dot_connect_dot_commands__pb2

Review Comment:
   I think this is the script that was checked in: https://github.com/apache/spark/blob/master/connector/connect/dev/generate_protos.sh



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] amaliujia commented on a diff in pull request #37993: [SPARK-40557][CONNECT] Update generated proto files for Spark Connect

Posted by GitBox <gi...@apache.org>.
amaliujia commented on code in PR #37993:
URL: https://github.com/apache/spark/pull/37993#discussion_r988542292


##########
python/pyspark/sql/connect/proto/base_pb2.py:
##########
@@ -28,26 +28,20 @@
 _sym_db = _symbol_database.Default()
 
 
-from pyspark.sql.connect.proto import (
-    commands_pb2 as spark_dot_connect_dot_commands__pb2,
-)
-from pyspark.sql.connect.proto import (
-    relations_pb2 as spark_dot_connect_dot_relations__pb2,
-)
+from pyspark.sql.connect.proto import commands_pb2 as spark_dot_connect_dot_commands__pb2

Review Comment:
   Hopefully we can figure out a better way in the future than manually generating protos for updates.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #37993: [CONNECT] [Cleanup] Update generated proto files for Spark Connect

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #37993:
URL: https://github.com/apache/spark/pull/37993#issuecomment-1257456288

   (Should probably need a separate JIRA for this)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #37993: [SPARK-40557][CONNECT] Update generated proto files for Spark Connect

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #37993:
URL: https://github.com/apache/spark/pull/37993#issuecomment-1258864096

   Merged to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] beliefer commented on a diff in pull request #37993: [SPARK-40557][CONNECT] Update generated proto files for Spark Connect

Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #37993:
URL: https://github.com/apache/spark/pull/37993#discussion_r990568669


##########
python/pyspark/sql/connect/proto/base_pb2.py:
##########
@@ -28,26 +28,20 @@
 _sym_db = _symbol_database.Default()
 
 
-from pyspark.sql.connect.proto import (
-    commands_pb2 as spark_dot_connect_dot_commands__pb2,
-)
-from pyspark.sql.connect.proto import (
-    relations_pb2 as spark_dot_connect_dot_relations__pb2,
-)
+from pyspark.sql.connect.proto import commands_pb2 as spark_dot_connect_dot_commands__pb2

Review Comment:
   > I think this is the script that was checked in: https://github.com/apache/spark/blob/master/connector/connect/dev/generate_protos.sh
   
   Thank you.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] amaliujia commented on pull request #37993: [SPARK-40557][CONNECT] Update generated proto files for Spark Connect

Posted by GitBox <gi...@apache.org>.
amaliujia commented on PR #37993:
URL: https://github.com/apache/spark/pull/37993#issuecomment-1259922406

   post + 1. Thanks for following up on this quickly!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on pull request #37993: [SPARK-40557] [CONNECT] [Cleanup] Update generated proto files for Spark Connect

Posted by GitBox <gi...@apache.org>.
grundprinzip commented on PR #37993:
URL: https://github.com/apache/spark/pull/37993#issuecomment-1257533571

   Created [SPARK-40557] to track.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] beliefer commented on a diff in pull request #37993: [SPARK-40557][CONNECT] Update generated proto files for Spark Connect

Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #37993:
URL: https://github.com/apache/spark/pull/37993#discussion_r984526908


##########
python/pyspark/sql/connect/proto/base_pb2.py:
##########
@@ -28,26 +28,20 @@
 _sym_db = _symbol_database.Default()
 
 
-from pyspark.sql.connect.proto import (
-    commands_pb2 as spark_dot_connect_dot_commands__pb2,
-)
-from pyspark.sql.connect.proto import (
-    relations_pb2 as spark_dot_connect_dot_relations__pb2,
-)
+from pyspark.sql.connect.proto import commands_pb2 as spark_dot_connect_dot_commands__pb2

Review Comment:
   Could you tell me how to regenerate these python files?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #37993: [SPARK-40557] [CONNECT] [Cleanup] Update generated proto files for Spark Connect

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #37993:
URL: https://github.com/apache/spark/pull/37993#issuecomment-1258555111

   Can one of the admins verify this patch?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on pull request #37993: [SPARK-40557][CONNECT] Update generated proto files for Spark Connect

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #37993:
URL: https://github.com/apache/spark/pull/37993#issuecomment-1258866543

   post +1, thanks for updating this!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #37993: [SPARK-40557][CONNECT] Update generated proto files for Spark Connect

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #37993:
URL: https://github.com/apache/spark/pull/37993#discussion_r988532139


##########
python/pyspark/sql/connect/proto/base_pb2.py:
##########
@@ -28,26 +28,20 @@
 _sym_db = _symbol_database.Default()
 
 
-from pyspark.sql.connect.proto import (
-    commands_pb2 as spark_dot_connect_dot_commands__pb2,
-)
-from pyspark.sql.connect.proto import (
-    relations_pb2 as spark_dot_connect_dot_relations__pb2,
-)
+from pyspark.sql.connect.proto import commands_pb2 as spark_dot_connect_dot_commands__pb2

Review Comment:
   Actually yeah. How do we generate this, @grundprinzip ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org