You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/12/19 07:06:38 UTC

[GitHub] [spark] HyukjinKwon opened a new pull request, #39123: [SPARK-41583][CONNECT][PROTOBUF] Add Spark Connect and protobuf into setup.py with specifying dependencies

HyukjinKwon opened a new pull request, #39123:
URL: https://github.com/apache/spark/pull/39123

   ### What changes were proposed in this pull request?
   
   This PR proposes to:
   
   - Add `pyspark.sql.connect` and `pyspark.sql.protobuf` to the PySpark package in PyPI.
   - Fix the documentation to specify the dependencies for Python Spark Connect client.
   
   ### Why are the changes needed?
   
   To guide users to use Spark Connect and Protobuf, and package these feature to be released properly.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, this exposes both `pyspark.sql.connect` and `pyspark.sql.protobuf` to the end users in PyPI package. In addition, this fixes the user-facing documentation about dependencies from Spark Connect.
   
   ### How was this patch tested?
   
   CI in this PR should test it out.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon closed pull request #39123: [SPARK-41583][CONNECT][PROTOBUF] Add Spark Connect and protobuf into setup.py with specifying dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #39123: [SPARK-41583][CONNECT][PROTOBUF] Add Spark Connect and protobuf into setup.py with specifying dependencies
URL: https://github.com/apache/spark/pull/39123


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #39123: [SPARK-41583][CONNECT][PROTOBUF] Add Spark Connect and protobuf into setup.py with specifying dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #39123:
URL: https://github.com/apache/spark/pull/39123#issuecomment-1357448291

   Build: https://github.com/HyukjinKwon/spark/actions/runs/3730423845


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #39123: [SPARK-41583][CONNECT][PROTOBUF] Add Spark Connect and protobuf into setup.py with specifying dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #39123:
URL: https://github.com/apache/spark/pull/39123#issuecomment-1358655672

   Merged to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #39123: [SPARK-41583][CONNECT][PROTOBUF] Add Spark Connect and protobuf into setup.py with specifying dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #39123:
URL: https://github.com/apache/spark/pull/39123#discussion_r1051869903


##########
python/docs/source/getting_started/install.rst:
##########
@@ -50,6 +50,8 @@ If you want to install extra dependencies for a specific component, you can inst
     pip install pyspark[sql]
     # pandas API on Spark
     pip install pyspark[pandas_on_spark] plotly  # to plot your data, you can install plotly together.
+    # Spark Connect
+    pip install pyspark[connect]

Review Comment:
   This will be fixed soon.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] itholic commented on a diff in pull request #39123: [SPARK-41583][CONNECT][PROTOBUF] Add Spark Connect and protobuf into setup.py with specifying dependencies

Posted by GitBox <gi...@apache.org>.
itholic commented on code in PR #39123:
URL: https://github.com/apache/spark/pull/39123#discussion_r1052404201


##########
python/setup.py:
##########
@@ -113,6 +113,7 @@ def _supports_symlinks():
 # Also don't forget to update python/docs/source/getting_started/install.rst.
 _minimum_pandas_version = "1.0.5"

Review Comment:
   Maybe we can remove this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #39123: [SPARK-41583][CONNECT][PROTOBUF] Add Spark Connect and protobuf into setup.py with specifying dependencies

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #39123:
URL: https://github.com/apache/spark/pull/39123#discussion_r1052742314


##########
python/setup.py:
##########
@@ -113,6 +113,7 @@ def _supports_symlinks():
 # Also don't forget to update python/docs/source/getting_started/install.rst.
 _minimum_pandas_version = "1.0.5"
 _minimum_pyarrow_version = "1.0.0"
+_minimum_pandas_version = "1.48.1"

Review Comment:
   ```suggestion
   _minimum_grpc_version = "1.48.1"
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org