You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/12/17 10:03:14 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
HyukjinKwon opened a new pull request #30822:
URL: https://github.com/apache/spark/pull/30822
### What changes were proposed in this pull request?
This PR proposes to restructure and refine the Python dependency management page.
I lately wrote a blog post which will be published soon, and decided contribute some of the contents back to PySpark documentation.
FWIW, it has been reviewed by some tech writers and engineers.
I built the site for making the review easier: https://hyukjin-spark.readthedocs.io/en/stable/user_guide/python_packaging.html
### Why are the changes needed?
For better documentation.
### Does this PR introduce _any_ user-facing change?
It's doc change but only in unreleased bracnhs for now.
### How was this patch tested?
I manually built the docs as:
```bash
cd python/docs
make clean html
open
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-747801518
Thank you guys. Merged to master and bracnh-3.1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-747822990
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37573/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-747374824
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37552/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #30822:
URL: https://github.com/apache/spark/pull/30822#discussion_r545069144
##########
File path: python/docs/source/user_guide/python_packaging.rst
##########
@@ -118,87 +116,139 @@ Example using ``SparkSession.builder``:
from app import main
os.environ['PYSPARK_PYTHON'] = "./environment/bin/python"
- spark = SparkSession.builder.master("...").config("spark.archives", "pyspark_env.tar.gz#environment").getOrCreate()
+ spark = SparkSession.builder.config(
+ "spark.archives", # 'spark.yarn.dist.archives' in Yarn.
+ "pyspark_conda_env.tar.gz#environment").getOrCreate()
main(spark)
-Example with ``pyspark`` shell:
+For a pyspark shell:
.. code-block:: bash
export PYSPARK_DRIVER_PYTHON=python
export PYSPARK_PYTHON=./environment/bin/python
- pyspark --master=... --archives pyspark_env.tar.gz#environment
+ pyspark --archives pyspark_conda_env.tar.gz#environment
-Using PEX
----------
+Using Virtualenv
+----------------
-`PEX <https://github.com/pantsbuild/pex>`_ is a library for generating ``.pex`` (Python EXecutable) files.
-A PEX file is a self-contained executable Python environment. It can be seen as the Python equivalent of Java uber-JARs (a.k.a. fat JARs).
+`Virtualenv <https://virtualenv.pypa.io/en/latest/>`_ is a Python tool to create isolated Python environments.
+Since Python 3.3, a subset of its features has been integrated into Python as a standard library under
+the `venv <https://docs.python.org/3/library/venv.html>`_ module. PySpark users can use virtualenv to manage
+Python dependencies in their clusters by using `venv-pack <https://jcristharif.com/venv-pack/index.html>`_
+in a similar way as conda-pack.
-You need to build the PEX file somewhere with all your requirements and then upload it to each Spark executor.
+A virtual environment to use on both driver and executor can be created as demonstrated below.
+It packs the current virtual environment to an archive file, and It self-contains both Python interpreter
+and the dependencies.
-Using CLI to Build PEX file
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: bash
- pex pyspark==3.0.1 pyarrow==0.15.1 pandas==0.25.3 -o myarchive.pex
+ python -m venv pyspark_venv
+ source pyspark_venv/bin/activate
+ pip install pyarrow pandas venv-pack
+ venv-pack -o pyspark_venv.tar.gz
+You can directly pass/unpack the archive file and enable the environment on executors by leveraging
+the ``--archives`` option or ``spark.archives`` configuration (``spark.yarn.dist.archives`` in Yarn).
Review comment:
Very minor but Yarn -> YARN
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-747376792
**[Test build #132952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132952/testReport)** for PR 30822 at commit [`414b135`](https://github.com/apache/spark/commit/414b135ba13bb5469761cdd66d1198c755ca03e5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #30822:
URL: https://github.com/apache/spark/pull/30822#discussion_r545302806
##########
File path: python/docs/source/user_guide/python_packaging.rst
##########
@@ -51,65 +51,63 @@ Here is the script ``app.py`` from the previous example that will be executed on
main(SparkSession.builder.getOrCreate())
-There are multiple ways to ship the dependencies to the cluster:
+There are multiple ways to manage Python dependencies in the cluster:
- Using PySpark Native Features
-- Using Zipped Virtual Environment
+- Using Conda
+- Using Virtualenv
- Using PEX
Using PySpark Native Features
-----------------------------
PySpark allows to upload Python files (``.py``), zipped Python packages (``.zip``), and Egg files (``.egg``)
-to the executors by setting the configuration setting ``spark.submit.pyFiles`` or by directly
-calling :meth:`pyspark.SparkContext.addPyFile`.
+to the executors by:
-This is an easy way to ship additional custom Python code to the cluster. You can just add individual files or zip whole
-packages and upload them. Using :meth:`pyspark.SparkContext.addPyFile` allows to upload code
-even after having started your job.
+- Setting the configuration setting ``spark.submit.pyFiles``
+- Ssing ``--py-files`` option in Spark scripts
Review comment:
Ssing? Setting?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-747872788
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132970/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-747339418
cc @srowen, @zero323, @viirya, @dongjoon-hyun, @erikerlandson, would you mind taking a quick look when you guys find some time?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-747865282
**[Test build #132970 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132970/testReport)** for PR 30822 at commit [`68323e2`](https://github.com/apache/spark/commit/68323e28514228a2497ade477f456b4fda08b1e2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-747822990
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37573/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-747370629
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-747402628
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #30822:
URL: https://github.com/apache/spark/pull/30822#discussion_r545308451
##########
File path: python/docs/source/user_guide/python_packaging.rst
##########
@@ -118,87 +116,139 @@ Example using ``SparkSession.builder``:
from app import main
os.environ['PYSPARK_PYTHON'] = "./environment/bin/python"
- spark = SparkSession.builder.master("...").config("spark.archives", "pyspark_env.tar.gz#environment").getOrCreate()
+ spark = SparkSession.builder.config(
+ "spark.archives", # 'spark.yarn.dist.archives' in Yarn.
+ "pyspark_conda_env.tar.gz#environment").getOrCreate()
main(spark)
-Example with ``pyspark`` shell:
+For a pyspark shell:
.. code-block:: bash
export PYSPARK_DRIVER_PYTHON=python
export PYSPARK_PYTHON=./environment/bin/python
- pyspark --master=... --archives pyspark_env.tar.gz#environment
+ pyspark --archives pyspark_conda_env.tar.gz#environment
-Using PEX
----------
+Using Virtualenv
+----------------
-`PEX <https://github.com/pantsbuild/pex>`_ is a library for generating ``.pex`` (Python EXecutable) files.
-A PEX file is a self-contained executable Python environment. It can be seen as the Python equivalent of Java uber-JARs (a.k.a. fat JARs).
+`Virtualenv <https://virtualenv.pypa.io/en/latest/>`_ is a Python tool to create isolated Python environments.
+Since Python 3.3, a subset of its features has been integrated into Python as a standard library under
+the `venv <https://docs.python.org/3/library/venv.html>`_ module. PySpark users can use virtualenv to manage
+Python dependencies in their clusters by using `venv-pack <https://jcristharif.com/venv-pack/index.html>`_
+in a similar way as conda-pack.
-You need to build the PEX file somewhere with all your requirements and then upload it to each Spark executor.
+A virtual environment to use on both driver and executor can be created as demonstrated below.
+It packs the current virtual environment to an archive file, and It self-contains both Python interpreter
+and the dependencies.
-Using CLI to Build PEX file
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: bash
- pex pyspark==3.0.1 pyarrow==0.15.1 pandas==0.25.3 -o myarchive.pex
+ python -m venv pyspark_venv
+ source pyspark_venv/bin/activate
+ pip install pyarrow pandas venv-pack
+ venv-pack -o pyspark_venv.tar.gz
+You can directly pass/unpack the archive file and enable the environment on executors by leveraging
+the ``--archives`` option or ``spark.archives`` configuration (``spark.yarn.dist.archives`` in Yarn).
-Invoking the PEX file will by default invoke the Python interpreter. pyarrow, pandas and pyspark will be included in the PEX file.
+For ``spark-submit``, you can use it by running the command as follows. Also, notice that
+``PYSPARK_DRIVER_PYTHON`` is not necessary in Kubernetes or Yarn cluster modes.
.. code-block:: bash
- ./myarchive.pex
- Python 3.6.6 (default, Jan 26 2019, 16:53:05)
- (InteractiveConsole)
- >>> import pyarrow
- >>> import pandas
- >>> import pyspark
- >>>
+ export PYSPARK_DRIVER_PYTHON=python
+ export PYSPARK_PYTHON=./environment/bin/python
+ spark-submit --archives pyspark_venv.tar.gz#environment app.py
-This can also be done directly with the Python API. For more information on how to build PEX files,
-please refer to `Building .pex files <https://pex.readthedocs.io/en/stable/buildingpex.html>`_
+For regular Python shells or notebooks:
-Upload to Spark Executors
-~~~~~~~~~~~~~~~~~~~~~~~~~
+.. code-block:: bash
-The upload can be done by setting ``--files`` option in spark-submit or setting ``spark.files`` configuration (``spark.yarn.dist.files`` on YARN)
-and changing the ``PYSPARK_PYTHON`` environment variable to change the Python interpreter to the PEX executable on each executor.
+ import os
+ from pyspark.sql import SparkSession
+ from app import main
-..
- TODO: we should also document the way on other cluster modes.
+ os.environ['PYSPARK_PYTHON'] = "./environment/bin/python"
+ spark = SparkSession.builder.config(
+ "spark.archives", # 'spark.yarn.dist.archives' in Yarn.
+ "pyspark_venv.tar.gz#environment").getOrCreate()
+ main(spark)
-Example with ``spark-submit`` on YARN:
+In the case of a pyspark shell:
.. code-block:: bash
export PYSPARK_DRIVER_PYTHON=python
- export PYSPARK_PYTHON=./myarchive.pex
- spark-submit --master=yarn --deploy-mode client --files myarchive.pex app.py
+ export PYSPARK_PYTHON=./environment/bin/python
+ pyspark --archives pyspark_venv.tar.gz#environment
+
+
+Using PEX
+---------
-Example using ``SparkSession.builder`` on YARN:
+PySpark can also use `PEX <https://github.com/pantsbuild/pex>`_ to ship the Python packages
+together. PEX is a tool that creates a self-contained Python environment. This is similar
+to Conda or virtualenv, but a ``.pex`` file is executable by itself.
+
+The following example creates a ``.pex`` file for the driver and executor to use.
+The file contains the Python dependencies specified with the ``pex`` command.
+
+.. code-block:: bash
+
+ pip install pyarrow pandas pex
+ pex pyspark pyarrow pandas -o pyspark_pex_env.pex
+
+This file behaves similarly with a regular Python interpreter.
+
+.. code-block:: bash
+
+ ./pyspark_pex_env.pex -c "import pandas; print(pandas.__version__)"
+ 1.1.5
+
+However, ``.pex`` file does not include a Python interpreter itself under the hood so all
+nodes in a cluster should have the same Python interpreter installed.
+
+In order to transfer and use the ``.pex`` file in a cluster, you should ship it via the
+``spark.files`` configuration (``spark.yarn.dist.files`` in Yarn) or ``--files`` option because they are regular files instead
Review comment:
Yarn is here too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-747346742
**[Test build #132949 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132949/testReport)** for PR 30822 at commit [`414b135`](https://github.com/apache/spark/commit/414b135ba13bb5469761cdd66d1198c755ca03e5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-747355329
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132949/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-747786974
**[Test build #132970 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132970/testReport)** for PR 30822 at commit [`68323e2`](https://github.com/apache/spark/commit/68323e28514228a2497ade477f456b4fda08b1e2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-747355005
**[Test build #132949 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132949/testReport)** for PR 30822 at commit [`414b135`](https://github.com/apache/spark/commit/414b135ba13bb5469761cdd66d1198c755ca03e5).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #30822:
URL: https://github.com/apache/spark/pull/30822
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-747346742
**[Test build #132949 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132949/testReport)** for PR 30822 at commit [`414b135`](https://github.com/apache/spark/commit/414b135ba13bb5469761cdd66d1198c755ca03e5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-747786974
**[Test build #132970 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132970/testReport)** for PR 30822 at commit [`68323e2`](https://github.com/apache/spark/commit/68323e28514228a2497ade477f456b4fda08b1e2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-749912334
Just in case anyone is interested in, my blog post was published yesterday here https://databricks.com/blog/2020/12/22/how-to-manage-python-dependencies-in-pyspark.html :-).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-747355329
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-747390188
**[Test build #132952 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132952/testReport)** for PR 30822 at commit [`414b135`](https://github.com/apache/spark/commit/414b135ba13bb5469761cdd66d1198c755ca03e5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-747376792
**[Test build #132952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132952/testReport)** for PR 30822 at commit [`414b135`](https://github.com/apache/spark/commit/414b135ba13bb5469761cdd66d1198c755ca03e5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-747872788
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132970/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30822:
URL: https://github.com/apache/spark/pull/30822#issuecomment-747402627
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org