You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/12/17 12:57:28 UTC
[GitHub] [spark] srowen commented on a change in pull request #30822: [SPARK-33824][PYTHON][DOCS] Restructure and improve Python package management page
srowen commented on a change in pull request #30822:
URL: https://github.com/apache/spark/pull/30822#discussion_r545069144
##########
File path: python/docs/source/user_guide/python_packaging.rst
##########
@@ -118,87 +116,139 @@ Example using ``SparkSession.builder``:
from app import main
os.environ['PYSPARK_PYTHON'] = "./environment/bin/python"
- spark = SparkSession.builder.master("...").config("spark.archives", "pyspark_env.tar.gz#environment").getOrCreate()
+ spark = SparkSession.builder.config(
+ "spark.archives", # 'spark.yarn.dist.archives' in Yarn.
+ "pyspark_conda_env.tar.gz#environment").getOrCreate()
main(spark)
-Example with ``pyspark`` shell:
+For a pyspark shell:
.. code-block:: bash
export PYSPARK_DRIVER_PYTHON=python
export PYSPARK_PYTHON=./environment/bin/python
- pyspark --master=... --archives pyspark_env.tar.gz#environment
+ pyspark --archives pyspark_conda_env.tar.gz#environment
-Using PEX
----------
+Using Virtualenv
+----------------
-`PEX <https://github.com/pantsbuild/pex>`_ is a library for generating ``.pex`` (Python EXecutable) files.
-A PEX file is a self-contained executable Python environment. It can be seen as the Python equivalent of Java uber-JARs (a.k.a. fat JARs).
+`Virtualenv <https://virtualenv.pypa.io/en/latest/>`_ is a Python tool to create isolated Python environments.
+Since Python 3.3, a subset of its features has been integrated into Python as a standard library under
+the `venv <https://docs.python.org/3/library/venv.html>`_ module. PySpark users can use virtualenv to manage
+Python dependencies in their clusters by using `venv-pack <https://jcristharif.com/venv-pack/index.html>`_
+in a similar way as conda-pack.
-You need to build the PEX file somewhere with all your requirements and then upload it to each Spark executor.
+A virtual environment to use on both driver and executor can be created as demonstrated below.
+It packs the current virtual environment to an archive file, and It self-contains both Python interpreter
+and the dependencies.
-Using CLI to Build PEX file
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: bash
- pex pyspark==3.0.1 pyarrow==0.15.1 pandas==0.25.3 -o myarchive.pex
+ python -m venv pyspark_venv
+ source pyspark_venv/bin/activate
+ pip install pyarrow pandas venv-pack
+ venv-pack -o pyspark_venv.tar.gz
+You can directly pass/unpack the archive file and enable the environment on executors by leveraging
+the ``--archives`` option or ``spark.archives`` configuration (``spark.yarn.dist.archives`` in Yarn).
Review comment:
Very minor but Yarn -> YARN
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org