You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/08/11 20:55:57 UTC
[GitHub] [spark] rohitmishr1484 opened a new pull request #29410: [WIP][SPARK-32180][PYTHON][DOCS] Getting started-Installation guide for pyspark doc
rohitmishr1484 opened a new pull request #29410:
URL: https://github.com/apache/spark/pull/29410
# What changes were proposed in this pull request?
This PR proposes to add getting started- installation to new PySpark docs.
### Why are the changes needed?
Better documentation.
### Does this PR introduce _any_ user-facing change?
No. Documentation only.
### How was this patch tested?
By generating documents locally. Since its a WIP, I will retest it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r468999216
##########
File path: python/docs/source/getting_started/index.rst
##########
@@ -20,3 +20,13 @@
Getting Started
===============
+**PySpark** is the Python API for Spark.
Review comment:
I think we can remove this.
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,62 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+===============
Review comment:
Let's match the number of `=` with the word below:
```
============
Installation
```
##########
File path: python/docs/source/getting_started/index.rst
##########
@@ -20,3 +20,13 @@
Getting Started
===============
+**PySpark** is the Python API for Spark.
+
+This page lists an overview of the basic steps required to setup & get started with PySpark.
+
+.. toctree::
+ :maxdepth: 2
+
+ installation
+ package_overview
Review comment:
Let's remove `package_overview`, `quickstart` and the corresponding RST files. It's always better to leave a PR focused on one issue without other changes.
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,62 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+===============
+Installation
+===============
+
+Using Conda
+~~~~~~~~~~~~~~~~~~
+PySpark installation using `Conda <https://anaconda.org/conda-forge/pyspark>`_ can be performed using the below command::
+
+ conda install -c conda-forge pyspark
+
+Using PyPI
+~~~~~~~~~~
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_::
+
+ pip install pyspark
+
+Official release channel
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Different flavor of PySpark is available in `the official release channel <https://spark.apache.org/downloads.html>`__.
+Any suitable version can be downloaded and extracted as below::
+
+ tar xzvf spark-3.0.0-bin-hadoop2.7.tgz
+
+An important step is to ensure ``SPARK_HOME`` environment variable points to the directory where the code has been extracted. The next step is to properly define ``PYTHONPATH`` such that it can find the PySpark and Py4J under ``$SPARK_HOME/python/lib``::
+
+ cd spark-3.0.0-bin-hadoop2.7
+
+ export SPARK_HOME=`pwd`
+
+ export PYTHONPATH=$(ZIPS=("$SPARK_HOME"/python/lib/*.zip); IFS=:; echo "${ZIPS[*]}"):$PYTHONPATH
Review comment:
Let's add some words saying this is just one of many examples since there are many ways to add them into `PYTHONPATH`.
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,62 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+===============
+Installation
+===============
+
+Using Conda
Review comment:
Conda is actually not an official release channel of Apache Spark. We can remove this section for now.
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,62 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+===============
+Installation
+===============
+
+Using Conda
+~~~~~~~~~~~~~~~~~~
Review comment:
This, too.
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,62 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+===============
+Installation
+===============
+
+Using Conda
+~~~~~~~~~~~~~~~~~~
+PySpark installation using `Conda <https://anaconda.org/conda-forge/pyspark>`_ can be performed using the below command::
+
+ conda install -c conda-forge pyspark
+
+Using PyPI
+~~~~~~~~~~
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_::
+
+ pip install pyspark
+
+Official release channel
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Different flavor of PySpark is available in `the official release channel <https://spark.apache.org/downloads.html>`__.
+Any suitable version can be downloaded and extracted as below::
+
+ tar xzvf spark-3.0.0-bin-hadoop2.7.tgz
+
+An important step is to ensure ``SPARK_HOME`` environment variable points to the directory where the code has been extracted. The next step is to properly define ``PYTHONPATH`` such that it can find the PySpark and Py4J under ``$SPARK_HOME/python/lib``::
+
+ cd spark-3.0.0-bin-hadoop2.7
+
+ export SPARK_HOME=`pwd`
+
+ export PYTHONPATH=$(ZIPS=("$SPARK_HOME"/python/lib/*.zip); IFS=:; echo "${ZIPS[*]}"):$PYTHONPATH
+
+Installing from source
+~~~~~~~~~~~~~~~~~~~~~~
+
+To install PySpark from source, refer `Building Spark <https://spark.apache.org/docs/latest/building-spark.html>`__.
+
+* Steps for defining ``PYTHONPATH`` is same as described in `Official release channel` section above.
+
+
+Dependencies
+------------
+* Using PySpark requires the Spark JARs.
+* At its core PySpark depends on Py4J, but some additional sub-packages have their own extra requirements for some features (including NumPy, pandas, and PyArrow).
Review comment:
I would make a table like https://koalas.readthedocs.io/en/latest/getting_started/install.html#dependencies based on https://github.com/apache/spark/blob/master/python/setup.py#L206-L214
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-686445346
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r472917349
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,84 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Installation
+============
+
+Using PyPI
+~~~~~~~~~~
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_::
+
+ pip install pyspark
+
+Using Miniconda
+~~~~~~~~~~~~~~~
+Pyspark is available in Conda but it is not a part of an official release. Conda can be used to create a virtual environment from terminal as shown below::
+
+ conda create -n ENV_NAME python==PYTHON_VERSION
+
+After the virtual environment is created, it should be visible under the list of conda environments which can be seen using the following command::
+
+ conda env list
+
+The newly created environment can be accessed using the following command::
+
+ source activate ENV_NAME
+
+On Windows, the command is::
+
+ conda activate ENV_NAME
+
+PySpark can be installed in this newly created environment using PyPI as shown before::
+
+ pip install pyspark
+
+Official release channel
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Different flavor of PySpark is available in `the official release channel <https://spark.apache.org/downloads.html>`__.
+Any suitable version can be downloaded and extracted as below::
+
+ tar xzvf spark-3.0.0-bin-hadoop2.7.tgz
+
+An important step is to ensure ``SPARK_HOME`` environment variable points to the directory where the code has been extracted. The next step is to properly define ``PYTHONPATH`` such that it can find the PySpark and Py4J under ``$SPARK_HOME/python/lib``, one example of doing this is shown below::
+
+ cd spark-3.0.0-bin-hadoop2.7
+
+ export SPARK_HOME=`pwd`
+
+ export PYTHONPATH=$(ZIPS=("$SPARK_HOME"/python/lib/*.zip); IFS=:; echo "${ZIPS[*]}"):$PYTHONPATH
+
+Installing from source
+~~~~~~~~~~~~~~~~~~~~~~
+
+To install PySpark from source, refer `Building Spark <https://spark.apache.org/docs/latest/building-spark.html>`__.
+
+* Steps for defining ``PYTHONPATH`` is same as described in `Official release channel` section above.
Review comment:
I wouldn't make it as a bullet point
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-684767923
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-676087526
> Personally I have used only PyPI as an installation mechanism thus haven't tried the other three. A prerequisite for PyPI installation was the availability of JAVA 8 path in JAVA_HOME environment variable. Please let me know if that's something I need to add explicitly in the "Dependencies" section.
This is a good point .. we should mention Java is required. Currently, yes, Java 8 should be available in the system.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rohitmishr1484 commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
rohitmishr1484 commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r480980405
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,84 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Installation
+============
+
+Using PyPI
Review comment:
@HyukjinKwon , I looked at the link you suggested but since there is a section on "Installing From Source", adding this again in the suggested place seemed duplicate to me. Am I missing something?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-672576869
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-686463470
Oops, looks something went wrong during syncing. Feel free to close this and create new PR :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-674282565
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rohitmishr1484 edited a comment on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
rohitmishr1484 edited a comment on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-674294866
@HyukjinKwon,
Thanks for your helpful comment. I have done the suggested changes but if you still find something which requires modification, please let me know, I will update it.
One question:
The "dependencies" contain the version which is slightly different from 2.4.X setup.py. Shall we add for both or the current one is sufficient?
Thanks!!!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r472916065
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,84 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Installation
+============
+
+Using PyPI
+~~~~~~~~~~
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_::
+
+ pip install pyspark
+
+Using Miniconda
+~~~~~~~~~~~~~~~
+Pyspark is available in Conda but it is not a part of an official release. Conda can be used to create a virtual environment from terminal as shown below::
+
+ conda create -n ENV_NAME python==PYTHON_VERSION
Review comment:
While we're here, let's also mention about the supported Python versions (see https://github.com/apache/spark/blob/master/python/setup.py#L219-L221)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r482726122
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,120 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+============
+Installation
+============
+
+The official release channel is to download it from https://spark.apache.org/downloads.html but we can install it via ``pip`` as well from PyPI. PyPI installation is usually to use standalone locally or as a client to connect to a cluster.
+
+Instruction for downloading PySpark using PyPI, Conda, Official Release Channel and Source are available in this document.
+
+Python Version Supported
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Python 3.6, 3.7 and 3.8
+
+Using PyPI
+~~~~~~~~~~
+
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_
+
+.. code-block:: bash
+
+ pip install pyspark
+
+Using Conda
+~~~~~~~~~~~
+
+Conda is an open-source package management and environment management system which is a part of `Anaconda <https://docs.continuum.io/anaconda/>`_ distribution. It is both cross-platform and language agnostic.
+
+Conda can be used to create a virtual environment from terminal as shown below:
+
+.. code-block:: bash
+
+ conda create -n pyspark_env
+
+After the virtual environment is created, it should be visible under the list of conda environments which can be seen using the following command:
+
+.. code-block:: bash
+
+ conda env list
+
+The newly created environment can be accessed using the following command:
+
+.. code-block:: bash
+
+ conda activate pyspark_env
+
+In lower Conda version, the following command might be used:
+
+.. code-block:: bash
+
+ source activate pyspark_env
+
+PySpark installation using ``pip`` under Conda environment is official.
+
+PySpark can be installed in this newly created environment using PyPI as shown before:
+
+
+.. code-block:: bash
+
+ pip install pyspark
+
+`PySpark at Conda <https://anaconda.org/conda-forge/pyspark>`__ is not the official release.
+
+Official Release Channel
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Different flavor of PySpark is available in `the official release channel <https://spark.apache.org/downloads.html>`__.
+Any suitable version can be downloaded and extracted as below:
+
+.. code-block:: bash
+
+ tar xzvf spark-3.0.0-bin-hadoop2.7.tgz
+
+An important step is to ensure ``SPARK_HOME`` environment variable points to the directory where the code has been extracted.
+The next step is to properly define ``PYTHONPATH`` such that it can find the PySpark and
+Py4J under ``$SPARK_HOME/python/lib``, one example of doing this is shown below:
+
+.. code-block:: bash
+
+ cd spark-3.0.0-bin-hadoop2.7
+
+ export SPARK_HOME=`pwd`
+
+ export PYTHONPATH=$(ZIPS=("$SPARK_HOME"/python/lib/*.zip); IFS=:; echo "${ZIPS[*]}"):$PYTHONPATH
+
+Installing From Source
+~~~~~~~~~~~~~~~~~~~~~~
Review comment:
I think we should use `-` for all subsections here consistently.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rohitmishr1484 commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
rohitmishr1484 commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r470866980
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,62 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+===============
Review comment:
Done
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,62 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+===============
+Installation
+===============
+
+Using Conda
+~~~~~~~~~~~~~~~~~~
Review comment:
Done
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,62 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+===============
+Installation
+===============
+
+Using Conda
Review comment:
Added this line
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,62 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+===============
+Installation
+===============
+
+Using Conda
+~~~~~~~~~~~~~~~~~~
+PySpark installation using `Conda <https://anaconda.org/conda-forge/pyspark>`_ can be performed using the below command::
+
+ conda install -c conda-forge pyspark
+
+Using PyPI
+~~~~~~~~~~
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_::
+
+ pip install pyspark
+
+Official release channel
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Different flavor of PySpark is available in `the official release channel <https://spark.apache.org/downloads.html>`__.
+Any suitable version can be downloaded and extracted as below::
+
+ tar xzvf spark-3.0.0-bin-hadoop2.7.tgz
+
+An important step is to ensure ``SPARK_HOME`` environment variable points to the directory where the code has been extracted. The next step is to properly define ``PYTHONPATH`` such that it can find the PySpark and Py4J under ``$SPARK_HOME/python/lib``::
+
+ cd spark-3.0.0-bin-hadoop2.7
+
+ export SPARK_HOME=`pwd`
+
+ export PYTHONPATH=$(ZIPS=("$SPARK_HOME"/python/lib/*.zip); IFS=:; echo "${ZIPS[*]}"):$PYTHONPATH
Review comment:
Added this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r472904581
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,84 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Installation
+============
+
+Using PyPI
+~~~~~~~~~~
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_::
+
+ pip install pyspark
+
+Using Miniconda
+~~~~~~~~~~~~~~~
+Pyspark is available in Conda but it is not a part of an official release. Conda can be used to create a virtual environment from terminal as shown below::
+
+ conda create -n ENV_NAME python==PYTHON_VERSION
+
+After the virtual environment is created, it should be visible under the list of conda environments which can be seen using the following command::
+
+ conda env list
+
+The newly created environment can be accessed using the following command::
+
+ source activate ENV_NAME
+
+On Windows, the command is::
+
+ conda activate ENV_NAME
+
+PySpark can be installed in this newly created environment using PyPI as shown before::
+
+ pip install pyspark
+
+Official release channel
Review comment:
I think it should be `Official Release Channel` to be consistent with other title styles in Spark documentation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-672576869
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r482725887
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,120 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
Review comment:
Let's add two leading spaces here just to match with other files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-672568232
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r472910807
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,84 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Installation
+============
+
+Using PyPI
+~~~~~~~~~~
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_::
+
+ pip install pyspark
+
+Using Miniconda
+~~~~~~~~~~~~~~~
+Pyspark is available in Conda but it is not a part of an official release. Conda can be used to create a virtual environment from terminal as shown below::
Review comment:
Just to share the full context,
Using `pip` under Conda environment is official.
[PySpark at Conda](https://anaconda.org/conda-forge/pyspark) is not the official release.
I think you can say this ^ at the end this section as a FYI to end users.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-684775771
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128148/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r482722762
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,120 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+============
+Installation
+============
+
+The official release channel is to download it from https://spark.apache.org/downloads.html but we can install it via ``pip`` as well from PyPI. PyPI installation is usually to use standalone locally or as a client to connect to a cluster.
+
+Instruction for downloading PySpark using PyPI, Conda, Official Release Channel and Source are available in this document.
+
+Python Version Supported
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Python 3.6, 3.7 and 3.8
Review comment:
I think you can just say Python 3.6 and above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r482722565
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,120 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+============
+Installation
+============
+
+The official release channel is to download it from https://spark.apache.org/downloads.html but we can install it via ``pip`` as well from PyPI. PyPI installation is usually to use standalone locally or as a client to connect to a cluster.
+
+Instruction for downloading PySpark using PyPI, Conda, Official Release Channel and Source are available in this document.
+
+Python Version Supported
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Python 3.6, 3.7 and 3.8
+
+Using PyPI
+~~~~~~~~~~
+
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_
+
+.. code-block:: bash
+
+ pip install pyspark
+
+Using Conda
+~~~~~~~~~~~
+
+Conda is an open-source package management and environment management system which is a part of `Anaconda <https://docs.continuum.io/anaconda/>`_ distribution. It is both cross-platform and language agnostic.
+
+Conda can be used to create a virtual environment from terminal as shown below:
+
+.. code-block:: bash
+
+ conda create -n pyspark_env
+
+After the virtual environment is created, it should be visible under the list of conda environments which can be seen using the following command:
+
+.. code-block:: bash
+
+ conda env list
+
+The newly created environment can be accessed using the following command:
+
+.. code-block:: bash
+
+ conda activate pyspark_env
+
+In lower Conda version, the following command might be used:
+
+.. code-block:: bash
+
+ source activate pyspark_env
+
+PySpark installation using ``pip`` under Conda environment is official.
+
+PySpark can be installed in this newly created environment using PyPI as shown before:
+
+
+.. code-block:: bash
+
+ pip install pyspark
+
+`PySpark at Conda <https://anaconda.org/conda-forge/pyspark>`__ is not the official release.
+
+Official Release Channel
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Different flavor of PySpark is available in `the official release channel <https://spark.apache.org/downloads.html>`__.
+Any suitable version can be downloaded and extracted as below:
+
+.. code-block:: bash
+
+ tar xzvf spark-3.0.0-bin-hadoop2.7.tgz
+
+An important step is to ensure ``SPARK_HOME`` environment variable points to the directory where the code has been extracted.
+The next step is to properly define ``PYTHONPATH`` such that it can find the PySpark and
+Py4J under ``$SPARK_HOME/python/lib``, one example of doing this is shown below:
+
+.. code-block:: bash
+
+ cd spark-3.0.0-bin-hadoop2.7
+
+ export SPARK_HOME=`pwd`
+
+ export PYTHONPATH=$(ZIPS=("$SPARK_HOME"/python/lib/*.zip); IFS=:; echo "${ZIPS[*]}"):$PYTHONPATH
+
+Installing From Source
Review comment:
From -> from
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-684775763
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-674274381
**[Test build #127465 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127465/testReport)** for PR 29410 at commit [`5df88b4`](https://github.com/apache/spark/commit/5df88b4cd441116cb023d932a8023cd8ef068307).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-686466389
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-686444651
**[Test build #128258 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128258/testReport)** for PR 29410 at commit [`001814f`](https://github.com/apache/spark/commit/001814ff7e4fcbdb3602cc7588d47bb7e53b8bd9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r472899455
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,84 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Installation
+============
+
+Using PyPI
+~~~~~~~~~~
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_::
+
+ pip install pyspark
+
+Using Miniconda
+~~~~~~~~~~~~~~~
+Pyspark is available in Conda but it is not a part of an official release. Conda can be used to create a virtual environment from terminal as shown below::
+
+ conda create -n ENV_NAME python==PYTHON_VERSION
+
+After the virtual environment is created, it should be visible under the list of conda environments which can be seen using the following command::
+
+ conda env list
+
+The newly created environment can be accessed using the following command::
+
+ source activate ENV_NAME
+
+On Windows, the command is::
+
+ conda activate ENV_NAME
+
+PySpark can be installed in this newly created environment using PyPI as shown before::
+
+ pip install pyspark
+
+Official release channel
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Different flavor of PySpark is available in `the official release channel <https://spark.apache.org/downloads.html>`__.
+Any suitable version can be downloaded and extracted as below::
+
+ tar xzvf spark-3.0.0-bin-hadoop2.7.tgz
+
+An important step is to ensure ``SPARK_HOME`` environment variable points to the directory where the code has been extracted. The next step is to properly define ``PYTHONPATH`` such that it can find the PySpark and Py4J under ``$SPARK_HOME/python/lib``, one example of doing this is shown below::
+
+ cd spark-3.0.0-bin-hadoop2.7
+
+ export SPARK_HOME=`pwd`
+
+ export PYTHONPATH=$(ZIPS=("$SPARK_HOME"/python/lib/*.zip); IFS=:; echo "${ZIPS[*]}"):$PYTHONPATH
+
+Installing from source
+~~~~~~~~~~~~~~~~~~~~~~
+
+To install PySpark from source, refer `Building Spark <https://spark.apache.org/docs/latest/building-spark.html>`__.
+
+* Steps for defining ``PYTHONPATH`` is same as described in `Official release channel` section above.
+
+
+Dependencies
+------------
+============= =========================
+Package Minimum supported version
+============= =========================
+`pandas` 0.23.2
+`NumPy` 1.7
+`pyarrow` 0.15.1
+`Py4J` 0.10.9
Review comment:
Let's add another column to describe what is required and what is optional. `pandas` and `pyarrow` is optional for SQL related features. NumPy is required to use it in ML. `Py4J` is required to run PySpark.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r472918336
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,84 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Installation
+============
+
+Using PyPI
+~~~~~~~~~~
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_::
+
+ pip install pyspark
+
+Using Miniconda
+~~~~~~~~~~~~~~~
+Pyspark is available in Conda but it is not a part of an official release. Conda can be used to create a virtual environment from terminal as shown below::
+
+ conda create -n ENV_NAME python==PYTHON_VERSION
+
+After the virtual environment is created, it should be visible under the list of conda environments which can be seen using the following command::
+
+ conda env list
+
+The newly created environment can be accessed using the following command::
+
+ source activate ENV_NAME
+
+On Windows, the command is::
+
+ conda activate ENV_NAME
+
+PySpark can be installed in this newly created environment using PyPI as shown before::
+
+ pip install pyspark
+
+Official release channel
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Different flavor of PySpark is available in `the official release channel <https://spark.apache.org/downloads.html>`__.
+Any suitable version can be downloaded and extracted as below::
+
+ tar xzvf spark-3.0.0-bin-hadoop2.7.tgz
+
+An important step is to ensure ``SPARK_HOME`` environment variable points to the directory where the code has been extracted. The next step is to properly define ``PYTHONPATH`` such that it can find the PySpark and Py4J under ``$SPARK_HOME/python/lib``, one example of doing this is shown below::
+
+ cd spark-3.0.0-bin-hadoop2.7
+
+ export SPARK_HOME=`pwd`
+
+ export PYTHONPATH=$(ZIPS=("$SPARK_HOME"/python/lib/*.zip); IFS=:; echo "${ZIPS[*]}"):$PYTHONPATH
+
+Installing from source
+~~~~~~~~~~~~~~~~~~~~~~
+
+To install PySpark from source, refer `Building Spark <https://spark.apache.org/docs/latest/building-spark.html>`__.
+
+* Steps for defining ``PYTHONPATH`` is same as described in `Official release channel` section above.
+
+
+Dependencies
+------------
Review comment:
Let's also add an overview about required and optional dependencies (see https://github.com/apache/spark/pull/29410/files#r472899455)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rohitmishr1484 edited a comment on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
rohitmishr1484 edited a comment on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-672391444
Hi @HyukjinKwon,
I was not sure how to add you as a Reviewer for this Pull request, thus adding this comment. I would like to mention a few points:
1. Baseline description: I have used this pull request as a reference since this is my first pull request- #29385
or https://github.com/apache/spark/pull/29385.
2. Most of the information used for the "Installation Page" has come from Koalas documentation-https://koalas.readthedocs.io/en/latest/getting_started/install.html. Am I supposed to mention this as reference/credit anywhere in this documentation?
3. Personally I have used only PyPI as an installation mechanism thus haven't tried the other three. A prerequisite for PyPI installation was the availability of JAVA 8 path in JAVA_HOME environment variable. Please let me know if that's something I need to add explicitly mention in the "Dependencies" section.
4. Where ever necessary I have updated the links.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-684763538
**[Test build #128147 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128147/testReport)** for PR 29410 at commit [`4284fb4`](https://github.com/apache/spark/commit/4284fb41b5c31ca9ff1bc92f68d4acfee06b3d57).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29410: [SPARK-32180][PYSPARK][DOCS] Getting started-Installation guide for pyspark doc
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-672278185
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-686271379
@rohitmishr1484, thanks. Looks getting there. Can you resolve the conflicts? See also "The Review Process" at https://spark.apache.org/contributing.html to make this PR synced to the latest master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-684765175
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-672567923
**[Test build #127366 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127366/testReport)** for PR 29410 at commit [`e93b4e1`](https://github.com/apache/spark/commit/e93b4e187329649113bda7cbfc8dedbca719ebbf).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r482723501
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,120 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+============
+Installation
+============
+
+The official release channel is to download it from https://spark.apache.org/downloads.html but we can install it via ``pip`` as well from PyPI. PyPI installation is usually to use standalone locally or as a client to connect to a cluster.
Review comment:
Let's make it properly linked and slightly reword, for example:
```
The official release channel is to download it from `the Apache Spark website < https://spark.apache.org/downloads.html>`_.
Alternatively, you can also install it via pip from PyPI. PyPI installation is usually to use
standalone locally or as a client to connect to a cluster instead of setting a cluster up.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r472909194
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,84 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Installation
+============
+
+Using PyPI
+~~~~~~~~~~
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_::
+
+ pip install pyspark
+
+Using Miniconda
+~~~~~~~~~~~~~~~
+Pyspark is available in Conda but it is not a part of an official release. Conda can be used to create a virtual environment from terminal as shown below::
+
+ conda create -n ENV_NAME python==PYTHON_VERSION
+
+After the virtual environment is created, it should be visible under the list of conda environments which can be seen using the following command::
+
+ conda env list
+
+The newly created environment can be accessed using the following command::
+
+ source activate ENV_NAME
+
+On Windows, the command is::
+
+ conda activate ENV_NAME
+
+PySpark can be installed in this newly created environment using PyPI as shown before::
+
+ pip install pyspark
+
+Official release channel
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Different flavor of PySpark is available in `the official release channel <https://spark.apache.org/downloads.html>`__.
+Any suitable version can be downloaded and extracted as below::
+
+ tar xzvf spark-3.0.0-bin-hadoop2.7.tgz
+
+An important step is to ensure ``SPARK_HOME`` environment variable points to the directory where the code has been extracted. The next step is to properly define ``PYTHONPATH`` such that it can find the PySpark and Py4J under ``$SPARK_HOME/python/lib``, one example of doing this is shown below::
+
+ cd spark-3.0.0-bin-hadoop2.7
+
+ export SPARK_HOME=`pwd`
+
+ export PYTHONPATH=$(ZIPS=("$SPARK_HOME"/python/lib/*.zip); IFS=:; echo "${ZIPS[*]}"):$PYTHONPATH
+
+Installing from source
+~~~~~~~~~~~~~~~~~~~~~~
+
+To install PySpark from source, refer `Building Spark <https://spark.apache.org/docs/latest/building-spark.html>`__.
+
+* Steps for defining ``PYTHONPATH`` is same as described in `Official release channel` section above.
Review comment:
I think you can link the section above via RST style. Something like `` `Official Release Channel <#official-release-channel>`_ ``
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29410: [WIP][SPARK-32180][PYTHON][DOCS] Getting started-Installation guide for pyspark doc
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-672277585
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rohitmishr1484 closed pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
rohitmishr1484 closed pull request #29410:
URL: https://github.com/apache/spark/pull/29410
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-686444651
**[Test build #128258 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128258/testReport)** for PR 29410 at commit [`001814f`](https://github.com/apache/spark/commit/001814ff7e4fcbdb3602cc7588d47bb7e53b8bd9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-672567923
**[Test build #127366 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127366/testReport)** for PR 29410 at commit [`e93b4e1`](https://github.com/apache/spark/commit/e93b4e187329649113bda7cbfc8dedbca719ebbf).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r472901225
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,84 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Installation
+============
+
Review comment:
Let's write a short overview. For example, like [pandas'](https://pandas.pydata.org/pandas-docs/stable/getting_started/install.html#installation).
The official release channel is to download it from https://spark.apache.org/downloads.html but we can install it via pip as well from PyPi. PyPi installation is usually to use standalone locally or as a client to connect to a cluster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r472911147
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,84 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Installation
+============
+
+Using PyPI
+~~~~~~~~~~
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_::
+
+ pip install pyspark
+
+Using Miniconda
Review comment:
I think you can just say Conda instead of Miniconda
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rohitmishr1484 commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
rohitmishr1484 commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-684767965
@HyukjinKwon ,
Apology for the delay in updating the required changes. I have made the suggested changes and I pushed it for your review.
I can see a conflict test failed and trying to understand how to fix it. If I understanding correctly this will automatically be taken care while merging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rohitmishr1484 commented on pull request #29410: [SPARK-32180][PYSPARK][DOCS] Getting started-Installation guide for pyspark doc
Posted by GitBox <gi...@apache.org>.
rohitmishr1484 commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-672391444
Hi @HyukjinKwon,
I was not sure how to add you as a Reviewer for this Pull request, thus adding this comment. I would like to mention a few points:
1. Baseline description: I have used this pull request as a reference since this is my first pull request- #29385
or https://github.com/apache/spark/pull/29385.
2. Most of the information used for the "Installation Page" has come from Koalas documentation-https://koalas.readthedocs.io/en/latest/getting_started/install.html. Am I supposed to mention this as reference/credit anywhere in this documentation?
3. Personally I have used only PyPI as an installation mechanism thus haven't tried the other three. A prerequisite for PyPI installation was the availability of JAVA 8 path in JAVA_HOME environment variable. Please let me know if that's something I need to add explicitly in the "Dependencies" section.
4. Where ever necessary I have updated the links.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-684775763
Build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-684764329
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29410: [WIP][SPARK-32180][PYTHON][DOCS] Getting started-Installation guide for pyspark doc
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-672278185
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r472901489
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,84 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Installation
+============
+
+Using PyPI
+~~~~~~~~~~
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_::
Review comment:
I believe strictly we should add a space between the title and contents.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r472912056
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,84 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Installation
+============
+
+Using PyPI
+~~~~~~~~~~
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_::
+
+ pip install pyspark
+
+Using Miniconda
+~~~~~~~~~~~~~~~
+Pyspark is available in Conda but it is not a part of an official release. Conda can be used to create a virtual environment from terminal as shown below::
+
+ conda create -n ENV_NAME python==PYTHON_VERSION
Review comment:
Let's remove `python==PYTHON_VERSION`, and switch `ENV_NAME` to something like `pyspark_env` so users can copy and paste to run.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r472914766
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,84 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Installation
+============
+
+Using PyPI
+~~~~~~~~~~
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_::
+
+ pip install pyspark
Review comment:
I would use the explicit codeblock such as:
```
.. code-block:: bash
pip install pyspark
```
this can syntax-highlight as well IIRC.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-674282273
**[Test build #127465 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127465/testReport)** for PR 29410 at commit [`5df88b4`](https://github.com/apache/spark/commit/5df88b4cd441116cb023d932a8023cd8ef068307).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29410: [SPARK-32180][PYSPARK][DOCS] Getting started-Installation guide for pyspark doc
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-672566533
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r472915029
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,84 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Installation
+============
+
+Using PyPI
+~~~~~~~~~~
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_::
+
+ pip install pyspark
+
+Using Miniconda
+~~~~~~~~~~~~~~~
+Pyspark is available in Conda but it is not a part of an official release. Conda can be used to create a virtual environment from terminal as shown below::
+
+ conda create -n ENV_NAME python==PYTHON_VERSION
+
+After the virtual environment is created, it should be visible under the list of conda environments which can be seen using the following command::
+
+ conda env list
+
+The newly created environment can be accessed using the following command::
+
+ source activate ENV_NAME
+
+On Windows, the command is::
+
+ conda activate ENV_NAME
+
+PySpark can be installed in this newly created environment using PyPI as shown before::
+
+ pip install pyspark
+
+Official release channel
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Different flavor of PySpark is available in `the official release channel <https://spark.apache.org/downloads.html>`__.
+Any suitable version can be downloaded and extracted as below::
+
+ tar xzvf spark-3.0.0-bin-hadoop2.7.tgz
+
+An important step is to ensure ``SPARK_HOME`` environment variable points to the directory where the code has been extracted. The next step is to properly define ``PYTHONPATH`` such that it can find the PySpark and Py4J under ``$SPARK_HOME/python/lib``, one example of doing this is shown below::
+
+ cd spark-3.0.0-bin-hadoop2.7
+
Review comment:
Let's don't add the space (and do the `.. code-block:: bash` for this and other places).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rohitmishr1484 commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
rohitmishr1484 commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-674294866
@HyukjinKwon,
Thanks for your helpful comment. I have done the suggested changes but if you still find something which requires modification, please let me know, I will update it.
Thanks!!!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-674274794
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-674274794
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rohitmishr1484 commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
rohitmishr1484 commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r470866885
##########
File path: python/docs/source/getting_started/index.rst
##########
@@ -20,3 +20,13 @@
Getting Started
===============
+**PySpark** is the Python API for Spark.
+
+This page lists an overview of the basic steps required to setup & get started with PySpark.
+
+.. toctree::
+ :maxdepth: 2
+
+ installation
+ package_overview
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r472913018
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,84 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Installation
+============
+
+Using PyPI
+~~~~~~~~~~
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_::
+
+ pip install pyspark
+
+Using Miniconda
+~~~~~~~~~~~~~~~
+Pyspark is available in Conda but it is not a part of an official release. Conda can be used to create a virtual environment from terminal as shown below::
+
+ conda create -n ENV_NAME python==PYTHON_VERSION
+
+After the virtual environment is created, it should be visible under the list of conda environments which can be seen using the following command::
+
+ conda env list
+
+The newly created environment can be accessed using the following command::
+
+ source activate ENV_NAME
Review comment:
Now `source activate` is not encouraged anymore up to my knowledge, and they encourage to use `conda activate` in general, see also https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html.
We can remove `source activate ENV_NAME`. Alternative, we can also say `source activate` might have to be used in lower Conda.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rohitmishr1484 commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
rohitmishr1484 commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r470867265
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,62 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+===============
+Installation
+===============
+
+Using Conda
+~~~~~~~~~~~~~~~~~~
+PySpark installation using `Conda <https://anaconda.org/conda-forge/pyspark>`_ can be performed using the below command::
+
+ conda install -c conda-forge pyspark
+
+Using PyPI
+~~~~~~~~~~
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_::
+
+ pip install pyspark
+
+Official release channel
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Different flavor of PySpark is available in `the official release channel <https://spark.apache.org/downloads.html>`__.
+Any suitable version can be downloaded and extracted as below::
+
+ tar xzvf spark-3.0.0-bin-hadoop2.7.tgz
+
+An important step is to ensure ``SPARK_HOME`` environment variable points to the directory where the code has been extracted. The next step is to properly define ``PYTHONPATH`` such that it can find the PySpark and Py4J under ``$SPARK_HOME/python/lib``::
+
+ cd spark-3.0.0-bin-hadoop2.7
+
+ export SPARK_HOME=`pwd`
+
+ export PYTHONPATH=$(ZIPS=("$SPARK_HOME"/python/lib/*.zip); IFS=:; echo "${ZIPS[*]}"):$PYTHONPATH
+
+Installing from source
+~~~~~~~~~~~~~~~~~~~~~~
+
+To install PySpark from source, refer `Building Spark <https://spark.apache.org/docs/latest/building-spark.html>`__.
+
+* Steps for defining ``PYTHONPATH`` is same as described in `Official release channel` section above.
+
+
+Dependencies
+------------
+* Using PySpark requires the Spark JARs.
+* At its core PySpark depends on Py4J, but some additional sub-packages have their own extra requirements for some features (including NumPy, pandas, and PyArrow).
Review comment:
Added this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-684763538
**[Test build #128147 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128147/testReport)** for PR 29410 at commit [`4284fb4`](https://github.com/apache/spark/commit/4284fb41b5c31ca9ff1bc92f68d4acfee06b3d57).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-672576899
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127366/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-686445346
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r472911447
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,84 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Installation
+============
+
+Using PyPI
+~~~~~~~~~~
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_::
+
+ pip install pyspark
+
+Using Miniconda
+~~~~~~~~~~~~~~~
Review comment:
Likewise, let's add a space between the title and contents.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-672575936
**[Test build #127366 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127366/testReport)** for PR 29410 at commit [`e93b4e1`](https://github.com/apache/spark/commit/e93b4e187329649113bda7cbfc8dedbca719ebbf).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r472917673
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,84 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Installation
+============
+
+Using PyPI
Review comment:
You can borrow some words from https://spark.apache.org/docs/latest/building-spark.html#pyspark-pip-installable, and link it back.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r482722143
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,120 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+============
+Installation
+============
+
+The official release channel is to download it from https://spark.apache.org/downloads.html but we can install it via ``pip`` as well from PyPI. PyPI installation is usually to use standalone locally or as a client to connect to a cluster.
+
+Instruction for downloading PySpark using PyPI, Conda, Official Release Channel and Source are available in this document.
+
+Python Version Supported
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Python 3.6, 3.7 and 3.8
+
+Using PyPI
+~~~~~~~~~~
+
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_
+
+.. code-block:: bash
+
+ pip install pyspark
+
+Using Conda
+~~~~~~~~~~~
+
+Conda is an open-source package management and environment management system which is a part of `Anaconda <https://docs.continuum.io/anaconda/>`_ distribution. It is both cross-platform and language agnostic.
+
+Conda can be used to create a virtual environment from terminal as shown below:
+
+.. code-block:: bash
+
+ conda create -n pyspark_env
+
+After the virtual environment is created, it should be visible under the list of conda environments which can be seen using the following command:
+
+.. code-block:: bash
+
+ conda env list
+
+The newly created environment can be accessed using the following command:
+
+.. code-block:: bash
+
+ conda activate pyspark_env
+
+In lower Conda version, the following command might be used:
+
+.. code-block:: bash
+
+ source activate pyspark_env
+
+PySpark installation using ``pip`` under Conda environment is official.
+
+PySpark can be installed in this newly created environment using PyPI as shown before:
+
+
+.. code-block:: bash
+
+ pip install pyspark
+
+`PySpark at Conda <https://anaconda.org/conda-forge/pyspark>`__ is not the official release.
+
+Official Release Channel
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Different flavor of PySpark is available in `the official release channel <https://spark.apache.org/downloads.html>`__.
+Any suitable version can be downloaded and extracted as below:
+
+.. code-block:: bash
+
+ tar xzvf spark-3.0.0-bin-hadoop2.7.tgz
+
+An important step is to ensure ``SPARK_HOME`` environment variable points to the directory where the code has been extracted.
+The next step is to properly define ``PYTHONPATH`` such that it can find the PySpark and
+Py4J under ``$SPARK_HOME/python/lib``, one example of doing this is shown below:
+
+.. code-block:: bash
+
+ cd spark-3.0.0-bin-hadoop2.7
+
+ export SPARK_HOME=`pwd`
+
+ export PYTHONPATH=$(ZIPS=("$SPARK_HOME"/python/lib/*.zip); IFS=:; echo "${ZIPS[*]}"):$PYTHONPATH
+
+Installing From Source
+~~~~~~~~~~~~~~~~~~~~~~
+
+To install PySpark from source, refer `Building Spark <https://spark.apache.org/docs/latest/building-spark.html>`__.
+
+Steps for defining ``PYTHONPATH`` is same as described in `Official Release Channel <#official-release-channel>`_.
+
+Dependencies
+------------
+============= ========================= ====================
+Package Minimum supported version Required or Optional
Review comment:
Let's convert `Required or Optional` to `Note`.
`NumPy` is an optional dependency for ML module in PySpark.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r482721843
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,120 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+============
+Installation
+============
+
+The official release channel is to download it from https://spark.apache.org/downloads.html but we can install it via ``pip`` as well from PyPI. PyPI installation is usually to use standalone locally or as a client to connect to a cluster.
+
+Instruction for downloading PySpark using PyPI, Conda, Official Release Channel and Source are available in this document.
+
+Python Version Supported
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Python 3.6, 3.7 and 3.8
+
+Using PyPI
+~~~~~~~~~~
+
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_
+
+.. code-block:: bash
+
+ pip install pyspark
+
+Using Conda
+~~~~~~~~~~~
+
+Conda is an open-source package management and environment management system which is a part of `Anaconda <https://docs.continuum.io/anaconda/>`_ distribution. It is both cross-platform and language agnostic.
+
+Conda can be used to create a virtual environment from terminal as shown below:
+
+.. code-block:: bash
+
+ conda create -n pyspark_env
+
+After the virtual environment is created, it should be visible under the list of conda environments which can be seen using the following command:
+
+.. code-block:: bash
+
+ conda env list
+
+The newly created environment can be accessed using the following command:
+
+.. code-block:: bash
+
+ conda activate pyspark_env
+
+In lower Conda version, the following command might be used:
+
+.. code-block:: bash
+
+ source activate pyspark_env
+
+PySpark installation using ``pip`` under Conda environment is official.
+
+PySpark can be installed in this newly created environment using PyPI as shown before:
+
+
+.. code-block:: bash
+
+ pip install pyspark
+
+`PySpark at Conda <https://anaconda.org/conda-forge/pyspark>`__ is not the official release.
+
+Official Release Channel
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Different flavor of PySpark is available in `the official release channel <https://spark.apache.org/downloads.html>`__.
+Any suitable version can be downloaded and extracted as below:
+
+.. code-block:: bash
+
+ tar xzvf spark-3.0.0-bin-hadoop2.7.tgz
+
+An important step is to ensure ``SPARK_HOME`` environment variable points to the directory where the code has been extracted.
+The next step is to properly define ``PYTHONPATH`` such that it can find the PySpark and
+Py4J under ``$SPARK_HOME/python/lib``, one example of doing this is shown below:
+
+.. code-block:: bash
+
+ cd spark-3.0.0-bin-hadoop2.7
+
+ export SPARK_HOME=`pwd`
+
+ export PYTHONPATH=$(ZIPS=("$SPARK_HOME"/python/lib/*.zip); IFS=:; echo "${ZIPS[*]}"):$PYTHONPATH
+
+Installing From Source
+~~~~~~~~~~~~~~~~~~~~~~
+
+To install PySpark from source, refer `Building Spark <https://spark.apache.org/docs/latest/building-spark.html>`__.
+
+Steps for defining ``PYTHONPATH`` is same as described in `Official Release Channel <#official-release-channel>`_.
+
+Dependencies
+------------
+============= ========================= ====================
+Package Minimum supported version Required or Optional
+============= ========================= ====================
+`pandas` 0.23.2 Optional
+`NumPy` 1.7 Required
+`pyarrow` 0.15.1 Optional
+`Py4J` 0.10.9 Required
+============= ========================= ====================
+
+**Note**: A prerequisite for PySpark installation is the availability of ``JAVA 8`` and ``JAVA 8`` path in ``JAVA_HOME`` environment variable.
Review comment:
- ``` ``JAVA 8`` ... ``` -> Java 8 or 11 and ``JAVA_HOME`` properly set.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-684764329
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-684767923
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-686465615
**[Test build #128258 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128258/testReport)** for PR 29410 at commit [`001814f`](https://github.com/apache/spark/commit/001814ff7e4fcbdb3602cc7588d47bb7e53b8bd9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r482721489
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,120 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+============
+Installation
+============
+
+The official release channel is to download it from https://spark.apache.org/downloads.html but we can install it via ``pip`` as well from PyPI. PyPI installation is usually to use standalone locally or as a client to connect to a cluster.
+
+Instruction for downloading PySpark using PyPI, Conda, Official Release Channel and Source are available in this document.
+
+Python Version Supported
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Python 3.6, 3.7 and 3.8
+
+Using PyPI
+~~~~~~~~~~
+
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_
+
+.. code-block:: bash
+
+ pip install pyspark
+
+Using Conda
+~~~~~~~~~~~
+
+Conda is an open-source package management and environment management system which is a part of `Anaconda <https://docs.continuum.io/anaconda/>`_ distribution. It is both cross-platform and language agnostic.
+
+Conda can be used to create a virtual environment from terminal as shown below:
+
+.. code-block:: bash
+
+ conda create -n pyspark_env
+
+After the virtual environment is created, it should be visible under the list of conda environments which can be seen using the following command:
+
+.. code-block:: bash
+
+ conda env list
+
+The newly created environment can be accessed using the following command:
+
+.. code-block:: bash
+
+ conda activate pyspark_env
+
+In lower Conda version, the following command might be used:
+
+.. code-block:: bash
+
+ source activate pyspark_env
+
+PySpark installation using ``pip`` under Conda environment is official.
+
+PySpark can be installed in this newly created environment using PyPI as shown before:
+
+
+.. code-block:: bash
+
+ pip install pyspark
+
+`PySpark at Conda <https://anaconda.org/conda-forge/pyspark>`__ is not the official release.
+
+Official Release Channel
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Different flavor of PySpark is available in `the official release channel <https://spark.apache.org/downloads.html>`__.
+Any suitable version can be downloaded and extracted as below:
+
+.. code-block:: bash
+
+ tar xzvf spark-3.0.0-bin-hadoop2.7.tgz
+
+An important step is to ensure ``SPARK_HOME`` environment variable points to the directory where the code has been extracted.
+The next step is to properly define ``PYTHONPATH`` such that it can find the PySpark and
+Py4J under ``$SPARK_HOME/python/lib``, one example of doing this is shown below:
+
+.. code-block:: bash
+
+ cd spark-3.0.0-bin-hadoop2.7
+
Review comment:
Let's remove empty newlines here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-674282565
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r472917673
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,84 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Installation
+============
+
+Using PyPI
Review comment:
You can borrow some words from https://spark.apache.org/docs/latest/building-spark.html#pyspark-pip-installable
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r472904741
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,84 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Installation
+============
+
+Using PyPI
+~~~~~~~~~~
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_::
+
+ pip install pyspark
+
+Using Miniconda
+~~~~~~~~~~~~~~~
+Pyspark is available in Conda but it is not a part of an official release. Conda can be used to create a virtual environment from terminal as shown below::
Review comment:
Pyspark -> PySpark
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r482725031
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,120 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+============
+Installation
+============
+
+The official release channel is to download it from https://spark.apache.org/downloads.html but we can install it via ``pip`` as well from PyPI. PyPI installation is usually to use standalone locally or as a client to connect to a cluster.
+
+Instruction for downloading PySpark using PyPI, Conda, Official Release Channel and Source are available in this document.
+
+Python Version Supported
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Python 3.6, 3.7 and 3.8
+
+Using PyPI
+~~~~~~~~~~
+
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_
+
+.. code-block:: bash
+
+ pip install pyspark
+
+Using Conda
+~~~~~~~~~~~
+
+Conda is an open-source package management and environment management system which is a part of `Anaconda <https://docs.continuum.io/anaconda/>`_ distribution. It is both cross-platform and language agnostic.
+
+Conda can be used to create a virtual environment from terminal as shown below:
+
+.. code-block:: bash
+
+ conda create -n pyspark_env
+
+After the virtual environment is created, it should be visible under the list of conda environments which can be seen using the following command:
+
+.. code-block:: bash
+
+ conda env list
+
+The newly created environment can be accessed using the following command:
+
+.. code-block:: bash
+
+ conda activate pyspark_env
+
+In lower Conda version, the following command might be used:
+
+.. code-block:: bash
+
+ source activate pyspark_env
+
+PySpark installation using ``pip`` under Conda environment is official.
+
+PySpark can be installed in this newly created environment using PyPI as shown before:
+
+
+.. code-block:: bash
+
+ pip install pyspark
+
+`PySpark at Conda <https://anaconda.org/conda-forge/pyspark>`__ is not the official release.
+
+Official Release Channel
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Different flavor of PySpark is available in `the official release channel <https://spark.apache.org/downloads.html>`__.
Review comment:
I think we can have one underscore for the links. `__` -> `_`. You'd have to change other links in this page as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-674274381
**[Test build #127465 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127465/testReport)** for PR 29410 at commit [`5df88b4`](https://github.com/apache/spark/commit/5df88b4cd441116cb023d932a8023cd8ef068307).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r482724540
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,120 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+============
+Installation
+============
+
+The official release channel is to download it from https://spark.apache.org/downloads.html but we can install it via ``pip`` as well from PyPI. PyPI installation is usually to use standalone locally or as a client to connect to a cluster.
+
+Instruction for downloading PySpark using PyPI, Conda, Official Release Channel and Source are available in this document.
Review comment:
I would write it like such as:
```
This page includes the instructions for installing PySpark by using pip, Conda, downloading manually, and building it from the source.
```
Feel free to rephrase.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29410: [WIP][SPARK-32180][PYTHON][DOCS] Getting started-Installation guide for pyspark doc
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-672277585
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rohitmishr1484 commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
rohitmishr1484 commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r470866807
##########
File path: python/docs/source/getting_started/index.rst
##########
@@ -20,3 +20,13 @@
Getting Started
===============
+**PySpark** is the Python API for Spark.
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-672568232
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-684775723
**[Test build #128148 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128148/testReport)** for PR 29410 at commit [`30cbdfa`](https://github.com/apache/spark/commit/30cbdfad21677cc8e89c5261cfba32bed0badd99).
* This patch **fails MiMa tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-684767299
**[Test build #128148 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128148/testReport)** for PR 29410 at commit [`30cbdfa`](https://github.com/apache/spark/commit/30cbdfad21677cc8e89c5261cfba32bed0badd99).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r472915553
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,84 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Installation
+============
+
+Using PyPI
+~~~~~~~~~~
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_::
+
+ pip install pyspark
+
+Using Miniconda
+~~~~~~~~~~~~~~~
+Pyspark is available in Conda but it is not a part of an official release. Conda can be used to create a virtual environment from terminal as shown below::
Review comment:
BTW, feel free to insert newlines if the lines are too long. e.g.)
```
Pyspark is available in Conda but it is not a part of an official release. Conda can be used to create a virtual environment from terminal as shown below::
```
->
```
Pyspark is available in Conda but it is not a part of an official release.
Conda can be used to create a virtual environment from terminal as
shown below::
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rohitmishr1484 edited a comment on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
rohitmishr1484 edited a comment on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-672391444
Hi @HyukjinKwon,
I was not sure how to add you as a Reviewer for this Pull request, thus adding this comment. I would like to mention a few points:
1. Baseline description: I have used this pull request as a reference since this is my first pull request- #29385
2. Most of the information used for the "Installation Page" has come from Koalas documentation-https://koalas.readthedocs.io/en/latest/getting_started/install.html. Am I supposed to mention this as reference/credit anywhere in this documentation?
3. Personally I have used only PyPI as an installation mechanism thus haven't tried the other three. A prerequisite for PyPI installation was the availability of JAVA 8 path in JAVA_HOME environment variable. Please let me know if that's something I need to add explicitly in the "Dependencies" section.
4. Where ever necessary I have updated the links.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-676088974
> Most of the information used for the "Installation Page" has come from Koalas documentation-https://koalas.readthedocs.io/en/latest/getting_started/install.html. Am I supposed to mention this as reference/credit anywhere in this documentation?
I think no need. But I would like to say that Koalas installation page is not good enough .. I wrote all most of documentations by myself alone in Koalas so it's not complete yet .. 😢
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29410:
URL: https://github.com/apache/spark/pull/29410#discussion_r472916566
##########
File path: python/docs/source/getting_started/installation.rst
##########
@@ -0,0 +1,84 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Installation
+============
+
+Using PyPI
+~~~~~~~~~~
+PySpark installation using `PyPI <https://pypi.org/project/pyspark/>`_::
+
+ pip install pyspark
+
+Using Miniconda
+~~~~~~~~~~~~~~~
+Pyspark is available in Conda but it is not a part of an official release. Conda can be used to create a virtual environment from terminal as shown below::
Review comment:
I would also add a short overview of Conda
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29410: [SPARK-32180][PYTHON][DOCS] Installation page in Getting Started in PySpark documentation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29410:
URL: https://github.com/apache/spark/pull/29410#issuecomment-686466389
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org