You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2022/06/09 22:19:45 UTC
[spark] branch branch-3.3 updated: [SPARK-39431][DOCS][PYTHON] Update PySpark dependencies in Installation doc
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 0361ee8cd2f [SPARK-39431][DOCS][PYTHON] Update PySpark dependencies in Installation doc
0361ee8cd2f is described below
commit 0361ee8cd2f65eb99e690804c6d415c46c12dfc3
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Thu Jun 9 15:18:58 2022 -0700
[SPARK-39431][DOCS][PYTHON] Update PySpark dependencies in Installation doc
### What changes were proposed in this pull request?
This PR aims to update `PySpark dependencies` section in Installation document.
- https://dist.apache.org/repos/dist/dev/spark/v3.3.0-rc5-docs/_site/api/python/getting_started/install.html#dependencies
### Why are the changes needed?
Apache Spark 3.3 requires `numpy` 1.15.
https://github.com/apache/spark/blob/8765eea1c08bc58a0cfc22b7cfbc0b5645cc81f9/python/setup.py#L270-L274
https://github.com/apache/spark/blob/8765eea1c08bc58a0cfc22b7cfbc0b5645cc81f9/python/setup.py#L264-L265
So,
- We need to update `numpy` to 1.15 from 1.14 accordingly in documentation.
- We had better remove the duplicated NumPy packages (with two versions) because both `MLlib` and `pandas API on Spark` requires the same version.
- We should use package names consistently.
### Does this PR introduce _any_ user-facing change?
This is a doc-only change.
### How was this patch tested?
Manual review.
Closes #36825 from dongjoon-hyun/SPARK-39431.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit b5c7f34b576d25aec292c65e7565360d67142227)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
python/docs/source/getting_started/install.rst | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/python/docs/source/getting_started/install.rst b/python/docs/source/getting_started/install.rst
index 479f8bf7cc8..e5c1455da7a 100644
--- a/python/docs/source/getting_started/install.rst
+++ b/python/docs/source/getting_started/install.rst
@@ -155,12 +155,11 @@ Dependencies
Package Minimum supported version Note
============= ========================= ======================================
`pandas` 1.0.5 Optional for Spark SQL
-`NumPy` 1.7 Required for MLlib DataFrame-based API
`pyarrow` 1.0.0 Optional for Spark SQL
-`Py4J` 0.10.9.5 Required
+`py4j` 0.10.9.5 Required
`pandas` 1.0.5 Required for pandas API on Spark
`pyarrow` 1.0.0 Required for pandas API on Spark
-`Numpy` 1.14 Required for pandas API on Spark
+`numpy` 1.15 Required for pandas API on Spark and MLLib DataFrame-based API
============= ========================= ======================================
Note that PySpark requires Java 8 or later with ``JAVA_HOME`` properly set.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org