You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2022/08/20 00:44:30 UTC
[spark] branch master updated: [SPARK-39170][PS] Raise ImportError if pandas version mismatch when creating PS document "Supported APIs"
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new bbdbdfff6be [SPARK-39170][PS] Raise ImportError if pandas version mismatch when creating PS document "Supported APIs"
bbdbdfff6be is described below
commit bbdbdfff6be86bc5046417d68bab2c0ecdc3756b
Author: Yikun Jiang <yi...@gmail.com>
AuthorDate: Sat Aug 20 09:44:07 2022 +0900
[SPARK-39170][PS] Raise ImportError if pandas version mismatch when creating PS document "Supported APIs"
### What changes were proposed in this pull request?
Raise ImportError when creating PS document "Supported APIs" but pandas version mismatch
### Why are the changes needed?
The pyspark.pandas documentation "Supported APIs" will be auto-generated. ([SPARK-38961](https://issues.apache.org/jira/browse/SPARK-38961)). At this point, we need to verify the version of pandas.
Related: https://github.com/apache/spark/pull/36509#discussion_r871062738
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- CI passed
- Failed with 1.4.2:
```
$ pip install pandas==1.4.2
$ cd docs
$ SKIP_SCALADOC=1 SKIP_RDOC=1 SKIP_SQLDOC=1 bundle exec jekyll serve --watch
Traceback (most recent call last):
File "/Users/yikun/venv/lib/python3.9/site-packages/sphinx/config.py", line 319, in eval_config_file
execfile_(filename, namespace)
File "/Users/yikun/venv/lib/python3.9/site-packages/sphinx/util/pycompat.py", line 81, in execfile_
exec(code, _globals)
File "/Users/yikun/spark/python/docs/source/conf.py", line 32, in <module>
generate_supported_api(output_rst_file_path)
File "/Users/yikun/spark/python/pyspark/pandas/supported_api_gen.py", line 108, in generate_supported_api
raise ImportError(msg)
ImportError: Warning: Latest version of pandas (1.4.3) is required to generate the documentation; however, your version was 1.4.2
```
Closes #37583 from Yikun/SPARK-39170.
Authored-by: Yikun Jiang <yi...@gmail.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/pyspark/pandas/supported_api_gen.py | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/python/pyspark/pandas/supported_api_gen.py b/python/pyspark/pandas/supported_api_gen.py
index df4e11ebd8c..fc36545fe01 100644
--- a/python/pyspark/pandas/supported_api_gen.py
+++ b/python/pyspark/pandas/supported_api_gen.py
@@ -98,12 +98,14 @@ def generate_supported_api(output_rst_file_path: str) -> None:
Write supported APIs documentation.
"""
- if LooseVersion(pd.__version__) < LooseVersion("1.4.0"):
- warnings.warn(
- "Warning: Latest version of pandas(>=1.4.0) is required to generate the documentation; "
- + "however, your version was %s" % pd.__version__,
- UserWarning,
+ pandas_latest_version = "1.4.3"
+ if LooseVersion(pd.__version__) != LooseVersion(pandas_latest_version):
+ msg = (
+ "Warning: Latest version of pandas (%s) is required to generate the documentation; "
+ "however, your version was %s" % (pandas_latest_version, pd.__version__)
)
+ warnings.warn(msg, UserWarning)
+ raise ImportError(msg)
all_supported_status: Dict[Tuple[str, str], Dict[str, SupportedStatus]] = {}
for pd_module_group, ps_module_group in MODULE_GROUP_MATCH:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org