You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2021/07/23 03:29:19 UTC
[spark] branch branch-3.2 updated: [SPARK-36268][PYTHON] Set the
lowerbound of mypy version to 0.910
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new f169f05 [SPARK-36268][PYTHON] Set the lowerbound of mypy version to 0.910
f169f05 is described below
commit f169f056b45b1ac93518e66e14bad1d62738226b
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Fri Jul 23 12:28:16 2021 +0900
[SPARK-36268][PYTHON] Set the lowerbound of mypy version to 0.910
### What changes were proposed in this pull request?
This PR proposes to set the lowerbound of mypy version to use in the testing script.
### Why are the changes needed?
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141519/console
```
python/pyspark/mllib/tree.pyi:29: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/tree.pyi:38: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:34: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:42: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:48: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:54: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:76: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:124: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:165: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/clustering.pyi:45: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/clustering.pyi:72: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/classification.pyi:39: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/classification.pyi:52: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
Found 13 errors in 4 files (checked 314 source files)
1
```
Jenkins installed mypy at SPARK-32797 but seems the version installed is not same as GIthub Actions.
It seems difficult to make the codebase compatible with multiple mypy versions. Therefore, this PR sets the lowerbound.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
Jenkins job in this PR should test it out.
Also manually tested:
Without mypy:
```
...
flake8 checks passed.
The mypy command was not found. Skipping for now.
```
With mypy 0.812:
```
...
flake8 checks passed.
The minimum mypy version needs to be 0.910. Your current version is mypy 0.812. Skipping for now.
```
With mypy 0.910:
```
...
flake8 checks passed.
starting mypy test...
mypy checks passed.
all lint-python tests passed!
```
Closes #33487 from HyukjinKwon/SPARK-36268.
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
(cherry picked from commit d6bc8cd6816c54a73dcc6a6eb13b87dc4ed0482d)
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
dev/lint-python | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/dev/lint-python b/dev/lint-python
index 02b9706..8650058e 100755
--- a/dev/lint-python
+++ b/dev/lint-python
@@ -19,6 +19,7 @@
FLAKE8_BUILD="flake8"
# TODO(SPARK-34943): minimum version should be 3.8+
MINIMUM_FLAKE8="3.5.0"
+MINIMUM_MYPY="0.910"
MYPY_BUILD="mypy"
PYCODESTYLE_BUILD="pycodestyle"
MINIMUM_PYCODESTYLE="2.7.0"
@@ -127,13 +128,21 @@ function mypy_test {
local MYPY_REPORT=
local MYPY_STATUS=
- # TODO(SPARK-32797): Install mypy on the Jenkins CI workers
if ! hash "$MYPY_BUILD" 2> /dev/null; then
echo "The $MYPY_BUILD command was not found. Skipping for now."
return
fi
- echo "starting $MYPY_BUILD test..."
+ _MYPY_VERSION=($($MYPY_BUILD --version))
+ MYPY_VERSION="${_MYPY_VERSION[1]}"
+ EXPECTED_MYPY="$(satisfies_min_version $MYPY_VERSION $MINIMUM_MYPY)"
+
+ if [[ "$EXPECTED_MYPY" == "False" ]]; then
+ echo "The minimum mypy version needs to be $MINIMUM_MYPY. Your current version is $MYPY_VERSION. Skipping for now."
+ return
+ fi
+
+ echo "starting mypy test..."
MYPY_REPORT=$( ($MYPY_BUILD --config-file python/mypy.ini python/pyspark) 2>&1)
MYPY_STATUS=$?
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org