You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2024/05/01 17:42:37 UTC
(spark) branch master updated: [SPARK-48068][PYTHON] `mypy` should have `--python-executable` parameter
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 26c871f18030 [SPARK-48068][PYTHON] `mypy` should have `--python-executable` parameter
26c871f18030 is described below
commit 26c871f180306fbf86ce65f14f8e7a71f89885ed
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Wed May 1 10:42:26 2024 -0700
[SPARK-48068][PYTHON] `mypy` should have `--python-executable` parameter
### What changes were proposed in this pull request?
This PR aims to fix `mypy` failure by propagating `lint-python`'s `PYTHON_EXECUTABLE` to `mypy`'s parameter correctly.
### Why are the changes needed?
We assumed that `PYTHON_EXECUTABLE` is used for `dev/lint-python` like the following. That's not always guaranteed. We need to use `mypy`'s parameter to make it sure.
https://github.com/apache/spark/blob/ff401dde50343c9bbc1c49a0294272f2da7d01e2/.github/workflows/build_and_test.yml#L705
This patch is useful whose `python3` chooses one of multiple Python installation like our CI environment.
```
$ docker run -it --rm ghcr.io/apache/apache-spark-ci-image:master-8905641334 bash
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
root2ef6ce08d2c4:/# python3 --version
Python 3.10.12
root2ef6ce08d2c4:/# python3.9 --version
Python 3.9.19
```
For example, the following shows that `PYTHON_EXECUTABLE` is not considered by `mypy`.
```
root18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.9 mypy --python-executable=python3.11 --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l
3428
root18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.9 mypy --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l
1
root18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.11 mypy --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l
1
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #46314 from dongjoon-hyun/SPARK-48068.
Authored-by: Dongjoon Hyun <dh...@apple.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
dev/lint-python | 2 ++
1 file changed, 2 insertions(+)
diff --git a/dev/lint-python b/dev/lint-python
index 6bd843103bd7..b8703310bc4b 100755
--- a/dev/lint-python
+++ b/dev/lint-python
@@ -125,6 +125,7 @@ function mypy_annotation_test {
echo "starting mypy annotations test..."
MYPY_REPORT=$( ($MYPY_BUILD \
+ --python-executable $PYTHON_EXECUTABLE \
--namespace-packages \
--config-file python/mypy.ini \
--cache-dir /tmp/.mypy_cache/ \
@@ -184,6 +185,7 @@ function mypy_examples_test {
echo "starting mypy examples test..."
MYPY_REPORT=$( (MYPYPATH=python $MYPY_BUILD \
+ --python-executable $PYTHON_EXECUTABLE \
--namespace-packages \
--config-file python/mypy.ini \
--exclude "mllib/*" \
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org