You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2021/07/30 22:20:52 UTC
[spark] branch master updated: [SPARK-36345][INFRA] Update PySpark
GitHubAction docker image to 20210730
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 0e65ed5 [SPARK-36345][INFRA] Update PySpark GitHubAction docker image to 20210730
0e65ed5 is described below
commit 0e65ed5fb9c62671789a651a993abbb9f546367c
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Sat Jul 31 07:20:17 2021 +0900
[SPARK-36345][INFRA] Update PySpark GitHubAction docker image to 20210730
### What changes were proposed in this pull request?
This PR aims to upgrade PySpark GitHub Action job to use the latest docker image `20210730` having `sklearn` and `mlflow` additionally.
- https://github.com/dongjoon-hyun/ApacheSparkGitHubActionImage/commit/5ca94453d1108dfe40bceb8872387a1b19b0c783
```
$ docker run -it --rm dongjoon/apache-spark-github-action-image:20210730 python3.9 -m pip list | grep mlflow
mlflow 1.19.0
$ docker run -it --rm dongjoon/apache-spark-github-action-image:20210730 python3.9 -m pip list | grep sklearn
sklearn 0.0
```
### Why are the changes needed?
This will save the installation time.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the GitHub Action PySpark jobs.
Closes #33595 from dongjoon-hyun/SPARK-36345.
Authored-by: Dongjoon Hyun <dh...@apple.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
.github/workflows/build_and_test.yml | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml
index 17908ff..58487a4 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -186,7 +186,7 @@ jobs:
name: "Build modules: ${{ matrix.modules }}"
runs-on: ubuntu-20.04
container:
- image: dongjoon/apache-spark-github-action-image:20210602
+ image: dongjoon/apache-spark-github-action-image:20210730
strategy:
fail-fast: false
matrix:
@@ -252,8 +252,6 @@ jobs:
# Run the tests.
- name: Run tests
run: |
- # TODO(SPARK-36345): Install mlflow>=1.0 and sklearn in Python 3.9 of the base image
- python3.9 -m pip install 'mlflow>=1.0' sklearn
export PATH=$PATH:$HOME/miniconda/bin
./dev/run-tests --parallelism 1 --modules "$MODULES_TO_TEST"
- name: Upload test results to report
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org