You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/10/26 04:34:41 UTC
[GitHub] [spark] HyukjinKwon commented on a change in pull request #30066: [SPARK-33239][INFRA] Use pre-built image at GitHub Action SparkR job
HyukjinKwon commented on a change in pull request #30066:
URL: https://github.com/apache/spark/pull/30066#discussion_r511715313
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -260,6 +244,68 @@ jobs:
name: unit-tests-log-${{ matrix.modules }}--1.8-hadoop3.2-hive2.3
path: "**/target/unit-tests.log"
+ sparkr:
+ name: Build modules - sparkr
+ runs-on: ubuntu-20.04
+ container:
+ image: dongjoon/apache-spark-github-action-image:20201025
+ env:
+ HADOOP_PROFILE: hadoop3.2
+ HIVE_PROFILE: hive2.3
+ GITHUB_PREV_SHA: ${{ github.event.before }}
+ GITHUB_INPUT_BRANCH: ${{ github.event.inputs.target }}
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v2
+ # In order to fetch changed files
+ with:
+ fetch-depth: 0
+ - name: Merge dispatched input branch
+ if: ${{ github.event.inputs.target != '' }}
+ run: git merge --progress --ff-only origin/${{ github.event.inputs.target }}
+ # Cache local repositories. Note that GitHub Actions cache has a 2G limit.
+ - name: Cache Scala, SBT, Maven and Zinc
+ uses: actions/cache@v2
+ with:
+ path: |
+ build/apache-maven-*
+ build/zinc-*
+ build/scala-*
+ build/*.jar
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Maven local repository
+ uses: actions/cache@v2
+ with:
+ path: ~/.m2/repository
+ key: sparkr-maven-${{ hashFiles('**/pom.xml') }}
+ restore-keys: |
+ sparkr-maven-
+ - name: Cache Ivy local repository
+ uses: actions/cache@v2
+ with:
+ path: ~/.ivy2/cache
+ key: sparkr-ivy-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ restore-keys: |
+ sparkr-ivy-
+ - name: Run tests
+ run: |
+ mkdir -p ~/.m2
+ # The followings are also used by `r-lib/actions/setup-r` to avoid
+ # R issues at docker environment
+ export NOT_CRAN=true
Review comment:
Out of curiosity, does this make it fail any test without setting `NOT_CRAN=true`? It was addressed at https://github.com/apache/spark/commit/fc472bddd1d9c6a28e57e31496c0166777af597e and https://github.com/apache/spark/commit/f4b138082ff91be74b0f5bbe19cdb90dd9e5f131 before in our own scripts.
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -260,6 +244,68 @@ jobs:
name: unit-tests-log-${{ matrix.modules }}--1.8-hadoop3.2-hive2.3
path: "**/target/unit-tests.log"
+ sparkr:
+ name: Build modules - sparkr
+ runs-on: ubuntu-20.04
+ container:
+ image: dongjoon/apache-spark-github-action-image:20201025
+ env:
+ HADOOP_PROFILE: hadoop3.2
+ HIVE_PROFILE: hive2.3
+ GITHUB_PREV_SHA: ${{ github.event.before }}
+ GITHUB_INPUT_BRANCH: ${{ github.event.inputs.target }}
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v2
+ # In order to fetch changed files
+ with:
+ fetch-depth: 0
+ - name: Merge dispatched input branch
+ if: ${{ github.event.inputs.target != '' }}
+ run: git merge --progress --ff-only origin/${{ github.event.inputs.target }}
+ # Cache local repositories. Note that GitHub Actions cache has a 2G limit.
+ - name: Cache Scala, SBT, Maven and Zinc
+ uses: actions/cache@v2
+ with:
+ path: |
+ build/apache-maven-*
+ build/zinc-*
+ build/scala-*
+ build/*.jar
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Maven local repository
+ uses: actions/cache@v2
+ with:
+ path: ~/.m2/repository
+ key: sparkr-maven-${{ hashFiles('**/pom.xml') }}
+ restore-keys: |
+ sparkr-maven-
+ - name: Cache Ivy local repository
+ uses: actions/cache@v2
+ with:
+ path: ~/.ivy2/cache
+ key: sparkr-ivy-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ restore-keys: |
+ sparkr-ivy-
+ - name: Run tests
+ run: |
+ mkdir -p ~/.m2
+ # The followings are also used by `r-lib/actions/setup-r` to avoid
+ # R issues at docker environment
+ export NOT_CRAN=true
+ export TZ=UTC
+ export _R_CHECK_SYSTEM_CLOCK_=FALSE
+ ./dev/run-tests --parallelism 2 --modules sparkr
Review comment:
no big deal but shell we keep it consistent with the above?
```
./dev/run-tests --parallelism 2 --modules "$MODULES_TO_TEST"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org