You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/10/06 22:03:04 UTC
[GitHub] [spark] sarutak opened a new pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
sarutak opened a new pull request #29958:
URL: https://github.com/apache/spark/pull/29958
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
SPARK-32926 added a build test to GitHub Action for Scala 2.13 but it's only with Maven.
As SPARK-32873 reported, some compilation error happens only with SBT so I think we need to add another build test to GitHub Action for SBT.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
To ensure build test passes even with SBT for Scala 2.13.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
GitHub Action's job.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-705911154
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-704631954
**[Test build #129470 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129470/testReport)** for PR 29958 at commit [`5dec1ff`](https://github.com/apache/spark/commit/5dec1ffc9eaae07b6c71ebb2de4b64da87328390).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r502226000
##########
File path: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
##########
@@ -91,7 +91,7 @@ class KinesisBackedBlockRDD[T: ClassTag](
require(_blockIds.length == arrayOfseqNumberRanges.length,
"Number of blockIds is not equal to the number of sequence number ranges")
- override def isValid(): Boolean = true
+ override def isValid: Boolean = true
Review comment:
Yes. empty parameter list causes compilation error if we build for Scala 2.13 with SBT.
This is what I'd like to detect by CI.
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -294,28 +294,28 @@ jobs:
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 install
rm -rf ~/.m2/repository/org/apache/spark
- scala-213:
- name: Scala 2.13 build
+ scala-213-sbt:
+ name: Scala 2.13 build with SBT
runs-on: ubuntu-latest
steps:
- name: Checkout Spark repository
uses: actions/checkout@v2
- - name: Cache Maven local repository
+ - name: Cache Ivy local repository
uses: actions/cache@v2
with:
- path: ~/.m2/repository
- key: scala-213-maven-${{ hashFiles('**/pom.xml') }}
+ path: ~/.ivy2/cache
+ key: scala-213-ivy-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- scala-213-maven-
+ scala-213-ivy-
- name: Install Java 11
uses: actions/setup-java@v1
with:
java-version: 11
- - name: Build with Maven
+ - name: Build with SBT
run: |
- export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
- export MAVEN_CLI_OPTS="--no-transfer-progress"
+ mkdir -p ~/.ivy2/local
mkdir -p ~/.m2
Review comment:
`.m2` is required because `publishLocal` installs artifacts into `.m2`.
But, as I mentioned above, I think we can only do `compile` and `test:compile` so I'll remove this part.
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -294,28 +294,28 @@ jobs:
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 install
rm -rf ~/.m2/repository/org/apache/spark
- scala-213:
- name: Scala 2.13 build
+ scala-213-sbt:
+ name: Scala 2.13 build with SBT
runs-on: ubuntu-latest
steps:
- name: Checkout Spark repository
uses: actions/checkout@v2
- - name: Cache Maven local repository
+ - name: Cache Ivy local repository
uses: actions/cache@v2
with:
- path: ~/.m2/repository
- key: scala-213-maven-${{ hashFiles('**/pom.xml') }}
+ path: ~/.ivy2/cache
+ key: scala-213-ivy-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- scala-213-maven-
+ scala-213-ivy-
- name: Install Java 11
uses: actions/setup-java@v1
with:
java-version: 11
- - name: Build with Maven
+ - name: Build with SBT
run: |
- export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
- export MAVEN_CLI_OPTS="--no-transfer-progress"
+ mkdir -p ~/.ivy2/local
mkdir -p ~/.m2
Review comment:
The existing build job for Scala 2.13 with Maven does `install` so the new job does `publshLocal`.
`.m2` is required because `publishLocal` installs artifacts into `.m2`.
But I think we can only do `compile` and `test:compile` so I'll remove this part.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-704632468
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-706134526
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r502281616
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -294,28 +294,28 @@ jobs:
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 install
rm -rf ~/.m2/repository/org/apache/spark
- scala-213:
- name: Scala 2.13 build
+ scala-213-sbt:
+ name: Scala 2.13 build with SBT
runs-on: ubuntu-latest
steps:
- name: Checkout Spark repository
uses: actions/checkout@v2
- - name: Cache Maven local repository
+ - name: Cache Ivy local repository
uses: actions/cache@v2
with:
- path: ~/.m2/repository
- key: scala-213-maven-${{ hashFiles('**/pom.xml') }}
+ path: ~/.ivy2/cache
+ key: scala-213-ivy-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- scala-213-maven-
+ scala-213-ivy-
- name: Install Java 11
uses: actions/setup-java@v1
with:
java-version: 11
- - name: Build with Maven
+ - name: Build with SBT
run: |
- export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
- export MAVEN_CLI_OPTS="--no-transfer-progress"
+ mkdir -p ~/.ivy2/local
mkdir -p ~/.m2
Review comment:
I think we can remove `mkdir -p ~/.m2` here and `rm -rf ~/.m2/repository/org/apache/spark` as we don't cache `.m2` anyway here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-706067598
**[Test build #129581 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129581/testReport)** for PR 29958 at commit [`30b2ddd`](https://github.com/apache/spark/commit/30b2ddd0d69434379075488418551eb3400080af).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r500696490
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -319,3 +319,29 @@ jobs:
./dev/change-scala-version.sh 2.13
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 -Pscala-2.13 install
rm -rf ~/.m2/repository/org/apache/spark
+
+ scala-213-sbt:
+ name: Scala 2.13 build with SBT
Review comment:
> The (almost) same thing is tested at "Scala 2.13 build"
The main purpose of adding the build with SBT is not just testing with SBT. As I mentioned, some compilation error happens only with SBT so I'd like detect such error before any changes being merged.
I understand we have not enough resource for GitHub Action but if an additional build is added to the Jenkins regular job, it will take much more longer time doesn't it?
Are there any problem to replace the existing "Scala 2.13 build" with the new one?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-705928223
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r500667625
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -319,3 +319,29 @@ jobs:
./dev/change-scala-version.sh 2.13
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 -Pscala-2.13 install
rm -rf ~/.m2/repository/org/apache/spark
+
+ scala-213-sbt:
+ name: Scala 2.13 build with SBT
Review comment:
Hm, should we set this job in the PR builder? The (almost) same thing is tested at "Scala 2.13 build". I think we'll set a Jenkins regular job to run the tests at SPARK-33044 as well. I would avoid adding too many jobs here since we're not able to have more resources (see also SPARK-32264).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-704578671
**[Test build #129470 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129470/testReport)** for PR 29958 at commit [`5dec1ff`](https://github.com/apache/spark/commit/5dec1ffc9eaae07b6c71ebb2de4b64da87328390).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29958: [SPARK-33079][TESTS] Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r502780262
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -300,22 +300,18 @@ jobs:
steps:
- name: Checkout Spark repository
uses: actions/checkout@v2
- - name: Cache Maven local repository
+ - name: Cache Ivy local repository
uses: actions/cache@v2
with:
- path: ~/.m2/repository
- key: scala-213-maven-${{ hashFiles('**/pom.xml') }}
+ path: ~/.ivy2/cache
+ key: scala-213-ivy-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- scala-213-maven-
+ scala-213-ivy-
- name: Install Java 11
uses: actions/setup-java@v1
with:
java-version: 11
- - name: Build with Maven
+ - name: Build with SBT
run: |
- export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
- export MAVEN_CLI_OPTS="--no-transfer-progress"
- mkdir -p ~/.m2
./dev/change-scala-version.sh 2.13
- ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 -Pscala-2.13 install
- rm -rf ~/.m2/repository/org/apache/spark
+ ./build/sbt -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Djava.version=11 -Pscala-2.13 compile test:compile
Review comment:
The number of job is limited in github actions instead of each entire workflow run IIRC.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29958: [SPARK-33079][TESTS] Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-706490505
I'll merge this after leaving it some more days in case people have different thoughts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-704614981
cc @HyukjinKwon
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-705956082
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-706134526
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r500665758
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -319,3 +319,29 @@ jobs:
./dev/change-scala-version.sh 2.13
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 -Pscala-2.13 install
rm -rf ~/.m2/repository/org/apache/spark
+
+ scala-213-sbt:
Review comment:
@sarutak, can we actually fix the main job https://github.com/apache/spark/blob/5dec1ffc9eaae07b6c71ebb2de4b64da87328390/.github/workflows/build_and_test.yml#L16-L80 to run the tests?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #29958: [SPARK-33079][TESTS] Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-706722202
Can we keep Maven and SBT tests for 2.13? Maven is the build of reference anyway
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-706133560
**[Test build #129581 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129581/testReport)** for PR 29958 at commit [`30b2ddd`](https://github.com/apache/spark/commit/30b2ddd0d69434379075488418551eb3400080af).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-705956077
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #29958: [SPARK-33079][TESTS] Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #29958:
URL: https://github.com/apache/spark/pull/29958
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-705955584
**[Test build #129568 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129568/testReport)** for PR 29958 at commit [`47a4806`](https://github.com/apache/spark/commit/47a48063bbe56d5c84b4dc1fe7a871ef14de4ee2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r502278155
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -294,28 +294,28 @@ jobs:
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 install
rm -rf ~/.m2/repository/org/apache/spark
- scala-213:
- name: Scala 2.13 build
+ scala-213-sbt:
Review comment:
I would remove -sbt postfix
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -294,28 +294,28 @@ jobs:
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 install
rm -rf ~/.m2/repository/org/apache/spark
- scala-213:
- name: Scala 2.13 build
+ scala-213-sbt:
+ name: Scala 2.13 build with SBT
Review comment:
and here too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #29958: [SPARK-33079][TESTS] Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-706488365
> @sarutak It seems that the title and description of this pr need to be modified
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-706094800
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-705928223
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-706067598
**[Test build #129581 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129581/testReport)** for PR 29958 at commit [`30b2ddd`](https://github.com/apache/spark/commit/30b2ddd0d69434379075488418551eb3400080af).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-704594782
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34077/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r502223867
##########
File path: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
##########
@@ -91,7 +91,7 @@ class KinesisBackedBlockRDD[T: ClassTag](
require(_blockIds.length == arrayOfseqNumberRanges.length,
"Number of blockIds is not equal to the number of sequence number ranges")
- override def isValid(): Boolean = true
+ override def isValid: Boolean = true
Review comment:
Is this change related to the current pr?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29958: [SPARK-33079][TESTS] Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r502763048
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -300,22 +300,18 @@ jobs:
steps:
- name: Checkout Spark repository
uses: actions/checkout@v2
- - name: Cache Maven local repository
+ - name: Cache Ivy local repository
uses: actions/cache@v2
with:
- path: ~/.m2/repository
- key: scala-213-maven-${{ hashFiles('**/pom.xml') }}
+ path: ~/.ivy2/cache
+ key: scala-213-ivy-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- scala-213-maven-
+ scala-213-ivy-
- name: Install Java 11
uses: actions/setup-java@v1
with:
java-version: 11
- - name: Build with Maven
+ - name: Build with SBT
run: |
- export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
- export MAVEN_CLI_OPTS="--no-transfer-progress"
- mkdir -p ~/.m2
./dev/change-scala-version.sh 2.13
- ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 -Pscala-2.13 install
- rm -rf ~/.m2/repository/org/apache/spark
+ ./build/sbt -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Djava.version=11 -Pscala-2.13 compile test:compile
Review comment:
I think it wouldn't matter too much. If a job is finished quicker the next job can be triggered
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29958: [SPARK-33079][TESTS] Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r502750382
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -300,22 +300,18 @@ jobs:
steps:
- name: Checkout Spark repository
uses: actions/checkout@v2
- - name: Cache Maven local repository
+ - name: Cache Ivy local repository
uses: actions/cache@v2
with:
- path: ~/.m2/repository
- key: scala-213-maven-${{ hashFiles('**/pom.xml') }}
+ path: ~/.ivy2/cache
+ key: scala-213-ivy-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- scala-213-maven-
+ scala-213-ivy-
- name: Install Java 11
uses: actions/setup-java@v1
with:
java-version: 11
- - name: Build with Maven
+ - name: Build with SBT
run: |
- export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
- export MAVEN_CLI_OPTS="--no-transfer-progress"
- mkdir -p ~/.m2
./dev/change-scala-version.sh 2.13
- ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 -Pscala-2.13 install
- rm -rf ~/.m2/repository/org/apache/spark
+ ./build/sbt -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Djava.version=11 -Pscala-2.13 compile test:compile
Review comment:
So, we are going to remove `packaging` and `installation (Maven)` here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29958: [SPARK-33079][TESTS] Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r502780262
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -300,22 +300,18 @@ jobs:
steps:
- name: Checkout Spark repository
uses: actions/checkout@v2
- - name: Cache Maven local repository
+ - name: Cache Ivy local repository
uses: actions/cache@v2
with:
- path: ~/.m2/repository
- key: scala-213-maven-${{ hashFiles('**/pom.xml') }}
+ path: ~/.ivy2/cache
+ key: scala-213-ivy-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- scala-213-maven-
+ scala-213-ivy-
- name: Install Java 11
uses: actions/setup-java@v1
with:
java-version: 11
- - name: Build with Maven
+ - name: Build with SBT
run: |
- export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
- export MAVEN_CLI_OPTS="--no-transfer-progress"
- mkdir -p ~/.m2
./dev/change-scala-version.sh 2.13
- ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 -Pscala-2.13 install
- rm -rf ~/.m2/repository/org/apache/spark
+ ./build/sbt -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Djava.version=11 -Pscala-2.13 compile test:compile
Review comment:
The number of job is limited in github actions instead of each entire workload run IIRC.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-706094789
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34186/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-705923340
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34174/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-705911154
**[Test build #129568 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129568/testReport)** for PR 29958 at commit [`47a4806`](https://github.com/apache/spark/commit/47a48063bbe56d5c84b4dc1fe7a871ef14de4ee2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-704600461
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-706473841
@sarutak It seems that the title and description of this pr need to be modified
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r502279441
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -294,28 +294,28 @@ jobs:
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 install
rm -rf ~/.m2/repository/org/apache/spark
- scala-213:
- name: Scala 2.13 build
+ scala-213-sbt:
+ name: Scala 2.13 build with SBT
runs-on: ubuntu-latest
steps:
- name: Checkout Spark repository
uses: actions/checkout@v2
- - name: Cache Maven local repository
+ - name: Cache Ivy local repository
uses: actions/cache@v2
with:
- path: ~/.m2/repository
- key: scala-213-maven-${{ hashFiles('**/pom.xml') }}
+ path: ~/.ivy2/cache
+ key: scala-213-ivy-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- scala-213-maven-
+ scala-213-ivy-
- name: Install Java 11
uses: actions/setup-java@v1
with:
java-version: 11
- - name: Build with Maven
+ - name: Build with SBT
run: |
- export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
- export MAVEN_CLI_OPTS="--no-transfer-progress"
+ mkdir -p ~/.ivy2/local
Review comment:
I guess we don't have to do this. The test cases above are using SBT too but it doesn't create the directory.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29958: [SPARK-33079][TESTS] Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-709236168
Merged to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-705928212
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34174/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak edited a comment on pull request #29958: [SPARK-33079][TESTS] Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job
Posted by GitBox <gi...@apache.org>.
sarutak edited a comment on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-706488365
> @sarutak It seems that the title and description of this pr need to be modified
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-705928223
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-704600461
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r502284241
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -294,28 +294,28 @@ jobs:
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 install
rm -rf ~/.m2/repository/org/apache/spark
- scala-213:
- name: Scala 2.13 build
+ scala-213-sbt:
+ name: Scala 2.13 build with SBT
runs-on: ubuntu-latest
steps:
- name: Checkout Spark repository
uses: actions/checkout@v2
- - name: Cache Maven local repository
+ - name: Cache Ivy local repository
uses: actions/cache@v2
with:
- path: ~/.m2/repository
- key: scala-213-maven-${{ hashFiles('**/pom.xml') }}
+ path: ~/.ivy2/cache
+ key: scala-213-ivy-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- scala-213-maven-
+ scala-213-ivy-
- name: Install Java 11
uses: actions/setup-java@v1
with:
java-version: 11
- - name: Build with Maven
+ - name: Build with SBT
run: |
- export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
- export MAVEN_CLI_OPTS="--no-transfer-progress"
+ mkdir -p ~/.ivy2/local
mkdir -p ~/.m2
Review comment:
`.m2` is required because `publishLocal` installs artifacts into `.m2`.
But, as I mentioned above, I think we can only do `compile` and `test:compile` so I'll remove this part.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #29958: [SPARK-33079][TESTS] Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r502755016
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -300,22 +300,18 @@ jobs:
steps:
- name: Checkout Spark repository
uses: actions/checkout@v2
- - name: Cache Maven local repository
+ - name: Cache Ivy local repository
uses: actions/cache@v2
with:
- path: ~/.m2/repository
- key: scala-213-maven-${{ hashFiles('**/pom.xml') }}
+ path: ~/.ivy2/cache
+ key: scala-213-ivy-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- scala-213-maven-
+ scala-213-ivy-
- name: Install Java 11
uses: actions/setup-java@v1
with:
java-version: 11
- - name: Build with Maven
+ - name: Build with SBT
run: |
- export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
- export MAVEN_CLI_OPTS="--no-transfer-progress"
- mkdir -p ~/.m2
./dev/change-scala-version.sh 2.13
- ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 -Pscala-2.13 install
- rm -rf ~/.m2/repository/org/apache/spark
+ ./build/sbt -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Djava.version=11 -Pscala-2.13 compile test:compile
Review comment:
Can Maven build and SBT build be executed serially in one GitHub Action？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-705911154
**[Test build #129568 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129568/testReport)** for PR 29958 at commit [`47a4806`](https://github.com/apache/spark/commit/47a48063bbe56d5c84b4dc1fe7a871ef14de4ee2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r502284241
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -294,28 +294,28 @@ jobs:
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 install
rm -rf ~/.m2/repository/org/apache/spark
- scala-213:
- name: Scala 2.13 build
+ scala-213-sbt:
+ name: Scala 2.13 build with SBT
runs-on: ubuntu-latest
steps:
- name: Checkout Spark repository
uses: actions/checkout@v2
- - name: Cache Maven local repository
+ - name: Cache Ivy local repository
uses: actions/cache@v2
with:
- path: ~/.m2/repository
- key: scala-213-maven-${{ hashFiles('**/pom.xml') }}
+ path: ~/.ivy2/cache
+ key: scala-213-ivy-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- scala-213-maven-
+ scala-213-ivy-
- name: Install Java 11
uses: actions/setup-java@v1
with:
java-version: 11
- - name: Build with Maven
+ - name: Build with SBT
run: |
- export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
- export MAVEN_CLI_OPTS="--no-transfer-progress"
+ mkdir -p ~/.ivy2/local
mkdir -p ~/.m2
Review comment:
The existing build job for Scala 2.13 with Maven does `install` so the new job does `publshLocal`.
`.m2` is required because `publishLocal` installs artifacts into `.m2`.
But I think we can only do `compile` and `test:compile` so I'll remove this part.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-706094800
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-704600451
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34077/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29958: [SPARK-33079][TESTS] Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-706808750
@srowen, maybe i think we'll have to setup the regular Jenkins job for that like we have done for other profiles. I agree that it's best to keep both Maven and SBT build in Scala 2.13 (and actually we should ideally test all combinations like JDK 11 with running the tests) but the problem is limited resource. The resource is currently all shared in all ASF projects although ASF has an Enterprise plan.
Maybe we could focus on PR builders to catch the errors with a minimized set. For reference or other stuff, it might be better to set these jobs in regular Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r502223867
##########
File path: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
##########
@@ -91,7 +91,7 @@ class KinesisBackedBlockRDD[T: ClassTag](
require(_blockIds.length == arrayOfseqNumberRanges.length,
"Number of blockIds is not equal to the number of sequence number ranges")
- override def isValid(): Boolean = true
+ override def isValid: Boolean = true
Review comment:
Is this change related to the current pr?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r502226000
##########
File path: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
##########
@@ -91,7 +91,7 @@ class KinesisBackedBlockRDD[T: ClassTag](
require(_blockIds.length == arrayOfseqNumberRanges.length,
"Number of blockIds is not equal to the number of sequence number ranges")
- override def isValid(): Boolean = true
+ override def isValid: Boolean = true
Review comment:
Yes. empty parameter list causes compilation error if we build for Scala 2.13 with SBT.
This is what I'd like to detect by CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r502278155
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -294,28 +294,28 @@ jobs:
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 install
rm -rf ~/.m2/repository/org/apache/spark
- scala-213:
- name: Scala 2.13 build
+ scala-213-sbt:
Review comment:
I would remove -sbt postfix
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -294,28 +294,28 @@ jobs:
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 install
rm -rf ~/.m2/repository/org/apache/spark
- scala-213:
- name: Scala 2.13 build
+ scala-213-sbt:
+ name: Scala 2.13 build with SBT
Review comment:
and here too
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -294,28 +294,28 @@ jobs:
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 install
rm -rf ~/.m2/repository/org/apache/spark
- scala-213:
- name: Scala 2.13 build
+ scala-213-sbt:
+ name: Scala 2.13 build with SBT
runs-on: ubuntu-latest
steps:
- name: Checkout Spark repository
uses: actions/checkout@v2
- - name: Cache Maven local repository
+ - name: Cache Ivy local repository
uses: actions/cache@v2
with:
- path: ~/.m2/repository
- key: scala-213-maven-${{ hashFiles('**/pom.xml') }}
+ path: ~/.ivy2/cache
+ key: scala-213-ivy-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- scala-213-maven-
+ scala-213-ivy-
- name: Install Java 11
uses: actions/setup-java@v1
with:
java-version: 11
- - name: Build with Maven
+ - name: Build with SBT
run: |
- export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
- export MAVEN_CLI_OPTS="--no-transfer-progress"
+ mkdir -p ~/.ivy2/local
Review comment:
I guess we don't have to do this. The test cases above are using SBT too but it doesn't create the directory.
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -294,28 +294,28 @@ jobs:
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 install
rm -rf ~/.m2/repository/org/apache/spark
- scala-213:
- name: Scala 2.13 build
+ scala-213-sbt:
+ name: Scala 2.13 build with SBT
runs-on: ubuntu-latest
steps:
- name: Checkout Spark repository
uses: actions/checkout@v2
- - name: Cache Maven local repository
+ - name: Cache Ivy local repository
uses: actions/cache@v2
with:
- path: ~/.m2/repository
- key: scala-213-maven-${{ hashFiles('**/pom.xml') }}
+ path: ~/.ivy2/cache
+ key: scala-213-ivy-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- scala-213-maven-
+ scala-213-ivy-
- name: Install Java 11
uses: actions/setup-java@v1
with:
java-version: 11
- - name: Build with Maven
+ - name: Build with SBT
run: |
- export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
- export MAVEN_CLI_OPTS="--no-transfer-progress"
+ mkdir -p ~/.ivy2/local
mkdir -p ~/.m2
Review comment:
I think we can remove `mkdir -p ~/.m2` here and `rm -rf ~/.m2/repository/org/apache/spark` as we don't cache `.m2` anyway here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r500737481
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -319,3 +319,29 @@ jobs:
./dev/change-scala-version.sh 2.13
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 -Pscala-2.13 install
rm -rf ~/.m2/repository/org/apache/spark
+
+ scala-213-sbt:
+ name: Scala 2.13 build with SBT
Review comment:
> For example, JDK 11 is not being tested in GitHub Actions and Jenkins PR builder.
Yeah, I know but whether build is successfully finish is tested in GitHub Actions so I thought build test (not unit test) should be checked in GitHub Actions too for now.
Anyway, If we can replace the existing "Scala 2.13 build" with the new one, I'll do it.
@dongjoon-hyun Do you have any concern about replacing it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-705928223
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-705911154
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-704632468
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-704578671
**[Test build #129470 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129470/testReport)** for PR 29958 at commit [`5dec1ff`](https://github.com/apache/spark/commit/5dec1ffc9eaae07b6c71ebb2de4b64da87328390).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29958:
URL: https://github.com/apache/spark/pull/29958#issuecomment-706086811
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34186/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #29958: [SPARK-33079][TESTS] Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r502766685
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -300,22 +300,18 @@ jobs:
steps:
- name: Checkout Spark repository
uses: actions/checkout@v2
- - name: Cache Maven local repository
+ - name: Cache Ivy local repository
uses: actions/cache@v2
with:
- path: ~/.m2/repository
- key: scala-213-maven-${{ hashFiles('**/pom.xml') }}
+ path: ~/.ivy2/cache
+ key: scala-213-ivy-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- scala-213-maven-
+ scala-213-ivy-
- name: Install Java 11
uses: actions/setup-java@v1
with:
java-version: 11
- - name: Build with Maven
+ - name: Build with SBT
run: |
- export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
- export MAVEN_CLI_OPTS="--no-transfer-progress"
- mkdir -p ~/.m2
./dev/change-scala-version.sh 2.13
- ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 -Pscala-2.13 install
- rm -rf ~/.m2/repository/org/apache/spark
+ ./build/sbt -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Djava.version=11 -Pscala-2.13 compile test:compile
Review comment:
It seems that 2.13 build action runs faster than other actions. Can we consider this ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29958: [SPARK-33079][TESTS] Add Scala 2.13 build test in GitHub Action for SBT
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r500720741
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -319,3 +319,29 @@ jobs:
./dev/change-scala-version.sh 2.13
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 -Pscala-2.13 install
rm -rf ~/.m2/repository/org/apache/spark
+
+ scala-213-sbt:
+ name: Scala 2.13 build with SBT
Review comment:
I am okay with replacing the existing "Scala 2.13 build". Seems that'll catch the compilation errors more correctly.
> it will take much more longer time doesn't it?
It does takes longer to detect the errors. However, there are many other combinations to test and currently we can't test all combinations. So I think the default common-ish combination was picked to use it as a GitHub Actions and Jenkins PR builder. For example, JDK 11 is not being tested in GitHub Actions and Jenkins PR builder.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #29958: [SPARK-33079][TESTS] Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #29958:
URL: https://github.com/apache/spark/pull/29958#discussion_r502755016
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -300,22 +300,18 @@ jobs:
steps:
- name: Checkout Spark repository
uses: actions/checkout@v2
- - name: Cache Maven local repository
+ - name: Cache Ivy local repository
uses: actions/cache@v2
with:
- path: ~/.m2/repository
- key: scala-213-maven-${{ hashFiles('**/pom.xml') }}
+ path: ~/.ivy2/cache
+ key: scala-213-ivy-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- scala-213-maven-
+ scala-213-ivy-
- name: Install Java 11
uses: actions/setup-java@v1
with:
java-version: 11
- - name: Build with Maven
+ - name: Build with SBT
run: |
- export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
- export MAVEN_CLI_OPTS="--no-transfer-progress"
- mkdir -p ~/.m2
./dev/change-scala-version.sh 2.13
- ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 -Pscala-2.13 install
- rm -rf ~/.m2/repository/org/apache/spark
+ ./build/sbt -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Djava.version=11 -Pscala-2.13 compile test:compile
Review comment:
Can Maven build and SBT build be executed serially in one action？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org