You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/25 12:23:56 UTC
[GitHub] [spark] LuciferYang opened a new pull request #34381: [SPARK-37106][YARN] Pass all UTs in `yarn` with Java 17
LuciferYang opened a new pull request #34381:
URL: https://github.com/apache/spark/pull/34381
### What changes were proposed in this pull request?
There are 13 UTs failed when test with Java 17 due to
```
ERROR ApplicationMaster: Uncaught exception:
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)
at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:514)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:278)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:929)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:928)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:928)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: java.util.concurrent.ExecutionException: Boxed Error
at scala.concurrent.impl.Promise$.resolver(Promise.scala:87)
at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:79)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
at scala.concurrent.Promise.tryFailure(Promise.scala:112)
at scala.concurrent.Promise.tryFailure$(Promise.scala:112)
at scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:187)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:761)
Caused by: java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x2a48d10f) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x2a48d10f
at org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:213)
at org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)
at org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:110)
at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:348)
at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:287)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:336)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:278)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:463)
at org.apache.spark.deploy.yarn.YarnExternalShuffleDriver$.main(YarnShuffleIntegrationSuite.scala:130)
at org.apache.spark.deploy.yarn.YarnExternalShuffleDriver.main(YarnShuffleIntegrationSuite.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:742)
```
So this pr add `JavaModuleOptions.defaultModuleOptions()` by default for AM startup, and it will not affect Java 8 and Java 11 due to `JavaModuleOptions.defaultModuleOptions()` includes `-XX:+IgnoreUnrecognizedVMOptions`.
### Why are the changes needed?
Pass UT with JDK 17
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Pass the Jenkins or GitHub Action
- Manual test use Java17 build + Java17 run (both openjdk version "17" 2021-09-14 LTS)
Use Java 17 to run the following two commands
```
build/mvn clean install -DskipTests -pl resource-managers/yarn -am -Pyarn
build/mvn test -pl resource-managers/yarn -Pyarn
```
**Before**
```
Run completed in 5 minutes, 26 seconds.
Total number of tests run: 144
Suites: completed 19, aborted 0
Tests: succeeded 131, failed 13, canceled 1, ignored 0, pending 0
*** 13 TESTS FAILED ***
```
**After**
```
Run completed in 5 minutes, 52 seconds.
Total number of tests run: 144
Suites: completed 19, aborted 0
Tests: succeeded 144, failed 0, canceled 1, ignored 0, pending 0
All tests passed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #34381: [SPARK-37106][YARN] Pass all UTs in `resource-managers/yarn` with Java 17
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34381:
URL: https://github.com/apache/spark/pull/34381#issuecomment-950907135
cc @srowen @dongjoon-hyun @wangyum @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34381: [SPARK-37106][YARN] Pass all UTs in `resource-managers/yarn` with Java 17
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34381:
URL: https://github.com/apache/spark/pull/34381#discussion_r735790785
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
##########
@@ -933,6 +933,11 @@ private[spark] class Client(
val javaOpts = ListBuffer[String]()
+ // SPARK-37106: To start AM with Java 17, `JavaModuleOptions.defaultModuleOptions`
+ // is added by default. It will not affect Java 8 and Java 11 due to existence of
+ // `-XX:+IgnoreUnrecognizedVMOptions`.
+ javaOpts += JavaModuleOptions.defaultModuleOptions()
+
Review comment:
Never mind~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34381: [SPARK-37106][YARN] Pass all UTs in `resource-managers/yarn` with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34381:
URL: https://github.com/apache/spark/pull/34381#issuecomment-951016809
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49059/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34381: [SPARK-37106][YARN] Pass all UTs in `resource-managers/yarn` with Java 17
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34381:
URL: https://github.com/apache/spark/pull/34381#discussion_r735790311
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
##########
@@ -933,6 +933,11 @@ private[spark] class Client(
val javaOpts = ListBuffer[String]()
+ // SPARK-37106: To start AM with Java 17, `JavaModuleOptions.defaultModuleOptions`
+ // is added by default. It will not affect Java 8 and Java 11 due to existence of
+ // `-XX:+IgnoreUnrecognizedVMOptions`.
+ javaOpts += JavaModuleOptions.defaultModuleOptions()
+
Review comment:
Can we make this one liner?
```scala
- val javaOpts = ListBuffer[String]()
-
- // SPARK-37106: To start AM with Java 17, `JavaModuleOptions.defaultModuleOptions`
- // is added by default. It will not affect Java 8 and Java 11 due to existence of
- // `-XX:+IgnoreUnrecognizedVMOptions`.
+ val javaOpts = JavaModuleOptions.defaultModuleOptions()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34381: [SPARK-37106][YARN] Pass all UTs in `resource-managers/yarn` with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34381:
URL: https://github.com/apache/spark/pull/34381#issuecomment-951016809
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49059/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34381: [SPARK-37106][YARN] Pass all UTs in `resource-managers/yarn` with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34381:
URL: https://github.com/apache/spark/pull/34381#issuecomment-950904003
**[Test build #144588 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144588/testReport)** for PR 34381 at commit [`6a569b7`](https://github.com/apache/spark/commit/6a569b74cff037dd338ed58e365a5aab3ad60021).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34381: [SPARK-37106][YARN] Pass all UTs in `resource-managers/yarn` with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34381:
URL: https://github.com/apache/spark/pull/34381#issuecomment-950958760
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144588/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34381: [SPARK-37106][YARN] Pass all UTs in `resource-managers/yarn` with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34381:
URL: https://github.com/apache/spark/pull/34381#issuecomment-950927559
**[Test build #144588 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144588/testReport)** for PR 34381 at commit [`6a569b7`](https://github.com/apache/spark/commit/6a569b74cff037dd338ed58e365a5aab3ad60021).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #34381: [SPARK-37106][YARN] Pass all UTs in `resource-managers/yarn` with Java 17
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #34381:
URL: https://github.com/apache/spark/pull/34381
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34381: [SPARK-37106][YARN] Pass all UTs in `resource-managers/yarn` with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34381:
URL: https://github.com/apache/spark/pull/34381#issuecomment-950904003
**[Test build #144588 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144588/testReport)** for PR 34381 at commit [`6a569b7`](https://github.com/apache/spark/commit/6a569b74cff037dd338ed58e365a5aab3ad60021).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34381: [SPARK-37106][YARN] Pass all UTs in `resource-managers/yarn` with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34381:
URL: https://github.com/apache/spark/pull/34381#issuecomment-950945723
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49059/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34381: [SPARK-37106][YARN] Pass all UTs in `resource-managers/yarn` with Java 17
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34381:
URL: https://github.com/apache/spark/pull/34381#issuecomment-951119997
+1, LGTM. Merged to master for Apache Spark 3.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34381: [SPARK-37106][YARN] Pass all UTs in `resource-managers/yarn` with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34381:
URL: https://github.com/apache/spark/pull/34381#issuecomment-950984156
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49059/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34381: [SPARK-37106][YARN] Pass all UTs in `resource-managers/yarn` with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34381:
URL: https://github.com/apache/spark/pull/34381#issuecomment-950958760
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144588/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org