You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/14 12:21:35 UTC
[GitHub] [spark] LuciferYang opened a new pull request #34286: [SPARK-37008][SQL][TEST] Replace `UseCompressedOops` with `UseCompressedClassPointers` to pass `WholeStageCodegenSparkSubmitSuite` with Java 17
LuciferYang opened a new pull request #34286:
URL: https://github.com/apache/spark/pull/34286
### What changes were proposed in this pull request?
`UseCompressedOops` Option has some different behaviors between Java 8 and Java 17, if we execute command
```
java -XX:+PrintFlagsFinal -XX:-UseCompressedOops -version | grep Compressed
```
The result of Java 8 is
```
bool UseCompressedClassPointers = false {lp64_product}
bool UseCompressedOops := false {lp64_product}
```
The result of Java 11 is
```
bool UseCompressedClassPointers = false {lp64_product} {default}
bool UseCompressedOops = false {lp64_product} {command line}
```
The result of Java 17 is
```
bool UseCompressedClassPointers = true {product lp64_product} {ergonomic}
bool UseCompressedOops = false {product lp64_product} {command line}
```
We found that `-XX:-UseCompressedOops` and `-XX:-UseCompressedClassPointers` are no longer react in chain, so when we execute
```
mvn clean install -pl sql/core -am -Dtest=none -DwildcardSuites=org.apache.spark.sql.execution.WholeStageCodegenSparkSubmitSuite
```
with Java 17, the test failed due to `Exception in thread "main" org.scalatest.exceptions.TestFailedException: 16 was not greater than 16`, so the change of this pr is replace `UseCompressedOops` with `UseCompressedClassPointers` to pass `WholeStageCodegenSparkSubmitSuite` with Java 17 .
### Why are the changes needed?
Pass UT with JDK 17
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass the Jenkins or GitHub Action
- Manual test with Java 11 passed
- Manual test with Java 17
```
mvn clean install -pl sql/core -am -Dtest=none -DwildcardSuites=org.apache.spark.sql.execution.WholeStageCodegenSparkSubmitSuite
```
**Before**
```
2021-10-14 04:32:38.038 - stderr> Exception in thread "main" org.scalatest.exceptions.TestFailedException: 16 was not greater than 16
2021-10-14 04:32:38.038 - stderr> at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
2021-10-14 04:32:38.038 - stderr> at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
2021-10-14 04:32:38.038 - stderr> at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
2021-10-14 04:32:38.038 - stderr> at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
2021-10-14 04:32:38.038 - stderr> at org.apache.spark.sql.execution.WholeStageCodegenSparkSubmitSuite$.main(WholeStageCodegenSparkSubmitSuite.scala:82)
2021-10-14 04:32:38.038 - stderr> at org.apache.spark.sql.execution.WholeStageCodegenSparkSubmitSuite.main(WholeStageCodegenSparkSubmitSuite.scala)
2021-10-14 04:32:38.038 - stderr> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2021-10-14 04:32:38.038 - stderr> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
2021-10-14 04:32:38.038 - stderr> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2021-10-14 04:32:38.038 - stderr> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
2021-10-14 04:32:38.038 - stderr> at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
2021-10-14 04:32:38.038 - stderr> at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
2021-10-14 04:32:38.038 - stderr> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
2021-10-14 04:32:38.038 - stderr> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
2021-10-14 04:32:38.038 - stderr> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
2021-10-14 04:32:38.038 - stderr> at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
2021-10-14 04:32:38.038 - stderr> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
2021-10-14 04:32:38.038 - stderr> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```
**After**
```
Run completed in 11 seconds, 15 milliseconds.
Total number of tests run: 1
Suites: completed 2, aborted 0
Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34286: [SPARK-37008][SQL][TEST] Replace `UseCompressedOops` with `UseCompressedClassPointers` to pass `WholeStageCodegenSparkSubmitSuite` with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34286:
URL: https://github.com/apache/spark/pull/34286#issuecomment-943412105
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48740/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34286: [SPARK-37008][SQL][TEST] Replace `UseCompressedOops` with `UseCompressedClassPointers` to pass `WholeStageCodegenSparkSubmitSuite` with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34286:
URL: https://github.com/apache/spark/pull/34286#issuecomment-943412105
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48740/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #34286: [SPARK-37008][SQL][TEST] Replace `UseCompressedOops` with `UseCompressedClassPointers` to pass `WholeStageCodegenSparkSubmitSuite` with Java 17
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34286:
URL: https://github.com/apache/spark/pull/34286#discussion_r729123563
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSparkSubmitSuite.scala
##########
@@ -40,15 +40,26 @@ class WholeStageCodegenSparkSubmitSuite extends SparkSubmitTestUtils
val unusedJar = TestUtils.createJarWithClasses(Seq.empty)
// HotSpot JVM specific: Set up a local cluster with the driver/executor using mismatched
- // settings of UseCompressedOops JVM option.
+ // settings of UseCompressedClassPointers JVM option.
val argsForSparkSubmit = Seq(
"--class", WholeStageCodegenSparkSubmitSuite.getClass.getName.stripSuffix("$"),
"--master", "local-cluster[1,1,1024]",
"--driver-memory", "1g",
"--conf", "spark.ui.enabled=false",
"--conf", "spark.master.rest.enabled=false",
- "--conf", "spark.driver.extraJavaOptions=-XX:-UseCompressedOops",
- "--conf", "spark.executor.extraJavaOptions=-XX:+UseCompressedOops",
+ // SPARK-37008: The results of `Platform.BYTE_ARRAY_OFFSET` using different Java versions
+ // and different args as follows table:
+ // +------------------------------+--------+---------+
+ // | |Java 8 |Java 17 |
+ // +------------------------------+--------+---------+
+ // |-XX:-UseCompressedOops | 24 | 16 |
+ // |-XX:+UseCompressedOops | 16 | 16 |
+ // |-XX:-UseCompressedClassPointers| 24 | 24 |
+ // |-XX:+UseCompressedClassPointers| 16 | 16 |
+ // +-------------------------------+-------+---------+
+ // So SPARK-37008 replace `UseCompressedOops` with `UseCompressedClassPointers`.
+ "--conf", "spark.driver.extraJavaOptions=-XX:-UseCompressedClassPointers",
Review comment:
So, this is only necessary to make the test pass, because it's quite sensitive to the size in memory, but spark works otherwise without this flag right?
Can you set this flag before Java 17 - would the JVM accept it? I just hadn't seen it before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #34286: [SPARK-37008][SQL][TEST] Replace `UseCompressedOops` with `UseCompressedClassPointers` to pass `WholeStageCodegenSparkSubmitSuite` with Java 17
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #34286:
URL: https://github.com/apache/spark/pull/34286#issuecomment-944926976
Merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34286: [SPARK-37008][SQL][TEST] Replace `UseCompressedOops` with `UseCompressedClassPointers` to pass `WholeStageCodegenSparkSubmitSuite` with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34286:
URL: https://github.com/apache/spark/pull/34286#issuecomment-943412048
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48740/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34286: [SPARK-37008][SQL][TEST] Replace `UseCompressedOops` with `UseCompressedClassPointers` to pass `WholeStageCodegenSparkSubmitSuite` with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34286:
URL: https://github.com/apache/spark/pull/34286#issuecomment-943351678
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48740/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen closed pull request #34286: [SPARK-37008][SQL][TEST] Replace `UseCompressedOops` with `UseCompressedClassPointers` to pass `WholeStageCodegenSparkSubmitSuite` with Java 17
Posted by GitBox <gi...@apache.org>.
srowen closed pull request #34286:
URL: https://github.com/apache/spark/pull/34286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34286: [SPARK-37008][SQL][TEST] Replace `UseCompressedOops` with `UseCompressedClassPointers` to pass `WholeStageCodegenSparkSubmitSuite` with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34286:
URL: https://github.com/apache/spark/pull/34286#issuecomment-943308540
**[Test build #144261 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144261/testReport)** for PR 34286 at commit [`a8d053b`](https://github.com/apache/spark/commit/a8d053b9e897f419312a966d1d12c9bcdda7cd7c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34286: [SPARK-37008][SQL][TEST] Replace `UseCompressedOops` with `UseCompressedClassPointers` to pass `WholeStageCodegenSparkSubmitSuite` with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34286:
URL: https://github.com/apache/spark/pull/34286#issuecomment-943396096
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48739/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34286: [SPARK-37008][SQL][TEST] Replace `UseCompressedOops` with `UseCompressedClassPointers` to pass `WholeStageCodegenSparkSubmitSuite` with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34286:
URL: https://github.com/apache/spark/pull/34286#issuecomment-943402474
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48739/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #34286: [SPARK-37008][SQL][TEST] Replace `UseCompressedOops` with `UseCompressedClassPointers` to pass `WholeStageCodegenSparkSubmitSuite` with Java 17
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34286:
URL: https://github.com/apache/spark/pull/34286#discussion_r729953166
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSparkSubmitSuite.scala
##########
@@ -40,15 +40,26 @@ class WholeStageCodegenSparkSubmitSuite extends SparkSubmitTestUtils
val unusedJar = TestUtils.createJarWithClasses(Seq.empty)
// HotSpot JVM specific: Set up a local cluster with the driver/executor using mismatched
- // settings of UseCompressedOops JVM option.
+ // settings of UseCompressedClassPointers JVM option.
val argsForSparkSubmit = Seq(
"--class", WholeStageCodegenSparkSubmitSuite.getClass.getName.stripSuffix("$"),
"--master", "local-cluster[1,1,1024]",
"--driver-memory", "1g",
"--conf", "spark.ui.enabled=false",
"--conf", "spark.master.rest.enabled=false",
- "--conf", "spark.driver.extraJavaOptions=-XX:-UseCompressedOops",
- "--conf", "spark.executor.extraJavaOptions=-XX:+UseCompressedOops",
+ // SPARK-37008: The results of `Platform.BYTE_ARRAY_OFFSET` using different Java versions
+ // and different args as follows table:
+ // +------------------------------+--------+---------+
+ // | |Java 8 |Java 17 |
+ // +------------------------------+--------+---------+
+ // |-XX:-UseCompressedOops | 24 | 16 |
+ // |-XX:+UseCompressedOops | 16 | 16 |
+ // |-XX:-UseCompressedClassPointers| 24 | 24 |
+ // |-XX:+UseCompressedClassPointers| 16 | 16 |
+ // +-------------------------------+-------+---------+
+ // So SPARK-37008 replace `UseCompressedOops` with `UseCompressedClassPointers`.
+ "--conf", "spark.driver.extraJavaOptions=-XX:-UseCompressedClassPointers",
Review comment:
That seems better to be explicit. So this still works as intended on Java 8 right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34286: [SPARK-37008][SQL][TEST] Replace `UseCompressedOops` with `UseCompressedClassPointers` to pass `WholeStageCodegenSparkSubmitSuite` with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34286:
URL: https://github.com/apache/spark/pull/34286#issuecomment-943341731
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48739/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34286: [SPARK-37008][SQL][TEST] Replace `UseCompressedOops` with `UseCompressedClassPointers` to pass `WholeStageCodegenSparkSubmitSuite` with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34286:
URL: https://github.com/apache/spark/pull/34286#issuecomment-943306025
**[Test build #144260 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144260/testReport)** for PR 34286 at commit [`b8b8af0`](https://github.com/apache/spark/commit/b8b8af09bd8b7ddca359fcfb713303d044cdd6c5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34286: [SPARK-37008][SQL][TEST] Replace `UseCompressedOops` with `UseCompressedClassPointers` to pass `WholeStageCodegenSparkSubmitSuite` with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34286:
URL: https://github.com/apache/spark/pull/34286#issuecomment-943402474
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48739/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org