You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2021/10/21 15:38:56 UTC
[spark] branch master updated: [SPARK-37070][TEST] Pass all UTs in
`mllib-local` and `mllib` with Java 17
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 4148fb5 [SPARK-37070][TEST] Pass all UTs in `mllib-local` and `mllib` with Java 17
4148fb5 is described below
commit 4148fb58aada5bb7bc4835b39fe1baa07f9bacce
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Thu Oct 21 08:37:34 2021 -0700
[SPARK-37070][TEST] Pass all UTs in `mllib-local` and `mllib` with Java 17
### What changes were proposed in this pull request?
`Mockito` can't mock `j.u.Random` with Java 17 due to `module java.base does not export jdk.internal.util.random to unnamed module` and there are some UTs in `mllib-local` and `mllib` module failed with Java 17.
So the main change of this pr is add `--add-exports=java.base/jdk.internal.util.random=ALL-UNNAMED` to `extraTestJavaArgs` to make all UTs passed in `mllib-local` and `mllib` module.
### Why are the changes needed?
Pass UT with JDK 17
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass the Jenkins or GitHub Action
- Manual test with Java 17
```
mvn clean install -pl mllib-local
mvn clean install -pl mllib
```
**Before**
```
Run completed in 714 milliseconds.
Total number of tests run: 96
Suites: completed 9, aborted 0
Tests: succeeded 92, failed 4, canceled 0, ignored 0, pending 0
*** 4 TESTS FAILED ***
Run completed in 10 minutes, 54 seconds.
Total number of tests run: 1639
Suites: completed 206, aborted 0
Tests: succeeded 1635, failed 4, canceled 0, ignored 7, pending 0
*** 4 TESTS FAILED ***
```
All failed UTs has similar reasons:
```
- rand *** FAILED ***
org.mockito.exceptions.base.MockitoException: Mockito cannot mock this class: class java.util.Random.
Mockito can only mock non-private & non-final classes.
If you're not sure why you're getting this error, please report to the mailing list.
Java : 17
JVM vendor name : Azul Systems, Inc.
JVM vendor version : 17+35-LTS
JVM name : OpenJDK 64-Bit Server VM
JVM version : 17+35-LTS
JVM info : mixed mode, sharing
OS name : Mac OS X
OS version : 11.4
Underlying exception : java.lang.IllegalStateException: Cannot access annotation property public abstract boolean jdk.internal.util.random.RandomSupport$RandomGeneratorProperties.isStochastic()
at org.scalatestplus.mockito.MockitoSugar.mock(MockitoSugar.scala:73)
at org.scalatestplus.mockito.MockitoSugar.mock$(MockitoSugar.scala:72)
at org.scalatestplus.mockito.MockitoSugar$.mock(MockitoSugar.scala:155)
at org.apache.spark.ml.linalg.MatricesSuite.$anonfun$new$54(MatricesSuite.scala:807)
at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
...
Cause: java.lang.IllegalStateException: Cannot access annotation property public abstract boolean jdk.internal.util.random.RandomSupport$RandomGeneratorProperties.isStochastic()
at net.bytebuddy.description.annotation.AnnotationDescription$ForLoadedAnnotation.getValue(AnnotationDescription.java:663)
at net.bytebuddy.implementation.attribute.AnnotationAppender$Default.handle(AnnotationAppender.java:298)
at net.bytebuddy.implementation.attribute.AnnotationAppender$Default.doAppend(AnnotationAppender.java:361)
at net.bytebuddy.implementation.attribute.AnnotationAppender$Default.append(AnnotationAppender.java:338)
at net.bytebuddy.implementation.attribute.TypeAttributeAppender$ForInstrumentedType.apply(TypeAttributeAppender.java:93)
at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForCreation.create(TypeWriter.java:5652)
at net.bytebuddy.dynamic.scaffold.TypeWriter$Default.make(TypeWriter.java:2166)
at net.bytebuddy.dynamic.scaffold.subclass.SubclassDynamicTypeBuilder.make(SubclassDynamicTypeBuilder.java:232)
at net.bytebuddy.dynamic.scaffold.subclass.SubclassDynamicTypeBuilder.make(SubclassDynamicTypeBuilder.java:204)
at net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase.make(DynamicType.java:3595)
...
Cause: java.lang.IllegalAccessException: class net.bytebuddy.description.annotation.AnnotationDescription$ForLoadedAnnotation cannot access interface jdk.internal.util.random.RandomSupport$RandomGeneratorProperties (in module java.base) because module java.base does not export jdk.internal.util.random to unnamed module e55b13
at java.base/jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:392)
at java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:674)
at java.base/java.lang.reflect.Method.invoke(Method.java:560)
at net.bytebuddy.description.annotation.AnnotationDescription$ForLoadedAnnotation.getValue(AnnotationDescription.java:642)
at net.bytebuddy.implementation.attribute.AnnotationAppender$Default.handle(AnnotationAppender.java:298)
at net.bytebuddy.implementation.attribute.AnnotationAppender$Default.doAppend(AnnotationAppender.java:361)
at net.bytebuddy.implementation.attribute.AnnotationAppender$Default.append(AnnotationAppender.java:338)
at net.bytebuddy.implementation.attribute.TypeAttributeAppender$ForInstrumentedType.apply(TypeAttributeAppender.java:93)
at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForCreation.create(TypeWriter.java:5652)
at net.bytebuddy.dynamic.scaffold.TypeWriter$Default.make(TypeWriter.java:2166)
...
```
**After**
```
Run completed in 1 second, 7 milliseconds.
Total number of tests run: 96
Suites: completed 9, aborted 0
Tests: succeeded 96, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
Run completed in 10 minutes, 25 seconds.
Total number of tests run: 1639
Suites: completed 206, aborted 0
Tests: succeeded 1639, failed 0, canceled 0, ignored 7, pending 0
All tests passed.
```
Closes #34355 from LuciferYang/SPARK-37070.
Authored-by: yangjie01 <ya...@baidu.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
pom.xml | 6 ++++++
project/SparkBuild.scala | 6 +++++-
2 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index d2d4e8b..e6d47d7 100644
--- a/pom.xml
+++ b/pom.xml
@@ -285,6 +285,11 @@
<maven.build.timestamp.format>yyyy-MM-dd HH:mm:ss z</maven.build.timestamp.format>
<!-- SPARK-36796 for JDK-17 test-->
+ <!--
+ SPARK-37070 In order to enable the UTs in `mllib-local` and `mllib` to use `mockito`
+ to mock `j.u.Random`, "-add-exports=java.base/jdk.internal.util.random=ALL-UNNAMED"
+ is added. Should remove it when `mockito` can mock `j.u.Random` directly.
+ -->
<extraJavaTestArgs>
-XX:+IgnoreUnrecognizedVMOptions
--add-opens=java.base/java.lang=ALL-UNNAMED
@@ -298,6 +303,7 @@
--add-opens=java.base/sun.nio.cs=ALL-UNNAMED
--add-opens=java.base/sun.security.action=ALL-UNNAMED
--add-opens=java.base/sun.util.calendar=ALL-UNNAMED
+ --add-exports=java.base/jdk.internal.util.random=ALL-UNNAMED
</extraJavaTestArgs>
</properties>
<repositories>
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 6d6ca4e..10c6dea 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -1138,7 +1138,11 @@ object TestSettings {
"--add-opens=java.base/sun.nio.ch=ALL-UNNAMED",
"--add-opens=java.base/sun.nio.cs=ALL-UNNAMED",
"--add-opens=java.base/sun.security.action=ALL-UNNAMED",
- "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED").mkString(" ")
+ "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED",
+ // SPARK-37070 In order to enable the UTs in `mllib-local` and `mllib` to use `mockito`
+ // to mock `j.u.Random`, "-add-exports=java.base/jdk.internal.util.random=ALL-UNNAMED"
+ // is added. Should remove it when `mockito` can mock `j.u.Random` directly.
+ "--add-exports=java.base/jdk.internal.util.random=ALL-UNNAMED").mkString(" ")
s"-Xmx4g -Xss4m -XX:MaxMetaspaceSize=$metaspaceSize -XX:+UseParallelGC -XX:-UseDynamicNumberOfGCThreads -XX:ReservedCodeCacheSize=128m $extraTestJavaArgs"
.split(" ").toSeq
},
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org