You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Vinay Devadiga (Jira)" <ji...@apache.org> on 2023/05/06 05:15:00 UTC
[jira] [Comment Edited] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
[ https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17720086#comment-17720086 ]
Vinay Devadiga edited comment on HADOOP-18716 at 5/6/23 5:14 AM:
-----------------------------------------------------------------
Hi [~ayushtkn] i dont have band width to raise PR , but here are most of the fixes explanation here are there tested it out . Any body can raise PR .
h3. Appendix
#1.The first major error which we encountered was the following stack trace -
java.lang.NoClassDefFoundError: Could not initialize class com.google.inject.internal.cglib.core.$ReflectUtils
at com.google.inject.internal.cglib.reflect.$FastClassEmitter.<init>(FastClassEmitter.java:67)
at com.google.inject.internal.cglib.reflect.$FastClass$Generator.generateClass(FastClass.java:72)
This error seems to be the lack of support of the current guice version 4.0 for jre 17 . so after upgrading the guice version to 5.1.0 reduced the {*}errors by the 300{*}.
*Ref:*
[https://github.com/google/guice/issues/1536]
[https://github.com/google/guice/pull/1529/files]
--<guice.version>4.0</guice.version>
++<guice.version>5.1.0</guice.version>
------------------------------------------------------------------------------------------------------
2 . The second major error which we encountered the child jvms spawned by the maven do not have major add-opens which were causing various failures .
```
1. java.lang.NoClassDefFoundError: Could not initialize class
org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesAttempts,TestJerseyClient
2. java.io.IOException: 0.0.0.0:43529: Unexpected HTTP response: code=500 != 200, op=MKDIRS,
message=Server Error
4. org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
6. java.lang.Exception: Unexpected exception,
expected<org.apache.hadoop.yarn.webapp.WebAppException> but
was<java.lang.NoClassDefFoundError>
7. org.apache.hadoop.service.ServiceStateException:
org.apache.hadoop.yarn.exceptions.YarnException: Failed to initialize queues
8. org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.ExceptionInInitializerError
```
and many more
To fix this we use the arg line child property of
<maven-surefire-plugin.argLine>-Xmx2048m -XX:+HeapDumpOnOutOfMemoryError</maven-surefire-plugin.argLine>
We add the following - add-opens in the arg line and in the child projects
--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.math=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/java.text=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
This fix brought down the errors by 2000. So the current count is {*}reduced to 500{*}.
-------------------------------------------------------------------------------------------------------
3. There were various mockito exceptions of these following stack trace
java.lang.NullPointerException: Cannot enter synchronized block because "this.lock" is null
Mockito cannot mock this class: class com.sun.jersey.api.client.ClientResponse.
If you're not sure why you're getting this error, please report to the mailing list.
Java : 17
JVM vendor name : Amazon.com Inc.
JVM vendor version : 17.0.6+10-LTS
JVM name : OpenJDK 64-Bit Server VM
JVM version : 17.0.6+10-LTS
JVM info : mixed mode, sharing
OS name : Linux
OS version : 4.14.309-231.529.amzn2.x86_64
You are seeing this disclaimer because Mockito is configured to create inlined mocks.
You can learn about inline mocks and their limitations under item #39 of the Mockito class javadoc.
Underlying exception : org.mockito.exceptions.base.MockitoException: Could not modify all classes [class com.sun.jersey.api.client.ClientResponse, class java.lang.Object]
etc .These errors got fix after updating the mockito-core version and its scope
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>2.28.2</version>
<version>3.11.2</version>
<scope>provided</scope>
Other mockito exceptions are failing with following stack trace -
Caused by: java.lang.UnsatisfiedLinkError: jdk.internal.misc.Unsafe.registerNatives()V
at jdk.internal.misc.Unsafe.registerNatives(Native Method)
at jdk.internal.misc.Unsafe.<clinit>(Unsafe.java:57)
at java.base/java.lang.Class.forName0(Native Method)
This is surely because we have not access to {{Unsafe}} .
Ref: [https://github.com/jboss-javassist/javassist/issues/227]
[https://github.com/mockito/mockito/issues/1207]
[. https://github.com/mockito/mockito/issues/2164|https://github.com/mockito/mockito/issues/2164]
This brought down *error failures to 200* .
# There were some other minor errors like for [MetricsInvariantChecker resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/invariants/MetricsInvariantChecker.java] etc which used javascript engine for their testing, which is not supported in JRE 17 so the fix was add the following dependency and use graal.js as an alternative.
Ref:
[https://stackoverflow.com/questions/71481562/use-javascript-scripting-engine-in-java-17]
<dependency>
<groupId>org.graalvm.js</groupId>
<artifactId>js</artifactId>
<version>22.0.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.graalvm.js</groupId>
<artifactId>js-scriptengine</artifactId>
<version>22.0.0</version>
<scope>provided</scope>
</dependency> --this.scriptEngine = (Compilable) manager.getEngineByName("JavaScript");
++this.scriptEngine = (Compilable) manager.getEngineByName("graal.js");
There were also some other failures -
cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x59d016c9
This caused fixed by adding -
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED \
--add-opens=java.base/sun.nio.cs=ALL-UNNAMED \
Ref:
[https://stackoverflow.com/questions/73465937/apache-spark-3-3-0-breaks-on-java-17-with-cannot-access-class-sun-nio-ch-direct]
---------------------------------------------------------------------------------------------------
5.The next set of errors which we fixed were IPC failures for these following classes TestIPC,TestAsyncIPC and TestRPCServerShutdown .
We encountered the following stack trace -
Caused by: java.lang.NoClassDefFoundError: org/bouncycastle/jce/provider/BouncyCastleProvider
at io.netty.handler.ssl.util.SelfSignedCertificate.<init>(SelfSignedCertificate.java:244)
... 98 more
Caused by: java.lang.ClassNotFoundException: org.bouncycastle.jce.provider.BouncyCastleProvider
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:606)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
... 99 more
This was fixed by adding the following dependency which was missing JRE 17 runtime for Netty -{*}{{org.bouncycastle:bcpkix-jdk15on:1.65}}{*}
Ref: [https://github.com/netty/netty/issues/10317]
<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcprov-jdk15on</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcpkix-jdk15on</artifactId>
<scope>test</scope>
</dependency>
This reduced the {*}error count to 96{*}.
---------------------------------------------------------------------------------------------------------
These are very rough numbers but this solves most of the issue .
For some errors : fix is this have not tried it https://github.com/apache/hbase/pull/3443/files
----------------------------------------------------
Just trying to contribute to the community and my findings .
was (Author: JIRAUSER290527):
Hi [~ayushtkn] i dont have band width to raise PR , but here are most of the fixes explanation here are there tested it out . Any body can raise PR .
h3. Appendix
#1.The first major error which we encountered was the following stack trace -
java.lang.NoClassDefFoundError: Could not initialize class com.google.inject.internal.cglib.core.$ReflectUtils
at com.google.inject.internal.cglib.reflect.$FastClassEmitter.<init>(FastClassEmitter.java:67)
at com.google.inject.internal.cglib.reflect.$FastClass$Generator.generateClass(FastClass.java:72)
This error seems to be the lack of support of the current guice version 4.0 for jre 17 . so after upgrading the guice version to 5.1.0 reduced the {*}errors by the 300{*}.
*Ref:*
[https://github.com/google/guice/issues/1536]
[https://github.com/google/guice/pull/1529/files]
--<guice.version>4.0</guice.version>
++<guice.version>5.1.0</guice.version>
------------------------------------------------------------------------------------------------------
2 . The second major error which we encountered the child jvms spawned by the maven do not have major add-opens which were causing various failures .
```
1. java.lang.NoClassDefFoundError: Could not initialize class
org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesAttempts,TestJerseyClient
2. java.io.IOException: 0.0.0.0:43529: Unexpected HTTP response: code=500 != 200, op=MKDIRS,
message=Server Error
4. org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
6. java.lang.Exception: Unexpected exception,
expected<org.apache.hadoop.yarn.webapp.WebAppException> but
was<java.lang.NoClassDefFoundError>
7. org.apache.hadoop.service.ServiceStateException:
org.apache.hadoop.yarn.exceptions.YarnException: Failed to initialize queues
8. org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.ExceptionInInitializerError
```
and many more
To fix this we use the arg line child property of
<maven-surefire-plugin.argLine>-Xmx2048m -XX:+HeapDumpOnOutOfMemoryError</maven-surefire-plugin.argLine>
We add the following - add-opens in the arg line and in the child projects
--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.math=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/java.text=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
This fix brought down the errors by 2000. So the current count is {*}reduced to 500{*}.
-------------------------------------------------------------------------------------------------------
3. There were various mockito exceptions of these following stack trace
java.lang.NullPointerException: Cannot enter synchronized block because "this.lock" is null
Mockito cannot mock this class: class com.sun.jersey.api.client.ClientResponse.
If you're not sure why you're getting this error, please report to the mailing list.
Java : 17
JVM vendor name : Amazon.com Inc.
JVM vendor version : 17.0.6+10-LTS
JVM name : OpenJDK 64-Bit Server VM
JVM version : 17.0.6+10-LTS
JVM info : mixed mode, sharing
OS name : Linux
OS version : 4.14.309-231.529.amzn2.x86_64
You are seeing this disclaimer because Mockito is configured to create inlined mocks.
You can learn about inline mocks and their limitations under item #39 of the Mockito class javadoc.
Underlying exception : org.mockito.exceptions.base.MockitoException: Could not modify all classes [class com.sun.jersey.api.client.ClientResponse, class java.lang.Object]
etc .These errors got fix after updating the mockito-core version and its scope
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>2.28.2</version>
<version>3.11.2</version>
<scope>provided</scope>
Other mockito exceptions are failing with following stack trace -
Caused by: java.lang.UnsatisfiedLinkError: jdk.internal.misc.Unsafe.registerNatives()V
at jdk.internal.misc.Unsafe.registerNatives(Native Method)
at jdk.internal.misc.Unsafe.<clinit>(Unsafe.java:57)
at java.base/java.lang.Class.forName0(Native Method)
This is surely because we have not access to {{Unsafe}} .
Ref: [https://github.com/jboss-javassist/javassist/issues/227]
[https://github.com/mockito/mockito/issues/1207]
[. https://github.com/mockito/mockito/issues/2164|https://github.com/mockito/mockito/issues/2164]
This brought down *error failures to 200* .
# There were some other minor errors like for [MetricsInvariantChecker tests|https://code.amazon.com/packages/Aws157HadoopGit/blobs/4b4b79f64a65146acdc5586428e5438aa1639867/--/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/invariants/MetricsInvariantChecker.java] etc which used javascript engine for their testing, which is not supported in JRE 17 so the fix was add the following dependency and use graal.js as an alternative.
Ref:
[https://stackoverflow.com/questions/71481562/use-javascript-scripting-engine-in-java-17]
<dependency>
<groupId>org.graalvm.js</groupId>
<artifactId>js</artifactId>
<version>22.0.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.graalvm.js</groupId>
<artifactId>js-scriptengine</artifactId>
<version>22.0.0</version>
<scope>provided</scope>
</dependency> --this.scriptEngine = (Compilable) manager.getEngineByName("JavaScript");
++this.scriptEngine = (Compilable) manager.getEngineByName("graal.js");
There were also some other failures -
cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x59d016c9
This caused fixed by adding -
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED \
--add-opens=java.base/sun.nio.cs=ALL-UNNAMED \
Ref:
[https://stackoverflow.com/questions/73465937/apache-spark-3-3-0-breaks-on-java-17-with-cannot-access-class-sun-nio-ch-direct]
---------------------------------------------------------------------------------------------------
5.The next set of errors which we fixed were IPC failures for these following classes TestIPC,TestAsyncIPC and TestRPCServerShutdown .
We encountered the following stack trace -
Caused by: java.lang.NoClassDefFoundError: org/bouncycastle/jce/provider/BouncyCastleProvider
at io.netty.handler.ssl.util.SelfSignedCertificate.<init>(SelfSignedCertificate.java:244)
... 98 more
Caused by: java.lang.ClassNotFoundException: org.bouncycastle.jce.provider.BouncyCastleProvider
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:606)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
... 99 more
This was fixed by adding the following dependency which was missing JRE 17 runtime for Netty -{*}{{org.bouncycastle:bcpkix-jdk15on:1.65}}{*}
Ref: [https://github.com/netty/netty/issues/10317]
<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcprov-jdk15on</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcpkix-jdk15on</artifactId>
<scope>test</scope>
</dependency>
This reduced the {*}error count to 96{*}.
---------------------------------------------------------------------------------------------------------
These are very rough numbers but this solves most of the issue .
For some errors : fix is this have not tried it https://github.com/apache/hbase/pull/3443/files
----------------------------------------------------
Just trying to contribute to the community and my findings .
> [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
> ---------------------------------------------------------------------
>
> Key: HADOOP-18716
> URL: https://issues.apache.org/jira/browse/HADOOP-18716
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Vinay Devadiga
> Priority: Critical
>
> Compiled Hadoop - Hadoop branch 3.3.3
> mvn clean install - DskipTests=True
> Java_Home -> points to Java-8
> maven version - 3.8.8 (Quite latest)
>
> Ran various whole test suit on my private cloud environment -
> Changed Java_Home to Java-17
>
> mvn surefire:test
>
> Out of 22k tests - 2.5 k tests failed .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org