You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@livy.apache.org by Xiaowei Wang <xi...@gmail.com> on 2022/12/08 20:14:32 UTC
Livy interactive session is not compatible with Java 11
To whom it may concern,
We are building a Spark connector for our own database. Due to some
technical concerns, the connector jar is only compatible with JAVA 11 and
above. We plan to deploy the connector jar in Amazon EMR and use it in a
Jupyter notebook. Due to the compatibility, we must set up an EMR cluster
and run Spark+YARN+LIVY+HADOOP services with *JAVA 11*.
However, we can't even open an idle Spark session in the Jupyter Notebook
with Livy. Below is the exception we got from YARN and Livy
> YARN Diagnostics:
> Application application_1670453222303_0001 failed 1 times (global limit =2; local limit is =1) due to AM Container for appattempt_1670453222303_0001_000001 exited with exitCode: 13
> Failing this attempt.Diagnostics: [2022-12-07 22:56:20.869]Exception from container-launch.
> Container id: container_1670453222303_0001_01_000001
> Exit code: 13
>
> [2022-12-07 22:56:20.897]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
> Last 4096 bytes of prelaunch.err :
> Last 4096 bytes of stderr :
> veMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:742)
> )
> 22/12/07 22:56:20 ERROR ApplicationMaster: Uncaught exception:
> org.apache.spark.SparkException: Exception thrown in awaitResult:
> at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
> at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:514) ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
> at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:278) ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
> at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:929) ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
> at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:928) ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
> at javax.security.auth.Subject.doAs(Subject.java:423) ~[?:?]
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) ~[hadoop-client-api-3.3.3-amzn-0.jar:?]
> at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:928) ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
> at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
> Caused by: java.util.concurrent.ExecutionException: Boxed Error
> at scala.concurrent.impl.Promise$.resolver(Promise.scala:87) ~[scala-library-2.12.15.jar:?]
> at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:79) ~[scala-library-2.12.15.jar:?]
> at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284) ~[scala-library-2.12.15.jar:?]
> at scala.concurrent.Promise.tryFailure(Promise.scala:112) ~[scala-library-2.12.15.jar:?]
> at scala.concurrent.Promise.tryFailure$(Promise.scala:112) ~[scala-library-2.12.15.jar:?]
> at scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:187) ~[scala-library-2.12.15.jar:?]
> at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:761) ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]*Caused by: java.lang.IncompatibleClassChangeError: Inconsistent constant pool data in classfile for class* org/apache/livy/shaded/json4s/DefaultFormats. Method 'java.text.SimpleDateFormat $anonfun$df$1(org.apache.livy.shaded.json4s.DefaultFormats)' at index 156 is CONSTANT_MethodRef and should be CONSTANT_InterfaceMethodRef
> at org.apache.livy.shaded.json4s.DefaultFormats.$init$(Formats.scala:318) ~[?:?]
> at org.apache.livy.shaded.json4s.DefaultFormats$.<init>(Formats.scala:296) ~[?:?].
>
>
Our EMR cluster is configured to have 1 master and 3 workers. There should
always be enough resources. The cluster is on *emr-6.9.0, with Spark 3.3.0,
JupyterHub 1.4.1, Livy 0.7.1. installed.*
We've been blocked by this issue for a while, our investigation tends to
believe this is a compatibility problem between Livy and Java 11. Also we
only see the exception for interactive sessions; For batch sessions, Livy
works as expected with Java 11. Any feedback here can be super helpful to
us. Thank you in advance for all your help!
regards,
Xiaowei
Re: Livy interactive session is not compatible with Java 11
Posted by Xiaowei Wang <xi...@gmail.com>.
Thank you, Larry! I will try to see if I can update the Livy install in
EMR, if you have any tutorial on upgrading Livy, please let me know.
Also top the post here, to see if other thoughts. Thanks a lot!
On Thu, Dec 8, 2022 at 1:44 PM larry mccay <lm...@apache.org> wrote:
> It appears to me that the org.json4s.DefaultFormats class in the shaded
> jar for Livy 0.7.1 is incompatible.
> You could try upgrading that dependency within the shaded jar.
>
> Others may have more thoughts?
>
> On Thu, Dec 8, 2022 at 3:14 PM Xiaowei Wang <xi...@gmail.com> wrote:
>
>> To whom it may concern,
>>
>> We are building a Spark connector for our own database. Due to some
>> technical concerns, the connector jar is only compatible with JAVA 11 and
>> above. We plan to deploy the connector jar in Amazon EMR and use it in a
>> Jupyter notebook. Due to the compatibility, we must set up an EMR cluster
>> and run Spark+YARN+LIVY+HADOOP services with *JAVA 11*.
>>
>> However, we can't even open an idle Spark session in the Jupyter Notebook
>> with Livy. Below is the exception we got from YARN and Livy
>>
>>> YARN Diagnostics:
>>> Application application_1670453222303_0001 failed 1 times (global limit =2; local limit is =1) due to AM Container for appattempt_1670453222303_0001_000001 exited with exitCode: 13
>>> Failing this attempt.Diagnostics: [2022-12-07 22:56:20.869]Exception from container-launch.
>>> Container id: container_1670453222303_0001_01_000001
>>> Exit code: 13
>>>
>>> [2022-12-07 22:56:20.897]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
>>> Last 4096 bytes of prelaunch.err :
>>> Last 4096 bytes of stderr :
>>> veMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>> at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:742)
>>> )
>>> 22/12/07 22:56:20 ERROR ApplicationMaster: Uncaught exception:
>>> org.apache.spark.SparkException: Exception thrown in awaitResult:
>>> at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
>>> at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:514) ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
>>> at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:278) ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
>>> at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:929) ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
>>> at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:928) ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
>>> at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
>>> at javax.security.auth.Subject.doAs(Subject.java:423) ~[?:?]
>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) ~[hadoop-client-api-3.3.3-amzn-0.jar:?]
>>> at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:928) ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
>>> at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
>>> Caused by: java.util.concurrent.ExecutionException: Boxed Error
>>> at scala.concurrent.impl.Promise$.resolver(Promise.scala:87) ~[scala-library-2.12.15.jar:?]
>>> at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:79) ~[scala-library-2.12.15.jar:?]
>>> at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284) ~[scala-library-2.12.15.jar:?]
>>> at scala.concurrent.Promise.tryFailure(Promise.scala:112) ~[scala-library-2.12.15.jar:?]
>>> at scala.concurrent.Promise.tryFailure$(Promise.scala:112) ~[scala-library-2.12.15.jar:?]
>>> at scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:187) ~[scala-library-2.12.15.jar:?]
>>> at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:761) ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]*Caused by: java.lang.IncompatibleClassChangeError: Inconsistent constant pool data in classfile for class* org/apache/livy/shaded/json4s/DefaultFormats. Method 'java.text.SimpleDateFormat $anonfun$df$1(org.apache.livy.shaded.json4s.DefaultFormats)' at index 156 is CONSTANT_MethodRef and should be CONSTANT_InterfaceMethodRef
>>> at org.apache.livy.shaded.json4s.DefaultFormats.$init$(Formats.scala:318) ~[?:?]
>>> at org.apache.livy.shaded.json4s.DefaultFormats$.<init>(Formats.scala:296) ~[?:?].
>>>
>>>
>> Our EMR cluster is configured to have 1 master and 3 workers. There
>> should always be enough resources. The cluster is on *emr-6.9.0,
>> with Spark 3.3.0, JupyterHub 1.4.1, Livy 0.7.1. installed.*
>>
>> We've been blocked by this issue for a while, our investigation tends to
>> believe this is a compatibility problem between Livy and Java 11. Also we
>> only see the exception for interactive sessions; For batch sessions, Livy
>> works as expected with Java 11. Any feedback here can be super helpful to
>> us. Thank you in advance for all your help!
>>
>> regards,
>> Xiaowei
>>
>
--
inner peace
Re: Livy interactive session is not compatible with Java 11
Posted by larry mccay <lm...@apache.org>.
It appears to me that the org.json4s.DefaultFormats class in the shaded jar
for Livy 0.7.1 is incompatible.
You could try upgrading that dependency within the shaded jar.
Others may have more thoughts?
On Thu, Dec 8, 2022 at 3:14 PM Xiaowei Wang <xi...@gmail.com> wrote:
> To whom it may concern,
>
> We are building a Spark connector for our own database. Due to some
> technical concerns, the connector jar is only compatible with JAVA 11 and
> above. We plan to deploy the connector jar in Amazon EMR and use it in a
> Jupyter notebook. Due to the compatibility, we must set up an EMR cluster
> and run Spark+YARN+LIVY+HADOOP services with *JAVA 11*.
>
> However, we can't even open an idle Spark session in the Jupyter Notebook
> with Livy. Below is the exception we got from YARN and Livy
>
>> YARN Diagnostics:
>> Application application_1670453222303_0001 failed 1 times (global limit =2; local limit is =1) due to AM Container for appattempt_1670453222303_0001_000001 exited with exitCode: 13
>> Failing this attempt.Diagnostics: [2022-12-07 22:56:20.869]Exception from container-launch.
>> Container id: container_1670453222303_0001_01_000001
>> Exit code: 13
>>
>> [2022-12-07 22:56:20.897]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
>> Last 4096 bytes of prelaunch.err :
>> Last 4096 bytes of stderr :
>> veMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:742)
>> )
>> 22/12/07 22:56:20 ERROR ApplicationMaster: Uncaught exception:
>> org.apache.spark.SparkException: Exception thrown in awaitResult:
>> at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
>> at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:514) ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
>> at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:278) ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
>> at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:929) ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
>> at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:928) ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
>> at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
>> at javax.security.auth.Subject.doAs(Subject.java:423) ~[?:?]
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) ~[hadoop-client-api-3.3.3-amzn-0.jar:?]
>> at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:928) ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
>> at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
>> Caused by: java.util.concurrent.ExecutionException: Boxed Error
>> at scala.concurrent.impl.Promise$.resolver(Promise.scala:87) ~[scala-library-2.12.15.jar:?]
>> at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:79) ~[scala-library-2.12.15.jar:?]
>> at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284) ~[scala-library-2.12.15.jar:?]
>> at scala.concurrent.Promise.tryFailure(Promise.scala:112) ~[scala-library-2.12.15.jar:?]
>> at scala.concurrent.Promise.tryFailure$(Promise.scala:112) ~[scala-library-2.12.15.jar:?]
>> at scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:187) ~[scala-library-2.12.15.jar:?]
>> at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:761) ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]*Caused by: java.lang.IncompatibleClassChangeError: Inconsistent constant pool data in classfile for class* org/apache/livy/shaded/json4s/DefaultFormats. Method 'java.text.SimpleDateFormat $anonfun$df$1(org.apache.livy.shaded.json4s.DefaultFormats)' at index 156 is CONSTANT_MethodRef and should be CONSTANT_InterfaceMethodRef
>> at org.apache.livy.shaded.json4s.DefaultFormats.$init$(Formats.scala:318) ~[?:?]
>> at org.apache.livy.shaded.json4s.DefaultFormats$.<init>(Formats.scala:296) ~[?:?].
>>
>>
> Our EMR cluster is configured to have 1 master and 3 workers. There should
> always be enough resources. The cluster is on *emr-6.9.0, with Spark
> 3.3.0, JupyterHub 1.4.1, Livy 0.7.1. installed.*
>
> We've been blocked by this issue for a while, our investigation tends to
> believe this is a compatibility problem between Livy and Java 11. Also we
> only see the exception for interactive sessions; For batch sessions, Livy
> works as expected with Java 11. Any feedback here can be super helpful to
> us. Thank you in advance for all your help!
>
> regards,
> Xiaowei
>