You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@livy.apache.org by XU Feng <fe...@thalesgroup.com> on 2021/08/22 18:38:18 UTC

Question about LivyScalaClient

I am trying to use LivyScalaClient 0.7.1 with HDP 3.1.1. My Spark Application is packaged into a big self-contained jar file that does not have dependency on any other jar files, and the jar file is put into spark.yarn.archive directory.  The job fails in Yarn with the following exception,

21/08/22 11:41:07 INFO JobWrapper: Failed to run job 4aafeaad-b331-4f55-9cf5-5976f542a99c
org.apache.livy.shaded.kryo.kryo.KryoException: Unable to find class: org.apache.livy.scalaapi.LivyScalaClient$$anon$1
        at org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
        at org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
        at org.apache.livy.shaded.kryo.kryo.Kryo.readClass(Kryo.java:656)
        at org.apache.livy.shaded.kryo.kryo.Kryo.readClassAndObject(Kryo.java:767)
        at org.apache.livy.client.common.Serializer.deserialize(Serializer.java:63)
        at org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:39)
        at org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:27)
        at org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:64)
        at org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:45)
        at org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:27)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.livy.scalaapi.LivyScalaClient$$anon$1
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:136)
        ... 13 more

I am sure the Spark application jar file contains org.apache.livy.scalaapi.LivyScalaClient.

Thanks,

Feng

RE: Question about LivyScalaClient

Posted by XU Feng <fe...@thalesgroup.com>.
The problem is my jar file is big(250m) and LivyScalaClient fails to upload. The same error reported at https://community.cloudera.com/t5/Support-Questions/SizeConstraintExceededException-on-Using-Livy-jar-file-size/td-p/242223. Any suggestion?

Thanks,

Feng

From: XU Feng
Sent: Sunday, August 22, 2021 11:38 AM
To: user@livy.apache.org
Subject: Question about LivyScalaClient

I am trying to use LivyScalaClient 0.7.1 with HDP 3.1.1. My Spark Application is packaged into a big self-contained jar file that does not have dependency on any other jar files, and the jar file is put into spark.yarn.archive directory.  The job fails in Yarn with the following exception,

21/08/22 11:41:07 INFO JobWrapper: Failed to run job 4aafeaad-b331-4f55-9cf5-5976f542a99c
org.apache.livy.shaded.kryo.kryo.KryoException: Unable to find class: org.apache.livy.scalaapi.LivyScalaClient$$anon$1
        at org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
        at org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
        at org.apache.livy.shaded.kryo.kryo.Kryo.readClass(Kryo.java:656)
        at org.apache.livy.shaded.kryo.kryo.Kryo.readClassAndObject(Kryo.java:767)
        at org.apache.livy.client.common.Serializer.deserialize(Serializer.java:63)
        at org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:39)
        at org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:27)
        at org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:64)
        at org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:45)
        at org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:27)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.livy.scalaapi.LivyScalaClient$$anon$1
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:136)
        ... 13 more

I am sure the Spark application jar file contains org.apache.livy.scalaapi.LivyScalaClient.

Thanks,

Feng