Not able to run Java & Python livy examples.

[Ruchir Vasavada <ru...@gmail.com>]

Hi,

I am new to spark (and Java + Scala also) and still trying to setup the
standalone mode spark with livy job manager. I am not able to run the
examples given in the livy example directory for any of the languages. I am
sure I am missing something here, but not able to figure out what's
missing. The examples are working properly if run from Spark direcoty.

   - I am using Ubuntu 16.04.3.
   - Other Installed versions :
      - 1. Spark Version :  2.2.1


   - 2. Java Version :
         - openjdk version "1.8.0_162"
         - OpenJDK Runtime Environment (build
         1.8.0_162-8u162-b12-0ubuntu0.16.04.2-b12)
         - OpenJDK 64-Bit Server VM (build 25.162-b12, mixed mode)


   - 3. Livy Version : 0.6.0-incubating-SNAPSHOT


*Error in Java Example : *
1) setting java classpaths :

JAVA_CP=<path>/spark-core_<version>.jar:<path>/livy-api-0.6.0-incubating-SNAPSHOT.jar:<path>/livy-client-http-0.6.0-incubating-SNAPSHOT.jar:<path>/livy-examples-0.6.0-incubating-SNAPSHOT.jar:<path>/slf4j-1.7.25/slf4j-api-1.7.25.jar:<path>/slf4j-1.7.25/slf4j-simple-1.7.25.jar


2) running example from livy home direcoty :

java -cp $JAVA_CP org.apache.livy.examples.PiApp http://localhost:8998 2


3) Error :

Uploading livy-example jar to the SparkContext...
Exception in thread "main" java.util.concurrent.ExecutionException:
java.io.IOException: Bad Request: "requirement failed: Local path
/root/.livy-sessions/f2169a4c-41eb-4b1d-8b77-c9eaf853b949/livy-examples-0.6.0-incubating-SNAPSHOT.jar
cannot be added to user sessions."
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.livy.examples.PiApp.main(PiApp.java:87)
Caused by: java.io.IOException: Bad Request: "requirement failed: Local
path
/root/.livy-sessions/f2169a4c-41eb-4b1d-8b77-c9eaf853b949/livy-examples-0.6.0-incubating-SNAPSHOT.jar
cannot be added to user sessions."
at
org.apache.livy.client.http.LivyConnection.sendRequest(LivyConnection.java:229)
at org.apache.livy.client.http.LivyConnection.post(LivyConnection.java:192)
at org.apache.livy.client.http.HttpClient$2.call(HttpClient.java:152)
at org.apache.livy.client.http.HttpClient$2.call(HttpClient.java:149)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


4) I have added the path /root/.livy-sessions/ in the whitelist config
param but that did not solve the error.

*Error in Python example : *
1) python ./examples/src/main/python/pi_app.py http://localhost:8998 2

2) Error in the execution :

Exception: org.apache.livy.shaded.kryo.kryo.KryoException: Encountered
unregistered class ID: 254
org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)
org.apache.livy.shaded.kryo.kryo.Kryo.readClass(Kryo.java:656)
org.apache.livy.shaded.kryo.kryo.Kryo.readClassAndObject(Kryo.java:767)
org.apache.livy.client.common.Serializer.deserialize(Serializer.java:63)
org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:39)
org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:27)
org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:57)
org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:42)
org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:27)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)


Can anybody please tell me what is missing here in both the examples?

-- 
Thanks & Regards,
Ruchir