You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@livy.apache.org by Pavel Sivak <si...@gmail.com> on 2019/08/10 03:01:30 UTC
Creating session in livy with jars parameter
Hi,
My idea is to create livy session with my library in class path using
"jars" parameter.
I'm using REST API to create a session. After sending POST request I can
see that spark session is UP, I can use Spark UI and there's my jar in the
Environment TAB.
But status of the session in livy is "Starting"...
This is example from log file:
> 19/08/09 22:10:37 INFO driver.SparkEntries: Created Spark session.
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.livy.rsc.driver.JobWrapper.cancel(JobWrapper.java:90)
> at org.apache.livy.rsc.driver.RSCDriver.shutdown(RSCDriver.java:127)
> at org.apache.livy.rsc.driver.RSCDriver.run(RSCDriver.java:356)
> at org.apache.livy.rsc.driver.RSCDriverBootstrapper.main(RSCDriverBootstrapper.java:93)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
>
> Without my library livy session I'm getting livy session in Idle status.
Can you help me to figure this out?
Thanks
--
Best wishes,
Pavel Sivak
Re: Creating session in livy with jars parameter
Posted by Pavel Sivak <si...@gmail.com>.
>
> 19/08/12 08:13:46 INFO driver.SparkEntries: Spark context finished initialization in 737ms
> 19/08/12 08:13:46 INFO driver.SparkEntries: Created Spark session.
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.livy.rsc.driver.JobWrapper.cancel(JobWrapper.java:90)
> at org.apache.livy.rsc.driver.RSCDriver.shutdown(RSCDriver.java:127)
> at org.apache.livy.rsc.driver.RSCDriver.run(RSCDriver.java:356)
> at org.apache.livy.rsc.driver.RSCDriverBootstrapper.main(RSCDriverBootstrapper.java:93)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
> at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
>
On Mon, Aug 12, 2019 at 1:26 AM Saisai Shao <sa...@gmail.com> wrote:
> Can you please paste the exception?
>
> Pavel Sivak <si...@gmail.com> 于2019年8月12日周一 上午11:52写道：
>
>> Sure Saisai,
>> First step - run livy server. I can see that server is running and web
>> interface is available
>> Second step - session creation. Example of body in POST request
>>
>>> {"kind":"spark", "jars":["local:/path_to_jar/test.jar"]}
>>>
>> Session is *STARTING* status, I can see that exception in LOG, but SPARK
>> UI is available and I can see my jar in environment TAB
>>
>> But if I'm using some other library(I took guava for example) - session
>> is in *IDLE* status
>>
>>> {"kind":"spark", "jars":["local:/path_to_guava_jar/guava.jar"]}
>>
>>
>> I don't understand what can be the difference between my jar and some
>> other jars...
>>
>> On Sun, Aug 11, 2019 at 10:46 PM Saisai Shao <sa...@gmail.com>
>> wrote:
>>
>>> Would you mind listing the steps to reproduce your issue, and how do you
>>> use REST APIs?
>>>
>>> Thanks
>>> Saisai
>>>
>>> Pavel Sivak <si...@gmail.com> 于2019年8月10日周六 上午11:01写道：
>>>
>>>> Hi,
>>>> My idea is to create livy session with my library in class path using
>>>> "jars" parameter.
>>>> I'm using REST API to create a session. After sending POST request I
>>>> can see that spark session is UP, I can use Spark UI and there's my jar in
>>>> the Environment TAB.
>>>> But status of the session in livy is "Starting"...
>>>> This is example from log file:
>>>>
>>>>> 19/08/09 22:10:37 INFO driver.SparkEntries: Created Spark session.
>>>>> Exception in thread "main" java.lang.NullPointerException
>>>>> at org.apache.livy.rsc.driver.JobWrapper.cancel(JobWrapper.java:90)
>>>>> at org.apache.livy.rsc.driver.RSCDriver.shutdown(RSCDriver.java:127)
>>>>> at org.apache.livy.rsc.driver.RSCDriver.run(RSCDriver.java:356)
>>>>> at org.apache.livy.rsc.driver.RSCDriverBootstrapper.main(RSCDriverBootstrapper.java:93)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>> at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>>>>> at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
>>>>> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
>>>>> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
>>>>>
>>>>> Without my library livy session I'm getting livy session in Idle
>>>> status.
>>>> Can you help me to figure this out?
>>>> Thanks
>>>> --
>>>> Best wishes,
>>>> Pavel Sivak
>>>>
>>>
>>
>> --
>> Best wishes,
>> Pavel Sivak
>>
>
--
Best wishes,
Pavel Sivak
Re: Creating session in livy with jars parameter
Posted by Saisai Shao <sa...@gmail.com>.
Can you please paste the exception?
Pavel Sivak <si...@gmail.com> 于2019年8月12日周一 上午11:52写道：
> Sure Saisai,
> First step - run livy server. I can see that server is running and web
> interface is available
> Second step - session creation. Example of body in POST request
>
>> {"kind":"spark", "jars":["local:/path_to_jar/test.jar"]}
>>
> Session is *STARTING* status, I can see that exception in LOG, but SPARK
> UI is available and I can see my jar in environment TAB
>
> But if I'm using some other library(I took guava for example) - session is
> in *IDLE* status
>
>> {"kind":"spark", "jars":["local:/path_to_guava_jar/guava.jar"]}
>
>
> I don't understand what can be the difference between my jar and some
> other jars...
>
> On Sun, Aug 11, 2019 at 10:46 PM Saisai Shao <sa...@gmail.com>
> wrote:
>
>> Would you mind listing the steps to reproduce your issue, and how do you
>> use REST APIs?
>>
>> Thanks
>> Saisai
>>
>> Pavel Sivak <si...@gmail.com> 于2019年8月10日周六 上午11:01写道：
>>
>>> Hi,
>>> My idea is to create livy session with my library in class path using
>>> "jars" parameter.
>>> I'm using REST API to create a session. After sending POST request I can
>>> see that spark session is UP, I can use Spark UI and there's my jar in the
>>> Environment TAB.
>>> But status of the session in livy is "Starting"...
>>> This is example from log file:
>>>
>>>> 19/08/09 22:10:37 INFO driver.SparkEntries: Created Spark session.
>>>> Exception in thread "main" java.lang.NullPointerException
>>>> at org.apache.livy.rsc.driver.JobWrapper.cancel(JobWrapper.java:90)
>>>> at org.apache.livy.rsc.driver.RSCDriver.shutdown(RSCDriver.java:127)
>>>> at org.apache.livy.rsc.driver.RSCDriver.run(RSCDriver.java:356)
>>>> at org.apache.livy.rsc.driver.RSCDriverBootstrapper.main(RSCDriverBootstrapper.java:93)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>> at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>>>> at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
>>>> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
>>>> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
>>>>
>>>> Without my library livy session I'm getting livy session in Idle status.
>>> Can you help me to figure this out?
>>> Thanks
>>> --
>>> Best wishes,
>>> Pavel Sivak
>>>
>>
>
> --
> Best wishes,
> Pavel Sivak
>
Re: Creating session in livy with jars parameter
Posted by Pavel Sivak <si...@gmail.com>.
Sure Saisai,
First step - run livy server. I can see that server is running and web
interface is available
Second step - session creation. Example of body in POST request
> {"kind":"spark", "jars":["local:/path_to_jar/test.jar"]}
>
Session is *STARTING* status, I can see that exception in LOG, but SPARK UI
is available and I can see my jar in environment TAB
But if I'm using some other library(I took guava for example) - session is
in *IDLE* status
> {"kind":"spark", "jars":["local:/path_to_guava_jar/guava.jar"]}
I don't understand what can be the difference between my jar and some other
jars...
On Sun, Aug 11, 2019 at 10:46 PM Saisai Shao <sa...@gmail.com> wrote:
> Would you mind listing the steps to reproduce your issue, and how do you
> use REST APIs?
>
> Thanks
> Saisai
>
> Pavel Sivak <si...@gmail.com> 于2019年8月10日周六 上午11:01写道：
>
>> Hi,
>> My idea is to create livy session with my library in class path using
>> "jars" parameter.
>> I'm using REST API to create a session. After sending POST request I can
>> see that spark session is UP, I can use Spark UI and there's my jar in the
>> Environment TAB.
>> But status of the session in livy is "Starting"...
>> This is example from log file:
>>
>>> 19/08/09 22:10:37 INFO driver.SparkEntries: Created Spark session.
>>> Exception in thread "main" java.lang.NullPointerException
>>> at org.apache.livy.rsc.driver.JobWrapper.cancel(JobWrapper.java:90)
>>> at org.apache.livy.rsc.driver.RSCDriver.shutdown(RSCDriver.java:127)
>>> at org.apache.livy.rsc.driver.RSCDriver.run(RSCDriver.java:356)
>>> at org.apache.livy.rsc.driver.RSCDriverBootstrapper.main(RSCDriverBootstrapper.java:93)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>>> at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
>>> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
>>> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
>>>
>>> Without my library livy session I'm getting livy session in Idle status.
>> Can you help me to figure this out?
>> Thanks
>> --
>> Best wishes,
>> Pavel Sivak
>>
>
--
Best wishes,
Pavel Sivak
Re: Creating session in livy with jars parameter
Posted by Saisai Shao <sa...@gmail.com>.
Would you mind listing the steps to reproduce your issue, and how do you
use REST APIs?
Thanks
Saisai
Pavel Sivak <si...@gmail.com> 于2019年8月10日周六 上午11:01写道：
> Hi,
> My idea is to create livy session with my library in class path using
> "jars" parameter.
> I'm using REST API to create a session. After sending POST request I can
> see that spark session is UP, I can use Spark UI and there's my jar in the
> Environment TAB.
> But status of the session in livy is "Starting"...
> This is example from log file:
>
>> 19/08/09 22:10:37 INFO driver.SparkEntries: Created Spark session.
>> Exception in thread "main" java.lang.NullPointerException
>> at org.apache.livy.rsc.driver.JobWrapper.cancel(JobWrapper.java:90)
>> at org.apache.livy.rsc.driver.RSCDriver.shutdown(RSCDriver.java:127)
>> at org.apache.livy.rsc.driver.RSCDriver.run(RSCDriver.java:356)
>> at org.apache.livy.rsc.driver.RSCDriverBootstrapper.main(RSCDriverBootstrapper.java:93)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>> at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
>> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
>> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
>>
>> Without my library livy session I'm getting livy session in Idle status.
> Can you help me to figure this out?
> Thanks
> --
> Best wishes,
> Pavel Sivak
>