Access to driver side log in cluster mode in Livy

[Swapnil Shinde <sw...@gmail.com>]

Hello
    I am new to Livy and still exploring but looking for ways to get driver
logs in cluster mode. Currently, we have bulky edge nodes which are used to
submit spark applications on YARN. We use client mode instead of
recommended cluster mode due to easy access of realtime driver side logs.
Livy would help us to remove need of bulky edge nodes instead submit spark
apps from anywhere in cluster mode. The only problem I am facing is not
having an access to driver side logs.
I would appreciate if someone help me understand if there is a way to get
driver side logs in Livy. It will be interesting to learn how other
people/companies use Livy + Spark + logging services too.

Thanks
Swapnil

Re: Access to driver side log in cluster mode in Livy

[Marco Gaido <ma...@gmail.com>]

Hi,
please for questions related to how to use Livy, use only the users list
and not the dev one.
Depending on your needs, there may be different answers to your question.
First, I'd recommend you to check the documentation of Livy's REST API, you
might find what you need there:
 -
https://livy.incubator.apache.org/docs/latest/rest-api.html#get-sessionssessionidlog
 -
https://livy.incubator.apache.org/docs/latest/rest-api.html#get-batchesbatchidlog
Second, you can always read the logs using YARN, since the Spark
application is executed on YARN.
Thanks,
Marco

Il giorno gio 7 mar 2019 alle ore 16:57 Swapnil Shinde <
swapnilushinde@gmail.com> ha scritto:

> Hello
>     I am new to Livy and still exploring but looking for ways to get driver
> logs in cluster mode. Currently, we have bulky edge nodes which are used to
> submit spark applications on YARN. We use client mode instead of
> recommended cluster mode due to easy access of realtime driver side logs.
> Livy would help us to remove need of bulky edge nodes instead submit spark
> apps from anywhere in cluster mode. The only problem I am facing is not
> having an access to driver side logs.
> I would appreciate if someone help me understand if there is a way to get
> driver side logs in Livy. It will be interesting to learn how other
> people/companies use Livy + Spark + logging services too.
>
> Thanks
> Swapnil
>