You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@livy.apache.org by Abbass <am...@talend.com> on 2018/06/24 20:37:12 UTC

job execution logs

Guys any idea about where Livy keeps job execution logs (Spark driver
logs) ?

I saw a couple of lines here :
https://github.com/apache/incubator-livy/blob/master/server/src/main/scala/org/apache/livy/server/batch/BatchSession.scala#L90
to redirect stdout & stderr somewhere.

Is there anyway to persist these logs on the filesystem ?

Thanks,

As a recipient of an email from Talend, your contact personal data will be on our systems. Please see our contacts privacy notice at Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>

Re: job execution logs

Posted by Abbass <am...@talend.com>.

On 25/06/2018 22:29, Meisam Fathi wrote:
> If you are using YARN the driver logs are always available at YARN
> resource manager.
YARN keeps the driver logs when you run in yarn-cluster mode, I'm more
interested in yarn-client and spark local modes.
> You can also see the logs from /sessions/{sessionid}/logs
My main question is whether Livy persists the logs somewhere on the
filesystem or they are simply gone when the server is restarted ?
>
> Thanks,
> Meisam
>
> On Sun, Jun 24, 2018 at 1:37 PM Abbass <amarouni@talend.com
> <ma...@talend.com>> wrote:
>
>     Guys any idea about where Livy keeps job execution logs (Spark driver
>     logs) ?
>
>     I saw a couple of lines here :
>     https://github.com/apache/incubator-livy/blob/master/server/src/main/scala/org/apache/livy/server/batch/BatchSession.scala#L90
>     <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dlivy_blob_master_server_src_main_scala_org_apache_livy_server_batch_BatchSession.scala-23L90&d=DwMFaQ&c=2w5q_42kFG40MI2alLPgJw&r=R8lliU6MvJkmSrhIl9IYEn3PxiKTyx5QGHNHOPuQ8U8&m=jSNqaj5fG6zT6_psC7NSSakPlE6z4ZthTd5g9iiuWss&s=3X1a15N68PV0i2Nt-4IGK_7weNHi1a-za5N_2JFeuFc&e=>
>     to redirect stdout & stderr somewhere.
>
>     Is there anyway to persist these logs on the filesystem ?
>
>     Thanks,
>
>     As a recipient of an email from Talend, your contact personal data
>     will be on our systems. Please see our contacts privacy notice at
>     Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
>
>

Re: job execution logs

Posted by Meisam Fathi <me...@gmail.com>.

If you are using YARN the driver logs are always available at YARN resource
manager.
You can also see the logs from /sessions/{sessionid}/logs

Thanks,
Meisam

On Sun, Jun 24, 2018 at 1:37 PM Abbass <am...@talend.com> wrote:

> Guys any idea about where Livy keeps job execution logs (Spark driver
> logs) ?
>
> I saw a couple of lines here :
>
> https://github.com/apache/incubator-livy/blob/master/server/src/main/scala/org/apache/livy/server/batch/BatchSession.scala#L90
> to redirect stdout & stderr somewhere.
>
> Is there anyway to persist these logs on the filesystem ?
>
> Thanks,
>
> As a recipient of an email from Talend, your contact personal data will be
> on our systems. Please see our contacts privacy notice at Talend, Inc. <
> https://www.talend.com/contacts-privacy-policy/>
>
>
>