You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@livy.apache.org by Praveen Muthusamy <pr...@gmail.com> on 2018/12/11 12:11:07 UTC
Retain livy batch session info longer
Hi,
Is there any setting in livy server that we can keep, so that the batch
session info will be retained longer ? When a batch application is
submitted and gets successful, that session is cleared, there is no way for
a external program to get the status of this batch session post that.
Regards,
Praveen M
Re: Retain livy batch session info longer
Posted by Marco Gaido <ma...@gmail.com>.
Hi Meisam,
the first property you mentioned is already available (please see LIVY-240).
Thanks,
Marco
Il giorno ven 14 dic 2018 alle ore 02:06 Meisam Fathi <
meisam.fathi@gmail.com> ha scritto:
> I propose the following configurations to support this feature:
>
> # How long to retain an inactive interactive session before cleaning it up
> livy.server.session.timeout
>
> # How long to retain an interactive session that ran successfully
> livy.server.session.success.retaintion
>
> # How long to retain an interactive session that did NOT run successfully
> livy.server.session.failure.retaintion
>
> # How long to retain an batch session that ran successfully
> livy.server.batch.success.retaintion
>
> # How long to retain an batch session that did NOT ran successfully
> livy.server.batch.failure.retaintion
>
> We added a variation of these configs to Livy here at PayPal, which helps
> us manage our Spark jobs with more control. Particularly, these configs
> help us clean up interactive sessions created on Notebooks more
> efficiently. They also help us clean up failed jobs quicker.
>
> If there is an interest, I can send a PR.
>
> On Tue, Dec 11, 2018 at 4:11 AM Praveen Muthusamy <pr...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Is there any setting in livy server that we can keep, so that the batch
> > session info will be retained longer ? When a batch application is
> > submitted and gets successful, that session is cleared, there is no way
> for
> > a external program to get the status of this batch session post that.
> >
> > Regards,
> > Praveen M
> >
>
Re: Retain livy batch session info longer
Posted by Meisam Fathi <me...@gmail.com>.
I propose the following configurations to support this feature:
# How long to retain an inactive interactive session before cleaning it up
livy.server.session.timeout
# How long to retain an interactive session that ran successfully
livy.server.session.success.retaintion
# How long to retain an interactive session that did NOT run successfully
livy.server.session.failure.retaintion
# How long to retain an batch session that ran successfully
livy.server.batch.success.retaintion
# How long to retain an batch session that did NOT ran successfully
livy.server.batch.failure.retaintion
We added a variation of these configs to Livy here at PayPal, which helps
us manage our Spark jobs with more control. Particularly, these configs
help us clean up interactive sessions created on Notebooks more
efficiently. They also help us clean up failed jobs quicker.
If there is an interest, I can send a PR.
On Tue, Dec 11, 2018 at 4:11 AM Praveen Muthusamy <pr...@gmail.com>
wrote:
> Hi,
>
> Is there any setting in livy server that we can keep, so that the batch
> session info will be retained longer ? When a batch application is
> submitted and gets successful, that session is cleared, there is no way for
> a external program to get the status of this batch session post that.
>
> Regards,
> Praveen M
>
Re: Retain livy batch session info longer
Posted by Meisam Fathi <me...@gmail.com>.
I propose the following configurations to support this feature:
# How long to retain an inactive interactive session before cleaning it up
livy.server.session.timeout
# How long to retain an interactive session that ran successfully
livy.server.session.success.retaintion
# How long to retain an interactive session that did NOT run successfully
livy.server.session.failure.retaintion
# How long to retain an batch session that ran successfully
livy.server.batch.success.retaintion
# How long to retain an batch session that did NOT ran successfully
livy.server.batch.failure.retaintion
We added a variation of these configs to Livy here at PayPal, which helps
us manage our Spark jobs with more control. Particularly, these configs
help us clean up interactive sessions created on Notebooks more
efficiently. They also help us clean up failed jobs quicker.
If there is an interest, I can send a PR.
On Tue, Dec 11, 2018 at 4:11 AM Praveen Muthusamy <pr...@gmail.com>
wrote:
> Hi,
>
> Is there any setting in livy server that we can keep, so that the batch
> session info will be retained longer ? When a batch application is
> submitted and gets successful, that session is cleared, there is no way for
> a external program to get the status of this batch session post that.
>
> Regards,
> Praveen M
>