You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@livy.apache.org by kant kodali <ka...@gmail.com> on 2017/12/17 04:43:19 UTC

How to scale livy servers?

Hi All,

I am wondering how to scale livy servers? say

1) I have a couple of livy servers that are submitting jobs and say one of
them crashes the session id's again start from 0 which can coincide with
the non-faulty running livy servers. I think it would be nice to have
session id's as UUID. isn't it?

2) Is there a way to get job progress periodically or get notified if it
dies and so on ?

Thanks!

Re: How to scale livy servers?

Posted by ka...@gmail.com.
Hi,

Thanks for the response. Session/job status you mean from the spark ui port because I don't see any endpoint like that in https://livy.incubator.apache.org/docs/latest/rest-api.html

Thanks!

Sent from my iPhone

> On Dec 17, 2017, at 7:03 PM, Meisam Fathi <me...@gmail.com> wrote:
> 
>  
>> 1) I have a couple of livy servers that are submitting jobs and say one of them crashes the session id's again start from 0 which can coincide with the non-faulty running livy servers. I think it would be nice to have session id's as UUID. isn't it?
>  
> If you enable recovery, the session IDs won't restart from 0 after recovery.
>  
>> 2) Is there a way to get job progress periodically or get notified if it dies and so on ?
> 
>  In the REST API, not as far as I know. You have to poll session/job status.
> 
> Thanks,
> Meisam

Re: How to scale livy servers?

Posted by amarouni <am...@talend.com>.

On 18/12/2017 04:03, Meisam Fathi wrote:
>  
>
>     1) I have a couple of livy servers that are submitting jobs and
>     say one of them crashes the session id's again start from 0 which
>     can coincide with the non-faulty running livy servers. I think it
>     would be nice to have session id's as UUID. isn't it?
>
>  
> If you enable recovery, the session IDs won't restart from 0 after
> recovery.
+1 for changing the ids, incremental ids are not a good practice from an
API design standpoint (and security). UUIDs are easy to implement and
make it easier to avoid confusion.
>  
>
>     2) Is there a way to get job progress periodically or get notified
>     if it dies and so on ?
>
>
>  In the REST API, not as far as I know. You have to poll session/job
> status.
>
> Thanks,
> Meisam

Re: How to scale livy servers?

Posted by Meisam Fathi <me...@gmail.com>.
> 1) I have a couple of livy servers that are submitting jobs and say one of
> them crashes the session id's again start from 0 which can coincide with
> the non-faulty running livy servers. I think it would be nice to have
> session id's as UUID. isn't it?
>

If you enable recovery, the session IDs won't restart from 0 after recovery.


> 2) Is there a way to get job progress periodically or get notified if it
> dies and so on ?
>

 In the REST API, not as far as I know. You have to poll session/job status.

Thanks,
Meisam