You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@livy.apache.org by kant kodali <ka...@gmail.com> on 2018/03/11 18:57:59 UTC
What happens if Livy server crashes ? All the spark jobs are gone?
Hi All,
When my live server crashes it looks like all my spark jobs are gone. I am
trying to see how I can make it more resilient? other words, I would like
spark jobs that were spawned by Livy to be running even if my Livy server
crashes because in theory Livy server can crash anytime and Spark Jobs
should run for weeks or months in my case. How can I achieve this?
Thanks!
Re: What happens if Livy server crashes ? All the spark jobs are gone?
Posted by kant kodali <ka...@gmail.com>.
I have the following but I am unable to successfully submit a job though
livy in cluster mode.
here are my settings
# spark-defaults.conf
spark.master yarn
#livy.conf
livy.spark.master=yarn
livy.spark.deploy-mode = cluster
livy.server.recovery.mode = recovery
livy.server.recovery.state-store = zookeeper
livy.server.recovery.state-store.url = localhost:2181
Anything wrong with this conf?
Thanks!
On Tue, Mar 20, 2018 at 5:38 PM, kant kodali <ka...@gmail.com> wrote:
> got it! is it livy.spark.deploy-mode=yarn-cluster or livy.spark.deploy-mode
> = cluster ? Sorry to ask this question. I couldn't find it in docs or the
> comments in livy.conf and I am using livy 0.4.0
>
> On Tue, Mar 20, 2018 at 5:01 PM, Meisam Fathi <me...@gmail.com>
> wrote:
>
>> If you are running on cluster mode, the application should keep running
>> on YRAN.
>>
>> On Tue, Mar 20, 2018 at 3:34 PM kant kodali <ka...@gmail.com> wrote:
>>
>>> @Meisam Fathi I am running with yarn and zookeeper as a state store. I
>>> spawned a job via livy that reads from kafka and writes to Kafka
>>> but the moment I kill the livy server the job also is getting killed.
>>> not sure why? I believe once the livy server crashes the spark context also
>>> get's killed so do I need to need to set the livy.spark.deploy.mode ? if
>>> so, what value should I set it to?
>>>
>>>
>>> On Mon, Mar 12, 2018 at 12:30 PM, Meisam Fathi <me...@gmail.com>
>>> wrote:
>>>
>>>> On YARN, your application keeps running even if the launcher fails. So
>>>> after recovery, Livy reconnects to the application. On Spark standalone, I
>>>> am not sure what happens to the application of the launcher fails.
>>>>
>>>> Thanks,
>>>> Meisam
>>>>
>>>> On Mon, Mar 12, 2018 at 10:34 AM kant kodali <ka...@gmail.com>
>>>> wrote:
>>>>
>>>>> can someone please explain how YARN helps here? And why not spark
>>>>> master?
>>>>>
>>>>> On Mon, Mar 12, 2018 at 3:41 AM, Matteo Durighetto <
>>>>> m.durighetto@miriade.it> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> 2018-03-12 9:58 GMT+01:00 kant kodali <ka...@gmail.com>:
>>>>>>
>>>>>>> Sorry I see there is a recovery mode and also I can set state store
>>>>>>> to zookeeper but looks like I need YARN? because I get the error message
>>>>>>> below
>>>>>>>
>>>>>>> "requirement failed: Session recovery requires YARN"
>>>>>>>
>>>>>>>
>>>>>>> I am using spark standalone and I don't use YARN anywhere in my
>>>>>>> cluster. is there any other option for recovery in this case?
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Mar 11, 2018 at 11:57 AM, kant kodali <ka...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> When my live server crashes it looks like all my spark jobs are
>>>>>>>> gone. I am trying to see how I can make it more resilient? other words, I
>>>>>>>> would like spark jobs that were spawned by Livy to be running even if my
>>>>>>>> Livy server crashes because in theory Livy server can crash anytime and
>>>>>>>> Spark Jobs should run for weeks or months in my case. How can I achieve
>>>>>>>> this?
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>>
>>>>>>> Hello,
>>>>>> to enable recovery in Livy you need Spark on YARN
>>>>>>
>>>>>> ( https://spark.apache.org/docs/latest/running-on-yarn.html )
>>>>>>
>>>>>>
>>>>>>
>>>>>> Kind Regards
>>>>>>
>>>>>
>>>>>
>>>
>
Re: What happens if Livy server crashes ? All the spark jobs are gone?
Posted by kant kodali <ka...@gmail.com>.
got it! is it livy.spark.deploy-mode=yarn-cluster or livy.spark.deploy-mode
= cluster ? Sorry to ask this question. I couldn't find it in docs or the
comments in livy.conf and I am using livy 0.4.0
On Tue, Mar 20, 2018 at 5:01 PM, Meisam Fathi <me...@gmail.com>
wrote:
> If you are running on cluster mode, the application should keep running on
> YRAN.
>
> On Tue, Mar 20, 2018 at 3:34 PM kant kodali <ka...@gmail.com> wrote:
>
>> @Meisam Fathi I am running with yarn and zookeeper as a state store. I
>> spawned a job via livy that reads from kafka and writes to Kafka
>> but the moment I kill the livy server the job also is getting killed. not
>> sure why? I believe once the livy server crashes the spark context also
>> get's killed so do I need to need to set the livy.spark.deploy.mode ? if
>> so, what value should I set it to?
>>
>>
>> On Mon, Mar 12, 2018 at 12:30 PM, Meisam Fathi <me...@gmail.com>
>> wrote:
>>
>>> On YARN, your application keeps running even if the launcher fails. So
>>> after recovery, Livy reconnects to the application. On Spark standalone, I
>>> am not sure what happens to the application of the launcher fails.
>>>
>>> Thanks,
>>> Meisam
>>>
>>> On Mon, Mar 12, 2018 at 10:34 AM kant kodali <ka...@gmail.com> wrote:
>>>
>>>> can someone please explain how YARN helps here? And why not spark
>>>> master?
>>>>
>>>> On Mon, Mar 12, 2018 at 3:41 AM, Matteo Durighetto <
>>>> m.durighetto@miriade.it> wrote:
>>>>
>>>>>
>>>>>
>>>>> 2018-03-12 9:58 GMT+01:00 kant kodali <ka...@gmail.com>:
>>>>>
>>>>>> Sorry I see there is a recovery mode and also I can set state store
>>>>>> to zookeeper but looks like I need YARN? because I get the error message
>>>>>> below
>>>>>>
>>>>>> "requirement failed: Session recovery requires YARN"
>>>>>>
>>>>>>
>>>>>> I am using spark standalone and I don't use YARN anywhere in my
>>>>>> cluster. is there any other option for recovery in this case?
>>>>>>
>>>>>>
>>>>>> On Sun, Mar 11, 2018 at 11:57 AM, kant kodali <ka...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> When my live server crashes it looks like all my spark jobs are
>>>>>>> gone. I am trying to see how I can make it more resilient? other words, I
>>>>>>> would like spark jobs that were spawned by Livy to be running even if my
>>>>>>> Livy server crashes because in theory Livy server can crash anytime and
>>>>>>> Spark Jobs should run for weeks or months in my case. How can I achieve
>>>>>>> this?
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>>
>>>>>> Hello,
>>>>> to enable recovery in Livy you need Spark on YARN
>>>>>
>>>>> ( https://spark.apache.org/docs/latest/running-on-yarn.html )
>>>>>
>>>>>
>>>>>
>>>>> Kind Regards
>>>>>
>>>>
>>>>
>>
Re: What happens if Livy server crashes ? All the spark jobs are gone?
Posted by Meisam Fathi <me...@gmail.com>.
If you are running on cluster mode, the application should keep running on
YRAN.
On Tue, Mar 20, 2018 at 3:34 PM kant kodali <ka...@gmail.com> wrote:
> @Meisam Fathi I am running with yarn and zookeeper as a state store. I
> spawned a job via livy that reads from kafka and writes to Kafka
> but the moment I kill the livy server the job also is getting killed. not
> sure why? I believe once the livy server crashes the spark context also
> get's killed so do I need to need to set the livy.spark.deploy.mode ? if
> so, what value should I set it to?
>
>
> On Mon, Mar 12, 2018 at 12:30 PM, Meisam Fathi <me...@gmail.com>
> wrote:
>
>> On YARN, your application keeps running even if the launcher fails. So
>> after recovery, Livy reconnects to the application. On Spark standalone, I
>> am not sure what happens to the application of the launcher fails.
>>
>> Thanks,
>> Meisam
>>
>> On Mon, Mar 12, 2018 at 10:34 AM kant kodali <ka...@gmail.com> wrote:
>>
>>> can someone please explain how YARN helps here? And why not spark master?
>>>
>>> On Mon, Mar 12, 2018 at 3:41 AM, Matteo Durighetto <
>>> m.durighetto@miriade.it> wrote:
>>>
>>>>
>>>>
>>>> 2018-03-12 9:58 GMT+01:00 kant kodali <ka...@gmail.com>:
>>>>
>>>>> Sorry I see there is a recovery mode and also I can set state store to
>>>>> zookeeper but looks like I need YARN? because I get the error message below
>>>>>
>>>>> "requirement failed: Session recovery requires YARN"
>>>>>
>>>>>
>>>>> I am using spark standalone and I don't use YARN anywhere in my
>>>>> cluster. is there any other option for recovery in this case?
>>>>>
>>>>>
>>>>> On Sun, Mar 11, 2018 at 11:57 AM, kant kodali <ka...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> When my live server crashes it looks like all my spark jobs are gone.
>>>>>> I am trying to see how I can make it more resilient? other words, I would
>>>>>> like spark jobs that were spawned by Livy to be running even if my Livy
>>>>>> server crashes because in theory Livy server can crash anytime and Spark
>>>>>> Jobs should run for weeks or months in my case. How can I achieve this?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>> Hello,
>>>> to enable recovery in Livy you need Spark on YARN
>>>>
>>>> ( https://spark.apache.org/docs/latest/running-on-yarn.html )
>>>>
>>>>
>>>>
>>>> Kind Regards
>>>>
>>>
>>>
>
Re: What happens if Livy server crashes ? All the spark jobs are gone?
Posted by kant kodali <ka...@gmail.com>.
@Meisam Fathi I am running with yarn and zookeeper as a state store. I
spawned a job via livy that reads from kafka and writes to Kafka
but the moment I kill the livy server the job also is getting killed. not
sure why? I believe once the livy server crashes the spark context also
get's killed so do I need to need to set the livy.spark.deploy.mode ? if
so, what value should I set it to?
On Mon, Mar 12, 2018 at 12:30 PM, Meisam Fathi <me...@gmail.com>
wrote:
> On YARN, your application keeps running even if the launcher fails. So
> after recovery, Livy reconnects to the application. On Spark standalone, I
> am not sure what happens to the application of the launcher fails.
>
> Thanks,
> Meisam
>
> On Mon, Mar 12, 2018 at 10:34 AM kant kodali <ka...@gmail.com> wrote:
>
>> can someone please explain how YARN helps here? And why not spark master?
>>
>> On Mon, Mar 12, 2018 at 3:41 AM, Matteo Durighetto <
>> m.durighetto@miriade.it> wrote:
>>
>>>
>>>
>>> 2018-03-12 9:58 GMT+01:00 kant kodali <ka...@gmail.com>:
>>>
>>>> Sorry I see there is a recovery mode and also I can set state store to
>>>> zookeeper but looks like I need YARN? because I get the error message below
>>>>
>>>> "requirement failed: Session recovery requires YARN"
>>>>
>>>>
>>>> I am using spark standalone and I don't use YARN anywhere in my
>>>> cluster. is there any other option for recovery in this case?
>>>>
>>>>
>>>> On Sun, Mar 11, 2018 at 11:57 AM, kant kodali <ka...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> When my live server crashes it looks like all my spark jobs are gone.
>>>>> I am trying to see how I can make it more resilient? other words, I would
>>>>> like spark jobs that were spawned by Livy to be running even if my Livy
>>>>> server crashes because in theory Livy server can crash anytime and Spark
>>>>> Jobs should run for weeks or months in my case. How can I achieve this?
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>> Hello,
>>> to enable recovery in Livy you need Spark on YARN
>>>
>>> ( https://spark.apache.org/docs/latest/running-on-yarn.html )
>>>
>>>
>>>
>>> Kind Regards
>>>
>>
>>
Re: What happens if Livy server crashes ? All the spark jobs are gone?
Posted by Meisam Fathi <me...@gmail.com>.
On YARN, your application keeps running even if the launcher fails. So
after recovery, Livy reconnects to the application. On Spark standalone, I
am not sure what happens to the application of the launcher fails.
Thanks,
Meisam
On Mon, Mar 12, 2018 at 10:34 AM kant kodali <ka...@gmail.com> wrote:
> can someone please explain how YARN helps here? And why not spark master?
>
> On Mon, Mar 12, 2018 at 3:41 AM, Matteo Durighetto <
> m.durighetto@miriade.it> wrote:
>
>>
>>
>> 2018-03-12 9:58 GMT+01:00 kant kodali <ka...@gmail.com>:
>>
>>> Sorry I see there is a recovery mode and also I can set state store to
>>> zookeeper but looks like I need YARN? because I get the error message below
>>>
>>> "requirement failed: Session recovery requires YARN"
>>>
>>>
>>> I am using spark standalone and I don't use YARN anywhere in my cluster.
>>> is there any other option for recovery in this case?
>>>
>>>
>>> On Sun, Mar 11, 2018 at 11:57 AM, kant kodali <ka...@gmail.com>
>>> wrote:
>>>
>>>> Hi All,
>>>>
>>>> When my live server crashes it looks like all my spark jobs are gone. I
>>>> am trying to see how I can make it more resilient? other words, I would
>>>> like spark jobs that were spawned by Livy to be running even if my Livy
>>>> server crashes because in theory Livy server can crash anytime and Spark
>>>> Jobs should run for weeks or months in my case. How can I achieve this?
>>>>
>>>> Thanks!
>>>>
>>>>
>>> Hello,
>> to enable recovery in Livy you need Spark on YARN
>>
>> ( https://spark.apache.org/docs/latest/running-on-yarn.html )
>>
>>
>>
>> Kind Regards
>>
>
>
Re: What happens if Livy server crashes ? All the spark jobs are gone?
Posted by kant kodali <ka...@gmail.com>.
can someone please explain how YARN helps here? And why not spark master?
On Mon, Mar 12, 2018 at 3:41 AM, Matteo Durighetto <m....@miriade.it>
wrote:
>
>
> 2018-03-12 9:58 GMT+01:00 kant kodali <ka...@gmail.com>:
>
>> Sorry I see there is a recovery mode and also I can set state store to
>> zookeeper but looks like I need YARN? because I get the error message below
>>
>> "requirement failed: Session recovery requires YARN"
>>
>>
>> I am using spark standalone and I don't use YARN anywhere in my cluster.
>> is there any other option for recovery in this case?
>>
>>
>> On Sun, Mar 11, 2018 at 11:57 AM, kant kodali <ka...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> When my live server crashes it looks like all my spark jobs are gone. I
>>> am trying to see how I can make it more resilient? other words, I would
>>> like spark jobs that were spawned by Livy to be running even if my Livy
>>> server crashes because in theory Livy server can crash anytime and Spark
>>> Jobs should run for weeks or months in my case. How can I achieve this?
>>>
>>> Thanks!
>>>
>>>
>> Hello,
> to enable recovery in Livy you need Spark on YARN
>
> ( https://spark.apache.org/docs/latest/running-on-yarn.html )
>
>
>
> Kind Regards
>
Re: What happens if Livy server crashes ? All the spark jobs are gone?
Posted by Matteo Durighetto <m....@miriade.it>.
2018-03-12 9:58 GMT+01:00 kant kodali <ka...@gmail.com>:
> Sorry I see there is a recovery mode and also I can set state store to
> zookeeper but looks like I need YARN? because I get the error message below
>
> "requirement failed: Session recovery requires YARN"
>
>
> I am using spark standalone and I don't use YARN anywhere in my cluster.
> is there any other option for recovery in this case?
>
>
> On Sun, Mar 11, 2018 at 11:57 AM, kant kodali <ka...@gmail.com> wrote:
>
>> Hi All,
>>
>> When my live server crashes it looks like all my spark jobs are gone. I
>> am trying to see how I can make it more resilient? other words, I would
>> like spark jobs that were spawned by Livy to be running even if my Livy
>> server crashes because in theory Livy server can crash anytime and Spark
>> Jobs should run for weeks or months in my case. How can I achieve this?
>>
>> Thanks!
>>
>>
> Hello,
to enable recovery in Livy you need Spark on YARN
( https://spark.apache.org/docs/latest/running-on-yarn.html )
Kind Regards
Re: What happens if Livy server crashes ? All the spark jobs are gone?
Posted by kant kodali <ka...@gmail.com>.
Sorry I see there is a recovery mode and also I can set state store to
zookeeper but looks like I need YARN? because I get the error message below
"requirement failed: Session recovery requires YARN"
I am using spark standalone and I don't use YARN anywhere in my cluster. is
there any other option for recovery in this case?
On Sun, Mar 11, 2018 at 11:57 AM, kant kodali <ka...@gmail.com> wrote:
> Hi All,
>
> When my live server crashes it looks like all my spark jobs are gone. I am
> trying to see how I can make it more resilient? other words, I would like
> spark jobs that were spawned by Livy to be running even if my Livy server
> crashes because in theory Livy server can crash anytime and Spark Jobs
> should run for weeks or months in my case. How can I achieve this?
>
> Thanks!
>
>