You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Priya Ch <le...@gmail.com> on 2014/10/14 13:33:46 UTC
Default spark.deploy.recoveryMode
Hi Spark users/experts,
In Spark source code (Master.scala & Worker.scala), when registering the
worker with master, I see the usage of *persistenceEngine*. When we don't
specify spark.deploy.recovery mode explicitly, what is the default value
used ? This recovery mode is used to persists and restore the application &
worker details.
I see when recovery mode not specified explicitly,
*BlackHolePersistenceEngine* being used. Am i right ?
Thanks,
Padma Ch
Re: Default spark.deploy.recoveryMode
Posted by Prashant Sharma <sc...@gmail.com>.
So if you need those features you can go ahead and setup one of Filesystem
or zookeeper options. Please take a look at:
http://spark.apache.org/docs/latest/spark-standalone.html.
Prashant Sharma
On Wed, Oct 15, 2014 at 3:25 PM, Chitturi Padma <
learnings.chitturi@gmail.com> wrote:
> which means the details are not persisted and hence any failures in
> workers and master wouldnt start the daemons normally ..right ?
>
> On Wed, Oct 15, 2014 at 12:17 PM, Prashant Sharma [via Apache Spark User
> List] <[hidden email] <http://user/SendEmail.jtp?type=node&node=16483&i=0>
> > wrote:
>
>> [Removing dev lists]
>>
>> You are absolutely correct about that.
>>
>> Prashant Sharma
>>
>>
>>
>> On Tue, Oct 14, 2014 at 5:03 PM, Priya Ch <[hidden email]
>> <http://user/SendEmail.jtp?type=node&node=16468&i=0>> wrote:
>>
>>> Hi Spark users/experts,
>>>
>>> In Spark source code (Master.scala & Worker.scala), when registering
>>> the worker with master, I see the usage of *persistenceEngine*. When we
>>> don't specify spark.deploy.recovery mode explicitly, what is the default
>>> value used ? This recovery mode is used to persists and restore the
>>> application & worker details.
>>>
>>> I see when recovery mode not specified explicitly,
>>> *BlackHolePersistenceEngine* being used. Am i right ?
>>>
>>>
>>> Thanks,
>>> Padma Ch
>>>
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the
>> discussion below:
>>
>> http://apache-spark-user-list.1001560.n3.nabble.com/Default-spark-deploy-recoveryMode-tp16375p16468.html
>> To start a new topic under Apache Spark User List, email [hidden email]
>> <http://user/SendEmail.jtp?type=node&node=16483&i=1>
>> To unsubscribe from Apache Spark User List, click here.
>> NAML
>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: Default spark.deploy.recoveryMode
> <http://apache-spark-user-list.1001560.n3.nabble.com/Default-spark-deploy-recoveryMode-tp16375p16483.html>
> Sent from the Apache Spark User List mailing list archive
> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>
Re: Default spark.deploy.recoveryMode
Posted by Chitturi Padma <le...@gmail.com>.
which means the details are not persisted and hence any failures in workers
and master wouldnt start the daemons normally ..right ?
On Wed, Oct 15, 2014 at 12:17 PM, Prashant Sharma [via Apache Spark User
List] <ml...@n3.nabble.com> wrote:
> [Removing dev lists]
>
> You are absolutely correct about that.
>
> Prashant Sharma
>
>
>
> On Tue, Oct 14, 2014 at 5:03 PM, Priya Ch <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=16468&i=0>> wrote:
>
>> Hi Spark users/experts,
>>
>> In Spark source code (Master.scala & Worker.scala), when registering
>> the worker with master, I see the usage of *persistenceEngine*. When we
>> don't specify spark.deploy.recovery mode explicitly, what is the default
>> value used ? This recovery mode is used to persists and restore the
>> application & worker details.
>>
>> I see when recovery mode not specified explicitly,
>> *BlackHolePersistenceEngine* being used. Am i right ?
>>
>>
>> Thanks,
>> Padma Ch
>>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Default-spark-deploy-recoveryMode-tp16375p16468.html
> To start a new topic under Apache Spark User List, email
> ml-node+s1001560n1h76@n3.nabble.com
> To unsubscribe from Apache Spark User List, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=bGVhcm5pbmdzLmNoaXR0dXJpQGdtYWlsLmNvbXwxfC03NzExMjUwMg==>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Default-spark-deploy-recoveryMode-tp16375p16483.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Default spark.deploy.recoveryMode
Posted by Prashant Sharma <sc...@gmail.com>.
[Removing dev lists]
You are absolutely correct about that.
Prashant Sharma
On Tue, Oct 14, 2014 at 5:03 PM, Priya Ch <le...@gmail.com>
wrote:
> Hi Spark users/experts,
>
> In Spark source code (Master.scala & Worker.scala), when registering the
> worker with master, I see the usage of *persistenceEngine*. When we don't
> specify spark.deploy.recovery mode explicitly, what is the default value
> used ? This recovery mode is used to persists and restore the application &
> worker details.
>
> I see when recovery mode not specified explicitly,
> *BlackHolePersistenceEngine* being used. Am i right ?
>
>
> Thanks,
> Padma Ch
>