Multi master Spark

[Pradeep Ch <pr...@gmail.com>]

Hi,

I want to enable Spark Master HA in spark. Documentation specifies that we
can do this with the help of Zookeepers. But what I am worried is how to
configure one master with the other and similarly how do workers know that
the have two masters? where do you specify the multi-master information?

Thanks for the help.

Thanks,
Pradeep

Re: Multi master Spark

[Aaron Davidson <il...@gmail.com>]

It is as Jagat said. The Masters do not need to know about one another, as
ZooKeeper manages their implicit communication. As for Workers (and
applications, such as spark-shell), once a Worker is registered with
*some *Master,
its metadata is stored in ZooKeeper such that if another Master is elected,
it informs all current Workers (and applications), so they need not have a
priori knowledge of all future Masters.


On Wed, Apr 9, 2014 at 3:46 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:

> ah.
>
> standalone HA master was added in 0.9.0. Same logic, but Spark-native.
>
>
> On Wed, Apr 9, 2014 at 3:31 PM, Pradeep Ch <pr...@gmail.com>wrote:
>
>> Thanks Dmitriy. But I want multi master support when running spark
>> standalone. Also I want to know if this multi master thing works if I use
>> spark-shell.
>>
>>
>> On Wed, Apr 9, 2014 at 3:26 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>
>>> The only way i know to do this is to use mesos with zookeepers. you
>>> specify zookeeper url as spark url that contains multiple zookeeper hosts.
>>> Multiple mesos masters are then elected thru zookeeper leader election
>>> until current leader dies; at which point mesos will elect another master
>>> (if still left).
>>>
>>> iirc, in this mode spark master never runs, only master slaves are being
>>> spun by mesos slaves directly.
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Apr 9, 2014 at 3:08 PM, Pradeep Ch <pr...@gmail.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> I want to enable Spark Master HA in spark. Documentation specifies that
>>>> we can do this with the help of Zookeepers. But what I am worried is how to
>>>> configure one master with the other and similarly how do workers know that
>>>> the have two masters? where do you specify the multi-master information?
>>>>
>>>> Thanks for the help.
>>>>
>>>> Thanks,
>>>> Pradeep
>>>>
>>>
>>>
>>
>

Re: Multi master Spark

[Dmitriy Lyubimov <dl...@gmail.com>]

ah.

standalone HA master was added in 0.9.0. Same logic, but Spark-native.


On Wed, Apr 9, 2014 at 3:31 PM, Pradeep Ch <pr...@gmail.com>wrote:

> Thanks Dmitriy. But I want multi master support when running spark
> standalone. Also I want to know if this multi master thing works if I use
> spark-shell.
>
>
> On Wed, Apr 9, 2014 at 3:26 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>
>> The only way i know to do this is to use mesos with zookeepers. you
>> specify zookeeper url as spark url that contains multiple zookeeper hosts.
>> Multiple mesos masters are then elected thru zookeeper leader election
>> until current leader dies; at which point mesos will elect another master
>> (if still left).
>>
>> iirc, in this mode spark master never runs, only master slaves are being
>> spun by mesos slaves directly.
>>
>>
>>
>>
>>
>> On Wed, Apr 9, 2014 at 3:08 PM, Pradeep Ch <pr...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I want to enable Spark Master HA in spark. Documentation specifies that
>>> we can do this with the help of Zookeepers. But what I am worried is how to
>>> configure one master with the other and similarly how do workers know that
>>> the have two masters? where do you specify the multi-master information?
>>>
>>> Thanks for the help.
>>>
>>> Thanks,
>>> Pradeep
>>>
>>
>>
>

Re: Multi master Spark

[Pradeep Ch <pr...@gmail.com>]

Thanks Dmitriy. But I want multi master support when running spark
standalone. Also I want to know if this multi master thing works if I use
spark-shell.


On Wed, Apr 9, 2014 at 3:26 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:

> The only way i know to do this is to use mesos with zookeepers. you
> specify zookeeper url as spark url that contains multiple zookeeper hosts.
> Multiple mesos masters are then elected thru zookeeper leader election
> until current leader dies; at which point mesos will elect another master
> (if still left).
>
> iirc, in this mode spark master never runs, only master slaves are being
> spun by mesos slaves directly.
>
>
>
>
>
> On Wed, Apr 9, 2014 at 3:08 PM, Pradeep Ch <pr...@gmail.com>wrote:
>
>> Hi,
>>
>> I want to enable Spark Master HA in spark. Documentation specifies that
>> we can do this with the help of Zookeepers. But what I am worried is how to
>> configure one master with the other and similarly how do workers know that
>> the have two masters? where do you specify the multi-master information?
>>
>> Thanks for the help.
>>
>> Thanks,
>> Pradeep
>>
>
>

Re: Multi master Spark

[Dmitriy Lyubimov <dl...@gmail.com>]

The only way i know to do this is to use mesos with zookeepers. you specify
zookeeper url as spark url that contains multiple zookeeper hosts. Multiple
mesos masters are then elected thru zookeeper leader election until current
leader dies; at which point mesos will elect another master (if still
left).

iirc, in this mode spark master never runs, only master slaves are being
spun by mesos slaves directly.





On Wed, Apr 9, 2014 at 3:08 PM, Pradeep Ch <pr...@gmail.com>wrote:

> Hi,
>
> I want to enable Spark Master HA in spark. Documentation specifies that we
> can do this with the help of Zookeepers. But what I am worried is how to
> configure one master with the other and similarly how do workers know that
> the have two masters? where do you specify the multi-master information?
>
> Thanks for the help.
>
> Thanks,
> Pradeep
>

Re: Multi master Spark

[Jagat Singh <ja...@gmail.com>]

Hello Pradeep,

Quoting from

https://spark.apache.org/docs/0.9.0/spark-standalone.html

In order to schedule new applications or add Workers to the cluster, they
need to know the IP address of the current leader. This can be accomplished
by simply passing in a list of Masters where you used to pass in a single
one. For example, you might start your SparkContext pointing to
spark://host1:port1,host2:port2. This would cause your SparkContext to try
registering with both Masters - if host1 goes down, this configuration
would still be correct as we'd find the new leader, host2.

Thanks,

Jagat Singh


On Thu, Apr 10, 2014 at 8:08 AM, Pradeep Ch <pr...@gmail.com>wrote:

> Hi,
>
> I want to enable Spark Master HA in spark. Documentation specifies that we
> can do this with the help of Zookeepers. But what I am worried is how to
> configure one master with the other and similarly how do workers know that
> the have two masters? where do you specify the multi-master information?
>
> Thanks for the help.
>
> Thanks,
> Pradeep
>