You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by kant kodali <ka...@gmail.com> on 2017/01/25 01:37:12 UTC
How many spark masters and do I need to tolerate one failure in one
DC and two AZ?
How many spark masters and zookeeper servers do I need to tolerate one
failure in one DC that has two availability zones ? Note: The one failure
that I want to tolerate can be in either availability zone.
Here is my understanding so far. please correct me If I am wrong?
for Zookeeper I would need 2F+1 server to tolerate F failures so in my case
that would be 3 however If one of the availability zone is down then I
would be left with only one zookeeper server (assuming the AZ that has two
zookeeper servers goes down) therefore I would need at least 4 zookeeper
servers two on each availability zone to tolerate a failure of one node and
to tolerate a failure of one availability zone.
And the number of spark master(standalone mode) would be 2. one on each
availability zone such that if one of the spark master in one of
availability zones or the entire availability zone goes down the zookeeper
ensemble will be ale to elect the spark master.
Is this correct so far?
Thanks,
kant
Re: How many spark masters and do I need to tolerate one failure in
one DC and two AZ?
Posted by kant kodali <ka...@gmail.com>.
I think I made a mistake...I would need at least N/2 + 1 nodes available
all the time to reach quorum and be able do leader election within
zookeeper ensemble
Given that I don't know ahead of time which availability zone is going to
go down I guess I cant really tolerate one AZ going down within the two
availability zones that are available in my DC.
On Tue, Jan 24, 2017 at 5:37 PM, kant kodali <ka...@gmail.com> wrote:
> How many spark masters and zookeeper servers do I need to tolerate one
> failure in one DC that has two availability zones ? Note: The one failure
> that I want to tolerate can be in either availability zone.
>
> Here is my understanding so far. please correct me If I am wrong?
>
> for Zookeeper I would need 2F+1 server to tolerate F failures so in my
> case that would be 3 however If one of the availability zone is down then I
> would be left with only one zookeeper server (assuming the AZ that has two
> zookeeper servers goes down) therefore I would need at least 4 zookeeper
> servers two on each availability zone to tolerate a failure of one node and
> to tolerate a failure of one availability zone.
>
> And the number of spark master(standalone mode) would be 2. one on each
> availability zone such that if one of the spark master in one of
> availability zones or the entire availability zone goes down the zookeeper
> ensemble will be ale to elect the spark master.
>
> Is this correct so far?
>
> Thanks,
> kant
>
>
>
>
>