You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by David Alves <da...@gmail.com> on 2010/12/06 16:43:04 UTC
A few questions on zookeeper
Hi
I'm a recent user of zookeeper (awesome work btw) and although I've (most of) the wiki and already have some running code I still have some questions.
1- Feasibility of a two node cluster:
Q- I know zookeeper runs over (a approx. version of) paxos and tolerates f failures in 2f+1 nodes but would it be possible to use in a 2 server + 1 quorum disk deployment (much like windows clusters do)? The objective would be to use ZK both for distributed (processing nodes >2) and single processing node (2 nodes active-passive) highly available deployments.
2- Meta-ZK cluster group membership/management.
Zk already provides a group membership construct by allowing to register/unregister ephemeral nodes and by deleting them on failed sessions, but would it be possible to do so for the actual zookeeper ensemble? For instance I'd like to make nodes add themselves to the ZK ensemble based on its size (for instance keep adding nodes based if ensemble size < 5) and if a node that is part of the ensemble fails then a new node would take its place. Is this possible?
Regards
David
Re: A few questions on zookeeper
Posted by Jared Cantwell <ja...@gmail.com>.
David,
For your second point, take a look at
https://issues.apache.org/jira/browse/ZOOKEEPER-107. Unfortunately,
Zookeeper does not support this feature yet.
~Jared
On Mon, Dec 6, 2010 at 10:43 AM, David Alves <da...@gmail.com> wrote:
> Hi
>
> I'm a recent user of zookeeper (awesome work btw) and although I've
> (most of) the wiki and already have some running code I still have some
> questions.
> 1- Feasibility of a two node cluster:
> Q- I know zookeeper runs over (a approx. version of) paxos
> and tolerates f failures in 2f+1 nodes but would it be possible to use in a
> 2 server + 1 quorum disk deployment (much like windows clusters do)? The
> objective would be to use ZK both for distributed (processing nodes >2) and
> single processing node (2 nodes active-passive) highly available
> deployments.
> 2- Meta-ZK cluster group membership/management.
> Zk already provides a group membership construct by allowing
> to register/unregister ephemeral nodes and by deleting them on failed
> sessions, but would it be possible to do so for the actual zookeeper
> ensemble? For instance I'd like to make nodes add themselves to the ZK
> ensemble based on its size (for instance keep adding nodes based if ensemble
> size < 5) and if a node that is part of the ensemble fails then a new node
> would take its place. Is this possible?
>
> Regards
> David
>
>
>
>
Re: A few questions on zookeeper
Posted by David Alves <da...@gmail.com>.
Hi Ted
Thank you again for your insight.
Regarding the rolling reconfigure solution... Is there any documentation on this procedure?
Regarding the two zk instances on a single machine... wouldn't that make that machine a SPOF? the objective here was to maintain a sustainable two machine config for a single server (2 nodes active-passive) highly available deployment, and take advantage of the fact that zk is already being used for larger (<3 active nodes) deployments.
regards
David
On Dec 7, 2010, at 8:38 PM, Ted Dunning wrote:
> You can make "quorum disk alike" happen by running two ZK instances on one
> of your two machines.
>
> You can also do this dynamically by doing the standard rolling reconfigure
> trick.
>
> On Tue, Dec 7, 2010 at 12:32 PM, David Alves <da...@gmail.com> wrote:
>
>> Hi Ted and Jared
>>
>> Thank you for your replies. Sorry in advance if my questions are too
>> absurd.
>> I know that, currently, with two servers the cluster would have a
>> higher failure probability (since the ensemble started with three and it
>> needs a two nodes majority, if any one fails there is no longer a majority
>> and the cluster hangs right?).
>> The purpose of the quorum disk would be to function has a tie
>> breaker in even node deployments (namely two), by representing an external,
>> storage only, resource where cluster state would be maintained. I know there
>> are some paxos-based deployments that use this technique, although I'm not
>> sufficiently into the internals of zookeeper to assert whether it would
>> work.
>> A question about an alternative... would it be possible to change
>> the majority function (dynamically) in two node deployments?
>> Regarding zookeeper-107, that is very interesting as we were
>> considering using zookeeper in an "elastic" setting. Is there progress on
>> this issue? can I help?
>>
>> Regards
>> David
>>
>> On Dec 6, 2010, at 5:02 PM, Ted Dunning wrote:
>>
>> David,
>>
>> For your second point, take a look at
>> https://issues.apache.org/jira/browse/ZOOKEEPER-107. Unfortunately,
>> Zookeeper does not support this feature yet.
>>
>> ~Jared
>>
>>> You can definitely do this, but your reliability will be different from
>> what
>>> might be expected. The probability of the cluster hanging due to a
>> failure
>>> is higher than for a single machine. The probability of data loss will
>> be
>>> lower than for a single machine. Similarly, any maintenance that
>> requires
>>> one of the two machines to be down will cause your cluster to hang.
>>>
>>> Can you clarify what you mean by 2 server + 1 quorum disk?
>>>
>>> Btw... one way around the downtime during maintenance is to reconfigure
>> the
>>> cluster on the fly to use a single server
>>> during the maintenance window. You will still have a short window of
>> freeze
>>> because you need to take down the
>>> server that is exiting the cluster first before bouncing the other server
>>> with the new config. If your maintenance period is less than a few
>> minutes,
>>> this isn't usually worthwhile.
>>>
>>> On Mon, Dec 6, 2010 at 7:43 AM, David Alves <da...@gmail.com>
>> wrote:
>>>
>>>> 1- Feasibility of a two node cluster:
>>>> Q- I know zookeeper runs over (a approx. version of) paxos
>>>> and tolerates f failures in 2f+1 nodes but would it be possible to use
>> in a
>>>> 2 server + 1 quorum disk deployment (much like windows clusters do)? The
>>>> objective would be to use ZK both for distributed (processing nodes >2)
>> and
>>>> single processing node (2 nodes active-passive) highly available
>>>> deployments.
>>>>
>>
>>
Re: A few questions on zookeeper
Posted by Ted Dunning <te...@gmail.com>.
You can make "quorum disk alike" happen by running two ZK instances on one
of your two machines.
You can also do this dynamically by doing the standard rolling reconfigure
trick.
On Tue, Dec 7, 2010 at 12:32 PM, David Alves <da...@gmail.com> wrote:
> Hi Ted and Jared
>
> Thank you for your replies. Sorry in advance if my questions are too
> absurd.
> I know that, currently, with two servers the cluster would have a
> higher failure probability (since the ensemble started with three and it
> needs a two nodes majority, if any one fails there is no longer a majority
> and the cluster hangs right?).
> The purpose of the quorum disk would be to function has a tie
> breaker in even node deployments (namely two), by representing an external,
> storage only, resource where cluster state would be maintained. I know there
> are some paxos-based deployments that use this technique, although I'm not
> sufficiently into the internals of zookeeper to assert whether it would
> work.
> A question about an alternative... would it be possible to change
> the majority function (dynamically) in two node deployments?
> Regarding zookeeper-107, that is very interesting as we were
> considering using zookeeper in an "elastic" setting. Is there progress on
> this issue? can I help?
>
> Regards
> David
>
> On Dec 6, 2010, at 5:02 PM, Ted Dunning wrote:
>
> David,
>
> For your second point, take a look at
> https://issues.apache.org/jira/browse/ZOOKEEPER-107. Unfortunately,
> Zookeeper does not support this feature yet.
>
> ~Jared
>
> > You can definitely do this, but your reliability will be different from
> what
> > might be expected. The probability of the cluster hanging due to a
> failure
> > is higher than for a single machine. The probability of data loss will
> be
> > lower than for a single machine. Similarly, any maintenance that
> requires
> > one of the two machines to be down will cause your cluster to hang.
> >
> > Can you clarify what you mean by 2 server + 1 quorum disk?
> >
> > Btw... one way around the downtime during maintenance is to reconfigure
> the
> > cluster on the fly to use a single server
> > during the maintenance window. You will still have a short window of
> freeze
> > because you need to take down the
> > server that is exiting the cluster first before bouncing the other server
> > with the new config. If your maintenance period is less than a few
> minutes,
> > this isn't usually worthwhile.
> >
> > On Mon, Dec 6, 2010 at 7:43 AM, David Alves <da...@gmail.com>
> wrote:
> >
> >> 1- Feasibility of a two node cluster:
> >> Q- I know zookeeper runs over (a approx. version of) paxos
> >> and tolerates f failures in 2f+1 nodes but would it be possible to use
> in a
> >> 2 server + 1 quorum disk deployment (much like windows clusters do)? The
> >> objective would be to use ZK both for distributed (processing nodes >2)
> and
> >> single processing node (2 nodes active-passive) highly available
> >> deployments.
> >>
>
>
Re: A few questions on zookeeper
Posted by David Alves <da...@gmail.com>.
Hi Ted and Jared
Thank you for your replies. Sorry in advance if my questions are too absurd.
I know that, currently, with two servers the cluster would have a higher failure probability (since the ensemble started with three and it needs a two nodes majority, if any one fails there is no longer a majority and the cluster hangs right?).
The purpose of the quorum disk would be to function has a tie breaker in even node deployments (namely two), by representing an external, storage only, resource where cluster state would be maintained. I know there are some paxos-based deployments that use this technique, although I'm not sufficiently into the internals of zookeeper to assert whether it would work.
A question about an alternative... would it be possible to change the majority function (dynamically) in two node deployments?
Regarding zookeeper-107, that is very interesting as we were considering using zookeeper in an "elastic" setting. Is there progress on this issue? can I help?
Regards
David
On Dec 6, 2010, at 5:02 PM, Ted Dunning wrote:
David,
For your second point, take a look at
https://issues.apache.org/jira/browse/ZOOKEEPER-107. Unfortunately,
Zookeeper does not support this feature yet.
~Jared
> You can definitely do this, but your reliability will be different from what
> might be expected. The probability of the cluster hanging due to a failure
> is higher than for a single machine. The probability of data loss will be
> lower than for a single machine. Similarly, any maintenance that requires
> one of the two machines to be down will cause your cluster to hang.
>
> Can you clarify what you mean by 2 server + 1 quorum disk?
>
> Btw... one way around the downtime during maintenance is to reconfigure the
> cluster on the fly to use a single server
> during the maintenance window. You will still have a short window of freeze
> because you need to take down the
> server that is exiting the cluster first before bouncing the other server
> with the new config. If your maintenance period is less than a few minutes,
> this isn't usually worthwhile.
>
> On Mon, Dec 6, 2010 at 7:43 AM, David Alves <da...@gmail.com> wrote:
>
>> 1- Feasibility of a two node cluster:
>> Q- I know zookeeper runs over (a approx. version of) paxos
>> and tolerates f failures in 2f+1 nodes but would it be possible to use in a
>> 2 server + 1 quorum disk deployment (much like windows clusters do)? The
>> objective would be to use ZK both for distributed (processing nodes >2) and
>> single processing node (2 nodes active-passive) highly available
>> deployments.
>>
Re: A few questions on zookeeper
Posted by Ted Dunning <te...@gmail.com>.
You can definitely do this, but your reliability will be different from what
might be expected. The probability of the cluster hanging due to a failure
is higher than for a single machine. The probability of data loss will be
lower than for a single machine. Similarly, any maintenance that requires
one of the two machines to be down will cause your cluster to hang.
Can you clarify what you mean by 2 server + 1 quorum disk?
Btw... one way around the downtime during maintenance is to reconfigure the
cluster on the fly to use a single server
during the maintenance window. You will still have a short window of freeze
because you need to take down the
server that is exiting the cluster first before bouncing the other server
with the new config. If your maintenance period is less than a few minutes,
this isn't usually worthwhile.
On Mon, Dec 6, 2010 at 7:43 AM, David Alves <da...@gmail.com> wrote:
> 1- Feasibility of a two node cluster:
> Q- I know zookeeper runs over (a approx. version of) paxos
> and tolerates f failures in 2f+1 nodes but would it be possible to use in a
> 2 server + 1 quorum disk deployment (much like windows clusters do)? The
> objective would be to use ZK both for distributed (processing nodes >2) and
> single processing node (2 nodes active-passive) highly available
> deployments.
>