You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Jonathan Gray <jg...@apache.org> on 2009/07/17 21:48:18 UTC
Multiple ZK clusters or a single, shared cluster?
Hey guys,
Been using ZK indirectly for a few months now in the HBase and Katta
realms. Both of these applications make it really easy so you don't
have to be involved much with managing your ZK cluster to support it.
I'm now using ZK for a bunch of things internally, so now I'm manually
configuring, starting, and managing a cluster.
What advice is there about whether I should be sharing a single cluster
between all my applications, or running separate ones for each use?
I've been told that it's strongly recommended to run your ZK nodes
separately from the application using them (this is actually what we're
telling new users over in HBase, though a majority of installations will
likely co-host them with DataNodes and RegionServers).
I don't have the resources to maintain a separate 3+ node ZK cluster for
each of my applications, so this is not really an option. I'm trying to
decide if I should have HBase running/managing it's own ZK cluster that
is co-located with some of the regionservers (there will be ample
memory, but ZK will not have a dedicated disk), or if I should be
pointing it to a dedicated 3 node ZK cluster.
I would then also have Katta pointing at this same shared cluster (or a
separate cluster would be co-located with katta nodes). Same for my
application; could share nodes with the app servers or pointed at a
single ZK cluster.
Trade-offs I should be aware of? Current best practices?
Any help would be much appreciated. Thanks.
Jonathan Gray
Re: Leader Elections
Posted by Flavio Junqueira <fp...@yahoo-inc.com>.
For the partial subscription, my take right now is that it should be
part of the registration procedure. When an observer joins, it
contacts some ensemble server (a follower or the current leader) and
appends a path to the initial message it sends to this server. This
path corresponds to the subtree it is interested in. The server simply
filters the updates it sends to the observer based on the path prefix.
With respect to offline modes, we have considered having a replica
switching to read-only when it is not part of a quorum (ZOOKEEPER-40).
Dealing with reconciliation is messy, and I would say that we do not
want to go down this path at this moment. Others might have a
different opinion, though.
-Flavio
On Jul 20, 2009, at 2:46 PM, Henry Robinson wrote:
> I think partial subscription for an Observer would be easy to do - I
> don't
> think it will make it into 368 which is big enough already, but it
> would not
> be an enormous amount of work. The main thing to do is to figure out
> the
> protocol for subscription; probably just a new message type. That
> said, it
> would require some careful stepping around the sync code in order to
> make
> sure that the the Observer knows what the latest zxid is even if it
> doesn't
> know the full history. Very do-able though.
>
> 'Offline' modes for parts of the graph are more challenging; we
> would need
> to think hard about the right way to implement this.
>
> I had imagined that Observers would be a good integration point for a
> third-party pubsub system (like TIB or something) via a plugin
> mechanism. In
> my opinion I think it's important for ZK not to try to become a
> general
> pubsub framework which is not its core goal, although I can't speak
> for the
> committers. That said, rudimentary subscription is a good idea to
> prevent
> unnecessary WAN traffic.
>
> The idea of having ensembles subscribe to each other is a bit tricky;
> essentially it would require one ensemble to mirror the other with one
> ensemble acting as the master with a netsplit putting the slave
> ensemble
> into read-only mode (or removing the mounted subtree). Again, I
> think it
> could be done but would be a big feature.
>
> Henry
>
> On Mon, Jul 20, 2009 at 10:21 PM, Scott Carey
> <sc...@richrelevance.com>wrote:
>
>> Todd has put it much more eloquently. Comments below:
>>
>> On 7/20/09 11:50 AM, "Todd Greenwood" <to...@audiencescience.com>
>> wrote:
>>
>>> Flavio, Ted, Henry, Scott, this would perfectly well for my use case
>>> provided:
>>>
>>> SINGLE ENSEMBLE:
>>> GROUP A : ZK Servers w/ read/write AND Leader Elections
>>> GROUP B : ZK Servers w/ read/write W/O Leader Elections
>>>
>>> So, we can craft this via Observers and Hiererarchial Quorum groups?
>>> Great. Problem solved.
>>>
>>> When will this be production ready? :o)
>>>
>>> --------------------
>>>
>>> Scott brought up a multi-feature that is very interesting for me.
>>> Namely:
>>>
>>> 1. Offline ZK servers that sync & merge on reconnect
>>>
>>> The offline servers seems conceptually simple, it's kind of like a
>>> messaging system. However, the merge and resolve step when two
>>> servers
>>> reconnect might be challenging. Cool idea though.
>>
>> Yes, this is very useful for WAN use cases. I've already done
>> something
>> like it with a hack:
>> Ensemble A "Master/Central"
>> "Remote Proxy" N -- embeds its own ZK, and runs two clients. One
>> Client
>> connects to Ensemble A and watches a partial sub-graph, propagating
>> that
>> into its local embedded ZK server. This subgraph is read-only for
>> those
>> that access the Proxy. A second client accesses the local ZK
>> server and
>> monitors a different subgraph, which it propagates to the Master
>> ensemble.
>> This is writeable by clients accessing the Proxy and on the Master
>> ensemble
>> is only written to by this Proxy.
>>
>> The above is all application enforced. There are constraints on
>> what sort
>> of
>> things can be built with this, but for the subset of use cases I
>> need over
>> WAN, its more than enough.
>>
>>>
>>> 2. Partial memory graph subscriptions
>>>
>>> The second idea is partial memory graph subscriptions. This would
>>> enable
>>> virtual ensembles to interract on the same physical ensemble. For
>>> my use
>>> case, this would prevent unnecessary cross talk between nodes on a
>>> WAN,
>>> allowing me to define the subsets of the memory graph that need to
>>> be
>>> replicated, and to whom. This would be a huge scalability win for
>>> WAN
>>> use cases.
>>
>> Yes, a more general partial graph subscription / ownership
>> framework would
>> allow for not just better WAN scalability but also (and more
>> critically
>> IMO)
>> higher reliability. Often, some large subset of application
>> functionality
>> is local to one network, and a minority is global and in need of WAN
>> communication. In this case, when the WAN breaks one wishes that
>> local
>> functionality to continue to function, and only those parts truly
>> dependant
>> on external events to be interrupted.
>> Currently one has to have separate ensembles to partition data and
>> clunky
>> 'bridge' code to intercommunicate.
>>
>> It would certainly be more natural if two ZK ensembles could
>> register with
>> each other, in a 'partial sub-graph publish/subscribe' framework.
>> It could
>> almost be like file system mounting:
>> To subscribe:
>> subscribe otherEnsemble:port/path/to/otherstuff /localpath/to/
>> mount/into
>>
>> Publishing is the same thing -- think of it as a request for a
>> remote ZK
>> cluster to subscribe to the local ZK's data.
>>
>>
>>
>>
>>>
>>> -Todd
>>>
>>> -----Original Message-----
>>> From: Scott Carey [mailto:scott@richrelevance.com]
>>> Sent: Monday, July 20, 2009 11:00 AM
>>> To: zookeeper-user@hadoop.apache.org
>>> Subject: Re: Leader Elections
>>>
>>> Observers would be awesome especially with a couple enhancements /
>>> extensions:
>>>
>>> An option for the observers to enter a special state if the WAN link
>>> goes down to the "master" cluster. A read-only option would be
>>> great.
>>> However, allowing certain types of writes to continue on a limited
>>> basis
>>> would be highly valuable as well. An observer could "own" a special
>>> node and its subnodes. Only these subnodes would be writable by the
>>> observer when there was a session break to the master cluster, and
>>> the
>>> master cluster would take all the changes when the link is
>>> reestablished. Essentially, it is a portion of the hierarchy that
>>> is
>>> writable only by a specitfic observer, and read-only for others.
>>> The purpose of this would be for when the WAN link goes down to the
>>> "master" ZKs for certain types of use cases - status updates or
>>> other
>>> changes local to the observer that are strictly read-only outside
>>> the
>>> Observer's 'realm'.
>>>
>>>
>>> On 7/19/09 12:16 PM, "Henry Robinson" <he...@cloudera.com> wrote:
>>>
>>> You can. See ZOOKEEPER-368 - at first glance it sounds like
>>> observers
>>> will
>>> be a good fit for your requirements.
>>>
>>> Do bear in mind that the patch on the jira is only for discussion
>>> purposes;
>>> I would not consider it currently fit for production use. I hope
>>> to put
>>> up a
>>> much better patch this week.
>>>
>>> Henry
>>>
>>> On Sat, Jul 18, 2009 at 7:38 PM, Ted Dunning <te...@gmail.com>
>>> wrote:
>>>
>>>> Can you submit updates via an observer?
>>>>
>>>> On Sat, Jul 18, 2009 at 6:38 AM, Flavio Junqueira <fpj@yahoo-inc.com
>>>> >
>>>> wrote:
>>>>
>>>>> 2- Observers: you could have one computing center containing an
>>> ensemble
>>>>> and observers around the edge just learning committed values.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Ted Dunning, CTO
>>>> DeepDyve
>>>>
>>>
>>>
>>
>>
Re: Leader Elections
Posted by Henry Robinson <he...@cloudera.com>.
I think partial subscription for an Observer would be easy to do - I don't
think it will make it into 368 which is big enough already, but it would not
be an enormous amount of work. The main thing to do is to figure out the
protocol for subscription; probably just a new message type. That said, it
would require some careful stepping around the sync code in order to make
sure that the the Observer knows what the latest zxid is even if it doesn't
know the full history. Very do-able though.
'Offline' modes for parts of the graph are more challenging; we would need
to think hard about the right way to implement this.
I had imagined that Observers would be a good integration point for a
third-party pubsub system (like TIB or something) via a plugin mechanism. In
my opinion I think it's important for ZK not to try to become a general
pubsub framework which is not its core goal, although I can't speak for the
committers. That said, rudimentary subscription is a good idea to prevent
unnecessary WAN traffic.
The idea of having ensembles subscribe to each other is a bit tricky;
essentially it would require one ensemble to mirror the other with one
ensemble acting as the master with a netsplit putting the slave ensemble
into read-only mode (or removing the mounted subtree). Again, I think it
could be done but would be a big feature.
Henry
On Mon, Jul 20, 2009 at 10:21 PM, Scott Carey <sc...@richrelevance.com>wrote:
> Todd has put it much more eloquently. Comments below:
>
> On 7/20/09 11:50 AM, "Todd Greenwood" <to...@audiencescience.com> wrote:
>
> > Flavio, Ted, Henry, Scott, this would perfectly well for my use case
> > provided:
> >
> > SINGLE ENSEMBLE:
> > GROUP A : ZK Servers w/ read/write AND Leader Elections
> > GROUP B : ZK Servers w/ read/write W/O Leader Elections
> >
> > So, we can craft this via Observers and Hiererarchial Quorum groups?
> > Great. Problem solved.
> >
> > When will this be production ready? :o)
> >
> > --------------------
> >
> > Scott brought up a multi-feature that is very interesting for me.
> > Namely:
> >
> > 1. Offline ZK servers that sync & merge on reconnect
> >
> > The offline servers seems conceptually simple, it's kind of like a
> > messaging system. However, the merge and resolve step when two servers
> > reconnect might be challenging. Cool idea though.
>
> Yes, this is very useful for WAN use cases. I've already done something
> like it with a hack:
> Ensemble A "Master/Central"
> "Remote Proxy" N -- embeds its own ZK, and runs two clients. One Client
> connects to Ensemble A and watches a partial sub-graph, propagating that
> into its local embedded ZK server. This subgraph is read-only for those
> that access the Proxy. A second client accesses the local ZK server and
> monitors a different subgraph, which it propagates to the Master ensemble.
> This is writeable by clients accessing the Proxy and on the Master ensemble
> is only written to by this Proxy.
>
> The above is all application enforced. There are constraints on what sort
> of
> things can be built with this, but for the subset of use cases I need over
> WAN, its more than enough.
>
> >
> > 2. Partial memory graph subscriptions
> >
> > The second idea is partial memory graph subscriptions. This would enable
> > virtual ensembles to interract on the same physical ensemble. For my use
> > case, this would prevent unnecessary cross talk between nodes on a WAN,
> > allowing me to define the subsets of the memory graph that need to be
> > replicated, and to whom. This would be a huge scalability win for WAN
> > use cases.
>
> Yes, a more general partial graph subscription / ownership framework would
> allow for not just better WAN scalability but also (and more critically
> IMO)
> higher reliability. Often, some large subset of application functionality
> is local to one network, and a minority is global and in need of WAN
> communication. In this case, when the WAN breaks one wishes that local
> functionality to continue to function, and only those parts truly dependant
> on external events to be interrupted.
> Currently one has to have separate ensembles to partition data and clunky
> 'bridge' code to intercommunicate.
>
> It would certainly be more natural if two ZK ensembles could register with
> each other, in a 'partial sub-graph publish/subscribe' framework. It could
> almost be like file system mounting:
> To subscribe:
> subscribe otherEnsemble:port/path/to/otherstuff /localpath/to/mount/into
>
> Publishing is the same thing -- think of it as a request for a remote ZK
> cluster to subscribe to the local ZK's data.
>
>
>
>
> >
> > -Todd
> >
> > -----Original Message-----
> > From: Scott Carey [mailto:scott@richrelevance.com]
> > Sent: Monday, July 20, 2009 11:00 AM
> > To: zookeeper-user@hadoop.apache.org
> > Subject: Re: Leader Elections
> >
> > Observers would be awesome especially with a couple enhancements /
> > extensions:
> >
> > An option for the observers to enter a special state if the WAN link
> > goes down to the "master" cluster. A read-only option would be great.
> > However, allowing certain types of writes to continue on a limited basis
> > would be highly valuable as well. An observer could "own" a special
> > node and its subnodes. Only these subnodes would be writable by the
> > observer when there was a session break to the master cluster, and the
> > master cluster would take all the changes when the link is
> > reestablished. Essentially, it is a portion of the hierarchy that is
> > writable only by a specitfic observer, and read-only for others.
> > The purpose of this would be for when the WAN link goes down to the
> > "master" ZKs for certain types of use cases - status updates or other
> > changes local to the observer that are strictly read-only outside the
> > Observer's 'realm'.
> >
> >
> > On 7/19/09 12:16 PM, "Henry Robinson" <he...@cloudera.com> wrote:
> >
> > You can. See ZOOKEEPER-368 - at first glance it sounds like observers
> > will
> > be a good fit for your requirements.
> >
> > Do bear in mind that the patch on the jira is only for discussion
> > purposes;
> > I would not consider it currently fit for production use. I hope to put
> > up a
> > much better patch this week.
> >
> > Henry
> >
> > On Sat, Jul 18, 2009 at 7:38 PM, Ted Dunning <te...@gmail.com>
> > wrote:
> >
> >> Can you submit updates via an observer?
> >>
> >> On Sat, Jul 18, 2009 at 6:38 AM, Flavio Junqueira <fp...@yahoo-inc.com>
> >> wrote:
> >>
> >>> 2- Observers: you could have one computing center containing an
> > ensemble
> >>> and observers around the edge just learning committed values.
> >>
> >>
> >>
> >>
> >> --
> >> Ted Dunning, CTO
> >> DeepDyve
> >>
> >
> >
>
>
Re: Leader Elections
Posted by Flavio Junqueira <fp...@yahoo-inc.com>.
Henry's observation sounds right. I'd like to point out, though, that
for BCP it might be interesting in some cases to allow multiple groups
to contain prospective leaders. To tolerate any group of a set of
groups failing completely, you would need at least 3 groups, so it is
probably not applicable to your case of two groups.
On Jul 20, 2009, at 3:04 PM, Todd Greenwood wrote:
> Henry, cool. When youre patch is ready for testing, I'll devote some
> time to take a test pass on it.
>
> -----Original Message-----
> From: Henry Robinson [mailto:henry@cloudera.com]
> Sent: Monday, July 20, 2009 2:54 PM
> To: zookeeper-user@hadoop.apache.org
> Subject: Re: Leader Elections
>
> On Mon, Jul 20, 2009 at 7:50 PM, Todd Greenwood
> <to...@audiencescience.com>wrote:
>
>> Flavio, Ted, Henry, Scott, this would perfectly well for my use case
>> provided:
>>
>> SINGLE ENSEMBLE:
>> GROUP A : ZK Servers w/ read/write AND Leader Elections
>> GROUP B : ZK Servers w/ read/write W/O Leader Elections
>>
>> So, we can craft this via Observers and Hiererarchial Quorum groups?
>> Great. Problem solved.
>>
>> When will this be production ready? :o)
>>
>
> Looks to me like you don't even need hierarchical quorums for this -
> make
> everyone in group B an Observer and you're done.
>
> I've been working on this feature. Recently we've been discussing a
> proof-of-concept patch on the JIRA. I have nearly finished a less
> rough
> patch which I will submit for discussion and potentially commit this
> week.
> At that point it would be extremely helpful if you could help test the
> patch, and you can start considering it for production. To get into
> trunk I
> will have to write a comprehensive test suite and update the
> documentation,
> and then making sure all the boxes are ticked and no regressions are
> thrown
> up can take a little while.
>
> Henry
>
>
>
>
>>
>> --------------------
>>
>> Scott brought up a multi-feature that is very interesting for me.
>> Namely:
>>
>> 1. Offline ZK servers that sync & merge on reconnect
>>
>> The offline servers seems conceptually simple, it's kind of like a
>> messaging system. However, the merge and resolve step when two
>> servers
>> reconnect might be challenging. Cool idea though.
>>
>> 2. Partial memory graph subscriptions
>>
>> The second idea is partial memory graph subscriptions. This would
> enable
>> virtual ensembles to interract on the same physical ensemble. For my
> use
>> case, this would prevent unnecessary cross talk between nodes on a
> WAN,
>> allowing me to define the subsets of the memory graph that need to be
>> replicated, and to whom. This would be a huge scalability win for WAN
>> use cases.
>>
>> -Todd
>>
>> -----Original Message-----
>> From: Scott Carey [mailto:scott@richrelevance.com]
>> Sent: Monday, July 20, 2009 11:00 AM
>> To: zookeeper-user@hadoop.apache.org
>> Subject: Re: Leader Elections
>>
>> Observers would be awesome especially with a couple enhancements /
>> extensions:
>>
>> An option for the observers to enter a special state if the WAN link
>> goes down to the "master" cluster. A read-only option would be
>> great.
>> However, allowing certain types of writes to continue on a limited
> basis
>> would be highly valuable as well. An observer could "own" a special
>> node and its subnodes. Only these subnodes would be writable by the
>> observer when there was a session break to the master cluster, and
>> the
>> master cluster would take all the changes when the link is
>> reestablished. Essentially, it is a portion of the hierarchy that is
>> writable only by a specitfic observer, and read-only for others.
>> The purpose of this would be for when the WAN link goes down to the
>> "master" ZKs for certain types of use cases - status updates or other
>> changes local to the observer that are strictly read-only outside the
>> Observer's 'realm'.
>>
>>
>> On 7/19/09 12:16 PM, "Henry Robinson" <he...@cloudera.com> wrote:
>>
>> You can. See ZOOKEEPER-368 - at first glance it sounds like observers
>> will
>> be a good fit for your requirements.
>>
>> Do bear in mind that the patch on the jira is only for discussion
>> purposes;
>> I would not consider it currently fit for production use. I hope to
> put
>> up a
>> much better patch this week.
>>
>> Henry
>>
>> On Sat, Jul 18, 2009 at 7:38 PM, Ted Dunning <te...@gmail.com>
>> wrote:
>>
>>> Can you submit updates via an observer?
>>>
>>> On Sat, Jul 18, 2009 at 6:38 AM, Flavio Junqueira
> <fp...@yahoo-inc.com>
>>> wrote:
>>>
>>>> 2- Observers: you could have one computing center containing an
>> ensemble
>>>> and observers around the edge just learning committed values.
>>>
>>>
>>>
>>>
>>> --
>>> Ted Dunning, CTO
>>> DeepDyve
>>>
>>
>>
RE: Leader Elections
Posted by Todd Greenwood <to...@audiencescience.com>.
Henry, cool. When youre patch is ready for testing, I'll devote some
time to take a test pass on it.
-----Original Message-----
From: Henry Robinson [mailto:henry@cloudera.com]
Sent: Monday, July 20, 2009 2:54 PM
To: zookeeper-user@hadoop.apache.org
Subject: Re: Leader Elections
On Mon, Jul 20, 2009 at 7:50 PM, Todd Greenwood
<to...@audiencescience.com>wrote:
> Flavio, Ted, Henry, Scott, this would perfectly well for my use case
> provided:
>
> SINGLE ENSEMBLE:
> GROUP A : ZK Servers w/ read/write AND Leader Elections
> GROUP B : ZK Servers w/ read/write W/O Leader Elections
>
> So, we can craft this via Observers and Hiererarchial Quorum groups?
> Great. Problem solved.
>
> When will this be production ready? :o)
>
Looks to me like you don't even need hierarchical quorums for this -
make
everyone in group B an Observer and you're done.
I've been working on this feature. Recently we've been discussing a
proof-of-concept patch on the JIRA. I have nearly finished a less rough
patch which I will submit for discussion and potentially commit this
week.
At that point it would be extremely helpful if you could help test the
patch, and you can start considering it for production. To get into
trunk I
will have to write a comprehensive test suite and update the
documentation,
and then making sure all the boxes are ticked and no regressions are
thrown
up can take a little while.
Henry
>
> --------------------
>
> Scott brought up a multi-feature that is very interesting for me.
> Namely:
>
> 1. Offline ZK servers that sync & merge on reconnect
>
> The offline servers seems conceptually simple, it's kind of like a
> messaging system. However, the merge and resolve step when two servers
> reconnect might be challenging. Cool idea though.
>
> 2. Partial memory graph subscriptions
>
> The second idea is partial memory graph subscriptions. This would
enable
> virtual ensembles to interract on the same physical ensemble. For my
use
> case, this would prevent unnecessary cross talk between nodes on a
WAN,
> allowing me to define the subsets of the memory graph that need to be
> replicated, and to whom. This would be a huge scalability win for WAN
> use cases.
>
> -Todd
>
> -----Original Message-----
> From: Scott Carey [mailto:scott@richrelevance.com]
> Sent: Monday, July 20, 2009 11:00 AM
> To: zookeeper-user@hadoop.apache.org
> Subject: Re: Leader Elections
>
> Observers would be awesome especially with a couple enhancements /
> extensions:
>
> An option for the observers to enter a special state if the WAN link
> goes down to the "master" cluster. A read-only option would be great.
> However, allowing certain types of writes to continue on a limited
basis
> would be highly valuable as well. An observer could "own" a special
> node and its subnodes. Only these subnodes would be writable by the
> observer when there was a session break to the master cluster, and the
> master cluster would take all the changes when the link is
> reestablished. Essentially, it is a portion of the hierarchy that is
> writable only by a specitfic observer, and read-only for others.
> The purpose of this would be for when the WAN link goes down to the
> "master" ZKs for certain types of use cases - status updates or other
> changes local to the observer that are strictly read-only outside the
> Observer's 'realm'.
>
>
> On 7/19/09 12:16 PM, "Henry Robinson" <he...@cloudera.com> wrote:
>
> You can. See ZOOKEEPER-368 - at first glance it sounds like observers
> will
> be a good fit for your requirements.
>
> Do bear in mind that the patch on the jira is only for discussion
> purposes;
> I would not consider it currently fit for production use. I hope to
put
> up a
> much better patch this week.
>
> Henry
>
> On Sat, Jul 18, 2009 at 7:38 PM, Ted Dunning <te...@gmail.com>
> wrote:
>
> > Can you submit updates via an observer?
> >
> > On Sat, Jul 18, 2009 at 6:38 AM, Flavio Junqueira
<fp...@yahoo-inc.com>
> > wrote:
> >
> > > 2- Observers: you could have one computing center containing an
> ensemble
> > > and observers around the edge just learning committed values.
> >
> >
> >
> >
> > --
> > Ted Dunning, CTO
> > DeepDyve
> >
>
>
Re: Leader Elections
Posted by Henry Robinson <he...@cloudera.com>.
On Mon, Jul 20, 2009 at 7:50 PM, Todd Greenwood
<to...@audiencescience.com>wrote:
> Flavio, Ted, Henry, Scott, this would perfectly well for my use case
> provided:
>
> SINGLE ENSEMBLE:
> GROUP A : ZK Servers w/ read/write AND Leader Elections
> GROUP B : ZK Servers w/ read/write W/O Leader Elections
>
> So, we can craft this via Observers and Hiererarchial Quorum groups?
> Great. Problem solved.
>
> When will this be production ready? :o)
>
Looks to me like you don't even need hierarchical quorums for this - make
everyone in group B an Observer and you're done.
I've been working on this feature. Recently we've been discussing a
proof-of-concept patch on the JIRA. I have nearly finished a less rough
patch which I will submit for discussion and potentially commit this week.
At that point it would be extremely helpful if you could help test the
patch, and you can start considering it for production. To get into trunk I
will have to write a comprehensive test suite and update the documentation,
and then making sure all the boxes are ticked and no regressions are thrown
up can take a little while.
Henry
>
> --------------------
>
> Scott brought up a multi-feature that is very interesting for me.
> Namely:
>
> 1. Offline ZK servers that sync & merge on reconnect
>
> The offline servers seems conceptually simple, it's kind of like a
> messaging system. However, the merge and resolve step when two servers
> reconnect might be challenging. Cool idea though.
>
> 2. Partial memory graph subscriptions
>
> The second idea is partial memory graph subscriptions. This would enable
> virtual ensembles to interract on the same physical ensemble. For my use
> case, this would prevent unnecessary cross talk between nodes on a WAN,
> allowing me to define the subsets of the memory graph that need to be
> replicated, and to whom. This would be a huge scalability win for WAN
> use cases.
>
> -Todd
>
> -----Original Message-----
> From: Scott Carey [mailto:scott@richrelevance.com]
> Sent: Monday, July 20, 2009 11:00 AM
> To: zookeeper-user@hadoop.apache.org
> Subject: Re: Leader Elections
>
> Observers would be awesome especially with a couple enhancements /
> extensions:
>
> An option for the observers to enter a special state if the WAN link
> goes down to the "master" cluster. A read-only option would be great.
> However, allowing certain types of writes to continue on a limited basis
> would be highly valuable as well. An observer could "own" a special
> node and its subnodes. Only these subnodes would be writable by the
> observer when there was a session break to the master cluster, and the
> master cluster would take all the changes when the link is
> reestablished. Essentially, it is a portion of the hierarchy that is
> writable only by a specitfic observer, and read-only for others.
> The purpose of this would be for when the WAN link goes down to the
> "master" ZKs for certain types of use cases - status updates or other
> changes local to the observer that are strictly read-only outside the
> Observer's 'realm'.
>
>
> On 7/19/09 12:16 PM, "Henry Robinson" <he...@cloudera.com> wrote:
>
> You can. See ZOOKEEPER-368 - at first glance it sounds like observers
> will
> be a good fit for your requirements.
>
> Do bear in mind that the patch on the jira is only for discussion
> purposes;
> I would not consider it currently fit for production use. I hope to put
> up a
> much better patch this week.
>
> Henry
>
> On Sat, Jul 18, 2009 at 7:38 PM, Ted Dunning <te...@gmail.com>
> wrote:
>
> > Can you submit updates via an observer?
> >
> > On Sat, Jul 18, 2009 at 6:38 AM, Flavio Junqueira <fp...@yahoo-inc.com>
> > wrote:
> >
> > > 2- Observers: you could have one computing center containing an
> ensemble
> > > and observers around the edge just learning committed values.
> >
> >
> >
> >
> > --
> > Ted Dunning, CTO
> > DeepDyve
> >
>
>
Re: Leader Elections
Posted by Scott Carey <sc...@richrelevance.com>.
On 7/20/09 2:21 PM, "Scott Carey" <sc...@richrelevance.com> wrote:
>
> Yes, a more general partial graph subscription / ownership framework would
> allow for not just better WAN scalability but also (and more critically IMO)
> higher reliability. Often, some large subset of application functionality is
> local to one network, and a minority is global and in need of WAN
> communication. In this case, when the WAN breaks one wishes that local
> functionality to continue to function, and only those parts truly dependant on
> external events to be interrupted.
> Currently one has to have separate ensembles to partition data and clunky
> 'bridge' code to intercommunicate.
>
> It would certainly be more natural if two ZK ensembles could register with
> each other, in a 'partial sub-graph publish/subscribe' framework. It could
> almost be like file system mounting:
> To subscribe:
> subscribe otherEnsemble:port/path/to/otherstuff /localpath/to/mount/into
>
> Publishing is the same thing -- think of it as a request for a remote ZK
> cluster to subscribe to the local ZK's data.
>
>
Actually, there is one other thing I would add to the above -- subscription
persistence options. The two basic ones are persistence to disk where the
subgraph survives (read only) if the master is absent -- probably with a
special notification to readers that it is in this state, or it can be
ephemeral and drop if the session to the master breaks. There are a variety
of options around behavior when the connection to the master breaks.
>
>
>>
>> -Todd
>>
>> -----Original Message-----
>> From: Scott Carey [mailto:scott@richrelevance.com]
>> Sent: Monday, July 20, 2009 11:00 AM
>> To: zookeeper-user@hadoop.apache.org
>> Subject: Re: Leader Elections
>>
>> Observers would be awesome especially with a couple enhancements /
>> extensions:
>>
>> An option for the observers to enter a special state if the WAN link
>> goes down to the "master" cluster. A read-only option would be great.
>> However, allowing certain types of writes to continue on a limited basis
>> would be highly valuable as well. An observer could "own" a special
>> node and its subnodes. Only these subnodes would be writable by the
>> observer when there was a session break to the master cluster, and the
>> master cluster would take all the changes when the link is
>> reestablished. Essentially, it is a portion of the hierarchy that is
>> writable only by a specitfic observer, and read-only for others.
>> The purpose of this would be for when the WAN link goes down to the
>> "master" ZKs for certain types of use cases - status updates or other
>> changes local to the observer that are strictly read-only outside the
>> Observer's 'realm'.
>>
>>
>> On 7/19/09 12:16 PM, "Henry Robinson" <he...@cloudera.com> wrote:
>>
>> You can. See ZOOKEEPER-368 - at first glance it sounds like observers
>> will
>> be a good fit for your requirements.
>>
>> Do bear in mind that the patch on the jira is only for discussion
>> purposes;
>> I would not consider it currently fit for production use. I hope to put
>> up a
>> much better patch this week.
>>
>> Henry
>>
>> On Sat, Jul 18, 2009 at 7:38 PM, Ted Dunning <te...@gmail.com>
>> wrote:
>>
>>> Can you submit updates via an observer?
>>>
>>> On Sat, Jul 18, 2009 at 6:38 AM, Flavio Junqueira <fp...@yahoo-inc.com>
>>> wrote:
>>>
>>>> 2- Observers: you could have one computing center containing an
>> ensemble
>>>> and observers around the edge just learning committed values.
>>>
>>>
>>>
>>>
>>> --
>>> Ted Dunning, CTO
>>> DeepDyve
>>>
>>
>>
Re: Leader Elections
Posted by Scott Carey <sc...@richrelevance.com>.
Todd has put it much more eloquently. Comments below:
On 7/20/09 11:50 AM, "Todd Greenwood" <to...@audiencescience.com> wrote:
> Flavio, Ted, Henry, Scott, this would perfectly well for my use case
> provided:
>
> SINGLE ENSEMBLE:
> GROUP A : ZK Servers w/ read/write AND Leader Elections
> GROUP B : ZK Servers w/ read/write W/O Leader Elections
>
> So, we can craft this via Observers and Hiererarchial Quorum groups?
> Great. Problem solved.
>
> When will this be production ready? :o)
>
> --------------------
>
> Scott brought up a multi-feature that is very interesting for me.
> Namely:
>
> 1. Offline ZK servers that sync & merge on reconnect
>
> The offline servers seems conceptually simple, it's kind of like a
> messaging system. However, the merge and resolve step when two servers
> reconnect might be challenging. Cool idea though.
Yes, this is very useful for WAN use cases. I've already done something
like it with a hack:
Ensemble A "Master/Central"
"Remote Proxy" N -- embeds its own ZK, and runs two clients. One Client
connects to Ensemble A and watches a partial sub-graph, propagating that
into its local embedded ZK server. This subgraph is read-only for those
that access the Proxy. A second client accesses the local ZK server and
monitors a different subgraph, which it propagates to the Master ensemble.
This is writeable by clients accessing the Proxy and on the Master ensemble
is only written to by this Proxy.
The above is all application enforced. There are constraints on what sort of
things can be built with this, but for the subset of use cases I need over
WAN, its more than enough.
>
> 2. Partial memory graph subscriptions
>
> The second idea is partial memory graph subscriptions. This would enable
> virtual ensembles to interract on the same physical ensemble. For my use
> case, this would prevent unnecessary cross talk between nodes on a WAN,
> allowing me to define the subsets of the memory graph that need to be
> replicated, and to whom. This would be a huge scalability win for WAN
> use cases.
Yes, a more general partial graph subscription / ownership framework would
allow for not just better WAN scalability but also (and more critically IMO)
higher reliability. Often, some large subset of application functionality
is local to one network, and a minority is global and in need of WAN
communication. In this case, when the WAN breaks one wishes that local
functionality to continue to function, and only those parts truly dependant
on external events to be interrupted.
Currently one has to have separate ensembles to partition data and clunky
'bridge' code to intercommunicate.
It would certainly be more natural if two ZK ensembles could register with
each other, in a 'partial sub-graph publish/subscribe' framework. It could
almost be like file system mounting:
To subscribe:
subscribe otherEnsemble:port/path/to/otherstuff /localpath/to/mount/into
Publishing is the same thing -- think of it as a request for a remote ZK
cluster to subscribe to the local ZK's data.
>
> -Todd
>
> -----Original Message-----
> From: Scott Carey [mailto:scott@richrelevance.com]
> Sent: Monday, July 20, 2009 11:00 AM
> To: zookeeper-user@hadoop.apache.org
> Subject: Re: Leader Elections
>
> Observers would be awesome especially with a couple enhancements /
> extensions:
>
> An option for the observers to enter a special state if the WAN link
> goes down to the "master" cluster. A read-only option would be great.
> However, allowing certain types of writes to continue on a limited basis
> would be highly valuable as well. An observer could "own" a special
> node and its subnodes. Only these subnodes would be writable by the
> observer when there was a session break to the master cluster, and the
> master cluster would take all the changes when the link is
> reestablished. Essentially, it is a portion of the hierarchy that is
> writable only by a specitfic observer, and read-only for others.
> The purpose of this would be for when the WAN link goes down to the
> "master" ZKs for certain types of use cases - status updates or other
> changes local to the observer that are strictly read-only outside the
> Observer's 'realm'.
>
>
> On 7/19/09 12:16 PM, "Henry Robinson" <he...@cloudera.com> wrote:
>
> You can. See ZOOKEEPER-368 - at first glance it sounds like observers
> will
> be a good fit for your requirements.
>
> Do bear in mind that the patch on the jira is only for discussion
> purposes;
> I would not consider it currently fit for production use. I hope to put
> up a
> much better patch this week.
>
> Henry
>
> On Sat, Jul 18, 2009 at 7:38 PM, Ted Dunning <te...@gmail.com>
> wrote:
>
>> Can you submit updates via an observer?
>>
>> On Sat, Jul 18, 2009 at 6:38 AM, Flavio Junqueira <fp...@yahoo-inc.com>
>> wrote:
>>
>>> 2- Observers: you could have one computing center containing an
> ensemble
>>> and observers around the edge just learning committed values.
>>
>>
>>
>>
>> --
>> Ted Dunning, CTO
>> DeepDyve
>>
>
>
Re: Leader Elections
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
Both of the options that Scott mentioned are quite interesting. Quite a few
of our users are interested in these two features. I think for 2, we should
be able to use observers with a subscription to the master cluster with
interested in a special subtree. That avoids too much of cross talk.
Henry/Flavio, do you guys want to keep this in mind for Observers in (not
implement it in the jira but to generalize it in a way that partial
subscription can be done later)
http://issues.apache.org/jira/browse/ZOOKEEPER-368
Wherein an observer can just register with interest in a subtree and the
master cluster can avoid sending updates to zookeeper tree not in the
sbutree. This would be very helpful in a WAN setting where in only small
data needs to be up to date within different data centers.
thanks
mahadev
On 7/20/09 11:50 AM, "Todd Greenwood" <to...@audiencescience.com> wrote:
> Flavio, Ted, Henry, Scott, this would perfectly well for my use case
> provided:
>
> SINGLE ENSEMBLE:
> GROUP A : ZK Servers w/ read/write AND Leader Elections
> GROUP B : ZK Servers w/ read/write W/O Leader Elections
>
> So, we can craft this via Observers and Hiererarchial Quorum groups?
> Great. Problem solved.
>
> When will this be production ready? :o)
>
> --------------------
>
> Scott brought up a multi-feature that is very interesting for me.
> Namely:
>
> 1. Offline ZK servers that sync & merge on reconnect
>
> The offline servers seems conceptually simple, it's kind of like a
> messaging system. However, the merge and resolve step when two servers
> reconnect might be challenging. Cool idea though.
>
> 2. Partial memory graph subscriptions
>
> The second idea is partial memory graph subscriptions. This would enable
> virtual ensembles to interract on the same physical ensemble. For my use
> case, this would prevent unnecessary cross talk between nodes on a WAN,
> allowing me to define the subsets of the memory graph that need to be
> replicated, and to whom. This would be a huge scalability win for WAN
> use cases.
>
> -Todd
>
> -----Original Message-----
> From: Scott Carey [mailto:scott@richrelevance.com]
> Sent: Monday, July 20, 2009 11:00 AM
> To: zookeeper-user@hadoop.apache.org
> Subject: Re: Leader Elections
>
> Observers would be awesome especially with a couple enhancements /
> extensions:
>
> An option for the observers to enter a special state if the WAN link
> goes down to the "master" cluster. A read-only option would be great.
> However, allowing certain types of writes to continue on a limited basis
> would be highly valuable as well. An observer could "own" a special
> node and its subnodes. Only these subnodes would be writable by the
> observer when there was a session break to the master cluster, and the
> master cluster would take all the changes when the link is
> reestablished. Essentially, it is a portion of the hierarchy that is
> writable only by a specitfic observer, and read-only for others.
> The purpose of this would be for when the WAN link goes down to the
> "master" ZKs for certain types of use cases - status updates or other
> changes local to the observer that are strictly read-only outside the
> Observer's 'realm'.
>
>
> On 7/19/09 12:16 PM, "Henry Robinson" <he...@cloudera.com> wrote:
>
> You can. See ZOOKEEPER-368 - at first glance it sounds like observers
> will
> be a good fit for your requirements.
>
> Do bear in mind that the patch on the jira is only for discussion
> purposes;
> I would not consider it currently fit for production use. I hope to put
> up a
> much better patch this week.
>
> Henry
>
> On Sat, Jul 18, 2009 at 7:38 PM, Ted Dunning <te...@gmail.com>
> wrote:
>
>> Can you submit updates via an observer?
>>
>> On Sat, Jul 18, 2009 at 6:38 AM, Flavio Junqueira <fp...@yahoo-inc.com>
>> wrote:
>>
>>> 2- Observers: you could have one computing center containing an
> ensemble
>>> and observers around the edge just learning committed values.
>>
>>
>>
>>
>> --
>> Ted Dunning, CTO
>> DeepDyve
>>
>
RE: Leader Elections
Posted by Todd Greenwood <to...@audiencescience.com>.
Flavio, Ted, Henry, Scott, this would perfectly well for my use case
provided:
SINGLE ENSEMBLE:
GROUP A : ZK Servers w/ read/write AND Leader Elections
GROUP B : ZK Servers w/ read/write W/O Leader Elections
So, we can craft this via Observers and Hiererarchial Quorum groups?
Great. Problem solved.
When will this be production ready? :o)
--------------------
Scott brought up a multi-feature that is very interesting for me.
Namely:
1. Offline ZK servers that sync & merge on reconnect
The offline servers seems conceptually simple, it's kind of like a
messaging system. However, the merge and resolve step when two servers
reconnect might be challenging. Cool idea though.
2. Partial memory graph subscriptions
The second idea is partial memory graph subscriptions. This would enable
virtual ensembles to interract on the same physical ensemble. For my use
case, this would prevent unnecessary cross talk between nodes on a WAN,
allowing me to define the subsets of the memory graph that need to be
replicated, and to whom. This would be a huge scalability win for WAN
use cases.
-Todd
-----Original Message-----
From: Scott Carey [mailto:scott@richrelevance.com]
Sent: Monday, July 20, 2009 11:00 AM
To: zookeeper-user@hadoop.apache.org
Subject: Re: Leader Elections
Observers would be awesome especially with a couple enhancements /
extensions:
An option for the observers to enter a special state if the WAN link
goes down to the "master" cluster. A read-only option would be great.
However, allowing certain types of writes to continue on a limited basis
would be highly valuable as well. An observer could "own" a special
node and its subnodes. Only these subnodes would be writable by the
observer when there was a session break to the master cluster, and the
master cluster would take all the changes when the link is
reestablished. Essentially, it is a portion of the hierarchy that is
writable only by a specitfic observer, and read-only for others.
The purpose of this would be for when the WAN link goes down to the
"master" ZKs for certain types of use cases - status updates or other
changes local to the observer that are strictly read-only outside the
Observer's 'realm'.
On 7/19/09 12:16 PM, "Henry Robinson" <he...@cloudera.com> wrote:
You can. See ZOOKEEPER-368 - at first glance it sounds like observers
will
be a good fit for your requirements.
Do bear in mind that the patch on the jira is only for discussion
purposes;
I would not consider it currently fit for production use. I hope to put
up a
much better patch this week.
Henry
On Sat, Jul 18, 2009 at 7:38 PM, Ted Dunning <te...@gmail.com>
wrote:
> Can you submit updates via an observer?
>
> On Sat, Jul 18, 2009 at 6:38 AM, Flavio Junqueira <fp...@yahoo-inc.com>
> wrote:
>
> > 2- Observers: you could have one computing center containing an
ensemble
> > and observers around the edge just learning committed values.
>
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>
Re: Leader Elections
Posted by Scott Carey <sc...@richrelevance.com>.
Observers would be awesome especially with a couple enhancements / extensions:
An option for the observers to enter a special state if the WAN link goes down to the "master" cluster. A read-only option would be great.
However, allowing certain types of writes to continue on a limited basis would be highly valuable as well. An observer could "own" a special node and its subnodes. Only these subnodes would be writable by the observer when there was a session break to the master cluster, and the master cluster would take all the changes when the link is reestablished. Essentially, it is a portion of the hierarchy that is writable only by a specitfic observer, and read-only for others.
The purpose of this would be for when the WAN link goes down to the "master" ZKs for certain types of use cases - status updates or other changes local to the observer that are strictly read-only outside the Observer's 'realm'.
On 7/19/09 12:16 PM, "Henry Robinson" <he...@cloudera.com> wrote:
You can. See ZOOKEEPER-368 - at first glance it sounds like observers will
be a good fit for your requirements.
Do bear in mind that the patch on the jira is only for discussion purposes;
I would not consider it currently fit for production use. I hope to put up a
much better patch this week.
Henry
On Sat, Jul 18, 2009 at 7:38 PM, Ted Dunning <te...@gmail.com> wrote:
> Can you submit updates via an observer?
>
> On Sat, Jul 18, 2009 at 6:38 AM, Flavio Junqueira <fp...@yahoo-inc.com>
> wrote:
>
> > 2- Observers: you could have one computing center containing an ensemble
> > and observers around the edge just learning committed values.
>
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>
Re: Leader Elections
Posted by Henry Robinson <he...@cloudera.com>.
You can. See ZOOKEEPER-368 - at first glance it sounds like observers will
be a good fit for your requirements.
Do bear in mind that the patch on the jira is only for discussion purposes;
I would not consider it currently fit for production use. I hope to put up a
much better patch this week.
Henry
On Sat, Jul 18, 2009 at 7:38 PM, Ted Dunning <te...@gmail.com> wrote:
> Can you submit updates via an observer?
>
> On Sat, Jul 18, 2009 at 6:38 AM, Flavio Junqueira <fp...@yahoo-inc.com>
> wrote:
>
> > 2- Observers: you could have one computing center containing an ensemble
> > and observers around the edge just learning committed values.
>
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>
Re: Leader Elections
Posted by Ted Dunning <te...@gmail.com>.
Can you submit updates via an observer?
On Sat, Jul 18, 2009 at 6:38 AM, Flavio Junqueira <fp...@yahoo-inc.com> wrote:
> 2- Observers: you could have one computing center containing an ensemble
> and observers around the edge just learning committed values.
--
Ted Dunning, CTO
DeepDyve
Re: Leader Elections
Posted by Flavio Junqueira <fp...@yahoo-inc.com>.
Todd, I don't think a feature right now that allows you to do exactly
what you're requesting. However, we have been working on a couple of
features that might give you what you want:
1- Hierarchical quorums: this feature allows you to split servers into
groups (perhaps mapping groups to computing centers), and to assign
weights to servers. I believe we could constrain leader servers to be
the ones with weight greater than zero;
2- Observers: you could have one computing center containing an
ensemble and observers around the edge just learning committed values.
Does any of these help in your case?
-Flavio
On Jul 18, 2009, at 12:16 AM, Todd Greenwood wrote:
> Zookeeper-Users,
>
> I'm configuring a zookeeper ensemble that will have servers across a
> WAN. However, I'd like to constrain the leader elections to elected
> leaders only from a pool of zookeeper servers in a centralized
> computing
> center. In this scenario, zookeeper servers at the edge of the WAN can
> be members of the ensemble, but not leaders. I anticipate that this
> will
> perform significantly better than forcing traffic from one edge of the
> WAN to another, for the case of a leader elected to a WAN edge node.
>
> ------------------------------------------------------
> WAN
> [POD (1..N)]* <--> [ DC ]**
>
> * contains multiple zk servers, but no leaders
> ** contains multiple zk servers, each may be a leader
> ------------------------------------------------------
>
> Are there a configuration parameters that will accomplish this, or
> do I
> need to patch ZK?
>
> I'm a bit confused by the 3.2.0 admin documentation in this respect:
>
> http://hadoop.apache.org/zookeeper/docs/r3.2.0/zookeeperAdmin.html#sc_mo
> nitoring
>
> """
> leaderServes
>
> (Java system property: zookeeper.leaderServes)
>
> Leader accepts client connections. Default value is "yes". The
> leader machine coordinates updates. For higher update throughput at
> thes
> slight expense of read throughput the leader can be configured to not
> accept clients and focus on coordination. The default to this option
> is
> yes, which means that a leader will accept client connections.
> Note
>
> Turning on leader selection is highly recommended when you have
> more
> than three ZooKeeper servers in an ensemble.
>
> """
>
> Q: The leaderServers property seems to be an optimization that favors
> updates at the expense of reads. Fine. But the note below that doesn't
> make sense to me... What does it mean to turn on leader selection?
> Does
> this mean to configure leaderServes="yes"? Is serving the same as
> selecting? Perhaps there is a typo here...
>
> Q: The next items on this web page mention configuring groups and
> assigning a weight to the votes of machines in those groups. If I am
> defining an ensemble with groups of machines that I do not want
> elected
> as leaders, would I assign them to a group and set their voting weight
> to zero? Is that the expected practice?
>
> """
> group.x=nnnnn[:nnnnn]
>
> (No Java system property)
>
> Enables a hierarchical quorum construction."x" is a group
> identifier
> and the numbers following the "=" sign correspond to server
> identifiers.
> The left-hand side of the assignment is a colon-separated list of
> server
> identifiers. Note that groups must be disjoint and the union of all
> groups must be the ZooKeeper ensemble.
> weight.x=nnnnn
>
> (No Java system property)
>
> Used along with "group", it assigns a weight to a server when
> forming quorums. Such a value corresponds to the weight of a server
> when
> voting. There are a few parts of ZooKeeper that require voting such as
> leader election and the atomic broadcast protocol. By default the
> weight
> of server is 1. If the configuration defines groups, but not weights,
> then a value of 1 will be assigned to all servers.
> """
>
> -Todd
Leader Elections
Posted by Todd Greenwood <to...@audiencescience.com>.
Zookeeper-Users,
I'm configuring a zookeeper ensemble that will have servers across a
WAN. However, I'd like to constrain the leader elections to elected
leaders only from a pool of zookeeper servers in a centralized computing
center. In this scenario, zookeeper servers at the edge of the WAN can
be members of the ensemble, but not leaders. I anticipate that this will
perform significantly better than forcing traffic from one edge of the
WAN to another, for the case of a leader elected to a WAN edge node.
------------------------------------------------------
WAN
[POD (1..N)]* <--> [ DC ]**
* contains multiple zk servers, but no leaders
** contains multiple zk servers, each may be a leader
------------------------------------------------------
Are there a configuration parameters that will accomplish this, or do I
need to patch ZK?
I'm a bit confused by the 3.2.0 admin documentation in this respect:
http://hadoop.apache.org/zookeeper/docs/r3.2.0/zookeeperAdmin.html#sc_mo
nitoring
"""
leaderServes
(Java system property: zookeeper.leaderServes)
Leader accepts client connections. Default value is "yes". The
leader machine coordinates updates. For higher update throughput at thes
slight expense of read throughput the leader can be configured to not
accept clients and focus on coordination. The default to this option is
yes, which means that a leader will accept client connections.
Note
Turning on leader selection is highly recommended when you have more
than three ZooKeeper servers in an ensemble.
"""
Q: The leaderServers property seems to be an optimization that favors
updates at the expense of reads. Fine. But the note below that doesn't
make sense to me... What does it mean to turn on leader selection? Does
this mean to configure leaderServes="yes"? Is serving the same as
selecting? Perhaps there is a typo here...
Q: The next items on this web page mention configuring groups and
assigning a weight to the votes of machines in those groups. If I am
defining an ensemble with groups of machines that I do not want elected
as leaders, would I assign them to a group and set their voting weight
to zero? Is that the expected practice?
"""
group.x=nnnnn[:nnnnn]
(No Java system property)
Enables a hierarchical quorum construction."x" is a group identifier
and the numbers following the "=" sign correspond to server identifiers.
The left-hand side of the assignment is a colon-separated list of server
identifiers. Note that groups must be disjoint and the union of all
groups must be the ZooKeeper ensemble.
weight.x=nnnnn
(No Java system property)
Used along with "group", it assigns a weight to a server when
forming quorums. Such a value corresponds to the weight of a server when
voting. There are a few parts of ZooKeeper that require voting such as
leader election and the atomic broadcast protocol. By default the weight
of server is 1. If the configuration defines groups, but not weights,
then a value of 1 will be assigned to all servers.
"""
-Todd
Re: Multiple ZK clusters or a single, shared cluster?
Posted by Jonathan Gray <jg...@apache.org>.
Yup.
I would say that the use of ZK by HBase today is very minimal. Very few
writes at all, almost exclusively reads and still not that often.
Dedicated resources would not make much of a difference.
For users who are just testing, developing, or running clusters that are
not highly loaded, it is not as important to have dedicated nodes _yet_.
The next version of HBase, 0.21, we are planning on widely expanding
what we use ZK for. At that time, we will be writing much more often,
and reading constantly. Dedicated nodes/disks would be highly
recommended at that point.
Thanks for the explanation.
JG
Benjamin Reed wrote:
> you need a dedicated disk for the logDir, but not the dataDir. the
> reason is that the write to the log is in the critical path: we cannot
> commit changes until they have been synced to disk, so we want to make
> sure that we don't contend for the disk. the snapshots in the dataDir
> are done in an asynchronous thread, so it can be written to a disk used
> by other applications (usually the OS disk).
>
> the problem with running other processes on the zookeeper server is
> similar to the disk contention: if zookeeper starts contending with a
> runaway app for CPU and memory, we can start timing out because of
> starvation.
>
> by using a dedicated disk for logs and a dedicated machine you can get
> very high deterministic performance.
>
> make sense?
>
> ben
>
> Jonathan Gray wrote:
>> Thanks for the input.
>>
>> Honestly, I'm thinking I need to have separate clusters. The version
>> of ZK is one thing; but also for an application like HBase, we have
>> had periods where we needed to patch ZK before it became part of a
>> release. Keeping track of that on a shared cluster will be tricky,
>> if not impossible.
>>
>> And with a small development team and a very fast dev cycle, I'm a
>> little concerned about a runaway application hosing all the other
>> dependencies on ZK...
>>
>> What are the actual reasons for wanting a separate disk for ZK?
>> Strictly reliability purposes? Should that disk be dedicated to the
>> logDir but not the dataDir, or both?
>>
>> If I don't give it a dedicated disk or node, but it has 1GB of memory
>> and a core, what are the downsides? Are they just about reliability?
>> If I could run 5 or 7 zk nodes, but co-hosted with my HBase cluster,
>> is that really less reliable than 3 separate nodes, as long as the jvm
>> has sufficient resources? Or are there performance or usability
>> concerns as well?
>>
>> Sorry for all the questions, just trying to get the story straight so
>> that we don't spread misinformation to HBase users. Most users start
>> out on very small clusters, so dedicated ZK nodes are not a realistic
>> assumption... How big of a deal is that?
>>
>> JG
>>
>> Benjamin Reed wrote:
>>
>>> we designed zk to have high performance so that it can be shared by
>>> multiple applications. the main thing is that you use dedicated zk
>>> machines (with a dedicated disk for logging). once you have that in
>>> place, watch the load on your cluster, as long as you aren't
>>> saturating the cluster you should share.
>>>
>>> as you point out running multiple clusters is a hardware investment,
>>> plus you miss out on opportunities to improve reliability. for
>>> example, if you have three applications that have a cluster of 3 zk
>>> servers each, one failure will result in an outage. if instead of
>>> using the 9 servers you have the same three applications use a zk
>>> cluster with 7 servers you can tolerate three failures without an
>>> outage.
>>>
>>> the key of course is to make sure that you don't oversubscribe the
>>> server.
>>>
>>> ben
>>>
>>> Jonathan Gray wrote:
>>>
>>>> Hey guys,
>>>>
>>>> Been using ZK indirectly for a few months now in the HBase and Katta
>>>> realms. Both of these applications make it really easy so you don't
>>>> have to be involved much with managing your ZK cluster to support it.
>>>>
>>>> I'm now using ZK for a bunch of things internally, so now I'm
>>>> manually configuring, starting, and managing a cluster.
>>>>
>>>> What advice is there about whether I should be sharing a single
>>>> cluster between all my applications, or running separate ones for
>>>> each use?
>>>>
>>>> I've been told that it's strongly recommended to run your ZK nodes
>>>> separately from the application using them (this is actually what
>>>> we're telling new users over in HBase, though a majority of
>>>> installations will likely co-host them with DataNodes and
>>>> RegionServers).
>>>>
>>>> I don't have the resources to maintain a separate 3+ node ZK cluster
>>>> for each of my applications, so this is not really an option. I'm
>>>> trying to decide if I should have HBase running/managing it's own ZK
>>>> cluster that is co-located with some of the regionservers (there
>>>> will be ample memory, but ZK will not have a dedicated disk), or if
>>>> I should be pointing it to a dedicated 3 node ZK cluster.
>>>>
>>>> I would then also have Katta pointing at this same shared cluster
>>>> (or a separate cluster would be co-located with katta nodes). Same
>>>> for my application; could share nodes with the app servers or
>>>> pointed at a single ZK cluster.
>>>>
>>>> Trade-offs I should be aware of? Current best practices?
>>>>
>>>> Any help would be much appreciated. Thanks.
>>>>
>>>> Jonathan Gray
>>>>
>
Re: Multiple ZK clusters or a single, shared cluster?
Posted by Scott Carey <sc...@richrelevance.com>.
You don't need a dedicated disk for performance purposes if:
You use an SSD (which the decent ones have sub 0.1ms write latency)
Or
You have a battery backed cache RAID card with write-back cache enabled so that synchronous writes return nearly instantly.
Its just the synchronous write latency for write performance. If Linux with ext3 and the default mount options, this can be really bad if shared with other uses. Additionally a separate disk won't have to seek as much if only one app is writing to it.
Given the price of a raid card, and the fact that the logs don't need that much space, I'd go for the ~$370 Intel 32GB X-25E SSD if you are performance sensitive.
On 7/17/09 2:26 PM, "Benjamin Reed" <br...@yahoo-inc.com> wrote:
you need a dedicated disk for the logDir, but not the dataDir. the
reason is that the write to the log is in the critical path: we cannot
commit changes until they have been synced to disk, so we want to make
sure that we don't contend for the disk. the snapshots in the dataDir
are done in an asynchronous thread, so it can be written to a disk used
by other applications (usually the OS disk).
the problem with running other processes on the zookeeper server is
similar to the disk contention: if zookeeper starts contending with a
runaway app for CPU and memory, we can start timing out because of
starvation.
by using a dedicated disk for logs and a dedicated machine you can get
very high deterministic performance.
make sense?
ben
Jonathan Gray wrote:
> Thanks for the input.
>
> Honestly, I'm thinking I need to have separate clusters. The version of
> ZK is one thing; but also for an application like HBase, we have had
> periods where we needed to patch ZK before it became part of a release.
> Keeping track of that on a shared cluster will be tricky, if not
> impossible.
>
> And with a small development team and a very fast dev cycle, I'm a
> little concerned about a runaway application hosing all the other
> dependencies on ZK...
>
> What are the actual reasons for wanting a separate disk for ZK?
> Strictly reliability purposes? Should that disk be dedicated to the
> logDir but not the dataDir, or both?
>
> If I don't give it a dedicated disk or node, but it has 1GB of memory
> and a core, what are the downsides? Are they just about reliability?
> If I could run 5 or 7 zk nodes, but co-hosted with my HBase cluster, is
> that really less reliable than 3 separate nodes, as long as the jvm has
> sufficient resources? Or are there performance or usability concerns as
> well?
>
> Sorry for all the questions, just trying to get the story straight so
> that we don't spread misinformation to HBase users. Most users start
> out on very small clusters, so dedicated ZK nodes are not a realistic
> assumption... How big of a deal is that?
>
> JG
>
> Benjamin Reed wrote:
>
>> we designed zk to have high performance so that it can be shared by
>> multiple applications. the main thing is that you use dedicated zk
>> machines (with a dedicated disk for logging). once you have that in
>> place, watch the load on your cluster, as long as you aren't saturating
>> the cluster you should share.
>>
>> as you point out running multiple clusters is a hardware investment,
>> plus you miss out on opportunities to improve reliability. for example,
>> if you have three applications that have a cluster of 3 zk servers each,
>> one failure will result in an outage. if instead of using the 9 servers
>> you have the same three applications use a zk cluster with 7 servers you
>> can tolerate three failures without an outage.
>>
>> the key of course is to make sure that you don't oversubscribe the server.
>>
>> ben
>>
>> Jonathan Gray wrote:
>>
>>> Hey guys,
>>>
>>> Been using ZK indirectly for a few months now in the HBase and Katta
>>> realms. Both of these applications make it really easy so you don't
>>> have to be involved much with managing your ZK cluster to support it.
>>>
>>> I'm now using ZK for a bunch of things internally, so now I'm manually
>>> configuring, starting, and managing a cluster.
>>>
>>> What advice is there about whether I should be sharing a single
>>> cluster between all my applications, or running separate ones for each
>>> use?
>>>
>>> I've been told that it's strongly recommended to run your ZK nodes
>>> separately from the application using them (this is actually what
>>> we're telling new users over in HBase, though a majority of
>>> installations will likely co-host them with DataNodes and RegionServers).
>>>
>>> I don't have the resources to maintain a separate 3+ node ZK cluster
>>> for each of my applications, so this is not really an option. I'm
>>> trying to decide if I should have HBase running/managing it's own ZK
>>> cluster that is co-located with some of the regionservers (there will
>>> be ample memory, but ZK will not have a dedicated disk), or if I
>>> should be pointing it to a dedicated 3 node ZK cluster.
>>>
>>> I would then also have Katta pointing at this same shared cluster (or
>>> a separate cluster would be co-located with katta nodes). Same for my
>>> application; could share nodes with the app servers or pointed at a
>>> single ZK cluster.
>>>
>>> Trade-offs I should be aware of? Current best practices?
>>>
>>> Any help would be much appreciated. Thanks.
>>>
>>> Jonathan Gray
>>>
>>>
Re: Multiple ZK clusters or a single, shared cluster?
Posted by Benjamin Reed <br...@yahoo-inc.com>.
you need a dedicated disk for the logDir, but not the dataDir. the
reason is that the write to the log is in the critical path: we cannot
commit changes until they have been synced to disk, so we want to make
sure that we don't contend for the disk. the snapshots in the dataDir
are done in an asynchronous thread, so it can be written to a disk used
by other applications (usually the OS disk).
the problem with running other processes on the zookeeper server is
similar to the disk contention: if zookeeper starts contending with a
runaway app for CPU and memory, we can start timing out because of
starvation.
by using a dedicated disk for logs and a dedicated machine you can get
very high deterministic performance.
make sense?
ben
Jonathan Gray wrote:
> Thanks for the input.
>
> Honestly, I'm thinking I need to have separate clusters. The version of
> ZK is one thing; but also for an application like HBase, we have had
> periods where we needed to patch ZK before it became part of a release.
> Keeping track of that on a shared cluster will be tricky, if not
> impossible.
>
> And with a small development team and a very fast dev cycle, I'm a
> little concerned about a runaway application hosing all the other
> dependencies on ZK...
>
> What are the actual reasons for wanting a separate disk for ZK?
> Strictly reliability purposes? Should that disk be dedicated to the
> logDir but not the dataDir, or both?
>
> If I don't give it a dedicated disk or node, but it has 1GB of memory
> and a core, what are the downsides? Are they just about reliability?
> If I could run 5 or 7 zk nodes, but co-hosted with my HBase cluster, is
> that really less reliable than 3 separate nodes, as long as the jvm has
> sufficient resources? Or are there performance or usability concerns as
> well?
>
> Sorry for all the questions, just trying to get the story straight so
> that we don't spread misinformation to HBase users. Most users start
> out on very small clusters, so dedicated ZK nodes are not a realistic
> assumption... How big of a deal is that?
>
> JG
>
> Benjamin Reed wrote:
>
>> we designed zk to have high performance so that it can be shared by
>> multiple applications. the main thing is that you use dedicated zk
>> machines (with a dedicated disk for logging). once you have that in
>> place, watch the load on your cluster, as long as you aren't saturating
>> the cluster you should share.
>>
>> as you point out running multiple clusters is a hardware investment,
>> plus you miss out on opportunities to improve reliability. for example,
>> if you have three applications that have a cluster of 3 zk servers each,
>> one failure will result in an outage. if instead of using the 9 servers
>> you have the same three applications use a zk cluster with 7 servers you
>> can tolerate three failures without an outage.
>>
>> the key of course is to make sure that you don't oversubscribe the server.
>>
>> ben
>>
>> Jonathan Gray wrote:
>>
>>> Hey guys,
>>>
>>> Been using ZK indirectly for a few months now in the HBase and Katta
>>> realms. Both of these applications make it really easy so you don't
>>> have to be involved much with managing your ZK cluster to support it.
>>>
>>> I'm now using ZK for a bunch of things internally, so now I'm manually
>>> configuring, starting, and managing a cluster.
>>>
>>> What advice is there about whether I should be sharing a single
>>> cluster between all my applications, or running separate ones for each
>>> use?
>>>
>>> I've been told that it's strongly recommended to run your ZK nodes
>>> separately from the application using them (this is actually what
>>> we're telling new users over in HBase, though a majority of
>>> installations will likely co-host them with DataNodes and RegionServers).
>>>
>>> I don't have the resources to maintain a separate 3+ node ZK cluster
>>> for each of my applications, so this is not really an option. I'm
>>> trying to decide if I should have HBase running/managing it's own ZK
>>> cluster that is co-located with some of the regionservers (there will
>>> be ample memory, but ZK will not have a dedicated disk), or if I
>>> should be pointing it to a dedicated 3 node ZK cluster.
>>>
>>> I would then also have Katta pointing at this same shared cluster (or
>>> a separate cluster would be co-located with katta nodes). Same for my
>>> application; could share nodes with the app servers or pointed at a
>>> single ZK cluster.
>>>
>>> Trade-offs I should be aware of? Current best practices?
>>>
>>> Any help would be much appreciated. Thanks.
>>>
>>> Jonathan Gray
>>>
>>>
Re: Multiple ZK clusters or a single, shared cluster?
Posted by Jonathan Gray <jg...@apache.org>.
Thanks for the input.
Honestly, I'm thinking I need to have separate clusters. The version of
ZK is one thing; but also for an application like HBase, we have had
periods where we needed to patch ZK before it became part of a release.
Keeping track of that on a shared cluster will be tricky, if not
impossible.
And with a small development team and a very fast dev cycle, I'm a
little concerned about a runaway application hosing all the other
dependencies on ZK...
What are the actual reasons for wanting a separate disk for ZK?
Strictly reliability purposes? Should that disk be dedicated to the
logDir but not the dataDir, or both?
If I don't give it a dedicated disk or node, but it has 1GB of memory
and a core, what are the downsides? Are they just about reliability?
If I could run 5 or 7 zk nodes, but co-hosted with my HBase cluster, is
that really less reliable than 3 separate nodes, as long as the jvm has
sufficient resources? Or are there performance or usability concerns as
well?
Sorry for all the questions, just trying to get the story straight so
that we don't spread misinformation to HBase users. Most users start
out on very small clusters, so dedicated ZK nodes are not a realistic
assumption... How big of a deal is that?
JG
Benjamin Reed wrote:
> we designed zk to have high performance so that it can be shared by
> multiple applications. the main thing is that you use dedicated zk
> machines (with a dedicated disk for logging). once you have that in
> place, watch the load on your cluster, as long as you aren't saturating
> the cluster you should share.
>
> as you point out running multiple clusters is a hardware investment,
> plus you miss out on opportunities to improve reliability. for example,
> if you have three applications that have a cluster of 3 zk servers each,
> one failure will result in an outage. if instead of using the 9 servers
> you have the same three applications use a zk cluster with 7 servers you
> can tolerate three failures without an outage.
>
> the key of course is to make sure that you don't oversubscribe the server.
>
> ben
>
> Jonathan Gray wrote:
>> Hey guys,
>>
>> Been using ZK indirectly for a few months now in the HBase and Katta
>> realms. Both of these applications make it really easy so you don't
>> have to be involved much with managing your ZK cluster to support it.
>>
>> I'm now using ZK for a bunch of things internally, so now I'm manually
>> configuring, starting, and managing a cluster.
>>
>> What advice is there about whether I should be sharing a single
>> cluster between all my applications, or running separate ones for each
>> use?
>>
>> I've been told that it's strongly recommended to run your ZK nodes
>> separately from the application using them (this is actually what
>> we're telling new users over in HBase, though a majority of
>> installations will likely co-host them with DataNodes and RegionServers).
>>
>> I don't have the resources to maintain a separate 3+ node ZK cluster
>> for each of my applications, so this is not really an option. I'm
>> trying to decide if I should have HBase running/managing it's own ZK
>> cluster that is co-located with some of the regionservers (there will
>> be ample memory, but ZK will not have a dedicated disk), or if I
>> should be pointing it to a dedicated 3 node ZK cluster.
>>
>> I would then also have Katta pointing at this same shared cluster (or
>> a separate cluster would be co-located with katta nodes). Same for my
>> application; could share nodes with the app servers or pointed at a
>> single ZK cluster.
>>
>> Trade-offs I should be aware of? Current best practices?
>>
>> Any help would be much appreciated. Thanks.
>>
>> Jonathan Gray
>>
>
Re: Multiple ZK clusters or a single, shared cluster?
Posted by Ted Dunning <te...@gmail.com>.
This may be true for monster clusters being supported by a ZK cluster or for
data-intensive operations like high throughput queuing, but many
applications of ZK are incredibly low volume.
For instance, with Katta, there is a read-modify-write when a search engine
comes up, when a shard is assigned or loaded, a dozen or so when an index is
added and a few when the master fails. None of these operations occurs in
practice even once an hour.
I also log machine state across lots of machines once every few seconds
(partly to give ZK something to do). That results in dozens (at most) of
transactions per second.
I think that a huge proportion of Zk clusters are going to have transactions
per day rather than kilo transactions per second. At these rates, all you
have to worry about is simultaneous failures ... make sure that the ZK
cluster members are on separate power supplies and network switches. Beyond
that, everything is over-kill. You could run on a cluster of iPhones and
still be happy with performance.
On Fri, Jul 17, 2009 at 1:38 PM, Benjamin Reed <br...@yahoo-inc.com> wrote:
> the main thing is that you use dedicated zk machines (with a dedicated disk
> for logging). once you have that in place, watch the load on your cluster,
> as long as you aren't saturating the cluster you should share.
>
--
Ted Dunning, CTO
DeepDyve
Re: Multiple ZK clusters or a single, shared cluster?
Posted by Benjamin Reed <br...@yahoo-inc.com>.
we designed zk to have high performance so that it can be shared by
multiple applications. the main thing is that you use dedicated zk
machines (with a dedicated disk for logging). once you have that in
place, watch the load on your cluster, as long as you aren't saturating
the cluster you should share.
as you point out running multiple clusters is a hardware investment,
plus you miss out on opportunities to improve reliability. for example,
if you have three applications that have a cluster of 3 zk servers each,
one failure will result in an outage. if instead of using the 9 servers
you have the same three applications use a zk cluster with 7 servers you
can tolerate three failures without an outage.
the key of course is to make sure that you don't oversubscribe the server.
ben
Jonathan Gray wrote:
> Hey guys,
>
> Been using ZK indirectly for a few months now in the HBase and Katta
> realms. Both of these applications make it really easy so you don't
> have to be involved much with managing your ZK cluster to support it.
>
> I'm now using ZK for a bunch of things internally, so now I'm manually
> configuring, starting, and managing a cluster.
>
> What advice is there about whether I should be sharing a single cluster
> between all my applications, or running separate ones for each use?
>
> I've been told that it's strongly recommended to run your ZK nodes
> separately from the application using them (this is actually what we're
> telling new users over in HBase, though a majority of installations will
> likely co-host them with DataNodes and RegionServers).
>
> I don't have the resources to maintain a separate 3+ node ZK cluster for
> each of my applications, so this is not really an option. I'm trying to
> decide if I should have HBase running/managing it's own ZK cluster that
> is co-located with some of the regionservers (there will be ample
> memory, but ZK will not have a dedicated disk), or if I should be
> pointing it to a dedicated 3 node ZK cluster.
>
> I would then also have Katta pointing at this same shared cluster (or a
> separate cluster would be co-located with katta nodes). Same for my
> application; could share nodes with the app servers or pointed at a
> single ZK cluster.
>
> Trade-offs I should be aware of? Current best practices?
>
> Any help would be much appreciated. Thanks.
>
> Jonathan Gray
>
Re: Multiple ZK clusters or a single, shared cluster?
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
Hi Jonathan,
Regarding sharing ZooKeeper servers among applications, we actually
recommend doing that. We ask our users to have dedicated 3-5 node clusters
with dedicated machines(/disk) for zookeeper and lets applications share
those 3-5 nodes clusters.
But we do have have users (inside Yahoo!) wherein they want to run these
clusters for themselves. The reasons being:
- there usage of zookeeper (in terms of reads/writes per second and number
of clients - 5K or so) is pretty high and they do not want to share the
dedicated cluster with other properties inside Yahoo!
- they also want to manage upgrades/administration of cluster on there own
There might be more info on administering your own cluster at
http://hadoop.apache.org/zookeeper/docs/r3.2.0/zookeeperAdmin.html
Thanks
mahadev
On 7/17/09 12:48 PM, "Jonathan Gray" <jg...@apache.org> wrote:
> Hey guys,
>
> Been using ZK indirectly for a few months now in the HBase and Katta
> realms. Both of these applications make it really easy so you don't
> have to be involved much with managing your ZK cluster to support it.
>
> I'm now using ZK for a bunch of things internally, so now I'm manually
> configuring, starting, and managing a cluster.
>
> What advice is there about whether I should be sharing a single cluster
> between all my applications, or running separate ones for each use?
>
> I've been told that it's strongly recommended to run your ZK nodes
> separately from the application using them (this is actually what we're
> telling new users over in HBase, though a majority of installations will
> likely co-host them with DataNodes and RegionServers).
>
> I don't have the resources to maintain a separate 3+ node ZK cluster for
> each of my applications, so this is not really an option. I'm trying to
> decide if I should have HBase running/managing it's own ZK cluster that
> is co-located with some of the regionservers (there will be ample
> memory, but ZK will not have a dedicated disk), or if I should be
> pointing it to a dedicated 3 node ZK cluster.
>
> I would then also have Katta pointing at this same shared cluster (or a
> separate cluster would be co-located with katta nodes). Same for my
> application; could share nodes with the app servers or pointed at a
> single ZK cluster.
>
> Trade-offs I should be aware of? Current best practices?
>
> Any help would be much appreciated. Thanks.
>
> Jonathan Gray