You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Zsolt Beothy-Elo <zb...@talend.com> on 2011/03/02 15:57:42 UTC
Question about Clustered Setup in two data centers
Hi all,
in our product we currently implement to use ZooKeeper in conjunction with CXF to dynamically manage available services and endpoints. Unfortunately one of our customers is not very happy of having to run a minimum of three ZooKeeper server instances to ensure fail over. The customer has two data centers in different locations where data and applications are replicated and some big-ip appliance in front of the data centers. If one data center fails everything must still be operable. So he would prefer to only have two instances one in each data center. I would be grateful for some advise how to best cope with these contradicting requirements.
Cheers,
Zsolt Beothy-Elo
Re: Question about Clustered Setup in two data centers
Posted by Ted Dunning <te...@gmail.com>.
What is being coordinated with ZK?
What level of reliability is required? What fail-over time? What failure
modes are acceptable?
How will clients coming into the load balancer at one datacenter be
redirected to the other?
In my experience, reliability means many things. In one application I
worked on, full triplication of the data-centers was required and all
transactions were sent to all three. Behind the scenes lots of gnomes
worked to keep things consistent, but the client always got a full speed
response no matter what.
Other situations have required <1 day recovery with no loss of data.
Presumably your needs are between these cases.
On Wed, Mar 2, 2011 at 6:57 AM, Zsolt Beothy-Elo <zb...@talend.com>wrote:
> Hi all,
> in our product we currently implement to use ZooKeeper in conjunction with
> CXF to dynamically manage available services and endpoints. Unfortunately
> one of our customers is not very happy of having to run a minimum of three
> ZooKeeper server instances to ensure fail over. The customer has two data
> centers in different locations where data and applications are replicated
> and some big-ip appliance in front of the data centers. If one data center
> fails everything must still be operable. So he would prefer to only have two
> instances one in each data center. I would be grateful for some advise how
> to best cope with these contradicting requirements.
>
> Cheers,
> Zsolt Beothy-Elo
Re: Question about Clustered Setup in two data centers
Posted by Ritesh Tijoriwala <ti...@gmail.com>.
Hi Zsolt,
Do you have any operations that cross data centers besides when failover
happens and other data center takes over? Is only one data center "active"
at given time and other just for pure failover or both of them can be
active?
If only one of them is active, then why do you need ZooKeeper instances
across data center? For things that you replicate across data center, can
you afford them to be eventually consistent? Within a data center, you can
use ZooKeeper to reach agreement on a "value" and then once reached, you can
just blindly replicate this to other data center. Ofcourse, if the other
data center is also active, then you need to detect conflicts and resolve
them.
Thanks,
Ritesh
On Wed, Mar 2, 2011 at 8:19 AM, Mahadev Konar <ma...@apache.org> wrote:
> Hi Zsolt,
> Sorry to disappoint but there isnt any way of running zookeeper on 2
> machines and calling it reliable. You can run a set of 2 machines for
> a quorum of 3 but if one of the machine goes down then zookeeper goes
> down. Also, a data center of 2 has issues since which ever datacenter
> you place the max part of the quorum in, in case of death will caiuse
> the ZK cluster to fail. I am assuming you are talking about ZK cross
> data center.
>
> thanks
> mahadev
>
> On Wed, Mar 2, 2011 at 6:57 AM, Zsolt Beothy-Elo <zb...@talend.com>
> wrote:
> > Hi all,
> > in our product we currently implement to use ZooKeeper in conjunction
> with CXF to dynamically manage available services and endpoints.
> Unfortunately one of our customers is not very happy of having to run a
> minimum of three ZooKeeper server instances to ensure fail over. The
> customer has two data centers in different locations where data and
> applications are replicated and some big-ip appliance in front of the data
> centers. If one data center fails everything must still be operable. So he
> would prefer to only have two instances one in each data center. I would be
> grateful for some advise how to best cope with these contradicting
> requirements.
> >
> > Cheers,
> > Zsolt Beothy-Elo
>
Re: Question about Clustered Setup in two data centers
Posted by Mahadev Konar <ma...@apache.org>.
Hi Zsolt,
Sorry to disappoint but there isnt any way of running zookeeper on 2
machines and calling it reliable. You can run a set of 2 machines for
a quorum of 3 but if one of the machine goes down then zookeeper goes
down. Also, a data center of 2 has issues since which ever datacenter
you place the max part of the quorum in, in case of death will caiuse
the ZK cluster to fail. I am assuming you are talking about ZK cross
data center.
thanks
mahadev
On Wed, Mar 2, 2011 at 6:57 AM, Zsolt Beothy-Elo <zb...@talend.com> wrote:
> Hi all,
> in our product we currently implement to use ZooKeeper in conjunction with CXF to dynamically manage available services and endpoints. Unfortunately one of our customers is not very happy of having to run a minimum of three ZooKeeper server instances to ensure fail over. The customer has two data centers in different locations where data and applications are replicated and some big-ip appliance in front of the data centers. If one data center fails everything must still be operable. So he would prefer to only have two instances one in each data center. I would be grateful for some advise how to best cope with these contradicting requirements.
>
> Cheers,
> Zsolt Beothy-Elo
Re: Question about Clustered Setup in two data centers
Posted by Ritesh Tijoriwala <ti...@gmail.com>.
Hi Jesse,
Yes, you could do that. But it all depends on the frequency of operations
and how often you use the coordination service. If those are quite frequent,
then the HQ will most likely become a bottleneck. It is mostly not a good
idea to run coordination services across data centers for operations that
are frequent.
Thanks,
Ritesh
On Wed, Mar 2, 2011 at 9:56 AM, Jesse Kempf <je...@twitter.com> wrote:
> Hi Zsolt,
>
> If the customer has a computer room at their headquarters, you could keep
> two ZKs in each datacenter and a fifth ZK at their HQ. In that case you
> could lose a datacenter and still have quorum.
>
> Cheers,
> -Jesse
>
> On Mar 2, 2011, at 2:57 PM, Zsolt Beothy-Elo wrote:
>
> > Hi all,
> > in our product we currently implement to use ZooKeeper in conjunction
> with CXF to dynamically manage available services and endpoints.
> Unfortunately one of our customers is not very happy of having to run a
> minimum of three ZooKeeper server instances to ensure fail over. The
> customer has two data centers in different locations where data and
> applications are replicated and some big-ip appliance in front of the data
> centers. If one data center fails everything must still be operable. So he
> would prefer to only have two instances one in each data center. I would be
> grateful for some advise how to best cope with these contradicting
> requirements.
> >
> > Cheers,
> > Zsolt Beothy-Elo
>
>
Re: Question about Clustered Setup in two data centers
Posted by Zsolt Beothy-Elo <zb...@talend.com>.
Not sure customer will accept to deploy an infrastructure service to EC2 (or any other hosting service), but it's worth mentioning it. Still there will be questions about realibility, response times, firewalls, etc.... Anyway, thx for the quick answers on this mailing list. As soon as we have relased our product I will add it to the PoweredBy page in the wiki.
Zsolt
Am 02.03.2011 um 23:57 schrieb Ted Dunning:
A variant is to keep the tie-breaker in EC2. That can be made fairly secure, especially if all you are hosting is status information.
On Wed, Mar 2, 2011 at 2:51 PM, Zsolt Beothy-Elo <zb...@talend.com>> wrote:
Am 02.03.2011 um 18:56 schrieb Jesse Kempf:
> Hi Zsolt,
>
> If the customer has a computer room at their headquarters, you could keep two ZKs in each datacenter and a fifth ZK at their HQ. In that case you could lose a datacenter and still have quorum.
Running such an infrastructure service outside one of the data centers of course violates almost every policy (security, backup,...) that is in place at the customer. But we also had the idea :) And as Ritesh states computer at headquarter would likely become the bottleneck. Connection between centers is fast and reliable under almost all circumstances.
Zsolt
>
> Cheers,
> -Jesse
>
> On Mar 2, 2011, at 2:57 PM, Zsolt Beothy-Elo wrote:
>
>> Hi all,
>> in our product we currently implement to use ZooKeeper in conjunction with CXF to dynamically manage available services and endpoints. Unfortunately one of our customers is not very happy of having to run a minimum of three ZooKeeper server instances to ensure fail over. The customer has two data centers in different locations where data and applications are replicated and some big-ip appliance in front of the data centers. If one data center fails everything must still be operable. So he would prefer to only have two instances one in each data center. I would be grateful for some advise how to best cope with these contradicting requirements.
>>
>> Cheers,
>> Zsolt Beothy-Elo
>
Re: Question about Clustered Setup in two data centers
Posted by Ted Dunning <te...@gmail.com>.
A variant is to keep the tie-breaker in EC2. That can be made fairly
secure, especially if all you are hosting is status information.
On Wed, Mar 2, 2011 at 2:51 PM, Zsolt Beothy-Elo <zb...@talend.com>wrote:
>
> Am 02.03.2011 um 18:56 schrieb Jesse Kempf:
>
> > Hi Zsolt,
> >
> > If the customer has a computer room at their headquarters, you could keep
> two ZKs in each datacenter and a fifth ZK at their HQ. In that case you
> could lose a datacenter and still have quorum.
>
> Running such an infrastructure service outside one of the data centers of
> course violates almost every policy (security, backup,...) that is in place
> at the customer. But we also had the idea :) And as Ritesh states computer
> at headquarter would likely become the bottleneck. Connection between
> centers is fast and reliable under almost all circumstances.
>
> Zsolt
>
> >
> > Cheers,
> > -Jesse
> >
> > On Mar 2, 2011, at 2:57 PM, Zsolt Beothy-Elo wrote:
> >
> >> Hi all,
> >> in our product we currently implement to use ZooKeeper in conjunction
> with CXF to dynamically manage available services and endpoints.
> Unfortunately one of our customers is not very happy of having to run a
> minimum of three ZooKeeper server instances to ensure fail over. The
> customer has two data centers in different locations where data and
> applications are replicated and some big-ip appliance in front of the data
> centers. If one data center fails everything must still be operable. So he
> would prefer to only have two instances one in each data center. I would be
> grateful for some advise how to best cope with these contradicting
> requirements.
> >>
> >> Cheers,
> >> Zsolt Beothy-Elo
> >
>
>
Re: Question about Clustered Setup in two data centers
Posted by Zsolt Beothy-Elo <zb...@talend.com>.
Am 02.03.2011 um 18:56 schrieb Jesse Kempf:
> Hi Zsolt,
>
> If the customer has a computer room at their headquarters, you could keep two ZKs in each datacenter and a fifth ZK at their HQ. In that case you could lose a datacenter and still have quorum.
Running such an infrastructure service outside one of the data centers of course violates almost every policy (security, backup,...) that is in place at the customer. But we also had the idea :) And as Ritesh states computer at headquarter would likely become the bottleneck. Connection between centers is fast and reliable under almost all circumstances.
Zsolt
>
> Cheers,
> -Jesse
>
> On Mar 2, 2011, at 2:57 PM, Zsolt Beothy-Elo wrote:
>
>> Hi all,
>> in our product we currently implement to use ZooKeeper in conjunction with CXF to dynamically manage available services and endpoints. Unfortunately one of our customers is not very happy of having to run a minimum of three ZooKeeper server instances to ensure fail over. The customer has two data centers in different locations where data and applications are replicated and some big-ip appliance in front of the data centers. If one data center fails everything must still be operable. So he would prefer to only have two instances one in each data center. I would be grateful for some advise how to best cope with these contradicting requirements.
>>
>> Cheers,
>> Zsolt Beothy-Elo
>
Re: Question about Clustered Setup in two data centers
Posted by Jesse Kempf <je...@twitter.com>.
Hi Zsolt,
If the customer has a computer room at their headquarters, you could keep two ZKs in each datacenter and a fifth ZK at their HQ. In that case you could lose a datacenter and still have quorum.
Cheers,
-Jesse
On Mar 2, 2011, at 2:57 PM, Zsolt Beothy-Elo wrote:
> Hi all,
> in our product we currently implement to use ZooKeeper in conjunction with CXF to dynamically manage available services and endpoints. Unfortunately one of our customers is not very happy of having to run a minimum of three ZooKeeper server instances to ensure fail over. The customer has two data centers in different locations where data and applications are replicated and some big-ip appliance in front of the data centers. If one data center fails everything must still be operable. So he would prefer to only have two instances one in each data center. I would be grateful for some advise how to best cope with these contradicting requirements.
>
> Cheers,
> Zsolt Beothy-Elo