split ZK cluster across DCs?

[Ishaaq Chandy <is...@gmail.com>]

Hi all,
We're in the process of understanding the implications of splitting some of
the nodes of our system into a different DC. The DC will be situated in a
different continent to be geographically closer to some of our clients -
for network latency reasons.

Anyway, we're trying to work out how work with our ZK cluster in this
scenario. The way I see it we have 2 options:

1. Leave our existing ZK cluster as is on our primary DC, make the nodes on
the new DC make the hop to do comms with it
2. Add a couple of new local ZK nodes to the new DC and let all them sort
out the comms between them and the rest of the ZK cluster back in the
primary DC

I am leaning towards the latter option but was wondering if any of you have
any insights/experiences in this area that might help us make an informed
decision.

Thanks,
Ishaaq

Re: split ZK cluster across DCs?

[Ishaaq Chandy <is...@gmail.com>]

Ok, I guess that means we're better of with keeping all the zk nodes in our
main DC and letting our app nodes on the remote DC do the hop for ZK comms.

Thanks for the input!

On 29 September 2012 03:55, Camille Fournier <ca...@apache.org> wrote:

> Good point. The monitor and restart trick is if you need the second DC to
> be voting for BCP purposes (over 3 datacenters), but otherwise an Observer
> is the way to go.
>
> C
>
> On Fri, Sep 28, 2012 at 1:25 PM, Ted Dunning <te...@gmail.com>
> wrote:
>
> > I am not sure of any substantial advantage to the monitor and restart
> trick
> > over observers.  As far as I can tell, it mostly just makes the majority
> > more delicate.  The minority still has to send updates through to the
> other
> > data center.
> >
> > On Fri, Sep 28, 2012 at 10:57 AM, Camille Fournier <camille@apache.org
> > >wrote:
> >
> > > If these nodes in the remote DC just need to read, you can do this via
> > > observers in the remote DC. If they need to write as well, you're in a
> > > tough spot. If most of the clients will be in one data center, I would
> > > favor keeping a majority of nodes in that datacenter and forcing the
> > master
> > > to be there (via monitoring and restarting as necessary), so that you
> can
> > > get votes locally and have mostly faster writes. Otherwise you're just
> > > going to have a slower cluster than is optimal but that's the tradeoff
> > for
> > > supporting two datacenters.
> > >
> > > C
> > >
> > > On Fri, Sep 28, 2012 at 12:13 AM, Ishaaq Chandy <is...@gmail.com>
> > wrote:
> > >
> > > > Hi all,
> > > > We're in the process of understanding the implications of splitting
> > some
> > > of
> > > > the nodes of our system into a different DC. The DC will be situated
> > in a
> > > > different continent to be geographically closer to some of our
> clients
> > -
> > > > for network latency reasons.
> > > >
> > > > Anyway, we're trying to work out how work with our ZK cluster in this
> > > > scenario. The way I see it we have 2 options:
> > > >
> > > > 1. Leave our existing ZK cluster as is on our primary DC, make the
> > nodes
> > > on
> > > > the new DC make the hop to do comms with it
> > > > 2. Add a couple of new local ZK nodes to the new DC and let all them
> > sort
> > > > out the comms between them and the rest of the ZK cluster back in the
> > > > primary DC
> > > >
> > > > I am leaning towards the latter option but was wondering if any of
> you
> > > have
> > > > any insights/experiences in this area that might help us make an
> > informed
> > > > decision.
> > > >
> > > > Thanks,
> > > > Ishaaq
> > > >
> > >
> >
>

Re: split ZK cluster across DCs?

[Camille Fournier <ca...@apache.org>]

Good point. The monitor and restart trick is if you need the second DC to
be voting for BCP purposes (over 3 datacenters), but otherwise an Observer
is the way to go.

C

On Fri, Sep 28, 2012 at 1:25 PM, Ted Dunning <te...@gmail.com> wrote:

> I am not sure of any substantial advantage to the monitor and restart trick
> over observers.  As far as I can tell, it mostly just makes the majority
> more delicate.  The minority still has to send updates through to the other
> data center.
>
> On Fri, Sep 28, 2012 at 10:57 AM, Camille Fournier <camille@apache.org
> >wrote:
>
> > If these nodes in the remote DC just need to read, you can do this via
> > observers in the remote DC. If they need to write as well, you're in a
> > tough spot. If most of the clients will be in one data center, I would
> > favor keeping a majority of nodes in that datacenter and forcing the
> master
> > to be there (via monitoring and restarting as necessary), so that you can
> > get votes locally and have mostly faster writes. Otherwise you're just
> > going to have a slower cluster than is optimal but that's the tradeoff
> for
> > supporting two datacenters.
> >
> > C
> >
> > On Fri, Sep 28, 2012 at 12:13 AM, Ishaaq Chandy <is...@gmail.com>
> wrote:
> >
> > > Hi all,
> > > We're in the process of understanding the implications of splitting
> some
> > of
> > > the nodes of our system into a different DC. The DC will be situated
> in a
> > > different continent to be geographically closer to some of our clients
> -
> > > for network latency reasons.
> > >
> > > Anyway, we're trying to work out how work with our ZK cluster in this
> > > scenario. The way I see it we have 2 options:
> > >
> > > 1. Leave our existing ZK cluster as is on our primary DC, make the
> nodes
> > on
> > > the new DC make the hop to do comms with it
> > > 2. Add a couple of new local ZK nodes to the new DC and let all them
> sort
> > > out the comms between them and the rest of the ZK cluster back in the
> > > primary DC
> > >
> > > I am leaning towards the latter option but was wondering if any of you
> > have
> > > any insights/experiences in this area that might help us make an
> informed
> > > decision.
> > >
> > > Thanks,
> > > Ishaaq
> > >
> >
>

Re: split ZK cluster across DCs?

[Ted Dunning <te...@gmail.com>]

I am not sure of any substantial advantage to the monitor and restart trick
over observers.  As far as I can tell, it mostly just makes the majority
more delicate.  The minority still has to send updates through to the other
data center.

On Fri, Sep 28, 2012 at 10:57 AM, Camille Fournier <ca...@apache.org>wrote:

> If these nodes in the remote DC just need to read, you can do this via
> observers in the remote DC. If they need to write as well, you're in a
> tough spot. If most of the clients will be in one data center, I would
> favor keeping a majority of nodes in that datacenter and forcing the master
> to be there (via monitoring and restarting as necessary), so that you can
> get votes locally and have mostly faster writes. Otherwise you're just
> going to have a slower cluster than is optimal but that's the tradeoff for
> supporting two datacenters.
>
> C
>
> On Fri, Sep 28, 2012 at 12:13 AM, Ishaaq Chandy <is...@gmail.com> wrote:
>
> > Hi all,
> > We're in the process of understanding the implications of splitting some
> of
> > the nodes of our system into a different DC. The DC will be situated in a
> > different continent to be geographically closer to some of our clients -
> > for network latency reasons.
> >
> > Anyway, we're trying to work out how work with our ZK cluster in this
> > scenario. The way I see it we have 2 options:
> >
> > 1. Leave our existing ZK cluster as is on our primary DC, make the nodes
> on
> > the new DC make the hop to do comms with it
> > 2. Add a couple of new local ZK nodes to the new DC and let all them sort
> > out the comms between them and the rest of the ZK cluster back in the
> > primary DC
> >
> > I am leaning towards the latter option but was wondering if any of you
> have
> > any insights/experiences in this area that might help us make an informed
> > decision.
> >
> > Thanks,
> > Ishaaq
> >
>

Re: split ZK cluster across DCs?

[Camille Fournier <ca...@apache.org>]

If these nodes in the remote DC just need to read, you can do this via
observers in the remote DC. If they need to write as well, you're in a
tough spot. If most of the clients will be in one data center, I would
favor keeping a majority of nodes in that datacenter and forcing the master
to be there (via monitoring and restarting as necessary), so that you can
get votes locally and have mostly faster writes. Otherwise you're just
going to have a slower cluster than is optimal but that's the tradeoff for
supporting two datacenters.

C

On Fri, Sep 28, 2012 at 12:13 AM, Ishaaq Chandy <is...@gmail.com> wrote:

> Hi all,
> We're in the process of understanding the implications of splitting some of
> the nodes of our system into a different DC. The DC will be situated in a
> different continent to be geographically closer to some of our clients -
> for network latency reasons.
>
> Anyway, we're trying to work out how work with our ZK cluster in this
> scenario. The way I see it we have 2 options:
>
> 1. Leave our existing ZK cluster as is on our primary DC, make the nodes on
> the new DC make the hop to do comms with it
> 2. Add a couple of new local ZK nodes to the new DC and let all them sort
> out the comms between them and the rest of the ZK cluster back in the
> primary DC
>
> I am leaning towards the latter option but was wondering if any of you have
> any insights/experiences in this area that might help us make an informed
> decision.
>
> Thanks,
> Ishaaq
>