Fully redundant installation of ZooKeeper

[Kambiz Niktabar <ni...@yahoo.com.INVALID>]

Hi,
We are using ZooKepper ensembles to manage configuration files of our Solr environment. Basically our servers are distributed to two different physical locations (e.g. Site A and Site B) which every location has its own maintenance window for security patching, etc. That means during the maintenance time, it might happens that all servers in site A will be unavailable.Since the majority of ZooKeeper instances needs to be up and running to have ZooKeeper available, this forces us to perform maintenance on the site with minority instances manually.I wonder if there is any smart way to setup Zookeeper to have fully redundant service for these two location without involving any manual job?please advise
RegardsKambiz Niktabar 

Re: Fully redundant installation of ZooKeeper

[Alexander Shraer <sh...@gmail.com>]

Hi,

if you're using 3.5.0 you could consider reconfiguring the cluster, e.g.
 remove the servers under maintanance and add them back after you're done.
The removal is logical - they will still continue serving clients unless
shut down, its just these servers will not be counted - a majority of the
remaining servers will be needed.

Alex

On Fri, Jan 9, 2015 at 12:50 AM, Kambiz Niktabar <niktabar@yahoo.com.invalid
> wrote:

> Hi,
> We are using ZooKepper ensembles to manage configuration files of our Solr
> environment. Basically our servers are distributed to two different
> physical locations (e.g. Site A and Site B) which every location has its
> own maintenance window for security patching, etc. That means during the
> maintenance time, it might happens that all servers in site A will be
> unavailable.Since the majority of ZooKeeper instances needs to be up and
> running to have ZooKeeper available, this forces us to perform maintenance
> on the site with minority instances manually.I wonder if there is any smart
> way to setup Zookeeper to have fully redundant service for these two
> location without involving any manual job?please advise
> RegardsKambiz Niktabar

Re: Fully redundant installation of ZooKeeper

[Ayon Sinha <ay...@gmail.com>]

Hi,
Your best bet would be to spread the ZK servers among odd number of
groupings. e.g. spread them between Datacenter A,B and C so you can take
down an entire DC for maintenance. If you don't have another DC available,
maybe the data center is split with clouds so you can bring down one cloud
in a DC and leave the other one running. You get the idea. Essentially, if
you have N/2 and N/2 nodes in 2 DCs its impossible to have a functional ZK
when you take down one DC.

We are running 3,3,3, in 3 datacenters. 9 is the smallest number where we
could get a redundant ZK ensemble. 2,2,2 config can only handle failure of
2 nodes.

-Ayon

On Fri, Jan 9, 2015 at 12:50 AM, Kambiz Niktabar <niktabar@yahoo.com.invalid
> wrote:

> Hi,
> We are using ZooKepper ensembles to manage configuration files of our Solr
> environment. Basically our servers are distributed to two different
> physical locations (e.g. Site A and Site B) which every location has its
> own maintenance window for security patching, etc. That means during the
> maintenance time, it might happens that all servers in site A will be
> unavailable.Since the majority of ZooKeeper instances needs to be up and
> running to have ZooKeeper available, this forces us to perform maintenance
> on the site with minority instances manually.I wonder if there is any smart
> way to setup Zookeeper to have fully redundant service for these two
> location without involving any manual job?please advise
> RegardsKambiz Niktabar