Expanding a ZK quorum with replication

[Ian Friedman <ia...@flurry.com>]

Hey guys, looking for some advice on this issue: 

We'd like to move our Hbase cluster from 3 to 5 Zookeeper nodes. Normally this would be easily handled by a rolling restart. However we also have replication between our 2 production clusters enabled, which uses the ZK quorum of either side as the key. As far as I can tell there's no way to modify a replication peer while it's running, and we can't take downtime. Any ideas? Ther only one I've had so far is to break replication, start it back up with the new peer, and then run a copytable job to catch up. 

-- 
Ian Friedman

Re: Expanding a ZK quorum with replication

[Jeremy Carroll <ph...@gmail.com>]

You should be able to do a rolling restart from 3 => 5 without losing
quorum. As long as the replication peer can connect to any ZooKeeper node
which is in quorum, it should be OK.

I agree that you would want to add additional peers to the replication
agreement. I have never done this personally. I believe something to the
affect that you stated would work. Stop replication for a moment, modify
the replication peer, start it back up, then CopyTable for the difference.
Just spit balling here, but you would look into WAL replay tools perhaps as
well? https://issues.apache.org/jira/browse/HBASE-5604


On Fri, Jul 19, 2013 at 10:26 AM, Ian Friedman <ia...@flurry.com> wrote:

> Hey guys, looking for some advice on this issue:
>
> We'd like to move our Hbase cluster from 3 to 5 Zookeeper nodes. Normally
> this would be easily handled by a rolling restart. However we also have
> replication between our 2 production clusters enabled, which uses the ZK
> quorum of either side as the key. As far as I can tell there's no way to
> modify a replication peer while it's running, and we can't take downtime.
> Any ideas? Ther only one I've had so far is to break replication, start it
> back up with the new peer, and then run a copytable job to catch up.
>
> --
> Ian Friedman
>
>