zookeeper cluster change impact on kafka

[vijay <vi...@gmail.com>]

Hi Guys,

We have a 3 node zookeeper cluster and a kafka cluster (3 nodes) using this
zookeeper cluster. we want to migrate the zookeeper nodes to better boxes (
hardware improvements). we already setup 3 new nodes.

can some one tell me what is the safe way to do the migration.

1. what happens if we change the zookeeper connection string in kafka nodes
to point to  3 new ZK nodes and start the kafka nodes one by one?
2. Do we need to change the old and new zookeeper nodes configuration before
applying any changes on kafka nodes (zk clients)? like adding new nodes as
observers etc...
3. what happens if we change the kafka configuration to talk to all 6 ZK
Nodes and later shut down 3 old nodes and reconfigure the kafka to talk to 3
new ZK nodes.

Thanks for your help.

--Vijay



--
View this message in context: http://zookeeper-user.578899.n2.nabble.com/zookeeper-cluster-change-impact-on-kafka-tp7581711.html
Sent from the zookeeper-user mailing list archive at Nabble.com.

Re: zookeeper cluster change impact on kafka

[vijay <vi...@gmail.com>]

Thanks for the quick response Flavio,

Just to iterate what i understand.

1.  swap the zookeeper nodes one by one, add new node and remove one old
node: i have one more question here, what about kafka cluster configuration,
is it ok to keep old 3 zookeeper nodes as still there are 2 old nodes
available?

2.  just before swapping the 3rd node (leader node), i think we need to
change the kafka cluster configuration to point to new zookeeper nodes and
bounce the kafka cluster. is it correct approach or do we need to take a
brief outage and swap the 3rd (leader) node and restart the kafka cluster
with new ZK nodes configuration.

Thanks for your help.



--
View this message in context: http://zookeeper-user.578899.n2.nabble.com/zookeeper-cluster-change-impact-on-kafka-tp7581711p7581713.html
Sent from the zookeeper-user mailing list archive at Nabble.com.

Re: zookeeper cluster change impact on kafka

[Flavio Junqueira <fp...@yahoo.com.INVALID>]

I'm copying the Kafka user list here:

> We have a 3 node zookeeper cluster and a kafka cluster (3 nodes) using this
> zookeeper cluster. we want to migrate the zookeeper nodes to better boxes (
> hardware improvements). we already setup 3 new nodes.
> 
> can some one tell me what is the safe way to do the migration.
> 
> 1. what happens if we change the zookeeper connection string in kafka nodes
> to point to  3 new ZK nodes and start the kafka nodes one by one?

The new cluster will start with an empty state. Switching to a new cluster abruptly like you suggest will mean loss of data, which is bad. Also, you won't be able to switch all brokers at once, so for some time, you'll have brokers talking to different ensembles, which can lead to an inconsistent state as well.

> 2. Do we need to change the old and new zookeeper nodes configuration before
> applying any changes on kafka nodes (zk clients)? like adding new nodes as
> observers etc...

You need to introduce one new node at a time. In the same step that you introduce a node, you remove a node from the old ensemble. To do this, you need to change the server configuration at each step to remove a server and add a server, and wait until a quorum follows a leader before moving to the next step. This way you guarantee that the state of zookeeper in the old ensemble migrates to the new ensemble.

> 3. what happens if we change the kafka configuration to talk to all 6 ZK
> Nodes and later shut down 3 old nodes and reconfigure the kafka to talk to 3
> new ZK nodes.

You can get into an inconsistent state if you do this. There is a chance that it works, but I don't recommend it.

-Flavio

> 
> 
> 
> --
> View this message in context: http://zookeeper-user.578899.n2.nabble.com/zookeeper-cluster-change-impact-on-kafka-tp7581711.html
> Sent from the zookeeper-user mailing list archive at Nabble.com.

Re: zookeeper cluster change impact on kafka

[Flavio Junqueira <fp...@yahoo.com.INVALID>]

I'm copying the Kafka user list here:

> We have a 3 node zookeeper cluster and a kafka cluster (3 nodes) using this
> zookeeper cluster. we want to migrate the zookeeper nodes to better boxes (
> hardware improvements). we already setup 3 new nodes.
> 
> can some one tell me what is the safe way to do the migration.
> 
> 1. what happens if we change the zookeeper connection string in kafka nodes
> to point to  3 new ZK nodes and start the kafka nodes one by one?

The new cluster will start with an empty state. Switching to a new cluster abruptly like you suggest will mean loss of data, which is bad. Also, you won't be able to switch all brokers at once, so for some time, you'll have brokers talking to different ensembles, which can lead to an inconsistent state as well.

> 2. Do we need to change the old and new zookeeper nodes configuration before
> applying any changes on kafka nodes (zk clients)? like adding new nodes as
> observers etc...

You need to introduce one new node at a time. In the same step that you introduce a node, you remove a node from the old ensemble. To do this, you need to change the server configuration at each step to remove a server and add a server, and wait until a quorum follows a leader before moving to the next step. This way you guarantee that the state of zookeeper in the old ensemble migrates to the new ensemble.

> 3. what happens if we change the kafka configuration to talk to all 6 ZK
> Nodes and later shut down 3 old nodes and reconfigure the kafka to talk to 3
> new ZK nodes.

You can get into an inconsistent state if you do this. There is a chance that it works, but I don't recommend it.

-Flavio

> 
> 
> 
> --
> View this message in context: http://zookeeper-user.578899.n2.nabble.com/zookeeper-cluster-change-impact-on-kafka-tp7581711.html
> Sent from the zookeeper-user mailing list archive at Nabble.com.