Horizontal scaling a topic

[Corey Nolet <cj...@gmail.com>]

I have a cluster of 3 nodes and I've created a topic with some number of
partitions and some number of replicas, let's say 10 and 2, respectively.
Later, after I've got my 3 nodes fairly consumed with data in the 10
partitions, I want to add 2 more nodes to the mix to help balance out the
partitions/replicas of my topic across 5 physical nodes instead of just 3.

I was assuming Kafka would just notice the new node and auto-replicate
partitions to it but research is telling me that this probably isn't the
case. Let's say I want no data loss and I want Kafka to spread my 10
partitions across all 5 nodes. How would I do this currently?

Re: Horizontal scaling a topic

[Yury Ruchin <yu...@gmail.com>]

For Kafka 0.8.x[.x], refer to
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool
.

2015-04-23 23:20 GMT+03:00 Corey Nolet <cj...@gmail.com>:

> I have a cluster of 3 nodes and I've created a topic with some number of
> partitions and some number of replicas, let's say 10 and 2, respectively.
> Later, after I've got my 3 nodes fairly consumed with data in the 10
> partitions, I want to add 2 more nodes to the mix to help balance out the
> partitions/replicas of my topic across 5 physical nodes instead of just 3.
>
> I was assuming Kafka would just notice the new node and auto-replicate
> partitions to it but research is telling me that this probably isn't the
> case. Let's say I want no data loss and I want Kafka to spread my 10
> partitions across all 5 nodes. How would I do this currently?
>