Kafka 0.8.1.1 partition reassignment script

[Florian Dambrine <fl...@gumgum.com>]

Hi,

I am currently running a Kafka 0.8.1.1 cluster with 8 servers. I would like
to add a new broker in the cluster. Each kafka instance has 400Go of data
and we are using replication factor equals to 3 with 50 partitions for each
topics.

I have check the documentation and especially the section dealing with the
cluster expansion (
https://kafka.apache.org/documentation.html#basic_ops_cluster_expansion).

So far, I have tried the automatic way to reassign partitions using
the --topics-to-move-json-file
option on 1 small topic. Here is the command that I have ran:

content of topic-to-move.json:

{
"topics": [
{"topic": "SMALL_TOPIC"}],
  "version":1
}

command:

./bin/kafka-reassign-partitions.sh --zookeeper zkDns:2181
--topics-to-move-json-file topics-to-move.json --broker-list
"1,2,3,4,5,6,7,8,9" --generate

Has you can see I specified a broker list including all the broker from the
cluster (including the newly added kafka node number 9).

First of all is it the right way to do that? As I am using replication
factor = 3, I have to specify at least 3 brokers. I thought that specifying
the entire list of broker will keep the cluster balanced (and it did).

Here is my main question, I looked the Ganglia ui in order to see the
impact of this command and here is the networking graph (
http://s18.postimg.org/hokyoeyax/kafka_network.png)

As you can see on the screen-shot the network traffic goes pretty high.
That is why I am afraid to run this tool on bigger topics.

I also took a look into the manual way of reassigning partitions. the
problem is that I always have to specify 3 replicas even if I just want to
move only one to my new broker.

Do you know any tool or script to help me in this process?

Thanks for your help,

Florian

-- 
*Florian Dambrine*  |  Intern, Big Data
*GumGum* <http://www.gumgum.com/>  |  *Ads that stick*
209-797-3994  |  florian@gumgum.com

Fwd: Kafka 0.8.1.1 partition reassignment script

[Florian Dambrine <fl...@gumgum.com>]

Hi,

I am currently running a Kafka 0.8.1.1 cluster with 8 servers. I would like
to add a new broker in the cluster. Each kafka instance has 400Go of data
and we are using replication factor equals to 3 with 50 partitions for each
topics.

I have check the documentation and especially the section dealing with the
cluster expansion (
https://kafka.apache.org/documentation.html#basic_ops_cluster_expansion).

So far, I have tried the automatic way to reassign partitions using
the --topics-to-move-json-file
option on 1 small topic. Here is the command that I have ran:

content of topic-to-move.json:

{
"topics": [
{"topic": "SMALL_TOPIC"}],
  "version":1
}

command:

./bin/kafka-reassign-partitions.sh --zookeeper zkDns:2181
--topics-to-move-json-file topics-to-move.json --broker-list
"1,2,3,4,5,6,7,8,9" --generate

Has you can see I specified a broker list including all the broker from the
cluster (including the newly added kafka node number 9).

First of all is it the right way to do that? As I am using replication
factor = 3, I have to specify at least 3 brokers. I thought that specifying
the entire list of broker will keep the cluster balanced (and it did).

Here is my main question, I looked the Ganglia ui in order to see the
impact of this command and here is the networking graph (
http://s18.postimg.org/hokyoeyax/kafka_network.png)

As you can see on the screen-shot the network traffic goes pretty high.
That is why I am afraid to run this tool on bigger topics.

I also took a look into the manual way of reassigning partitions. the
problem is that I always have to specify 3 replicas even if I just want to
move only one to my new broker.

Do you know any tool or script to help me in this process?

Thanks for your help,

Florian

-- 
*Florian Dambrine*  |  Intern, Big Data
*GumGum* <http://www.gumgum.com/>  |  *Ads that stick*
209-797-3994  |  florian@gumgum.com