Kafka Reassignment - traffic is different for the same type of reassignment

[Yi Yin <yi...@gmail.com>]

Hello everyone,


I have a question about how reassignments work.

When I issue a reassignment for similar topicpartitions, the throughput
between the reassignments is very different even though all settings are
similar. There is a huge difference on when the partitions finish their
reassignment.

Here is my reassignment mapping:
{ "partitions": [
{ "topic": "my_topic", "partition": 100, "replicas": [1426,1425] },
{ "topic": "my_topic", "partition": 101, "replicas": [1476,1287] }
], "version": 1 }

I am moving partition 100 from 1035  to 1426.
I am moving partition 101 from 1037 to 1476.

Prior to the reassignment, 1035 and 1037 have the same network, and the
above partitions are the only partitions on this hosts.

When I kick off the reassignment, 1037's network out spikes 3x more than
1035. As a result, partition 101's reassignment finishes much sooner than
100.


our server.properties is identical across the whole cluster.

I'm curious as to why the two partition reassignment has drastically
different throughput. Their source and destination brokers are identical,
at no point was there saturation. And the two partitions have the same data
size.

If anyone has insights into this behaviour, please let me know how this can
happen.

Thank you!

Yi