[jira] [Created] (KAFKA-5335) Controller should batch updatePartitionReassignmentData() operation

["Dong Lin (JIRA)" <ji...@apache.org>]

Dong Lin created KAFKA-5335:
-------------------------------

             Summary: Controller should batch updatePartitionReassignmentData() operation
                 Key: KAFKA-5335
                 URL: https://issues.apache.org/jira/browse/KAFKA-5335
             Project: Kafka
          Issue Type: Bug
            Reporter: Dong Lin
            Assignee: Dong Lin


Currently controller will update partition reassignment data every time a partition in the reassignment is completed. It means that if user specifies a huge reassignment znode of size 1 MB to move 10K partitions, controller will need to write roughly 0.5 MB * 10000 = 5 GB data to zookeeper in order to complete this reassignment. This is because controller needs to write the remaining partitions to the znode every time a partition is completely moved.

This is problematic because such a huge reassignment may greatly slow down Kafka controller. Note that partition reassignment doesn't necessarily cause data movement between brokers because we may use it only to recorder the replica list of partitions to evenly distribute preferred leader.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)