[jira] [Created] (KAFKA-1829) Support adding new partitions on brokers without reassignment of existing partitions

["Allen Wang (JIRA)" <ji...@apache.org>]

Allen Wang created KAFKA-1829:
---------------------------------

             Summary: Support adding new partitions on brokers without reassignment of existing partitions
                 Key: KAFKA-1829
                 URL: https://issues.apache.org/jira/browse/KAFKA-1829
             Project: Kafka
          Issue Type: New Feature
    Affects Versions: 0.8.1.1
            Reporter: Allen Wang


One solution to deal with increasing incoming traffic to Kafka brokers is to add new broker instances and new partitions at the same time. This is possible if disk size is not the bottleneck. In this case, ideally the new partitions would be assigned to new brokers without any changes to existing partitions. The reason behind it is that moving existing partitions would cause more traffic and hence more load.

The way to handle this now is a multi step process:

- add partitions
- generate replica assignment for the new partitions on the new brokers
- execute the assignment

This is complicated. Also, adding partitions and assigning replicas is not atomic -- when partitions are added, they are already assigned using a default assignment algorithm.

This feature would greatly help auto scaling Kafka to deal with increasing load.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)