[jira] [Created] (KAFKA-4606) make getOffsetsTopicPartitionCount mutable

["Ryan P (JIRA)" <ji...@apache.org>]

Ryan P created KAFKA-4606:
-----------------------------

             Summary: make getOffsetsTopicPartitionCount mutable  
                 Key: KAFKA-4606
                 URL: https://issues.apache.org/jira/browse/KAFKA-4606
             Project: Kafka
          Issue Type: Improvement
          Components: consumer
            Reporter: Ryan P


Under certain circumstances users may wish to increase the size of their __consumer_offsets topic to spread the load as their consumer load increases. 

The current limitation, in which we try to prevent the topic's expansion makes this incredibly difficult. This is done intentionally to reduce the risk of unexpected behavior once the group id has been assigned a new coordinator. 

Instead Kafka should consider ways to handle this operation more gracefully with the following behaviors. 

1. Calculate the number of __consumer_offsets partitions from the metadata cache with each metadata update 

2. Force a rebalance after the new partitions are added to the topic. *I'm sure to take into consideration here as well. Alternatively the operation could be documented as risky emphasizing the behavior of the reset policy under these conditions.  

Currently the number of partitions for the consumer_offsets topic is fixed. This mean you will need to restart each broker to ensure that coordinator discovery happens as expected. That seems less than ideal to me for what seems to be a fairly artificial limitation of the consumer group implementation. 




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)