[jira] [Commented] (KAFKA-4606) make getOffsetsTopicPartitionCount mutable

["Ewen Cheslack-Postava (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-4606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15805984#comment-15805984 ] 

Ewen Cheslack-Postava commented on KAFKA-4606:
----------------------------------------------

It's not really an artificial limitation though, right? The way consumer groups are mapped to coordinators depends on this. If you allow changing the # of partitions then brokers would have to load and cache in memory the entire offsets topic to ensure that if the partition count changed they could still correctly handle offset fetch requests. Also, during the change you'd somehow need to ensure the change was made atomically across the entire cluster so you wouldn't have multiple brokers thinking they were coordinator for the same group. I don't think getOffsetsTopicPartitionCount is the biggest problem with expanding the number of partitions in the offsets topic...

> make getOffsetsTopicPartitionCount mutable  
> --------------------------------------------
>
>                 Key: KAFKA-4606
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4606
>             Project: Kafka
>          Issue Type: Improvement
>          Components: consumer
>            Reporter: Ryan P
>
> Under certain circumstances users may wish to increase the size of their __consumer_offsets topic to spread the load as their consumer load increases. 
> The current limitation, in which we try to prevent the topic's expansion makes this incredibly difficult. This is done intentionally to reduce the risk of unexpected behavior once the group id has been assigned a new coordinator. 
> Instead Kafka should consider ways to handle this operation more gracefully with the following behaviors. 
> 1. Calculate the number of __consumer_offsets partitions from the metadata cache with each metadata update 
> 2. Force a rebalance after the new partitions are added to the topic. *I'm sure to take into consideration here as well. Alternatively the operation could be documented as risky emphasizing the behavior of the reset policy under these conditions.  
> Currently the number of partitions for the consumer_offsets topic is fixed. This mean you will need to restart each broker to ensure that coordinator discovery happens as expected. That seems less than ideal to me for what seems to be a fairly artificial limitation of the consumer group implementation. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)