[jira] [Updated] (KAFKA-9590) Add read-only configuration to limit number of topic partitions

["Gokul Ramanan Subramanian (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-9590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gokul Ramanan Subramanian updated KAFKA-9590:
---------------------------------------------
    Description: 
Currently, a Kafka cluster does not limit the number of topic partitions despite the fact that on a given system, having more than a certain number of topic partitions (without even any messages written / read) renders the cluster unusable.

 

The ask here is to have a read-only Kafka configuration named something like *max.broker.partitions* (an integer, which applies at the broker level and defaults to INT.MAX) that the administrator the cluster can set to ensure smooth operation.

 

Things to keep in mind:
 # If a broker already has more than *max.broker.partitions* number of partitions at the time when the *max.broker.partitions* configuration is first applied on the broker, the broker should disallow any more topic partitions. This allows for graceful reassignments and in-place upgrade to a Kafka version that supports the *max.broker.partitions* configuration.
 # Different brokers can technically have different *max.broker.partitions* in case of a heterogenous cluster, where not all broker nodes operate with the same system limits (such as CPU, memory etc.)
 # When creating a topic partition, the controller has to identify brokers which will replicate the topic partition. Currently, the controller distributes the leadership of the topic partitions to ensure uniform load. This algorithm will need to account for the *max.broker.partitions* configuration.

 

  was:
Currently, a Kafka cluster does not limit the number of topic partitions despite the fact that on a given system, having more than a certain number of topic partitions (without even any messages written / read) renders the cluster unusable.

 

The ask here is to have a read-only Kafka configuration named something like *max.topic.partitions* (an integer, which applies at the broker level and defaults to INT.MAX) that the administrator the cluster can set to ensure smooth operation.

 

Things to keep in mind:
 # If a broker already has more than *max.topic.partitions* number of partitions at the time when the *max.topic.partitions* configuration is first applied on the broker, the broker should disallow any more topic partitions. This allows for graceful reassignments and in-place upgrade to a Kafka version that supports the *max.topic.partitions* configuration.
 # Different brokers can technically have different *max.topic.partitions* in case of a heterogenous cluster, where not all broker nodes operate with the same system limits (such as CPU, memory etc.)
 # When creating a topic partition, the controller has to identify brokers which will replicate the topic partition. Currently, the controller distributes the leadership of the topic partitions to ensure uniform load. This algorithm will need to account for the *max.topic.partitions* configuration.

 


> Add read-only configuration to limit number of topic partitions
> ---------------------------------------------------------------
>
>                 Key: KAFKA-9590
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9590
>             Project: Kafka
>          Issue Type: Improvement
>          Components: admin
>            Reporter: Gokul Ramanan Subramanian
>            Priority: Minor
>
> Currently, a Kafka cluster does not limit the number of topic partitions despite the fact that on a given system, having more than a certain number of topic partitions (without even any messages written / read) renders the cluster unusable.
>  
> The ask here is to have a read-only Kafka configuration named something like *max.broker.partitions* (an integer, which applies at the broker level and defaults to INT.MAX) that the administrator the cluster can set to ensure smooth operation.
>  
> Things to keep in mind:
>  # If a broker already has more than *max.broker.partitions* number of partitions at the time when the *max.broker.partitions* configuration is first applied on the broker, the broker should disallow any more topic partitions. This allows for graceful reassignments and in-place upgrade to a Kafka version that supports the *max.broker.partitions* configuration.
>  # Different brokers can technically have different *max.broker.partitions* in case of a heterogenous cluster, where not all broker nodes operate with the same system limits (such as CPU, memory etc.)
>  # When creating a topic partition, the controller has to identify brokers which will replicate the topic partition. Currently, the controller distributes the leadership of the topic partitions to ensure uniform load. This algorithm will need to account for the *max.broker.partitions* configuration.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)