[jira] [Created] (KAFKA-15118) Need for a Centralized Configuration Management System in Apache Kafka

["Jimmy Wang (Jira)" <ji...@apache.org>]

Jimmy Wang created KAFKA-15118:
----------------------------------

             Summary: Need for a Centralized Configuration Management System in Apache Kafka
                 Key: KAFKA-15118
                 URL: https://issues.apache.org/jira/browse/KAFKA-15118
             Project: Kafka
          Issue Type: Improvement
          Components: core
    Affects Versions: 2.7.1
            Reporter: Jimmy Wang


Hi all,

In our use of Apache Kafka, we found something strange following:

Despite our partitions being well-distributed across brokers, we noticed a significant discrepancy in disk usage between different brokers. Specifically, the same partition takes up different amounts of disk space on different brokers. Upon investigating, we found that the root cause of this discrepancy was a variation in {{log.retention.hours}} setting between different brokers.

On the one hand, we know that we should make sure that the server.properties should be same across the whole cluster. But on the other hand, could kafka provide guarantee or advance check to make sure all the configurations of brokers to be the same to avoid such situations?

Here is some of my opinions:
 # Provide centralized configuration center or just manage them in internal topics(kraft mode) or zookeeper like dynamic configuration.
 # Warn users about potential inconsistencies during the broker startup process.

Best regards,

Jimmy Wang



--
This message was sent by Atlassian Jira
(v8.20.10#820010)