[jira] [Commented] (KAFKA-15118) Need for a Centralized Configuration Management System in Apache Kafka

["Kirk True (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-15118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17737245#comment-17737245 ] 

Kirk True commented on KAFKA-15118:
-----------------------------------

[~jimmywang611] Thanks for the ticket. Centralized configuration is something we've talked about doing for {_}clients{_}, but brokers could benefit from this too. I'm not sure if this is something that should be handled inside Kafka or relegated to the orchestration layer outside of Kafka.

Either way, this is project would have a lot of ramifications, so a KIP would almost certainly be needed. Is that something that you plan to write up?

Thanks!

> Need for a Centralized Configuration Management System in Apache Kafka
> ----------------------------------------------------------------------
>
>                 Key: KAFKA-15118
>                 URL: https://issues.apache.org/jira/browse/KAFKA-15118
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 2.7.1
>            Reporter: Jimmy Wang
>            Priority: Major
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> Hi all,
> In our use of Apache Kafka, we found something strange following:
> Despite our partitions being well-distributed across brokers, we noticed a significant discrepancy in disk usage between different brokers. Specifically, the same partition takes up different amounts of disk space on different brokers. Upon investigating, we found that the root cause of this discrepancy was a variation in {{log.retention.hours}} setting between different brokers.
> On the one hand, we know that we should make sure that the server.properties should be same across the whole cluster. But on the other hand, could kafka provide guarantee or advance check to make sure all the configurations of brokers to be the same to avoid such situations?
> Here is some of my opinions:
>  # Provide centralized configuration center or just manage them in internal topics(kraft mode) or zookeeper like dynamic configuration.
>  # Warn users about potential inconsistencies during the broker startup process.
> Best regards,
> Jimmy Wang



--
This message was sent by Atlassian Jira
(v8.20.10#820010)