[jira] [Updated] (KAFKA-19805) Add acks dimension to BrokerTopicMetrics for produce requests

["伍学明 (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-19805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

伍学明 updated KAFKA-19805:
------------------------
    Fix Version/s:     (was: 4.1.0)

> Add acks dimension to BrokerTopicMetrics for produce requests
> -------------------------------------------------------------
>
>                 Key: KAFKA-19805
>                 URL: https://issues.apache.org/jira/browse/KAFKA-19805
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 4.1.0
>            Reporter: 伍学明
>            Assignee: 伍学明
>            Priority: Major
>
> h3. *Title*
> Add {{acks}} dimension to {{BrokerTopicMetrics}} to measure per-acks produce performance impact
> ----
> h3. *Summary*
> Currently, Kafka’s {{BrokerTopicMetrics}} only tracks produce metrics such as {{{}BytesInPerSec{}}}, {{{}MessagesInPerSec{}}}, and {{ProduceRequestsPerSec}} at the topic level. However, these metrics do not distinguish between different producer acknowledgment ({{{}acks{}}}) configurations ({{{}acks=0{}}}, {{{}acks=1{}}}, {{{}acks=-1{}}}).
> In high-throughput environments, different {{acks}} levels have significantly different impacts on broker CPU, I/O, and network utilization.
> This proposal introduces an additional {{acks}} label to existing topic-level metrics, enabling more granular visibility into broker performance under various producer reliability modes.
> ----
> h3. *Motivation*
> The current aggregated produce metrics make it difficult to assess the performance and stability implications of different {{acks}} settings on brokers.
> For example, asynchronous ({{{}acks=0{}}}) and fully acknowledged ({{{}acks=-1{}}}) produces can have very different effects on disk I/O, request queues, and replication latency, but these effects are hidden in current metrics.
> By introducing an {{acks}} dimension, operators and performance engineers can:
>  * Quantify the resource cost of different producer acknowledgment strategies.
>  * Analyze how {{acks}} configuration affects cluster throughput, replication load, and latency.
>  * Perform fine-grained benchmarking and capacity planning.
> ----
> h3. *Proposed Changes*
>  # *Extend {{BrokerTopicStats}}*
> Add a new {{perTopicAcksStats}} structure to track metrics per {{(topic, acks)}} combination:
> {{val perTopicAcksStats = new Pool[(String, Short), BrokerTopicMetrics](
>   Some((key) => new BrokerTopicMetrics(Some(s"${key._1},ack=${key._2}")))
> )}}
>  # *Instrument Produce Handling*
> In {{{}KafkaApis.handleProduceRequest{}}}, extract the producer {{acks}} value and record metrics accordingly:
> {{val ackVal = produceRequest.acks()
> brokerTopicStats.topicStats(topic).bytesInRate.mark(bytes)
> brokerTopicStats.topicAcksStats(topic, ackVal).bytesInRate.mark(bytes)}}
> The same logic applies to:
>  ** {{messagesInRate}}
>  ** {{produceRequestsRate}}
>  # *Automatic Metric Naming*
> Since {{BrokerTopicMetrics}} extends {{{}KafkaMetricsGroup{}}}, the new label will automatically generate JMX metrics like:
> {{kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=perf-test,ack=-1
> kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=perf-test,ack=1
> kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=perf-test,ack=0}}
>  # *Performance Considerations*
>  ** {{perTopicAcksStats}} uses lazy initialization and caching via {{Pool}} to avoid excessive metric object creation.
>  ** Expiration or cleanup logic can be added for inactive metrics.
> ----
> h3. *Example Metrics Output*
>  
> {{kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=perf-test,ack=0
> kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=perf-test,ack=1
> kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=perf-test,ack=-1}}
> ----
> h3. *Compatibility & Impact*
>  * No breaking changes to existing metrics.
>  * Existing metric names and topic-level aggregation remain unaffected.
>  * New metrics are additive and optional.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)