You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "伍学明 (Jira)" <ji...@apache.org> on 2025/10/17 05:55:00 UTC

[jira] [Created] (KAFKA-19805) Add acks dimension to BrokerTopicMetrics for produce requests

伍学明 created KAFKA-19805:
---------------------------

             Summary: Add acks dimension to BrokerTopicMetrics for produce requests
                 Key: KAFKA-19805
                 URL: https://issues.apache.org/jira/browse/KAFKA-19805
             Project: Kafka
          Issue Type: Improvement
          Components: core
    Affects Versions: 4.1.0
            Reporter: 伍学明
            Assignee: 伍学明
             Fix For: 4.1.0


h3. *Title*

Add {{acks}} dimension to {{BrokerTopicMetrics}} to measure per-acks produce performance impact
----
h3. *Summary*

Currently, Kafka’s {{BrokerTopicMetrics}} only tracks produce metrics such as {{{}BytesInPerSec{}}}, {{{}MessagesInPerSec{}}}, and {{ProduceRequestsPerSec}} at the topic level. However, these metrics do not distinguish between different producer acknowledgment ({{{}acks{}}}) configurations ({{{}acks=0{}}}, {{{}acks=1{}}}, {{{}acks=-1{}}}).
In high-throughput environments, different {{acks}} levels have significantly different impacts on broker CPU, I/O, and network utilization.
This proposal introduces an additional {{acks}} label to existing topic-level metrics, enabling more granular visibility into broker performance under various producer reliability modes.
----
h3. *Motivation*

The current aggregated produce metrics make it difficult to assess the performance and stability implications of different {{acks}} settings on brokers.
For example, asynchronous ({{{}acks=0{}}}) and fully acknowledged ({{{}acks=-1{}}}) produces can have very different effects on disk I/O, request queues, and replication latency, but these effects are hidden in current metrics.

By introducing an {{acks}} dimension, operators and performance engineers can:
 * Quantify the resource cost of different producer acknowledgment strategies.

 * Analyze how {{acks}} configuration affects cluster throughput, replication load, and latency.

 * Perform fine-grained benchmarking and capacity planning.

----
h3. *Proposed Changes*
 # *Extend {{BrokerTopicStats}}*
Add a new {{perTopicAcksStats}} structure to track metrics per {{(topic, acks)}} combination:

{{val perTopicAcksStats = new Pool[(String, Short), BrokerTopicMetrics](
  Some((key) => new BrokerTopicMetrics(Some(s"${key._1},ack=${key._2}")))
)}}
 # *Instrument Produce Handling*
In {{{}KafkaApis.handleProduceRequest{}}}, extract the producer {{acks}} value and record metrics accordingly:

{{val ackVal = produceRequest.acks()
brokerTopicStats.topicStats(topic).bytesInRate.mark(bytes)
brokerTopicStats.topicAcksStats(topic, ackVal).bytesInRate.mark(bytes)}}
The same logic applies to:

 ** {{messagesInRate}}

 ** {{produceRequestsRate}}

 # *Automatic Metric Naming*
Since {{BrokerTopicMetrics}} extends {{{}KafkaMetricsGroup{}}}, the new label will automatically generate JMX metrics like:

{{kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=perf-test,ack=-1
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=perf-test,ack=1
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=perf-test,ack=0}}
 # *Performance Considerations*

 ** {{perTopicAcksStats}} uses lazy initialization and caching via {{Pool}} to avoid excessive metric object creation.

 ** Expiration or cleanup logic can be added for inactive metrics.

----
h3. *Example Metrics Output*

 

{{kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=perf-test,ack=0
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=perf-test,ack=1
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=perf-test,ack=-1}}
----
h3. *Compatibility & Impact*
 * No breaking changes to existing metrics.

 * Existing metric names and topic-level aggregation remain unaffected.

 * New metrics are additive and optional.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)