You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "伍学明 (Jira)" <ji...@apache.org> on 2025/10/17 05:55:00 UTC
[jira] [Created] (KAFKA-19805) Add acks dimension to BrokerTopicMetrics for produce requests
伍学明 created KAFKA-19805:
---------------------------
Summary: Add acks dimension to BrokerTopicMetrics for produce requests
Key: KAFKA-19805
URL: https://issues.apache.org/jira/browse/KAFKA-19805
Project: Kafka
Issue Type: Improvement
Components: core
Affects Versions: 4.1.0
Reporter: 伍学明
Assignee: 伍学明
Fix For: 4.1.0
h3. *Title*
Add {{acks}} dimension to {{BrokerTopicMetrics}} to measure per-acks produce performance impact
----
h3. *Summary*
Currently, Kafka’s {{BrokerTopicMetrics}} only tracks produce metrics such as {{{}BytesInPerSec{}}}, {{{}MessagesInPerSec{}}}, and {{ProduceRequestsPerSec}} at the topic level. However, these metrics do not distinguish between different producer acknowledgment ({{{}acks{}}}) configurations ({{{}acks=0{}}}, {{{}acks=1{}}}, {{{}acks=-1{}}}).
In high-throughput environments, different {{acks}} levels have significantly different impacts on broker CPU, I/O, and network utilization.
This proposal introduces an additional {{acks}} label to existing topic-level metrics, enabling more granular visibility into broker performance under various producer reliability modes.
----
h3. *Motivation*
The current aggregated produce metrics make it difficult to assess the performance and stability implications of different {{acks}} settings on brokers.
For example, asynchronous ({{{}acks=0{}}}) and fully acknowledged ({{{}acks=-1{}}}) produces can have very different effects on disk I/O, request queues, and replication latency, but these effects are hidden in current metrics.
By introducing an {{acks}} dimension, operators and performance engineers can:
* Quantify the resource cost of different producer acknowledgment strategies.
* Analyze how {{acks}} configuration affects cluster throughput, replication load, and latency.
* Perform fine-grained benchmarking and capacity planning.
----
h3. *Proposed Changes*
# *Extend {{BrokerTopicStats}}*
Add a new {{perTopicAcksStats}} structure to track metrics per {{(topic, acks)}} combination:
{{val perTopicAcksStats = new Pool[(String, Short), BrokerTopicMetrics](
Some((key) => new BrokerTopicMetrics(Some(s"${key._1},ack=${key._2}")))
)}}
# *Instrument Produce Handling*
In {{{}KafkaApis.handleProduceRequest{}}}, extract the producer {{acks}} value and record metrics accordingly:
{{val ackVal = produceRequest.acks()
brokerTopicStats.topicStats(topic).bytesInRate.mark(bytes)
brokerTopicStats.topicAcksStats(topic, ackVal).bytesInRate.mark(bytes)}}
The same logic applies to:
** {{messagesInRate}}
** {{produceRequestsRate}}
# *Automatic Metric Naming*
Since {{BrokerTopicMetrics}} extends {{{}KafkaMetricsGroup{}}}, the new label will automatically generate JMX metrics like:
{{kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=perf-test,ack=-1
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=perf-test,ack=1
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=perf-test,ack=0}}
# *Performance Considerations*
** {{perTopicAcksStats}} uses lazy initialization and caching via {{Pool}} to avoid excessive metric object creation.
** Expiration or cleanup logic can be added for inactive metrics.
----
h3. *Example Metrics Output*
{{kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=perf-test,ack=0
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=perf-test,ack=1
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=perf-test,ack=-1}}
----
h3. *Compatibility & Impact*
* No breaking changes to existing metrics.
* Existing metric names and topic-level aggregation remain unaffected.
* New metrics are additive and optional.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)