[jira] [Comment Edited] (KAFKA-16129) Add integration test for KIP-977

["Jonny Vu Huy Mai (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-16129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18071102#comment-18071102 ] 

Jonny Vu Huy Mai edited comment on KAFKA-16129 at 4/4/26 6:57 PM:
------------------------------------------------------------------

Proposed solution: treat {{KAFKA-16129}} as blocked or rescope it to depend on the missing KIP-977 runtime baseline, rather than trying to land it as a tests-only task on the current tree. The current {{trunk}} checkout still does not include {{{}metrics.verbosity{}}}, the related dynamic broker-config wiring, or partition-level {{{}BrokerTopicMetrics{}}}, so there is no stable runtime surface to write meaningful integration coverage against yet. If maintainers want this issue to move forward now, the scope likely needs to expand to include that baseline work, or the issue should be narrowed explicitly to an agreed first slice such as JMX-only validation after the runtime support exists.

Questions:
 * Can {{KAFKA-16129}} include the missing KIP-977 runtime baseline, or should it remain strictly an integration-tests issue?
 * If phased delivery is acceptable, is a JMX-first landing enough for initial closure, or are Prometheus/OpenTelemetry checks required too?
 * Which KIP-977 semantics should be treated as authoritative for the first implementation: 3 metrics vs 10, {{name}} vs {{{}names{}}}, empty partition tag vs filtered partition tag, and restart-persistence behavior?
 * Should PR {{#20397}} be revived as prior art, or is a fresh implementation preferred?

 

If the proposal seems acceptable, I want to be assigned this issue


was (Author: JIRAUSER312845):
Proposed solution: treat {{KAFKA-16129}} as blocked or rescope it to depend on the missing KIP-977 runtime baseline, rather than trying to land it as a tests-only task on the current tree. The current {{trunk}} checkout still does not include {{{}metrics.verbosity{}}}, the related dynamic broker-config wiring, or partition-level {{{}BrokerTopicMetrics{}}}, so there is no stable runtime surface to write meaningful integration coverage against yet. If maintainers want this issue to move forward now, the scope likely needs to expand to include that baseline work, or the issue should be narrowed explicitly to an agreed first slice such as JMX-only validation after the runtime support exists.

Questions:
 * Can {{KAFKA-16129}} include the missing KIP-977 runtime baseline, or should it remain strictly an integration-tests issue?
 * If phased delivery is acceptable, is a JMX-first landing enough for initial closure, or are Prometheus/OpenTelemetry checks required too?
 * Which KIP-977 semantics should be treated as authoritative for the first implementation: 3 metrics vs 10, {{name}} vs {{{}names{}}}, empty partition tag vs filtered partition tag, and restart-persistence behavior?
 * Should PR {{#20397}} be revived as prior art, or is a fresh implementation preferred?

> Add integration test for KIP-977
> --------------------------------
>
>                 Key: KAFKA-16129
>                 URL: https://issues.apache.org/jira/browse/KAFKA-16129
>             Project: Kafka
>          Issue Type: Task
>            Reporter: Qichao Chu
>            Priority: Critical
>
> *{{metrics.verbosity}}* will be a new dynamic config introduced to control the verbosity(fan-out rate) of the metrics. It's a config with JSON format specifying the condition controlling fan-out of the metrics. If the value *{{high}}* is set for the *{{level}}* key of the configured JSON(see below for example values), high fan-out tags(e.g. {*}{{partition}}{*})will be added to metrics specified by the {{*name*}} filter and will apply to all the topics that meet the conditions in the {{*filters*}} section. In the *{{low}}* settings, these tags will be assigned with an empty value. We elected to make it central so that this implementation can be generalized in the future either into a library, or allow other means for centralized control.
> More details: https://cwiki.apache.org/confluence/display/KAFKA/KIP-977%3A+Partition-Level+Throughput+Metrics
>  
> The following 3 tests will be done for common metrics collectors: JMX, Prometheus, and OpenTelemetry.
>  # The partition tag can be observed from metrics if high verbosity is used
>  # The partition tag should result in an empty string or be filtered out by the metrics collector if default verbosity is used
>  # Dynamically setting the verbosity can result in the behavior defined in the above tests



--
This message was sent by Atlassian Jira
(v8.20.10#820010)