[jira] [Updated] (KAFKA-21078) Telemetry reporter pushes metrics whose sensors are disabled by metrics.recording.level

["Steven Schlansker (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steven Schlansker updated KAFKA-21078:
--------------------------------------
    Description: 
{{Sensor.add}} registers every metric with the registry and thus with every {{{}MetricsReporter{}}}, regardless of whether the sensor's {{RecordingLevel}} is enabled by {{{}metrics.recording.level{}}}. Only {{Sensor.record}} checks {{{}shouldRecord{}}}. As a result, an application at INFO still exposes many DEBUG-level metrics with a value of 0 (a {{Rate}} with no samples measures 0, not NaN).

{{KafkaMetricsCollector.collectMetric}} has two filters: the subscription prefix ({{{}MetricsEmitter.shouldEmitMetric{}}}) and "value is a Number". It has no notion of recording level and does not drop zero values. So the KIP-714 push carries every disabled metric with its full label set on every interval, and in delta mode it emits a fresh zero delta for each disabled sum.

For Kafka Streams (KIP-1076) the disabled DEBUG metrics are most of the payload: per-store per-operation rates ({{{}stream-state-metrics{}}}), per-node {{{}process-rate{}}}/{{{}process-total{}}}, and task-level {{{}process-*{}}}. In one production topology at INFO, {{org.apache.kafka.stream.state.*.rate}} alone was several thousand series per stream thread, all zero. This inflates the push toward {{telemetry.max.bytes}} (see the linked bug) and the cardinality of the receiving backend.

{{KafkaMetric}} does not know its sensor's level ({{{}MetricConfig.recordLevel(){}}} is the configured threshold, not the sensor's own level), so the collector cannot filter today. Options:
 * {{Sensor.add}} skips {{registry.registerMetric}} when {{shouldRecord()}} is false. This could be fragile if a user reconfigures the metric level, but also cleans up JMX.
 * {{Metrics}} exposes the sensor level for a metric, and {{KafkaMetricsCollector}} skips metrics whose sensor is disabled.

The first option changes what JMX shows and needs a KIP note. The second is local to the telemetry reporter.

  was:
{\{Sensor.add}} registers every metric with the registry and thus with every \{{MetricsReporter}}, regardless of whether the sensor's \{{RecordingLevel}} is enabled by \{{metrics.recording.level}}. Only \{{Sensor.record}} checks \{{shouldRecord}}. As a result, an application at INFO still exposes every DEBUG-level metric with a value of 0 (a \{{Rate}} with no samples measures 0, not NaN).

{\{KafkaMetricsCollector.collectMetric}} has two filters: the subscription prefix (\{{MetricsEmitter.shouldEmitMetric}}) and "value is a Number". It has no notion of recording level and does not drop zero values. So the KIP-714 push carries every disabled metric with its full label set on every interval, and in delta mode it emits a fresh zero delta for each disabled sum.

For Kafka Streams (KIP-1076) the disabled DEBUG metrics are most of the payload: per-store per-operation rates (\{{stream-state-metrics}}), per-node \{{process-rate}}/\{{process-total}}, and task-level \{{process-\*}}. In one production topology at INFO, \{{org.apache.kafka.stream.state.\*.rate}} alone was several thousand series per stream thread, all zero. This inflates the push toward \{{telemetry.max.bytes}} (see the linked bug) and the cardinality of the receiving backend.

{\{KafkaMetric}} does not know its sensor's level (\{{MetricConfig.recordLevel()}} is the configured threshold, not the sensor's own level), so the collector cannot filter today. Options:

* \{{Sensor.add}} skips \{{registry.registerMetric}} when \{{shouldRecord()}} is false. The level is fixed at construction, so nothing can later enable the sensor. This also cleans up JMX.
* \{{Metrics}} exposes the sensor level for a metric, and \{{KafkaMetricsCollector}} skips metrics whose sensor is disabled.

The first option changes what JMX shows and needs a KIP note. The second is local to the telemetry reporter.


> Telemetry reporter pushes metrics whose sensors are disabled by metrics.recording.level
> ---------------------------------------------------------------------------------------
>
>                 Key: KAFKA-21078
>                 URL: https://issues.apache.org/jira/browse/KAFKA-21078
>             Project: Kafka
>          Issue Type: Improvement
>          Components: clients, streams
>    Affects Versions: 4.3.1
>            Reporter: Steven Schlansker
>            Priority: Minor
>
> {{Sensor.add}} registers every metric with the registry and thus with every {{{}MetricsReporter{}}}, regardless of whether the sensor's {{RecordingLevel}} is enabled by {{{}metrics.recording.level{}}}. Only {{Sensor.record}} checks {{{}shouldRecord{}}}. As a result, an application at INFO still exposes many DEBUG-level metrics with a value of 0 (a {{Rate}} with no samples measures 0, not NaN).
> {{KafkaMetricsCollector.collectMetric}} has two filters: the subscription prefix ({{{}MetricsEmitter.shouldEmitMetric{}}}) and "value is a Number". It has no notion of recording level and does not drop zero values. So the KIP-714 push carries every disabled metric with its full label set on every interval, and in delta mode it emits a fresh zero delta for each disabled sum.
> For Kafka Streams (KIP-1076) the disabled DEBUG metrics are most of the payload: per-store per-operation rates ({{{}stream-state-metrics{}}}), per-node {{{}process-rate{}}}/{{{}process-total{}}}, and task-level {{{}process-*{}}}. In one production topology at INFO, {{org.apache.kafka.stream.state.*.rate}} alone was several thousand series per stream thread, all zero. This inflates the push toward {{telemetry.max.bytes}} (see the linked bug) and the cardinality of the receiving backend.
> {{KafkaMetric}} does not know its sensor's level ({{{}MetricConfig.recordLevel(){}}} is the configured threshold, not the sensor's own level), so the collector cannot filter today. Options:
>  * {{Sensor.add}} skips {{registry.registerMetric}} when {{shouldRecord()}} is false. This could be fragile if a user reconfigures the metric level, but also cleans up JMX.
>  * {{Metrics}} exposes the sensor level for a metric, and {{KafkaMetricsCollector}} skips metrics whose sensor is disabled.
> The first option changes what JMX shows and needs a KIP note. The second is local to the telemetry reporter.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)