You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Fabian Paul (Jira)" <ji...@apache.org> on 2020/03/11 16:23:00 UTC
[jira] [Updated] (FLINK-16553) KafkaFetcher topic/partition metrics
[ https://issues.apache.org/jira/browse/FLINK-16553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Fabian Paul updated FLINK-16553:
--------------------------------
Description:
When using the Kafka universal connector, currently not all KafkaFetcher metrics ([link|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetcherMetricsRegistry.java]) which are exposed through the KafkaConsumer are accessible within the Flink metrics system.
Especially, all metrics which are related to topics and partitions are not available. The KafkaConsumer internally only registers those metrics after it has fetched some records.
Unfortunately, at the moment Flink only checks the available metrics right after the initialization of the KafkaConsumer when no records are polled, yet.
was:
When using the Kafka universal connector, currently not all KafkaFetcher metrics ([link| https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetcherMetricsRegistry.java] which are exposed through the KafkaConsumer are accessible within the Flink metrics system.
Especially, all metrics which are related to topics and partitions are not available. The KafkaConsumer internally only registers those metrics after it has fetched some records.
Unfortunately, at the moment Flink only checks the available metrics right after the initialization of the KafkaConsumer when no records are polled, yet.
> KafkaFetcher topic/partition metrics
> ------------------------------------
>
> Key: FLINK-16553
> URL: https://issues.apache.org/jira/browse/FLINK-16553
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka, Runtime / Metrics
> Reporter: Fabian Paul
> Priority: Major
>
> When using the Kafka universal connector, currently not all KafkaFetcher metrics ([link|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetcherMetricsRegistry.java]) which are exposed through the KafkaConsumer are accessible within the Flink metrics system.
> Especially, all metrics which are related to topics and partitions are not available. The KafkaConsumer internally only registers those metrics after it has fetched some records.
> Unfortunately, at the moment Flink only checks the available metrics right after the initialization of the KafkaConsumer when no records are polled, yet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)