[jira] [Commented] (KAFKA-15192) Network thread receives exception when updating request metrics

["Divij Vaidya (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-15192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17744169#comment-17744169 ] 

Divij Vaidya commented on KAFKA-15192:
--------------------------------------

[~hudeqi] [~manyanda] Upgrading the metrics library is a major thing that we want to do for Apache Kafka for a long time. Please consider picking up (not just this ticket but upgrading the library itself, including the KIP).

This old metrics library is cause of many pains in Apache Kafka. The reason we haven't been able to upgrade it is because the newer versions are backward in-compatible. [~christo_lolov] did some digging and has some potential ideas on how to make it backward compatible. 

Another example is the ~4-5% CPU utilization by the histogram method used by this old library. See Processor#upgradeRequestMetrics in this flamegraph: [^latency-cpu.html]

> Network thread receives exception when updating request metrics
> ---------------------------------------------------------------
>
>                 Key: KAFKA-15192
>                 URL: https://issues.apache.org/jira/browse/KAFKA-15192
>             Project: Kafka
>          Issue Type: Bug
>          Components: metrics
>            Reporter: David Mao
>            Priority: Major
>         Attachments: latency-cpu.html
>
>
> We noticed an exception being thrown from the network threads when updating some of the request histograms. Example stack trace:
>  
> java.util.NoSuchElementException
> 	at java.util.concurrent.ConcurrentSkipListMap.firstKey(ConcurrentSkipListMap.java:2064)
> 	at com.yammer.metrics.stats.ExponentiallyDecayingSample.update(ExponentiallyDecayingSample.java:102)
> 	at com.yammer.metrics.stats.ExponentiallyDecayingSample.update(ExponentiallyDecayingSample.java:81)
> 	at com.yammer.metrics.core.Histogram.update(Histogram.java:110)
>  
> Searching the error I found a similar ticket resolved in Cassandra by updating their dropwizard dependency to pull in [https://github.com/dropwizard/metrics/pull/1436]. https://issues.apache.org/jira/browse/CASSANDRA-15472
> Kafka currently still uses yammer metrics, so we would need to take [https://cwiki.apache.org/confluence/display/KAFKA/KIP-510%3A+Metrics+library+upgrade] forward to upgrade to a dropwizard version that fixes this issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)