Need for histogram type for Kafka Connect JMX Metrics

[Kanupriya Batra <ka...@adobe.com.INVALID>]

Hi,



We are using kafka-connect framework in our use-case, and would also like to track metrics for connectors/task. But currently Kafka Connect JMX metrics are not supporting histogram types due to which we are not able to plot percentiles on a lot of important metrics like :
kafka_connect_sink_task_metrics_put_batch_avg_time_ms, or kafka_connect_sink_task_metrics_sink_record_read_rate.
I see there is a KIP Ticket (https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework

) tracking the same, but it’s an old one from 2017 and there isn’t any progress after that.
Could you let me know if there are plans to support percentile metrics soon, if not, how can I achieve it from my end.








Thanks,
Kanupriya

Re: Need for histogram type for Kafka Connect JMX Metrics

[mandeep gandhi <we...@gmail.com>]

>So, If the community is interested we can have the same approach in Kafka.
+1

On Thu, Mar 19, 2020 at 12:40 PM Nikolay Izhikov <ni...@apache.org>
wrote:

> Hello, Kanupriya
>
> Recently, I’ve implemented a histogram metric [1] in Apache Ignite.
> My implementation can work on a stream load.
>
> It just counts events that is fall into the predefined intervals(buckets).
> I think histogram can be useful to measure consumer lag or similar metrics.
>
> So, If the community is interested we can have the same approach in Kafka.
>
> [1]
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/metric/impl/HistogramMetricImpl.java
>
> > 18 марта 2020 г., в 13:29, Kanupriya Batra <ka...@adobe.com.INVALID>
> написал(а):
> >
> > Hi,
> >
> >
> >
> > We are using kafka-connect framework in our use-case, and would also
> like to track metrics for connectors/task. But currently Kafka Connect JMX
> metrics are not supporting histogram types due to which we are not able to
> plot percentiles on a lot of important metrics like :
> > kafka_connect_sink_task_metrics_put_batch_avg_time_ms, or
> kafka_connect_sink_task_metrics_sink_record_read_rate.
> > I see there is a KIP Ticket (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework
> >
> > ) tracking the same, but it’s an old one from 2017 and there isn’t any
> progress after that.
> > Could you let me know if there are plans to support percentile metrics
> soon, if not, how can I achieve it from my end.
> >
> >
> >
> >
> >
> >
> >
> >
> > Thanks,
> > Kanupriya
>
>

Re: Need for histogram type for Kafka Connect JMX Metrics

[Nikolay Izhikov <ni...@apache.org>]

Hello, Kanupriya

Recently, I’ve implemented a histogram metric [1] in Apache Ignite.
My implementation can work on a stream load.

It just counts events that is fall into the predefined intervals(buckets).
I think histogram can be useful to measure consumer lag or similar metrics.

So, If the community is interested we can have the same approach in Kafka.

[1] https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/metric/impl/HistogramMetricImpl.java

> 18 марта 2020 г., в 13:29, Kanupriya Batra <ka...@adobe.com.INVALID> написал(а):
> 
> Hi,
> 
> 
> 
> We are using kafka-connect framework in our use-case, and would also like to track metrics for connectors/task. But currently Kafka Connect JMX metrics are not supporting histogram types due to which we are not able to plot percentiles on a lot of important metrics like :
> kafka_connect_sink_task_metrics_put_batch_avg_time_ms, or kafka_connect_sink_task_metrics_sink_record_read_rate.
> I see there is a KIP Ticket (https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework
> 
> ) tracking the same, but it’s an old one from 2017 and there isn’t any progress after that.
> Could you let me know if there are plans to support percentile metrics soon, if not, how can I achieve it from my end.
> 
> 
> 
> 
> 
> 
> 
> 
> Thanks,
> Kanupriya