Re: Flink KafkaConsumer metrics in DataDog

[Debraj Manna <su...@gmail.com>]

We disabled kafka consumer metrics in flink via setting
register.consumer.metrics to false in KafkaSource and enabled
kafka consumer integration in datadog agent as explained here
<https://www.datadoghq.com/blog/monitor-kafka-with-datadog/#configure-the-agent>
.

On Fri, Aug 27, 2021 at 8:44 AM Debraj Manna <su...@gmail.com>
wrote:

> Yes we are also facing the same problem and not able to find any solution.
>
> On Thu, Aug 26, 2021 at 5:59 PM Chesnay Schepler <ch...@apache.org>
> wrote:
>
>> AFAIK this metric is directly forwarded from Kafka as-is; so Flink isn't
>> calculating anything.
>>
>> I suggest to reach out to the Kafka folks.
>>
>> On 25/08/2021 17:23, Shilpa Shankar wrote:
>>
>> Hello ,
>>
>> We have enabled DataDogHTTPReporter to fetch metrics on flink v1.13.1
>> running on kubernetes. The metric flink.operator.KafkaConsumer.records_lag_max
>> is not displaying accurate values. It also displays 0 most of the time and
>> when it does fetch a value, it seems to be wrong when I compare them with
>> the Kafka lag broker metrics.
>> Could you please let us know how these metrics are calculated? Are there
>> any configuration changes that need to be made to support the Kafka
>> Consumer metrics?
>>
>> # Datadog Integration
>> metrics.reporter.dghttp.class:
>> org.apache.flink.metrics.datadog.DatadogHttpReporter
>> metrics.reporter.dghttp.apikey: xxxx
>> metrics.reporter.dghttp.maxMetricsPerRequest: 1000
>> metrics.reporter.dghttp.tags:
>> flink-cluster:flink-noc-cluster,data-center:lab
>>
>> metrics.scope.jm: flink.jobmanager
>> metrics.scope.jm.job: flink.jobmanager.job
>> metrics.scope.tm: flink.taskmanager
>> metrics.scope.tm.job: flink.taskmanager.job
>> metrics.scope.task: flink.task
>>
>> Thanks,
>> Shilpa
>>
>>
>>