How to get the metrics details for streaming application.

[jitendra sharma <ji...@gmail.com>]

Hi team,

I have a streaming application in beam, i'm using counter to calculate the
number of records processed.Currently I need to send the counter metrics to
Datadog.

If it is a non-streaming application we can use metric query results to
send the metrics as it is a streaming application can't find a way for
this.Could anyone provide some suggestions?

Regards,
Jitendra Sharma

Re: How to get the metrics details for streaming application.

[Lars Almgren Schwartz <la...@tink.com>]

Hey,

It kind of depends on which runner you're using. We were using SparkRunner
and used the Spark GraphiteSink in combination with MetricsGraphiteSink (
https://github.com/apache/beam/pull/6569) to get metrics and then we pushed
those metrics to an exporter which was scraped by our Prometheus. No we're
using Flink, but I have not been able to setup something similar yet.

Regards,
Lars

On Thu, Feb 27, 2020 at 6:23 AM jitendra sharma <ji...@gmail.com>
wrote:

> Hi team,
>
> I have a streaming application in beam, i'm using counter to calculate the
> number of records processed.Currently I need to send the counter metrics to
> Datadog.
>
> If it is a non-streaming application we can use metric query results to
> send the metrics as it is a streaming application can't find a way for
> this.Could anyone provide some suggestions?
>
> Regards,
> Jitendra Sharma
>