You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Vijay Balakrishnan <bv...@gmail.com> on 2019/07/15 16:01:36 UTC

Converting Metrics from a Reporter to a Custom Events mapping

Hi,
I need to capture the Metrics sent from a Flink app to a Reporter and
transform them to an Events API format I have designed. I have been looking
at the Reporters(
https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#list-of-all-variables)
and have used them but what would be a best practice to capture this
metrics data to transform it ?

The folks using the Flink app still want to see their metrics in the Flink
Dashboard using their chosen(not sure yet what they chose-assuming
ConsoleReporter) Reporter. I need to capture those metrics, transform them
to my Events API format and send it to a Kinesis Stream.

We use Prometheus and InfluxDB in our environments for other purposes.

Should I use the SLF4J Reporter to dump the metrics into a log file/folder
and watch that with a Kinesis Agent and transform it somehow(?) and then
send it to the Kinesis data stream ?

TIA,

Re: Converting Metrics from a Reporter to a Custom Events mapping

Posted by Vijay Balakrishnan <bv...@gmail.com>.

Thanks for all your replies.Ended up using a StatsdReporter with Flink and
building a statsd plugin to transform the data to my required output format
and dump it into a folder that the Kinesis agent can then pick up.

On Tue, Jul 16, 2019 at 2:16 AM Chesnay Schepler <ch...@apache.org> wrote:

> You can configure multiple reporters, so just configured both the reporter
> that the app users want and your own which does the transformation and
> sending to kinesis.
>
> On 16/07/2019 09:37, Haibo Sun wrote:
>
> Hi,  Vijay
>
> Or can you implement a Reporter that transforms the metrics and sends them
> directly to a Kinesis Stream?
>
> Best,
> Haibo
>
> At 2019-07-16 00:01:36, "Vijay Balakrishnan" <bv...@gmail.com>
> <bv...@gmail.com> wrote:
>
> Hi,
> I need to capture the Metrics sent from a Flink app to a Reporter and
> transform them to an Events API format I have designed. I have been looking
> at the Reporters(
> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#list-of-all-variables)
> and have used them but what would be a best practice to capture this
> metrics data to transform it ?
>
> The folks using the Flink app still want to see their metrics in the Flink
> Dashboard using their chosen(not sure yet what they chose-assuming
> ConsoleReporter) Reporter. I need to capture those metrics, transform them
> to my Events API format and send it to a Kinesis Stream.
>
> We use Prometheus and InfluxDB in our environments for other purposes.
>
> Should I use the SLF4J Reporter to dump the metrics into a log file/folder
> and watch that with a Kinesis Agent and transform it somehow(?) and then
> send it to the Kinesis data stream ?
>
> TIA,
>
>
>

Re: Converting Metrics from a Reporter to a Custom Events mapping

Posted by Chesnay Schepler <ch...@apache.org>.

You can configure multiple reporters, so just configured both the 
reporter that the app users want and your own which does the 
transformation and sending to kinesis.

On 16/07/2019 09:37, Haibo Sun wrote:
> Hi, Vijay
>
> Or can you implement a Reporter that transforms the metrics and sends 
> them directly to a Kinesis Stream?
>
> Best,
> Haibo
>
> At 2019-07-16 00:01:36, "Vijay Balakrishnan" <bv...@gmail.com> wrote:
>
>     Hi,
>     I need to capture the Metrics sent from a Flink app to a Reporter
>     and transform them to an Events API format I have designed. I have
>     been looking at the
>     Reporters(https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#list-of-all-variables)
>     and have used them but what would be a best practice to capture
>     this metrics data to transform it ?
>
>     The folks using the Flink app still want to see their metrics in
>     the Flink Dashboard using their chosen(not sure yet what they
>     chose-assuming ConsoleReporter) Reporter. I need to capture those
>     metrics, transform them to my Events API format and send it to a
>     Kinesis Stream.
>
>     We use Prometheus and InfluxDB in our environments for other purposes.
>
>     Should I use the SLF4J Reporter to dump the metrics into a log
>     file/folder and watch that with a Kinesis Agent and transform it
>     somehow(?) and then send it to the Kinesis data stream ?
>
>     TIA,
>

Re:Converting Metrics from a Reporter to a Custom Events mapping

Posted by Haibo Sun <su...@163.com>.

Hi,  Vijay


Or can you implement a Reporter that transforms the metrics and sends them directly to a Kinesis Stream?


Best,
Haibo

At 2019-07-16 00:01:36, "Vijay Balakrishnan" <bv...@gmail.com> wrote:

Hi,

I need to capture the Metrics sent from a Flink app to a Reporter and transform them to an Events API format I have designed. I have been looking at the Reporters(https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#list-of-all-variables) and have used them but what would be a best practice to capture this metrics data to transform it ?


The folks using the Flink app still want to see their metrics in the Flink Dashboard using their chosen(not sure yet what they chose-assuming ConsoleReporter) Reporter. I need to capture those metrics, transform them to my Events API format and send it to a Kinesis Stream.


We use Prometheus and InfluxDB in our environments for other purposes.


Should I use the SLF4J Reporter to dump the metrics into a log file/folder and watch that with a Kinesis Agent and transform it somehow(?) and then send it to the Kinesis data stream ?


TIA,