Flink Count of Events using metric

[aj <aj...@gmail.com>]

Please help me with this:

https://stackoverflow.com/questions/62297467/flink-count-of-events-using-metric

I have a topic in Kafka where I am getting multiple types of events in JSON
format. I have created a file stream sink to write these events to S3 with
bucketing.

Now I want to publish an hourly count of each event as metrics to
Prometheus and publish a grafana dashboard over that.

So please help me how can I achieve hourly count for each event using Flink
metrics and publish to Prometheus.


-- 
Thanks & Regards,
Anuj Jain



<http://www.cse.iitm.ac.in/%7Eanujjain/>

Re: [EXTERNAL] Flink Count of Events using metric

["Slotterback, Chris" <Ch...@comcast.com>]

As the answer on SO suggests, Prometheus comes with lots of functionality to do what you’re requesting using just a simple count metric:

https://prometheus.io/docs/prometheus/latest/querying/functions/

If you want to implement the function on your own inside flink, you can make your own metrics, and add them to the metric group, and use the metrics inside rich functions. For example:

getRuntimeContext().getMetricGroup().addGroup(YOUR_METRIC_GROUP_NAME)
This will return you a org.apache.flink.metrics.MetricGroup. You can then use this MetricGroup to attach Metrics, such as counters, gauges (in the SO suggestion), and histograms. You can then interact with these Metrics, and they will expose your new metrics to your Prometheus exporter, for example.

Chris

From: aj <aj...@gmail.com>
Date: Tuesday, June 16, 2020 at 1:31 PM
To: user <us...@flink.apache.org>
Subject: [EXTERNAL] Flink Count of Events using metric

Please help me with this:

https://stackoverflow.com/questions/62297467/flink-count-of-events-using-metric<https://urldefense.com/v3/__https:/stackoverflow.com/questions/62297467/flink-count-of-events-using-metric__;!!CQl3mcHX2A!TRrPwe9O-e1gS06ETkdnp8Fxk1KmQO7N1ctDELZh5OeArodlxrTReVv4ZPHh5olSv7Sq5Pk$>

I have a topic in Kafka where I am getting multiple types of events in JSON format. I have created a file stream sink to write these events to S3 with bucketing.

Now I want to publish an hourly count of each event as metrics to Prometheus and publish a grafana dashboard over that.

So please help me how can I achieve hourly count for each event using Flink metrics and publish to Prometheus.


--
Thanks & Regards,
Anuj Jain