Re: [External] : How to visualize the results of Flink processing or aggregation?

[Fuyao Li <fu...@oracle.com>]

Hello Xiong,

You can expose monitors through Metric system of Flink.
https://ci.apache.org/projects/flink/flink-docs-stable/ops/metrics.html
Metrics can be exposed by metric reporter:
https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/metric_reporters.html
That includes Prometheus.

For the DataStream API, you can build customized metrics.
For the Flink SQL/Table API, you can only use the listed predefined metrics in Flink. There could be work arounds ways, but no direct way to supply Flink SQL customized metrics.

Best regards,
Fuyao

From: Xiong Qiang <wi...@gmail.com>
Date: Friday, March 26, 2021 at 09:15
To: user@flink.apache.org <us...@flink.apache.org>
Subject: [External] : How to visualize the results of Flink processing or aggregation?
Hi All,

I am new to Flink, so forgive me if it is a naive question.

The context is:
We have a data streaming coming in, and we will use Flink applications to do the processing or aggregations. After the processing or aggregation, we need some approaches to visualize the results, to either build a dashboard or setup alerts, for example, using Prometheus and Grafana.
However, after reading the documents (https://flink.apache.org/flink-architecture.html<https://urldefense.com/v3/__https:/flink.apache.org/flink-architecture.html__;!!GqivPVa7Brio!I_F5ufA_gXteug2Rt61sNT_ZisoJzyVoNxCGhTeaerx7AjyZq2DB6luYzsv-qn4$> and more links) and examples (https://ci.apache.org/projects/flink/flink-docs-release-1.12/try-flink/datastream_api.html<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/try-flink/datastream_api.html__;!!GqivPVa7Brio!I_F5ufA_gXteug2Rt61sNT_ZisoJzyVoNxCGhTeaerx7AjyZq2DB6luYTyZXJuA$>) (https://github.com/ververica/flink-training/blob/master/long-ride-alerts/src/solution/java/org/apache/flink/training/solutions/longrides/LongRidesSolution.java<https://urldefense.com/v3/__https:/github.com/ververica/flink-training/blob/master/long-ride-alerts/src/solution/java/org/apache/flink/training/solutions/longrides/LongRidesSolution.java__;!!GqivPVa7Brio!I_F5ufA_gXteug2Rt61sNT_ZisoJzyVoNxCGhTeaerx7AjyZq2DB6luYPVveCAs$>) , I am still not able to close the gap between Flink and a monitoring/dashboard tool, e.g. Prometheus/Grafana.

The question is:
How are processing results connected/sinked from Flink to Prometheus/Grafana? for example, in the fraud detection example, how is the account id = 3, send to Prometheus and Grafana, so that I have a dashboard showing there is one suspected account? In the taxi long rides example, how do I send the count of long rides from Flink to Prometheus/Grafana?

I understand there are sinks (https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/connectors/<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/dev/connectors/__;!!GqivPVa7Brio!I_F5ufA_gXteug2Rt61sNT_ZisoJzyVoNxCGhTeaerx7AjyZq2DB6luYGAHDnS8$>). However, I didn't see sinks for Prometheus.

Hope I made my question clear.

Thanks