Service discovery for Prometheus on YARN

[Dongwon Kim <ea...@gmail.com>]

Hi community,

Prior to my presentation [1], I'd like to share a Python script [2] to discover Flink clusters on YARN and let Prometheus know via its file-based service discovery mechanism [3].

Prometheus needs to pull metrics from Prometheus exporters running inside TaskManagers.
The problem is that, as also discussed in [4], we cannot know in advance hosts and ports to which Prometheus exporters are bound.
As Chesnay mentioned in [4], unless the information as to which the exporters are bound is exposed by Flink's REST APIs, the only way is to manually scrape logs of JM and TMs.
Then you need to let Prometheus know where to pull metrics from.

The script is intended to automate the manual process.
Please visit the git repository for the detailed information.

Best,

- Dongwon

[1] https://berlin-2018.flink-forward.org/conference-program/#real-time-driving-score-service-using-flink <https://berlin-2018.flink-forward.org/conference-program/#real-time-driving-score-service-using-flink>
[2] https://github.com/eastcirclek/flink-service-discovery <https://github.com/eastcirclek/flink-service-discovery>
[3] https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cfile_sd_config%3E <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#<file_sd_config>>
[4] http://mail-archives.apache.org/mod_mbox/flink-user/201801.mbox/%3Cb9b8868a-3026-689a-941f-ac20b42e6612@apache.org%3E <http://mail-archives.apache.org/mod_mbox/flink-user/201801.mbox/%3Cb9b8868a-3026-689a-941f-ac20b42e6612@apache.org%3E>