[GitHub] [flink] Draczech commented on issue #7820: [FLINK-11742][Metrics]Push metrics to Pushgateway without "instance"

[GitBox <gi...@apache.org>]

Draczech commented on issue #7820: [FLINK-11742][Metrics]Push metrics to Pushgateway without "instance"
URL: https://github.com/apache/flink/pull/7820#issuecomment-579782620
 
 
   Hi, has there been any progress on this topic? Correct me if I am wrong, as my understanding of flink is a bit worse than understanding of Prometheus...
   
   ### Current setup
   Multiple flink clusters, each running multiple scheduled flink jobs (with multiple task managers). I am using flink **1.9.1**. When I set `metrics.reporter.promgateway.jobName=myJob` and `metrics.reporter.promgateway.randomJobNameSuffix: false`, metrics are always replaced in push GW by the last updating machine as job manager and all task managers using the same `job` label and empty `instance` label. Metrics are pushed from different hosts, but the last push overwrites any previous data.
   
   If I use `metrics.reporter.promgateway.randomJobNameSuffix: true` every host pushes metrics with generated `job` label. So no metrics are overwritten in the GW. The problem is that there is no way how to clean the GW as each run of the flink job spams new set of metrics. If I set `metrics.reporter.promgateway.deleteOnShutdown: false` then only jobmanager metrics are removed from the GW once the job finishes. But taskmanager metrics are left there forever.
   
   So I have not yet found any configuration which would work for me without problems.
   
   ### Ideal situation
   The best case for prometheus labels would be `job`==flink job name, `instance`=ID of container, generated in deterministic manner, e.g. `jm1`, `tm1`-`tm9`, etc.
   
   This way, there wouldn't be any conflicts in push gateway during single run of the flink job. Cleaning of metrics would be easier as well.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services