Re: [PR] Optional removal of metrics from Prometheus PushGateway on shutdown (druid)

["abhishekrb19 (via GitHub)" <gi...@apache.org>]

abhishekrb19 commented on code in PR #14935:
URL: https://github.com/apache/druid/pull/14935#discussion_r1402566808


##########
extensions-contrib/prometheus-emitter/src/main/java/org/apache/druid/emitter/prometheus/PrometheusEmitter.java:
##########
@@ -207,6 +207,15 @@ public void close()
     } else {
       exec.shutdownNow();
       flush();
+
+      if (pushGateway != null && config.isDeleteOnShutdown()) {
+        try {
+          pushGateway.delete(config.getNamespace(), ImmutableMap.of(config.getNamespace(), identifier));

Review Comment:
   Sorry for the delayed review! :( 
   
   Thinking more, the missing metrics problem exists with the Prometheus `exporter` strategy (the pull model) - i.e., when the Druid servers are shutting down, and Prometheus hasn't scraped the metrics yet on time.
   
   I think what is proposed here suffices, and @BartMiki and I discussed this offline. In addition to the patch, he suggested adding a delay config, something like `waitForShutdownDelay`, so the peon tasks will wait for the configured time before the delete is issued at the time of shutdown, which seems reasonable.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org