[GitHub] [druid] divick commented on issue #12885: prometheus emitter extension doesn't emit any metrics

[GitBox <gi...@apache.org>]

divick commented on issue #12885:
URL: https://github.com/apache/druid/issues/12885#issuecomment-1273460452

   @holyachon To make it easier to debug and change configuration I have provisioned druid in k8s using druid-operator instead of logging to each server separately (master, data, query) and changing and restarting them. With that I can see the metrics __are__ getting emitted. I am guessing that may be it was not emitted earlier because metricsDimensions.json was not specified? Anyhow given now I can see some metrics being emitted but now I see other issues that it complain of unmapped metrics:
   
   ```
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/timeout/count]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/threads/started]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/threads/finished]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/threads/live]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/threads/liveDaemon]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/threads/livePeak]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/max]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/capacity]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/used]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/init]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/max]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/capacity]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/used]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/init]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/max]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/capacity]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/used]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/init]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/max]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/capacity]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/used]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/gc/mem/init]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/heapAlloc/bytes]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [avatica/server/AvaticaProtobufHandler/Handler/RequestTimings]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [avatica/server/AvaticaJsonHandler/Handler/RequestTimings]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [avatica/remote/ProtobufHandler/Handler/Serialization]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [avatica/remote/JsonHandler/Handler/Serialization]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/cpu/user]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/cpu/total]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/cpu/sys]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jvm/cpu/percent]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/delta/put/ok]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/delta/put/error]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/delta/put/oversized]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/total/put/ok]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/total/put/error]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/total/put/oversized]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/caffeine/delta/requests]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/caffeine/total/requests]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/caffeine/delta/loadTime]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/caffeine/total/loadTime]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/caffeine/delta/evictionBytes]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [query/cache/caffeine/total/evictionBytes]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jetty/numOpenConnections]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jetty/threadPool/total]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jetty/threadPool/idle]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jetty/threadPool/isLowOnThreads]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jetty/threadPool/min]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jetty/threadPool/max]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jetty/threadPool/queueSize]
   [MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric [jetty/threadPool/busy]
   ```
   
   This is seen on broker and other nodes too. As per docs even though, I have extended the metricDimensions.json so that these metrics are mapped. As an example to enable jvm threads metrics I have added org.apache.druid.java.util.metrics.JvmThreadsMonitor in the list of druid.monitoring.monitors list and extended the metricsDimensions as below:
   
   ```
         "jvm/threads/started" : { "dimensions" : [], "type" : "gauge" },
         "jvm/threads/finished" : { "dimensions" : [], "type" : "gauge" },
         "jvm/threads/live" : { "dimensions" : [], "type" : "gauge" },
   ```
   
   But then also I see these metrics being unmapped in the logs. Any thoughts? Should I create a separate issue on github for this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org