You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/09/04 08:14:34 UTC
[GitHub] [druid] 599166320 opened a new issue, #13025: Prometheus-emitter does not push metrics to the pushgateway server
599166320 opened a new issue, #13025:
URL: https://github.com/apache/druid/issues/13025
I used Prometheus push gateway to monitor the Druid process, and found that the push gateway server has never had Druid related metrics.
By looking at the source code, we can see that `Prometheus-emitter` does not call flush periodically.
Referring to `incluxdbemitter`, I added `scheduledexecutorservice`, called flush regularly, and fixed it.
### Affected Version
It seems that all versions have this problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #13025: Prometheus-emitter does not push metrics to the pushgateway server
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #13025:
URL: https://github.com/apache/druid/issues/13025#issuecomment-1236733613
which druid version are you using?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #13025: Prometheus-emitter does not push metrics to the pushgateway server
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #13025:
URL: https://github.com/apache/druid/issues/13025#issuecomment-1236755048
I took a look at the implementation, metrics are not pushed to the gateway until the emitter is going to closed. So, actually I think your gateway receives metrics from peon tasks, the only problem is that it doesn't receive metrics when task is running. If a task runs for a long time, this could be a problem.
There're many emitters that have a scheduler, such as KafkaEmitter, and GraphiteEmitter.
I'm wondering if we can abstract such scheduler mechanism to Emitter interface so that any implementations won't do it by itself and they won't forget to do it. This might be out of the scope of the fix of current problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] 599166320 commented on issue #13025: Prometheus-emitter does not push metrics to the pushgateway server
Posted by GitBox <gi...@apache.org>.
599166320 commented on issue #13025:
URL: https://github.com/apache/druid/issues/13025#issuecomment-1236708499
> So you mean metrics are not emitted from peon tasks? If so, can you check if this works?
>
> [#12242 (comment)](https://github.com/apache/druid/issues/12242#issuecomment-1216185702)
I just checked, and the prefix of `druid.indexer.fork.property `still doesn't work. Normally, pushgateway needs to continuously report metrics. However, no matter whether the prefix is added or not, the metrics is not continuously reported.
Below is the configuration I just checked
```
druid.indexer.fork.property.druid.extensions.loadList=["druid-s3-extensions","druid-histogram","mysql-metadata-storage","druid-multi-stage-query","druid-prometheus-emitter"]
druid.indexer.fork.property.druid.emitter=prometheus
druid.indexer.fork.property.druid.emitter.prometheus.strategy=pushgateway
druid.indexer.fork.property.druid.emitter.prometheus.pushGatewayAddress=localhost:9091
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] 599166320 commented on issue #13025: Prometheus-emitter does not push metrics to the pushgateway server
Posted by GitBox <gi...@apache.org>.
599166320 commented on issue #13025:
URL: https://github.com/apache/druid/issues/13025#issuecomment-1236741056
> which druid version are you using?
I tested two versions: 25.0.0-SNAPSHOT and 0.22.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 closed issue #13025: Prometheus-emitter does not push metrics to the pushgateway server
Posted by GitBox <gi...@apache.org>.
FrankChen021 closed issue #13025: Prometheus-emitter does not push metrics to the pushgateway server
URL: https://github.com/apache/druid/issues/13025
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] 599166320 commented on issue #13025: Prometheus-emitter does not push metrics to the pushgateway server
Posted by GitBox <gi...@apache.org>.
599166320 commented on issue #13025:
URL: https://github.com/apache/druid/issues/13025#issuecomment-1236604064
>
Thank you for your reply I used the `ps` command to check the parameters of Kafka peon. The pushgateway has been configured, but the metrics is still not reported to the pushgatewayserver.
<img width="1765" alt="image" src="https://user-images.githubusercontent.com/3204398/188382190-ce792b3a-c407-480f-9424-505929279f33.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #13025: Prometheus-emitter does not push metrics to the pushgateway server
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #13025:
URL: https://github.com/apache/druid/issues/13025#issuecomment-1236484342
This emitter is used by many users, and we receive some issues/PRs about this emitter, but never hear that it does not work.
I don't use this emitter, but according to current doc, `exporter` strategy should be used.
> Default strategy exporter would expose metrics for scraping purpose. Only peon task (short-lived jobs) need to use pushgateway strategy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] 599166320 commented on issue #13025: Prometheus-emitter does not push metrics to the pushgateway server
Posted by GitBox <gi...@apache.org>.
599166320 commented on issue #13025:
URL: https://github.com/apache/druid/issues/13025#issuecomment-1236774904
>
Your idea is great, the implementation logic of the scheduler should really be extracted, and others who implement other emitters can remind them when compiling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #13025: Prometheus-emitter does not push metrics to the pushgateway server
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #13025:
URL: https://github.com/apache/druid/issues/13025#issuecomment-1236635424
So you mean metrics are not emitted from peon tasks? If so, can you check if this works?
https://github.com/apache/druid/issues/12242#issuecomment-1216185702
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on issue #13025: Prometheus-emitter does not push metrics to the pushgateway server
Posted by GitBox <gi...@apache.org>.
kfaraz commented on issue #13025:
URL: https://github.com/apache/druid/issues/13025#issuecomment-1240255903
> There're many emitters that have a scheduler, such as KafkaEmitter, and GraphiteEmitter.
I'm wondering if we can abstract such scheduler mechanism to Emitter interface so that any implementations won't do it by itself and they won't forget to do it. This might be out of the scope of the fix of current problem.
Yes, @FrankChen021, we should extracted out this common requirement. `HttpPostEmitter` uses a separate `EmittingThread` too, and seems to have an involved batching logic. I suppose all of this should be consolidated in a common emitter utility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org