You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "schernic (via GitHub)" <gi...@apache.org> on 2023/05/22 18:49:34 UTC
[GitHub] [druid] schernic opened a new issue, #14327: Configurable Query Metric Collection - Disable/Sample Metric Generation to reduce CPU load
schernic opened a new issue, #14327:
URL: https://github.com/apache/druid/issues/14327
### Description
Provide a way to disable collection (not only emission) of query metrics. Alternatively, introduce sampling to the metric collection or make the collected metric types and dimensions configurable.
### Motivation
For certain workloads, we have experienced an unusually high CPU load on our historical nodes. Often times the load increased to the point that queries started to time out and Zookeeper considered the node unavailable. Out of curiosity I used a profile to look into the CPU hot spots and found that metric generation was causing most of that load. As we are not even emitting metrics I was looking for a way to disable metric collection altogether but found no way to configure that.
Nevertheless, checking the code I found a way to disable metric collection during runtime using JMX in order to check if that would really help with the high CPU load. It reduced the load on that historical by 80% which makes me believe that some features around making it possible to disable collection of query metrics or dimension could be valuable addition for others as well. Actually, I don't see why metric collection would be enabled by default if no emitter was configured.
Sampling or rate limiting of metric collections would also be useful options in cases like ours.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Configurable Query Metric Collection - Disable/Sample Metric Generation to reduce CPU load (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14327:
URL: https://github.com/apache/druid/issues/14327#issuecomment-1965563659
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Configurable Query Metric Collection - Disable/Sample Metric Generation to reduce CPU load (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #14327: Configurable Query Metric Collection - Disable/Sample Metric Generation to reduce CPU load
URL: https://github.com/apache/druid/issues/14327
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Configurable Query Metric Collection - Disable/Sample Metric Generation to reduce CPU load (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14327:
URL: https://github.com/apache/druid/issues/14327#issuecomment-2021694722
This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org