You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "GWphua (via GitHub)" <gi...@apache.org> on 2026/03/04 09:39:28 UTC
[PR] Documentation in Basic Cluster Tuning using Group By Metrics (druid)
GWphua opened a new pull request, #19083:
URL: https://github.com/apache/druid/pull/19083
<!-- Thanks for trying to help us make Apache Druid be the best it can be! Please fill out as much of the following information as is possible (where relevant, and remove it when irrelevant) to help make the intention and scope of this PR clear in order to ease review. -->
<!-- Please read the doc for contribution (https://github.com/apache/druid/blob/master/CONTRIBUTING.md) before making this PR. Also, once you open a PR, please _avoid using force pushes and rebasing_ since these make it difficult for reviewers to see what you've changed in response to their reviews. See [the 'If your pull request shows conflicts with master' section](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#if-your-pull-request-shows-conflicts-with-master) for more details. -->
Continuation of #18731 and #18934
<!-- Replace XXXX with the id of the issue fixed in this PR. Remove this section if there is no corresponding issue. Don't reference the issue in the title of this pull-request. -->
<!-- If you are a committer, follow the PR action item checklist for committers:
https://github.com/apache/druid/blob/master/dev/committer-instructions.md#pr-and-issue-action-item-checklist-for-committers. -->
### Description
Introduce documentations as requested in [here](https://github.com/apache/druid/pull/18934#issuecomment-3787658970) for Group By Metrics.
<hr>
##### Key changed/added classes in this PR
* `basic-cluster-tuning.md`
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [x] been self-reviewed.
- [x] added documentation for new or modified features or behaviors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Documentation in Basic Cluster Tuning using Group By Metrics (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #19083:
URL: https://github.com/apache/druid/pull/19083#discussion_r2893485382
##########
docs/operations/basic-cluster-tuning.md:
##########
@@ -334,6 +334,24 @@ Non-nested GroupBy queries require 1 merge buffer per query, while a nested Grou
The number of merge buffers determines the number of GroupBy queries that can be processed concurrently.
+#### Using metrics to tune GroupBy buffer configuration
+
+Druid can emit metrics that help you right-size merge buffers and related GroupBy configuration. These metrics are available when the `GroupByStatsMonitor` module is enabled by adding `org.apache.druid.server.metrics.GroupByStatsMonitor` to `druid.monitoring.monitors`. See the [metrics reference](metrics.md) for full details.
+
+##### Sizing `druid.processing.buffer.sizeBytes`
+
+- `mergeBuffer/maxBytesUsed`: peak merge buffer bytes used by any single GroupBy query within the emission period. If this value consistently approaches `druid.processing.buffer.sizeBytes`, consider increasing the buffer size.
+- `groupBy/maxSpilledBytes`: peak bytes spilled to disk by any single GroupBy query. Non-zero values indicate that merge buffers are too small to hold intermediate results in memory, causing disk spill. Increasing `druid.processing.buffer.sizeBytes` reduces spilling. You can also adjust `druid.query.groupBy.maxOnDiskStorage` to control how much spilling is allowed before a query fails.
Review Comment:
Good catch, added!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Documentation in Basic Cluster Tuning using Group By Metrics (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on PR #19083:
URL: https://github.com/apache/druid/pull/19083#issuecomment-4009916774
Thanks for the review @aho135
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Documentation in Basic Cluster Tuning using Group By Metrics (druid)
Posted by "aho135 (via GitHub)" <gi...@apache.org>.
aho135 commented on code in PR #19083:
URL: https://github.com/apache/druid/pull/19083#discussion_r2891463944
##########
docs/operations/basic-cluster-tuning.md:
##########
@@ -334,6 +334,24 @@ Non-nested GroupBy queries require 1 merge buffer per query, while a nested Grou
The number of merge buffers determines the number of GroupBy queries that can be processed concurrently.
+#### Using metrics to tune GroupBy buffer configuration
+
+Druid can emit metrics that help you right-size merge buffers and related GroupBy configuration. These metrics are available when the `GroupByStatsMonitor` module is enabled by adding `org.apache.druid.server.metrics.GroupByStatsMonitor` to `druid.monitoring.monitors`. See the [metrics reference](metrics.md) for full details.
+
+##### Sizing `druid.processing.buffer.sizeBytes`
+
+- `mergeBuffer/maxBytesUsed`: peak merge buffer bytes used by any single GroupBy query within the emission period. If this value consistently approaches `druid.processing.buffer.sizeBytes`, consider increasing the buffer size.
+- `groupBy/maxSpilledBytes`: peak bytes spilled to disk by any single GroupBy query. Non-zero values indicate that merge buffers are too small to hold intermediate results in memory, causing disk spill. Increasing `druid.processing.buffer.sizeBytes` reduces spilling. You can also adjust `druid.query.groupBy.maxOnDiskStorage` to control how much spilling is allowed before a query fails.
+
+##### Sizing `druid.processing.numMergeBuffers`
+
+- `mergeBuffer/pendingRequests`: number of queries waiting to acquire merge buffers. Persistently non-zero values indicate merge buffer pool exhaustion; consider increasing `druid.processing.numMergeBuffers`.
+- `mergeBuffer/maxAcquisitionTimeNs`: peak time in nanoseconds that any single GroupBy query waited to acquire merge buffers. High values suggest contention on the merge buffer pool; increasing `druid.processing.numMergeBuffers` can reduce wait times.
+
+##### Sizing `druid.query.groupBy.maxMergingDictionarySize`
+
+- `groupBy/maxMergeDictionarySize`: peak on-heap merge dictionary size in bytes for any single GroupBy query. If this approaches `druid.query.groupBy.maxMergingDictionarySize`, queries may spill to disk or fail. If this happens, consider increasing the dictionary size limit.
Review Comment:
```suggestion
- `groupBy/maxMergeDictionarySize`: peak on-heap merge dictionary size in bytes for any single GroupBy query. If this approaches `druid.query.groupBy.maxMergingDictionarySize`, queries may spill to disk if `druid.query.groupBy.maxOnDiskStorage` is configured or fail. If this happens, consider increasing the dictionary size limit.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Documentation in Basic Cluster Tuning using Group By Metrics (druid)
Posted by "aho135 (via GitHub)" <gi...@apache.org>.
aho135 commented on code in PR #19083:
URL: https://github.com/apache/druid/pull/19083#discussion_r2893646659
##########
docs/operations/basic-cluster-tuning.md:
##########
@@ -334,6 +334,25 @@ Non-nested GroupBy queries require 1 merge buffer per query, while a nested Grou
The number of merge buffers determines the number of GroupBy queries that can be processed concurrently.
+#### Using metrics to tune GroupBy buffer configuration
+
+Druid can emit metrics that help you right-size merge buffers and related GroupBy configuration. These metrics are available when the `GroupByStatsMonitor` module is enabled by adding `org.apache.druid.server.metrics.GroupByStatsMonitor` to `druid.monitoring.monitors`. See the [metrics reference](metrics.md) for full details.
+
+##### Sizing `druid.processing.buffer.sizeBytes`
+
+- `mergeBuffer/maxBytesUsed`: peak merge buffer bytes used by any single GroupBy query within the emission period. If this value consistently approaches `druid.processing.buffer.sizeBytes`, consider increasing the buffer size.
+- `groupBy/maxSpilledBytes`: peak bytes spilled to disk by any single GroupBy query. Non-zero values indicate that merge buffers are too small to hold intermediate results in memory, causing disk spill. Increasing `druid.processing.buffer.sizeBytes` reduces spilling. You can also adjust `druid.query.groupBy.maxOnDiskStorage` to control how much spilling is allowed before a query fails.
+- `groupBy/spilledQueries`: how often GroupBy queries are spilling to disk. Non-zero values may indicate that your buffer size is too small and should be increased to avoid performance issues caused by excessive spilling.
Review Comment:
```suggestion
- `groupBy/spilledQueries`: number of GroupBy queries spilled to disk within the emission period. Non-zero values may indicate that your buffer size is too small and should be increased to avoid performance issues caused by excessive spilling.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Documentation in Basic Cluster Tuning using Group By Metrics (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua merged PR #19083:
URL: https://github.com/apache/druid/pull/19083
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Documentation in Basic Cluster Tuning using Group By Metrics (druid)
Posted by "aho135 (via GitHub)" <gi...@apache.org>.
aho135 commented on code in PR #19083:
URL: https://github.com/apache/druid/pull/19083#discussion_r2891444383
##########
docs/operations/basic-cluster-tuning.md:
##########
@@ -334,6 +334,24 @@ Non-nested GroupBy queries require 1 merge buffer per query, while a nested Grou
The number of merge buffers determines the number of GroupBy queries that can be processed concurrently.
+#### Using metrics to tune GroupBy buffer configuration
+
+Druid can emit metrics that help you right-size merge buffers and related GroupBy configuration. These metrics are available when the `GroupByStatsMonitor` module is enabled by adding `org.apache.druid.server.metrics.GroupByStatsMonitor` to `druid.monitoring.monitors`. See the [metrics reference](metrics.md) for full details.
+
+##### Sizing `druid.processing.buffer.sizeBytes`
+
+- `mergeBuffer/maxBytesUsed`: peak merge buffer bytes used by any single GroupBy query within the emission period. If this value consistently approaches `druid.processing.buffer.sizeBytes`, consider increasing the buffer size.
+- `groupBy/maxSpilledBytes`: peak bytes spilled to disk by any single GroupBy query. Non-zero values indicate that merge buffers are too small to hold intermediate results in memory, causing disk spill. Increasing `druid.processing.buffer.sizeBytes` reduces spilling. You can also adjust `druid.query.groupBy.maxOnDiskStorage` to control how much spilling is allowed before a query fails.
Review Comment:
Thanks for the doc update @GWphua!
Here we could call out that operators can monitor `groupBy/spilledQueries` as an indicator to increase buffer size
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org