[GitHub] [druid] drewdahlke opened a new issue #12108: Max Intermediate Aggregator Size vs Complex Types

[GitBox <gi...@apache.org>]

drewdahlke opened a new issue #12108:
URL: https://github.com/apache/druid/issues/12108


   Hi, after extending druid with new aggregators I've found that AggregatorFactory.getMaxIntermediateSize(), used for memory estimation, has a huge impact on query performance. Smaller values are faster, but if your aggregator's intermediate results exceed the threshold then it fails. It's easy to dial that in with primitives, but some data sketches (eg frequent items) vary in size tremendously depending on the shape/quantity of input data it was fed. For folks extending druid to add new aggregators using complex types, are there any tricks to dealing with intermediate result sizes that greatly vary in size? Maybe this is a feature request to make that setting a soft limit/optimization rather than hard limit? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org