[GitHub] gianm commented on issue #6743: IncrementalIndex generally overestimates theta sketch size

[GitBox <gi...@apache.org>]

gianm commented on issue #6743: IncrementalIndex generally overestimates theta sketch size
URL: https://github.com/apache/incubator-druid/issues/6743#issuecomment-448030739
 
 
   @AlexanderSaydakov One weird thing about how the ingestion-time row size estimator works is that it's using on-heap Aggregators, but it estimates size via the AggregatorFactory `getMaxIntermediateSize()` method that is meant to be the max size of a buffer needed for the BufferAggregator.
   
   One thing we could do is add a method to Aggregator (the on-heap interface) that estimates the current on-heap footprint. It can vary in a way that BufferAggregator, currently, can't.
   
   Another thing we could do is implement growable BufferAggregators and switch IncrementalIndex to use those instead. It has come up before, see proposal from @himanshug in https://github.com/apache/incubator-druid/issues/2963, but isn't implemented.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org