[GitHub] surekhasaharan commented on issue #5583: 'maxBytesInMemory' tuningConfig introduced for ingestion tasks

[GitBox <gi...@apache.org>]

surekhasaharan commented on issue #5583: 'maxBytesInMemory' tuningConfig introduced for ingestion tasks
URL: https://github.com/apache/incubator-druid/pull/5583#issuecomment-463069510
 
 
   @hellobabygogo there is a comment in the code about this [here](https://github.com/apache/incubator-druid/blob/master/server/src/main/java/org/apache/druid/segment/indexing/TuningConfig.java#L41). Some more details, while analyzing the heap dump on OOMEs, we found that two `OnheapIncrementalIndex` instances roughly took 2/3 of retained heap, with each instance roughly taking 1/3 of heap. So we realized that our original default which was 1/3(jvm mem) may be too aggressive, since rows or bytes in memory trackers track only the _active_ index, but not the one being _flushed to disk_. So the total used memory could be upto double of what is being tracked (as there's always one active index and potentially one being flushed to disk).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org