[GitHub] kaijianding opened a new pull request #6699: release mmap immediately after merge indexes

[GitBox <gi...@apache.org>]

kaijianding opened a new pull request #6699: release mmap immediately after merge indexes
URL: https://github.com/apache/incubator-druid/pull/6699
 
 
   When coordinator is down or loading segments slowly, or realtime node populates segments too fast(like pulling data from long time ago), there will be many many merged files and many many hydrant segments be loaded as mmap during the merge process. There will be many many intermediate objects, like DirectBufferR, while these hydrant segments' queryableIndex can't be unloaded immediately after the merge process due to they are may be used by a query.
   
   As a result, both mmap and heap size will grow to a large number and finally killed by container(in my case is yarn) or OOM(it indeed happened in my product environment).
   
   A better approach is that: separate query and merge process, alway load indexes from hydrant files and close the loaded indexes after merge success/fail to release mmap and intermediate objects

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org