[GitHub] kaijianding commented on a change in pull request #6699: release mmap immediately after merge indexes

[GitBox <gi...@apache.org>]

kaijianding commented on a change in pull request #6699: release mmap immediately after merge indexes
URL: https://github.com/apache/incubator-druid/pull/6699#discussion_r239678427
 
 

 ##########
 File path: server/src/main/java/org/apache/druid/segment/realtime/appenderator/AppenderatorImpl.java
 ##########
 @@ -716,8 +717,20 @@ private DataSegment mergeAndPush(final SegmentIdentifier identifier, final Sink
           closer.register(segmentAndCloseable.rhs);
         }
 
-        mergedFile = indexMerger.mergeQueryableIndex(
-            indexes,
 
 Review comment:
   The QueryableIndexes are always new objects loaded from file, and closed when IndexMerger.mergeSegmentFiles() is done. This behavior frees the mmap usage increased during the merge process.
   Currently we use indexMerger.mergeQueryableIndex(indexes) in RealtimePlumber, the indexes are not closed to explicitly release the increased mmap because these indexes are still used by queries. Then the mmap usage always increases until abondonSegment() is called which only happens when handoff succeeds.
   If handoff is slow or coordinator is not working properly, the mmap usage will keep increasing, this is a problem.
   
   This PR separates the QueryableIndex used by query and used by merge process, the we can close the QueryableIndexes used by merge process to release mmap, and leave the QueryableIndexes used by query untouched.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org