[GitHub] [druid] gianm commented on pull request #10986: DruidSegmentReader: Close segments properly.

[GitBox <gi...@apache.org>]

gianm commented on pull request #10986:
URL: https://github.com/apache/druid/pull/10986#issuecomment-817953054


   When I run the tests locally, they crash due to HLL collectors referencing mmapped memory from a QueryableIndex that has been closed. I think we'll want to do one of the following:
   
   - Change the contract of `intermediateRowIterator` to say that you must fully dispose of all objects before closing the iterator, and make sure all callers adhere to it.
   - Implement a way to clone complex objects that may reference mmapped memory, and have DruidSegmentReader use that method, so the objects will keep working properly after the segment they came from is unmapped.
   - Implement some form of refcounting for objects that reference mmapped memory.
   
   To me the first and third ones seem like a boondoggle, because these objects can go on a pretty wild journey after the iterator closes. So the second (cloning the objects) seems cleanest. But it will have a performance hit that the other two would not. If the performance hit is acceptable, though, then that one seems best.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org