[GitHub] [pinot] klsince edited a comment on pull request #7301: mark and sweep indices for V3 segment format

[GitBox <gi...@apache.org>]

klsince edited a comment on pull request #7301:
URL: https://github.com/apache/pinot/pull/7301#issuecomment-901608648


   Thanks for reviewing!
   
   > Quick question - Are we also handling de-fragmentation as well ? ....
   
   Yes. The general idea in this PR is to copy indices still set in table config to a temp file, placed them contiguously there, and then rename the temp file back to the index file, and update the index_map accordingly. The cleanup is done at file level, w/o need to track position info into the PinotDataBuffer. 
   
   iiuc, I think the major difference from the idea in your prototype, is the use of a temp file to simplify the cleanup and do cleanup in one batch at the end of segment reloading.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org