[GitHub] [pinot] jtao15 commented on issue #7413: Race conditions between segment merge/roll-up and purge (or convertToRawIndex) tasks

[GitBox <gi...@apache.org>]

jtao15 commented on issue #7413:
URL: https://github.com/apache/pinot/issues/7413#issuecomment-917290051


   > Does it make sense to have some sort of segment level locks so that the same set of segments are not picked up by multiple tasks? If we can come up with a lock-free algorithms on this, that will be even better.
   > 
   > Alternatively, some config value of not doing merge/rollup if the retention is just N hours/days away (if this is the only race condition).
   
   The segment zk metadata version will achieve the segment level lock? If we always check the version when replacing/deleting the segments, only one task will succeed. The other replacing/deleting request can retry later.
   
   I think scheduling part will not cause any issue, but the clean up part of merge/roll-up does. By adding merge/roll-up task, the retention manager now need to clean up merged segment to free the disk space. Say we merge A and B to C. After merging is done, the retention manager will remove A and B (even if they didn't pass the retention threshold, they should be removed since they are not serving queries anymore). If A and B are scheduled with purge task, the race condition can happen.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org