[GitHub] [pinot] Jackie-Jiang commented on issue #10973: Minion upload after purge may leave segment in an inconsistent state

["Jackie-Jiang (via GitHub)" <gi...@apache.org>]

Jackie-Jiang commented on issue #10973:
URL: https://github.com/apache/pinot/issues/10973#issuecomment-1612143722

   I think the race condition happens because moving segment file to deep store and updating ZK metadata cannot be done atomically. There is another potential race condition: crc in ZK metadata doesn't match the segment crc in deep store. This applies to any segment upload, not only to purge task.
   When a segment is newly added, we add the segment to IS as the last step, so worst case we might leave an orphan segment in ZK metadata or deep store. But when refreshing a segment, this mismatch might cause the segment being unavailable.
   I don't see an easy way to make these 2 operations atomic, so we can focus on how to make it recoverable (eventual consistency)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org