[GitHub] [pinot] mcvsubbu opened a new issue, #10973: Minion upload after purge may leave segment in an inconsistent state

["mcvsubbu (via GitHub)" <gi...@apache.org>]

mcvsubbu opened a new issue, #10973:
URL: https://github.com/apache/pinot/issues/10973

   This is a race condition that we have observed during the following sequence of events.
   - Minion purge task downloads a segment and purges some records
   - The task tries to upload the segment back to the controller via controller API 
   - The controller happens to field a restart while moving the file to deepstore (we use LocalFS, but this may happen in other FS as well,  not sure).
   
   In this case, it is possible that the deep store does not have a copy of the segment, but the segment exists in idealstate, and the segment metadata exists as well.
   
   We have not worked out the details of the exact sub-steps during the upload process that lets the segment reach this state. It could be because LocalFS first deletes the segment in deep store, and then copies the new segment from local file over to deep store, and the restart happens between these two operations.
   
   Also, not sure if minion retries the push, but even if it does so, it probably fails because the segment is in inconsistent state (or because concurrent push is enabled).
   
   We will be digging more into this, and proposing fixes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] Jackie-Jiang commented on issue #10973: Minion upload after purge may leave segment in an inconsistent state

["Jackie-Jiang (via GitHub)" <gi...@apache.org>]

Jackie-Jiang commented on issue #10973:
URL: https://github.com/apache/pinot/issues/10973#issuecomment-1612143722

   I think the race condition happens because moving segment file to deep store and updating ZK metadata cannot be done atomically. There is another potential race condition: crc in ZK metadata doesn't match the segment crc in deep store. This applies to any segment upload, not only to purge task.
   When a segment is newly added, we add the segment to IS as the last step, so worst case we might leave an orphan segment in ZK metadata or deep store. But when refreshing a segment, this mismatch might cause the segment being unavailable.
   I don't see an easy way to make these 2 operations atomic, so we can focus on how to make it recoverable (eventual consistency)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org