[GitHub] [pinot] Jackie-Jiang opened a new issue, #9738: Add sync/async mode when deleting segments

[GitBox <gi...@apache.org>]

Jackie-Jiang opened a new issue, #9738:
URL: https://github.com/apache/pinot/issues/9738

   Currently delete segments API delete segments asynchronously (`PinotHelixResourceManager.deleteSegments()`), but delete table API delete segments synchronously (`PinotHelixResourceManager.deleteOfflineTable()` and `PinotHelixResourceManager.deleteRealtimeTable()`). Instead, we should provide a flag so that user can decide how to delete the segments.
   
   When deleting a segment, we delete the segment from ideal state first, then delete ZK metadata and file in deep store. The second step can happen both synchronously and asynchronously. The reason why we want to delete the ZK metadata and deep store file at the same time is because we need the download address from the ZK metadata to point to the deep store file so that it won't left untracked. We should try to delete deep store file first, then delete the ZK metadata to ensure the file is always cleaned up.
   
   To make the process symmetric, when adding the segment, we should first create the ZK metadata, then copy the segment to deep store, in the end add the segment to the ideal state.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org