[GitHub] [pinot] jackjlli commented on pull request #10238: Fix race condition on removing aged deleted segments

["jackjlli (via GitHub)" <gi...@apache.org>]

jackjlli commented on PR #10238:
URL: https://github.com/apache/pinot/pull/10238#issuecomment-1421586088

   > Does it run into exception or just logged some warning? I'd suggest not introducing the lead controller concept into this module, but simply handle the case of file deleted during the checks because lead controller change is not guaranteed to be visible to all controllers at the same time.
   
   Yes, we ran into some exception that multiple controllers were trying to delete for the same table in Prod. 
   There is a 5-min delay config in lead controller resource, which means there would be lead controller changes only when there is some discrepancy between IS and EV for 5 mins.
   I think in this case it should be fine despite the lead controller change hasn't reached the controllers, as we just need a dedicated controller to clean up the aged segments. Even if one controller is permanently removed out of the cluster, it's okay to have the aged segments stay for another 5 mins and then another new controller got assigned to take the segment deletion. 
   
   I'm fine with capturing the exception within a try-catch block and proceed with the rest of aged segment deletion for the rest of the tables.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org