[GitHub] [incubator-doris] zenoyang edited a comment on pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler

[GitBox <gi...@apache.org>]

zenoyang edited a comment on pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448#issuecomment-900910166


   > Hi @zenoyang thanks for your PR. This is indeed a problem.
   > But there some thing that you need to know:
   > 
   > 1. The `labelCleaner` thread is running on all FEs, each FE will clean its own `deleteInfos`. So don't need to write a edit log `OP_REMOVE_OLD_DELETE_INFO` to "tell" other FE to clean the delete infos.
   > 2. Once the `labelCleaner` thread cleans the delete infos on each FE. What we need to take care is to clean the delete infos in meta replay logic. Eg, after calling `DeleteHandler.read()` method, we should clean the delete info actively, so that the meta image won't contains expired delete infos.
   > 
   > This logic is a bit convoluted. If you don't understand something, welcome to communicate.
   
   Thank you very much for @morningman 's review, I really didn't consider it thoroughly before. Now I have sorted out the new plan.
   
   If the EditLog synchronization logic is canceled, each FE regularly cleans up DeleteInfo, which is relatively simple to implement, but the currentTimeMs of each FE is different when cleaning, so there will be inconsistencies in DeleteInfo in each FE, although the inconsistency seems to have little effect.
   
   So my plan is:
   - Execute regularly on the Master FE, and notify other FEs to execute through EditLog, all FEs sharing the same timestamp will not cause inconsistencies.
   - When an FE plays back metadata, after the deleteHandler is initialized, the DeleteInfo cleanup logic is triggered, and other FEs are notified to clean up through EditLog to ensure consistency.
   
   I hope you can help to see if there are any problems with this, thank you very much


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org