[GitHub] [hudi] nsivabalan commented on pull request #4035: [HUDI-2792] Configure metadata payload consistency check

[GitBox <gi...@apache.org>]

nsivabalan commented on pull request #4035:
URL: https://github.com/apache/hudi/pull/4035#issuecomment-975529517


   @vinothchandar @prashantwason : I was half way through fixing finalize write to also delete extra marker files, but on 2nd thought, wondering if we really need this fix for deleting the extra marker files in finalize write. 
   so, far I have moved the finalize write before updating metadata table. but to delete the extra markers, we need to fix timeline server based which is not trivial (need to add an endpoint). this is not the main issue.
   
   Anyways, coming back to my point:  
   This is a very tiny widow we are trying to solve here, and a large window anyways will have the gap we are looking to fix. 
   for eg, this will help us only when a partially failed commit is getting rolledback, and it failed just after writing to metadata table, but before committing to data table. In every other scenario, this is not going to help. 
   - Incase, the commit failed even before writing to metadata table, this fix may not help. 
   - If this represents a restore operation, we will do list based rollback. And so marker files does not matter. 
   
   So, I feel like we are not gaining much by fixing this tiny window. Let me know your opinion. 
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org