You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/11/30 07:24:04 UTC

[GitHub] [hudi] bithw1 opened a new issue #2288: [SUPPORT]What does deltacommit.requested,deltacommit.inflight mean

bithw1 opened a new issue #2288:
URL: https://github.com/apache/hudi/issues/2288


   Hi,
   
   I do serveral upsert on MOR table. When I look at the HDFS ,the are several pieces of delta commits
   
   ```
   .hoodie/20201122100057.deltacommit
   .hoodie/20201122100057.deltacommit.inflight
   .hoodie/20201122100057.deltacommit.requested
   .hoodie/20201122100101.deltacommit
   .hoodie/20201122100101.deltacommit.inflight
   .hoodie/20201122100101.deltacommit.requested
   ```
   
   I think 
   1.  20201122100101.deltacommit.requested means a delta commit with commitId `20201122100101`  is `requested to do` delta commit
   
   2. 20201122100101.deltacommit.inflight means a delta commit with commitId `20201122100101 is `doing` delta commit
   
   3. 20201122100101.deltacommit means a delta commit with commitId `20201122100101 is `done` delta commit
   
   I would ask why these 3 files are there? Does it mean every state of delta commit(reqeusted, inflight, and done) will be there? and old state will not be deleted when subsequent state is being processing?
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] bithw1 closed issue #2288: [SUPPORT]What does deltacommit.requested,deltacommit.inflight mean

Posted by GitBox <gi...@apache.org>.

bithw1 closed issue #2288:
URL: https://github.com/apache/hudi/issues/2288


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] bvaradar commented on issue #2288: [SUPPORT]What does deltacommit.requested,deltacommit.inflight mean

Posted by GitBox <gi...@apache.org>.

bvaradar commented on issue #2288:
URL: https://github.com/apache/hudi/issues/2288#issuecomment-735939031


   Yes, these are metadata files used to track the status of operations and is needed to perform rollbacks if needed. Instead of keeping one file, Hudi tracks them in separate files to avoid using rename() operations which are not guaranteed to be atomic across all kinds of storage layers. 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] bithw1 commented on issue #2288: [SUPPORT]What does deltacommit.requested,deltacommit.inflight mean

Posted by GitBox <gi...@apache.org>.

bithw1 commented on issue #2288:
URL: https://github.com/apache/hudi/issues/2288#issuecomment-736181454


   Thanks @bvaradar  for the good explanation.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org