[I] Hudi Commit Callback and commit in a single transaction [hudi]

["hudi-bot (via GitHub)" <gi...@apache.org>]

hudi-bot opened a new issue, #14800:
URL: https://github.com/apache/hudi/issues/14800

   Hello,
   
   I am using Hudi Commit callbacks to call an internal service. As per my understanding, the service is called after the commit on the dataset and if there is a failure in the callback service we would not rollback the commit.
   
   The service which we call saves the commit time in a database which is accessed by multiple pipelines to get the incremental delta. For example, when there are 4 commits in hudi dataset, we register 4 commit timestamps in the database. The pipelines that need the incremental delta, run at different frequencies and use this database to fetch new data after their respective runs. 
   
   For this to work well, we need the hudi commit and call back to be atomic in a single transaction. Otherwise on callback failures, there may be data in the hudi dataset which may not be registered in the DB.
   
   Please can you let me know if this can be supported and if there is a way to achieve this with the current implementation. We do have retries set up and are not expecting failures but we want to keep the hudi commits in sync with what we register in the DB.
   
    
   
   Thanks.
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-1947
   - Type: New Feature


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Hudi Commit Callback and commit in a single transaction [hudi]

["nsivabalan (via GitHub)" <gi...@apache.org>]

nsivabalan commented on issue #14800:
URL: https://github.com/apache/hudi/issues/14800#issuecomment-4795261375

   Closing for inactivity. JIRA: [HUDI-1947](https://issues.apache.org/jira/browse/HUDI-1947). This issue has had no activity for over two years and the GitHub thread has no replies. If this is still relevant, please open a fresh issue with reproduction details on a recent release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Hudi Commit Callback and commit in a single transaction [hudi]

["nsivabalan (via GitHub)" <gi...@apache.org>]

nsivabalan closed issue #14800: Hudi Commit Callback and commit in a single transaction
URL: https://github.com/apache/hudi/issues/14800


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org