You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 08:35:56 UTC
[I] Fix multi-writer with deltastreamer checkpointing [hudi]
hudi-bot opened a new issue, #16136:
URL: https://github.com/apache/hudi/issues/16136
As of now, we store checkpoints in commit metadata while writing via deltastreamer.
To support multiple writers (multiple deltastreamers), we added support sometime back where in the checkpoint will be a map to store multiple entries with key referring to writer identifier.
{ \{ "writer1" = "checkpointVal1"},
{ "writer2" = "checkpointVal2"} }
But this incurs some additional locking since everytime when new checkpoint needs to be updated, we have to reload the timeline and fetch the latest known commit metadata.
Instead we can de-couple the checkpoint.
Each writer only update its own checkpoint. and while parsing/fetching the latest known checkpoint for a writer, we might need to walk back in the timeline and find the right checkpoint value.
For eg:
commit1 by writer1: commit metadata ➝ \{writer1 = checkpointVal1}
commit2 by writer2: commit metadata ➝ \{writer2 = checkpointValA}
commit2 by writer1: To fetch latest checkpoint for writer1, we walk back the timeline and fetch the checkpoint of interest. So, even though latest commit metadata might have checkpoint, its key refers to writer2. And so we might need to go back and fetch the checkpoint from commit1.
and finally writer1 will update the commit metadata to \{writer1 = checkpointVal2}
btw, Please check when was the multiple checkpoint support was added. if it was added before 0.13.0, we need to ensure its backwards compatible as well. if not, we are good. Just fixing the exiting solution would suffice.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6609
- Type: Improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix multi-writer with deltastreamer checkpointing [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #16136: Fix multi-writer with deltastreamer checkpointing
URL: https://github.com/apache/hudi/issues/16136
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix multi-writer with deltastreamer checkpointing [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #16136:
URL: https://github.com/apache/hudi/issues/16136#issuecomment-3630171983
### Linked PR(s)
- https://github.com/apache/hudi/pull/9312
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix multi-writer with deltastreamer checkpointing [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #16136:
URL: https://github.com/apache/hudi/issues/16136#issuecomment-4795541760
Closing as resolved. Tracked in [HUDI-6609](https://issues.apache.org/jira/browse/HUDI-6609). The change landed in PR [#9312](https://github.com/apache/hudi/pull/9312) (commit `bc583b415868`). Please reopen if you can still reproduce on the latest release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org