You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 08:28:43 UTC
[I] Ensure there is no data duplication with spark streaming writes [hudi]
hudi-bot opened a new issue, #16108:
URL: https://github.com/apache/hudi/issues/16108
w/ spark-streaming writes, we can deduce first batch using batchId vs an existing batch which got resumed after a long long time.
we should guarantee idempotency by deducing the batch Id
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6561
- Type: Improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Ensure there is no data duplication with spark streaming writes [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on issue #16108:
URL: https://github.com/apache/hudi/issues/16108#issuecomment-4892554188
This issue was reviewed as part of the JIRA-migrated backlog triage.
Findings: This appears to be fixed. Idempotent Spark streaming writes keyed by batch id were implemented in PR #6098 (HUDI-4389, "Make HoodieStreamingSink idempotent") and hardened by #7383 / #7783 / #8558. HoodieStreamingSink now records the batch id as `_hudi_streaming_sink_checkpoint` in commit metadata and skips any already-committed micro-batch via canSkipBatch() (checking latestCommittedBatchId >= incomingBatchId per streaming checkpoint identifier), giving exactly-once micro-batch processing — see HoodieStreamingSink.scala (canSkipBatch / SINK_CHECKPOINT_KEY). This has been in place since release-0.13.0.
If you're still seeing duplication with Spark streaming writes on a recent version, please reopen with repro steps (Hudi version, write config, and how batches are replayed).
Closing as fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Ensure there is no data duplication with spark streaming writes [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 closed issue #16108: Ensure there is no data duplication with spark streaming writes
URL: https://github.com/apache/hudi/issues/16108
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org