You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 10:17:11 UTC
[I] Couldn't restart streams when using Spark Structured Streaming when Kafka offset goes out of range [hudi]
hudi-bot opened a new issue, #16528:
URL: https://github.com/apache/hudi/issues/16528
When using spark structured streaming with kafka and writing data in Hudi,. when jobs sometimes cant keep up with the input rate and fails as the kafka offset goes out of range (i.e earliest kafka messages are cleaned up due to the retention policy) and when we try to restart the job by clearing the previous checkpoint and consume from latest offset we see that the batches are skipped by the 'HoodieStreamingSink'.
There is no way to restart these streams again currently.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-7912
- Type: Bug
- Fix version(s):
- 0.16.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Couldn't restart streams when using Spark Structured Streaming when Kafka offset goes out of range [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #16528:
URL: https://github.com/apache/hudi/issues/16528#issuecomment-4795618037
Closing as resolved. Tracked in [HUDI-7912](https://issues.apache.org/jira/browse/HUDI-7912) (fixVersions: 0.16.0). Please reopen if you can still reproduce on the latest release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Couldn't restart streams when using Spark Structured Streaming when Kafka offset goes out of range [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #16528: Couldn't restart streams when using Spark Structured Streaming when Kafka offset goes out of range
URL: https://github.com/apache/hudi/issues/16528
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org