You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "HuangZhenQiu (via GitHub)" <gi...@apache.org> on 2026/01/30 00:42:11 UTC
[I] Silent data lose in Flink hudi writer when commit fail [hudi]
HuangZhenQiu opened a new issue, #18051:
URL: https://github.com/apache/hudi/issues/18051
### Bug Description
**What happened:**
For Flink Hudi ingestion jobs configured restart strategy, if it failed after checkpoint complete before Hudi commit finish, the job will restart with completed checkpoint in which kafka offset already proceeded.
**What you expected:**
In this case, we should at least prevent job to do the internal restart and let the job fail directly. It gives platform to do the additional state check and make the checkpoint and hudi commit consistent before restarting the job.
**Steps to reproduce:**
1.
2.
3.
### Environment
**Hudi version:**
**Query engine:** (Spark/Flink/Trino etc)
**Relevant configs:**
### Logs and Stack Trace
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Silent data lose in Flink hudi writer when commit fail [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #18051:
URL: https://github.com/apache/hudi/issues/18051#issuecomment-3822236039
There is already a fix for `#2`: https://github.com/apache/hudi/pull/12150
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Silent data lose in Flink hudi writer when commit fail [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on issue #18051:
URL: https://github.com/apache/hudi/issues/18051#issuecomment-3822071242
Make sense. Let me create the fixing PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Silent data lose in Flink hudi writer when commit fail [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #18051:
URL: https://github.com/apache/hudi/issues/18051#issuecomment-3821843008
> if it failed after checkpoint complete before Hudi commit finish
the pipeline will recommit the metadata during restart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Silent data lose in Flink hudi writer when commit fail [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #18051:
URL: https://github.com/apache/hudi/issues/18051#issuecomment-3821987160
the failue usually is caused by abnormal data payload, we can check `write.ignore.failed` to decide whether to throw?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Silent data lose in Flink hudi writer when commit fail [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 closed issue #18051: Silent data lose in Flink hudi writer when commit fail
URL: https://github.com/apache/hudi/issues/18051
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Silent data lose in Flink hudi writer when commit fail [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on issue #18051:
URL: https://github.com/apache/hudi/issues/18051#issuecomment-3821850974
Updated the Hudi version. We are still running in Hudi 0.14 internally. From the log, I think there was a rollback and job restart with last completed checkpoint, it caused data loss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Silent data lose in Flink hudi writer when commit fail [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on issue #18051:
URL: https://github.com/apache/hudi/issues/18051#issuecomment-3821965318
Agree. It is a right fix. Currently, I am thinking how can I throw an exception to prevent the internal restart of a flink job. For 99% internal restart reduce downtime, for this case, we want to fail job directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Silent data lose in Flink hudi writer when commit fail [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #18051:
URL: https://github.com/apache/hudi/issues/18051#issuecomment-3822224921
Let's hold on for the fix, the current state of the throwing behavior is engine specific, in Spark streaming, there is a option named `hoodie.datasource.write.streaming.ignore.failed.batch`, which is checked right before the instant commit, it works because the txn is sync while the Fink does not work because ckp is async.
and there are two ways to fix the issue:
1. collect the write failures after each file write and throw;
2. throws directly in the write handle.
We also need to consider how to standardize this behavior(like unifying the namings), opition `#1` got the minimum changes but I think `#2` maybe more proper fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Silent data lose in Flink hudi writer when commit fail [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #18051:
URL: https://github.com/apache/hudi/issues/18051#issuecomment-3821945376
I think it is another case, that is when during instant commit, if errors are found in the write status, Flink will not commit the instant and rolls it back instead, that would cause the data loss and needs a fix, the solution might be just throwing in the write function when errors are dected just like what we do to the append function: https://github.com/apache/hudi/pull/13417, instead of logging, we should throw.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org