You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 04:47:11 UTC
[I] Failed to execute rollback due to HoodieIOException: Could not delete instant [hudi]
hudi-bot opened a new issue, #15067:
URL: https://github.com/apache/hudi/issues/15067
Deltastreamer continuous mode writing to COW table with async clustering and cleaning.
{code:java}
org.apache.hudi.exception.HoodieRollbackException: Failed to rollback file:/Users/ethan/Work/scripts/mt_rollout_testing/deploy_b_single_writer_async_services/b3_ds_cow_010mt_011mt_conf/test_table commits 20220314165647208
at org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:695)
at org.apache.hudi.client.BaseHoodieWriteClient.rollbackFailedWrites(BaseHoodieWriteClient.java:1037)
at org.apache.hudi.client.BaseHoodieWriteClient.tryUpgrade(BaseHoodieWriteClient.java:1404)
at org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1302)
at org.apache.hudi.client.SparkRDDWriteClient.insert(SparkRDDWriteClient.java:174)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:574)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:329)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:656)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieIOException: Could not delete instant [==>20220314165647208__commit__INFLIGHT]
at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.deleteInstantFile(HoodieActiveTimeline.java:250)
at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.deletePending(HoodieActiveTimeline.java:201)
at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.deleteInflightAndRequestedInstant(BaseRollbackActionExecutor.java:270)
at org.apache.hudi.table.action.rollback.CopyOnWriteRollbackActionExecutor.executeRollback(CopyOnWriteRollbackActionExecutor.java:90)
at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.doRollbackAndGetStats(BaseRollbackActionExecutor.java:218)
at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.runRollback(BaseRollbackActionExecutor.java:115)
at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.execute(BaseRollbackActionExecutor.java:144)
at org.apache.hudi.table.HoodieSparkCopyOnWriteTable.rollback(HoodieSparkCopyOnWriteTable.java:346)
at org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:680)
... 11 more {code}
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3648
- Type: Bug
- Fix version(s):
- 1.1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Failed to execute rollback due to HoodieIOException: Could not delete instant [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #15067:
URL: https://github.com/apache/hudi/issues/15067#issuecomment-4795309365
Triage: this looks **fixed** on current master.
The contemporaneous fix is [HUDI-3604 / PR #5114](https://github.com/apache/hudi/pull/5114) — commit `484b3407e00c` (2022-03-26) — "Adjust the order of timeline changes in rollbacks". That change:
1. Moved `deleteInflightAndRequestedInstant(...)` inside the transaction lock together with `finishRollback(...)`.
2. Reordered to: write rollback metadata → delete inflight commit → transition rollback `inflight → completed`, so a partial crash leaves a re-driveable `rollback.requested`.
Follow-on hardening makes the originally reported failure mode (continuous Deltastreamer + COW + async clustering & cleaning racing the upgrade-time `rollbackFailedWrites`) unreachable on master:
- Subsequent reaffirmation of the ordering invariant in HUDI-6736 (#9521).
- Locking improvements in HUDI-9387 (commit-time created and persisted in one transaction).
- Recent rollback-concurrency fixes: PR #18089 (detect rollback conflicts with ongoing commits), PR #18093 (don't delete pending rollback plan on read failure), PR #18033 (allow MDT bootstrap when pending commits are being rolled back), HUDI-9559 (#13497, account for failed deletion in rollback execution).
Current code: see `BaseRollbackActionExecutor.finishRollback` (`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/BaseRollbackActionExecutor.java#L268-L298`) — `txnManager.beginStateChange(...)` now wraps both the metadata write and `deleteInflightAndRequestedInstant`.
If anyone is still able to reproduce on a recent release (≥0.12, ideally master), please share a repro and re-open. Otherwise this can be closed.
— triaged via static-code review, not a runtime repro.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Failed to execute rollback due to HoodieIOException: Could not delete instant [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #15067: Failed to execute rollback due to HoodieIOException: Could not delete instant
URL: https://github.com/apache/hudi/issues/15067
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org