You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "YinChunGuang (via GitHub)" <gi...@apache.org> on 2026/07/16 03:09:29 UTC
[I] [Flink] Metadata table write failure during deltacommit causes permanent job failure with restart loop [hudi]
YinChunGuang opened a new issue, #19301:
URL: https://github.com/apache/hudi/issues/19301
h2. Summary
When a Flink streaming writer encounters a transient HDFS error during metadata table (MDT) log file write, the entire deltacommit fails. Because the MDT write happens before `saveAsComplete`, the data table's inflight instant is never transitioned to complete. On restart, the coordinator retries from the oldest pending instant, but the inflight file may no longer exist on HDFS, causing a permanent restart failure loop that exhausts all retry attempts and kills the job.
h2. Environment
|| Component || Version ||
| Apache Flink | 1.18.1 |
| Apache Hudi | 1.1.1 |
| Hadoop | 3.1.1 |
| JDK | 1.8 |
| Table Type | MOR (Merge On Read) |
| Index Type | BUCKET |
| Metadata Table | Enabled (default) |
h2. Table Configuration
{code:sql}
CREATE TABLE `t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock`(
...
)
PARTITIONED BY (`dt`)
WITH (
'connector' = 'hudi',
'table.type' = 'MERGE_ON_READ',
'index.type' = 'BUCKET',
'hoodie.metadata.enable' = 'true', -- MDT enabled
'hoodie.cleaner.policy' = 'KEEP_LATEST_COMMITS',
'hoodie.cleaner.commits.retained' = '10',
'hoodie.clean.manual.batch.enable' = 'true', -- Auto clean OFF
'hoodie.archive.manual.batch.enable' = 'true', -- Auto archive OFF
'compaction.tasks' = '1',
'compaction.async.enabled' = 'true',
'compaction.delta_commits' = '20',
'compaction.trigger.strategy' = 'num_commits',
'write.tasks' = '8',
'changelog.enabled' = 'true',
'write.precombine' = 'true',
'write.ignore.failed' = 'true'
);
{code}
h2. Failure Timeline
h3. Phase 1: MDT Write Failure → Global Job Failure
# A high-priority compaction completes at *02:53:51*
# The next deltacommit `20260710185353327` starts at *02:53:51*
# The MDT log file is created successfully:
{code}
02:53:51,332 INFO DFSClient [] - DFSClient_createFile_..._FOR_CREATE [.metadata/.hoodie_mt_log_...]
{code}
# 4.4 seconds later, while writing to the MDT log file, `hsync` fails:
{code}
02:53:55,728 WARN DFSClient [] - Caught exception in hsync: Holder DFSClient_-563620450 does not have any open files
02:53:55,758 WARN DFSClient [] - DataStreamer Exception: InterruptedException
02:53:55,760 WARN DFSClient [] - Abandoning all blocks in the pipeline
{code}
# The {{HoodieAppendException}} propagates up through {{BaseHoodieWriteClient.commit()}}:
{code}
02:53:55,761 ERROR BaseFlinkCommitActionExecutor [] - Failed to commit deltacommit at 20260710185353327
{code}
# The uncaught exception triggers Flink's global failure mechanism:
{code}
02:53:55,787 INFO NonThrownExecutor [] - Executing global failure.
02:53:55,794 INFO ExecutionGraph [] - Switching TaskExecutionJobVertex from RUNNING to FAILED
{code}
h3. Phase 2: Permanent Restart Failure Loop
After restart, the coordinator finds the incomplete instant and retries, but the inflight file was cleaned up or lost during the failed attempt:
{code}
02:59:15 INFO StreamWriteOperatorCoordinator [] - Coordinator starts.
02:59:15 INFO StreamWriteOperatorCoordinator [] - Found the earliest pending deltacommit instant: 20260710185353327
02:59:15 INFO StreamWriteOperatorCoordinator [] - New checkpoint triggers the pending instant commit: 20260710185353327
02:59:15 ERROR StreamWriteOperatorCoordinator [] - Executor execute action [commitInstants] failed.
org.apache.hudi.exception.HoodieIOException: Failed to commit deltacommit instant 20260710185353327
Caused by: org.apache.hudi.exception.HoodieIOException: Cannot find inflight file for ...20260710185353327.deltacommit.inflight
{code}
This repeats for *10 restart attempts over 30 minutes*. All fail with the same error. At *03:26:41* the job reaches terminal state FAILED.
h2. Log Evidence
h3. 1. MDT log file created (successful)
{code}
02:53:51,332 INFO DFSClient [] - DFSClient_createFile_DFSClient_-563620450_1246882867_SYNC_CREATE_FOR_CREATE_ONLY [.metadata/.hoodie_mt_log_20260710185353327_0-1586-02-000003055]
{code}
h3. 2. hsync fails after 4.4 seconds
{code}
02:53:55,728 WARN DFSClient [] - Caught exception in hsync: Holder DFSClient_-563620450 does not have any open files
java.io.IOException: The file /tmp/.../ .metadata/.hoodie_mt_log_... has been closed and is unavailable for write
{code}
h3. 3. DataNode pipeline broken
{code}
02:53:55,758 WARN DFSClient [] - DataStreamer Exception: InterruptedException
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:853)
at org.apache.hudi.org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:16861)
02:53:55,760 WARN DFSClient [] - Abandoning all blocks in the pipeline
{code}
h3. 4. Commit fails
{code}
02:53:55,761 ERROR BaseFlinkCommitActionExecutor [] - Failed to commit deltacommit at 20260710185353327
org.apache.hudi.exception.HoodieAppendException: Failed to close writer for metadata log file
Caused by: java.io.IOException: The file has been closed and is unavailable for write
{code}
h3. 5. Flink global failure
{code}
02:53:55,787 INFO NonThrownExecutor [] - Executing global failure.
02:53:55,794 INFO ExecutionGraph [] - Switching from RUNNING to FAILED. Failure Error: org.apache.hudi.exception.HoodieAppendException
{code}
h3. 6. Restart fails — inflight file missing
{code}
02:59:15 ERROR StreamWriteOperatorCoordinator [] - Executor execute action [commitInstants] failed.
Caused by: org.apache.hudi.exception.HoodieIOException: Cannot find inflight file for ...20260710185353327.deltacommit.inflight
{code}
h3. 7. Terminal state FAILED
{code}
03:26:41,727 INFO JobMaster [] - Job job_f6d4153a4696d9b1f29c924dc8704f9b reached terminal state FAILED.
{code}
h2. Root Cause Analysis
h3. Why the first deltacommit failed
The {{BaseHoodieWriteClient.commit()}} method writes to the metadata table *before* calling {{saveAsComplete}} on the data table timeline. When the MDT write fails (in this case due to an HDFS DataNode pipeline break), {{saveAsComplete}} is never reached, leaving the data table's deltacommit stuck in INFLIGHT state.
Additionally, in {{commitStats()}}, the {{catch}} block only catches {{IOException}}, but {{HoodieAppendException}} extends {{RuntimeException}}, so it bypasses the error handling and propagates directly to Flink's {{NonThrownExecutor}}, triggering a global job failure.
{code:java}
// BaseHoodieWriteClient.commit() — line ~320
// ① Write MDT (BEFORE saveAsComplete)
writeToMetadataTable(skipStreamingWritesToMetadataTable, table, instantTime,
tableWriteStats.getMetadataTableWriteStats(), metadata);
// ② Never reached if ① fails
activeTimeline.saveAsComplete(false,
table.getMetaClient().createNewInstant(
HoodieInstant.State.INFLIGHT, commitActionType, instantTime),
Option.of(metadata), ...);
{code}
{code:java}
// BaseHoodieWriteClient.commitStats() — line ~279
try {
commit(table, commitActionType, instantTime, metadata,
tableWriteStats, skipStreamingWritesToMetadataTable);
} catch (IOException e) {
// HoodieAppendException extends RuntimeException — NOT caught here!
throw new HoodieCommitException(...);
}
// HoodieAppendException escapes to Flink's NonThrownExecutor → global failure
{code}
h3. Why every restart attempt fails
The {{StreamWriteOperatorCoordinator}} retries from the oldest pending instant on each restart. However, the inflight file for that instant no longer exists — it was either cleaned up during the failed write or lost due to the pipeline failure. The {{ActiveTimelineV2.transitionStateToComplete()}} method requires the inflight file to exist, so it throws:
{code:java}
// ActiveTimelineV2.transitionStateToComplete()
Preconditions.checkArgument(isInflight(inflightInstant),
"Unable to find instant (%s)", inflightInstant);
{code}
h3. Why Flink is more affected than Spark
{{SparkRDDWriteClient}} overrides {{writeToMetadataTable()}} with streaming-aware handling. {{HoodieFlinkWriteClient}} does *not* override this method — it uses the base class default which always performs synchronous MDT writes. Furthermore, Flink hardcodes {{skipStreamingWritesToMetadataTable = false}}, meaning MDT writes are never skipped.
h2. Proposed Fixes
h3. Fix 1: Tolerate MDT write failure (BaseHoodieWriteClient.commit)
Wrap the MDT write in a try-catch block so its failure does not block {{saveAsComplete}} on the data table:
{code:java}
protected void commit(HoodieTable table, String commitActionType, String instantTime,
HoodieCommitMetadata metadata, TableWriteStats tableWriteStats,
boolean skipStreamingWritesToMetadataTable) throws IOException {
finalizeWrite(table, instantTime, tableWriteStats.getDataTableWriteStats());
saveInternalSchema(table, instantTime, metadata);
try {
writeToMetadataTable(skipStreamingWritesToMetadataTable, table, instantTime,
tableWriteStats.getMetadataTableWriteStats(), metadata);
} catch (Exception e) {
LOG.error("Failed to write to metadata table for instant {}, "
+ "proceeding with data table commit. MDT can be recovered via bootstrap.",
instantTime, e);
}
if (isLockRequiredForActiveTimeline(table.getMetaClient())) {
this.txnManager.beginStateChange(
Option.of(table.getMetaClient().createNewInstant(
HoodieInstant.State.INFLIGHT, commitActionType, instantTime)));
}
activeTimeline.saveAsComplete(false,
table.getMetaClient().createNewInstant(
HoodieInstant.State.INFLIGHT, commitActionType, instantTime),
Option.of(metadata), tableWriteStats.getRuntimeStats());
}
{code}
h3. Fix 2: Skip missing inflight files on restart (StreamWriteOperatorCoordinator)
When the inflight file is missing, skip the instant instead of failing:
{code:java}
// StreamWriteOperatorCoordinator.commitInstants()
if (allInstants.isEmpty()) {
// no pending commits, skip
return;
}
HoodieInstant instant = allInstants.get(0);
// Skip if inflight file no longer exists (e.g., cleaned up after a previous failed attempt)
if (!writeClient.getTable().getMetaClient().getActiveTimeline()
.filter(inflights -> inflights.contains(instant)).isPresent()) {
LOG.warn("Skipping instant {} — inflight file not found in active timeline", instant);
return;
}
{code}
h3. Fix 3: Catch HoodieAppendException in commitStats
Extend the catch block to include {{RuntimeException}} or specifically {{HoodieAppendException}}:
{code:java}
try {
commit(table, commitActionType, instantTime, metadata,
tableWriteStats, skipStreamingWritesToMetadataTable);
} catch (IOException | HoodieAppendException e) {
throw new HoodieCommitException(
"Failed to commit " + instantTime + " time=" + commitTime, e);
}
{code}
h2. Current Workaround
{code:sql}
ALTER TABLE t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock
SET TBLPROPERTIES (
'hoodie.metadata.enable' = 'false'
);
{code}
This disables MDT writes entirely, preventing the coupling issue. It is safe for single-writer scenarios using BUCKET index with compaction enabled.
**additionLog**
`2026-07-16 03:02:37,282 INFO org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2 [] - Completed [==>20260716025337008__deltacommit__INFLIGHT]
2026-07-16 03:02:37,287 INFO org.apache.hudi.common.fs.FSUtils [] - Removed directory at hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/.hoodie/.temp/20260716025337008
2026-07-16 03:02:37,288 INFO org.apache.hudi.client.BaseHoodieWriteClient [] - Committed 20260716025337008
2026-07-16 03:02:37,293 INFO org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2 [] - Marking instant complete [==>20260716025337008__deltacommit__INFLIGHT]
2026-07-16 03:02:37,294 INFO org.apache.hudi.client.transaction.TransactionManager [] - State change ending for action instant Option{val=[==>20260716025337008__deltacommit__INFLIGHT]}
2026-07-16 03:02:37,295 INFO org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock, Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@1efbc60b[Write locks = 1, Read locks = 0], Thread meta-event-handle, In-process lock state RELEASING
2026-07-16 03:02:37,295 INFO org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock, Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@1efbc60b[Write locks = 0, Read locks = 0], Thread meta-event-handle, In-process lock state RELEASED
2026-07-16 03:02:37,296 INFO org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock, Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@1efbc60b[Write locks = 0, Read locks = 0], Thread meta-event-handle, In-process lock state ALREADY_RELEASED
2026-07-16 03:02:37,297 INFO org.apache.hudi.client.transaction.lock.LockManager [] - Released connection created for acquiring lock
2026-07-16 03:02:37,297 INFO org.apache.hudi.client.transaction.TransactionManager [] - State change ended for action instant Option{val=[==>20260716025337008__deltacommit__INFLIGHT]}
2026-07-16 03:02:37,297 INFO org.apache.hudi.client.heartbeat.HoodieHeartbeatClient [] - Stopping heartbeat for instant 20260716025337008
2026-07-16 03:02:37,298 INFO org.apache.hudi.client.heartbeat.HoodieHeartbeatClient [] - Stopped heartbeat for instant 20260716025337008
2026-07-16 03:02:37,300 INFO org.apache.hudi.client.heartbeat.HeartbeatUtils [] - Deleted the heartbeat for instant 20260716025337008
2026-07-16 03:02:37,300 INFO org.apache.hudi.client.heartbeat.HoodieHeartbeatClient [] - Deleted heartbeat file for instant 20260716025337008
2026-07-16 03:02:37,301 ERROR org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor executes action [commits the instant 20260716030218327] error
java.lang.IllegalArgumentException: File hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/timeline/20260716025337008.deltacommit.inflight does not exist!
at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.transitionStateToComplete(ActiveTimelineV2.java:563) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:171) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:162) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:178) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:321) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:276) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:244) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:107) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:74) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:226) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:617) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:602) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$commitInstants$10(StreamWriteOperatorCoordinator.java:557) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_281]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_281]
at java.util.concurrent.ConcurrentSkipListMap$EntrySpliterator.forEachRemaining(ConcurrentSkipListMap.java:3535) ~[?:1.8.0_281]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_281]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_281]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_281]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_281]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_281]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstants(StreamWriteOperatorCoordinator.java:557) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$3(StreamWriteOperatorCoordinator.java:299) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:131) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_281]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_281]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281]
2026-07-16 03:02:37,304 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Trying to recover from a global failure.
org.apache.flink.util.FlinkException: Global failure triggered by OperatorCoordinator for 'bucket_write: yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock' (operator b754eda8ef527d9494a0b43e9bf7abf3).
at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:624) ~[flink-dist-1.18.1.jar:1.18.1]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$start$0(StreamWriteOperatorCoordinator.java:230) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.utils.NonThrownExecutor.handleException(NonThrownExecutor.java:143) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:134) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_281]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_281]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_281]
Caused by: org.apache.hudi.exception.HoodieException: Executor executes action [commits the instant 20260716030218327] error
... 6 more
Caused by: java.lang.IllegalArgumentException: File hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/timeline/20260716025337008.deltacommit.inflight does not exist!
at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.transitionStateToComplete(ActiveTimelineV2.java:563) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:171) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:162) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:178) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:321) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:276) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:244) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:107) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:74) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:226) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:617) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:602) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$commitInstants$10(StreamWriteOperatorCoordinator.java:557) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_281]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_281]
at java.util.concurrent.ConcurrentSkipListMap$EntrySpliterator.forEachRemaining(ConcurrentSkipListMap.java:3535) ~[?:1.8.0_281]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_281]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_281]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_281]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_281]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_281]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstants(StreamWriteOperatorCoordinator.java:557) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$3(StreamWriteOperatorCoordinator.java:299) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:131) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
... 3 more
2026-07-16 03:02:37,305 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - 4 tasks will be restarted to recover from a global failure.
2026-07-16 03:02:37,306 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job insert-into_hoodie_catalog.yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (a9775e03fcf62008b13b248ee8693bb9) switched from state RUNNING to RESTARTING.
2026-07-16 03:02:37,307 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - compact_plan_generate (1/1) (f49a8f6332f3f91794ef2cae46a713ef_2c190601d2e331d8fbe808700bcf6c4e_0_2) switched from RUNNING to CANCELING.
2026-07-16 03:02:37,308 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - compact_task -> Sink: compact_commit (1/1) (f49a8f6332f3f91794ef2cae46a713ef_edd9ec76dbb7cd545d93de35f574c84e_0_2) switched from RUNNING to CANCELING.
2026-07-16 03:02:37,308 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: source_kafka_01[1] -> Calc[2] -> ConstraintEnforcer[3] -> row_data_to_hoodie_record (1/1) (f49a8f6332f3f91794ef2cae46a713ef_cbc357ccb763df2852fee8c4fc7d55f2_0_2) switched from RUNNING to CANCELING.
2026-07-16 03:02:37,308 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing registered reader after failure for subtask 0 (#2) of source Source: source_kafka_01[1].
2026-07-16 03:02:37,309 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - bucket_write: yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (1/1) (f49a8f6332f3f91794ef2cae46a713ef_b754eda8ef527d9494a0b43e9bf7abf3_0_2) switched from RUNNING to CANCELING.
2026-07-16 03:02:37,325 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - compact_plan_generate (1/1) (f49a8f6332f3f91794ef2cae46a713ef_2c190601d2e331d8fbe808700bcf6c4e_0_2) switched from CANCELING to CANCELED.
2026-07-16 03:02:37,326 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - compact_task -> Sink: compact_commit (1/1) (f49a8f6332f3f91794ef2cae46a713ef_edd9ec76dbb7cd545d93de35f574c84e_0_2) switched from CANCELING to CANCELED.
2026-07-16 03:02:37,330 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - bucket_write: yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (1/1) (f49a8f6332f3f91794ef2cae46a713ef_b754eda8ef527d9494a0b43e9bf7abf3_0_2) switched from CANCELING to CANCELED.
2026-07-16 03:02:37,335 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: source_kafka_01[1] -> Calc[2] -> ConstraintEnforcer[3] -> row_data_to_hoodie_record (1/1) (f49a8f6332f3f91794ef2cae46a713ef_cbc357ccb763df2852fee8c4fc7d55f2_0_2) switched from CANCELING to CANCELED.
2026-07-16 03:02:37,336 INFO org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Clearing resource requirements of job a9775e03fcf62008b13b248ee8693bb9
2026-07-16 03:02:37,337 INFO org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedTaskManagerTracker [] - Clear all pending allocations for job a9775e03fcf62008b13b248ee8693bb9.
2026-07-16 03:03:37,311 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job insert-into_hoodie_catalog.yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (a9775e03fcf62008b13b248ee8693bb9) switched from state RESTARTING to RUNNING.
2026-07-16 03:03:37,312 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Restoring job a9775e03fcf62008b13b248ee8693bb9 from Checkpoint 74 @ 1784142138303 for a9775e03fcf62008b13b248ee8693bb9 located at hdfs://rtcluster/user/flink/checkpoints/a9775e03fcf62008b13b248ee8693bb9/chk-74.
2026-07-16 03:03:37,313 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - No master state to restore
2026-07-16 03:03:37,313 INFO org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator [] - Resetting coordinator to checkpoint.
2026-07-16 03:03:37,314 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Closing SourceCoordinator for source Source: source_kafka_01[1].
2026-07-16 03:03:37,317 INFO org.apache.flink.kafka.shaded.org.apache.kafka.common.utils.AppInfoParser [] - App info kafka.admin.client for hudi-11-test-1-ym3_11_mem94-enumerator-admin-client unregistered
2026-07-16 03:03:37,318 INFO org.apache.flink.kafka.shaded.org.apache.kafka.common.metrics.Metrics [] - Metrics scheduler closed
2026-07-16 03:03:37,319 INFO org.apache.flink.kafka.shaded.org.apache.kafka.common.metrics.Metrics [] - Closing reporter org.apache.flink.kafka.shaded.org.apache.kafka.common.metrics.JmxReporter
2026-07-16 03:03:37,319 INFO org.apache.flink.kafka.shaded.org.apache.kafka.common.metrics.Metrics [] - Metrics reporters closed
2026-07-16 03:03:37,320 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source coordinator for source Source: source_kafka_01[1] closed.
2026-07-16 03:03:37,321 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Restoring SplitEnumerator of source Source: source_kafka_01[1] from checkpoint.
2026-07-16 03:03:37,322 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Starting split enumerator for source Source: source_kafka_01[1].
2026-07-16 03:03:37,332 INFO org.apache.flink.kafka.shaded.org.apache.kafka.clients.admin.AdminClientConfig [] - AdminClientConfig values:
auto.include.jmx.reporter = true
bootstrap.servers = [bd-dw-kafka-n1.bestpay.mid:9092, bd-dw-kafka-n2.bestpay.mid:9092, bd-dw-kafka-n3.bestpay.mid:9092, bd-dw-kafka-n4.bestpay.mid:9092, bd-dw-kafka-n5.bestpay.mid:9092]
client.dns.lookup = use_all_dns_ips
client.id = hudi-11-test-1-ym3_11_mem94-enumerator-admin-client
connections.max.idle.ms = 300000
default.api.timeout.ms = 60000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 2147483647
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.connect.timeout.ms = null
sasl.login.read.timeout.ms = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.login.retry.backoff.max.ms = 10000
sasl.login.retry.backoff.ms = 100
sasl.mechanism = GSSAPI
sasl.oauthbearer.clock.skew.seconds = 30
sasl.oauthbearer.expected.audience = null
sasl.oauthbearer.expected.issuer = null
sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
sasl.oauthbearer.jwks.endpoint.url = null
sasl.oauthbearer.scope.claim.name = scope
sasl.oauthbearer.sub.claim.name = sub
sasl.oauthbearer.token.endpoint.url = null
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.2
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Flink] Metadata table write failure during deltacommit causes permanent job failure with restart loop [hudi]
Posted by "YinChunGuang (via GitHub)" <gi...@apache.org>.
YinChunGuang commented on issue #19301:
URL: https://github.com/apache/hudi/issues/19301#issuecomment-5001548469
@danny0405 Sounds cool ,thank you very much ! I merged the pr , and rerun the job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Flink] Metadata table write failure during deltacommit causes permanent job failure with restart loop [hudi]
Posted by "YinChunGuang (via GitHub)" <gi...@apache.org>.
YinChunGuang commented on issue #19301:
URL: https://github.com/apache/hudi/issues/19301#issuecomment-4987773864
hoodie.properties
2026-07-16 03:08:08,235 INFO org.apache.hudi.common.table.view.FileSystemViewManager [] - Creating View Manager with storage type MEMORY.
2026-07-16 03:08:08,235 INFO org.apache.hudi.keygen.constant.KeyGeneratorType [] - No key generator type is set properly
2026-07-16 03:08:08,236 INFO org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder [] - Building HoodieLogFormat Writer
2026-07-16 03:08:08,237 INFO org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder [] - Computing the next log version for files-0000-0 in hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/files
2026-07-16 03:08:08,237 INFO org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder [] - Computed the next log version for files-0000-0 in hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/files as 1 with write-token 0-1-0
2026-07-16 03:08:08,237 INFO org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder [] - HoodieLogFile on path hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/files/.files-0000-0_20260716025337008.log.1_0-1-0
2026-07-16 03:08:08,245 INFO org.apache.hudi.table.marker.DirectWriteMarkers [] - [direct] Created marker file hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/.hoodie/.temp/20260716025337008/files/.files-0000-0_20260716025337008.log.1_0-1-0.marker.CREATE in 2 ms
2026-07-16 03:08:08,247 INFO org.apache.hudi.common.table.log.HoodieLogFormatWriter [] - Created a new log file: HoodieLogFile{pathStr='hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/files/.files-0000-0_20260716025337008.log.1_0-1-0', fileLen=0}
2026-07-16 03:08:08,329 INFO org.apache.hudi.io.HoodieAppendHandle [] - AppendHandle for partitionPath files filePath files/.files-0000-0_20260716025337008.log.1_0-1-0, took 93 ms.
2026-07-16 03:08:08,364 INFO org.apache.hudi.common.table.HoodieTableConfig [] - Loading table properties from hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/.hoodie/hoodie.properties
2026-07-16 03:08:08,365 INFO org.apache.hudi.client.BaseHoodieWriteClient [] - Committing 20260716025337008 action deltacommit
2026-07-16 03:08:08,367 INFO org.apache.hudi.common.table.HoodieTableConfig [] - Loading table properties from hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/.hoodie/hoodie.properties
2026-07-16 03:08:08,368 INFO org.apache.hudi.common.table.view.FileSystemViewManager [] - Creating View Manager with storage type MEMORY.
2026-07-16 03:08:08,368 INFO org.apache.hudi.common.util.CommitUtils [] - Creating metadata for UPSERT_PREPPED numWriteStats:1 numReplaceFileIds:0
2026-07-16 03:08:08,369 INFO org.apache.hudi.common.table.HoodieTableConfig [] - Loading table properties from hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/.hoodie/hoodie.properties
2026-07-16 03:08:08,370 INFO org.apache.hudi.common.table.view.FileSystemViewManager [] - Creating View Manager with storage type MEMORY.
2026-07-16 03:08:08,371 INFO org.apache.hudi.client.BaseHoodieWriteClient [] - Committing 20260716025337008 action deltacommit
2026-07-16 03:08:08,381 INFO org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2 [] - Marking instant complete [==>20260716025337008__deltacommit__INFLIGHT]
2026-07-16 03:08:08,443 INFO org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2 [] - Created new file for toInstant: hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/.hoodie/timeline/20260716025337008_20260716030808382.deltacommit
2026-07-16 03:08:08,444 INFO org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2 [] - Completed [==>20260716025337008__deltacommit__INFLIGHT]
2026-07-16 03:08:08,447 INFO org.apache.hudi.common.fs.FSUtils [] - Removed directory at hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/.hoodie/.temp/20260716025337008
2026-07-16 03:08:08,448 INFO org.apache.hudi.client.BaseHoodieWriteClient [] - Committed 20260716025337008
2026-07-16 03:08:08,454 INFO org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2 [] - Marking instant complete [==>20260716025337008__deltacommit__INFLIGHT]
2026-07-16 03:08:08,454 INFO org.apache.hudi.client.transaction.TransactionManager [] - State change ending for action instant Option{val=[==>20260716025337008__deltacommit__INFLIGHT]}
2026-07-16 03:08:08,455 INFO org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock, Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@1efbc60b[Write locks = 1, Read locks = 0], Thread meta-event-handle, In-process lock state RELEASING
2026-07-16 03:08:08,455 INFO org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock, Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@1efbc60b[Write locks = 0, Read locks = 0], Thread meta-event-handle, In-process lock state RELEASED
2026-07-16 03:08:08,456 INFO org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock, Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@1efbc60b[Write locks = 0, Read locks = 0], Thread meta-event-handle, In-process lock state ALREADY_RELEASED
2026-07-16 03:08:08,456 INFO org.apache.hudi.client.transaction.lock.LockManager [] - Released connection created for acquiring lock
2026-07-16 03:08:08,456 INFO org.apache.hudi.client.transaction.TransactionManager [] - State change ended for action instant Option{val=[==>20260716025337008__deltacommit__INFLIGHT]}
2026-07-16 03:08:08,457 INFO org.apache.hudi.client.heartbeat.HoodieHeartbeatClient [] - Stopping heartbeat for instant 20260716025337008
2026-07-16 03:08:08,457 INFO org.apache.hudi.client.heartbeat.HoodieHeartbeatClient [] - Stopped heartbeat for instant 20260716025337008
2026-07-16 03:08:08,458 INFO org.apache.hudi.client.heartbeat.HeartbeatUtils [] - Deleted the heartbeat for instant 20260716025337008
2026-07-16 03:08:08,459 INFO org.apache.hudi.client.heartbeat.HoodieHeartbeatClient [] - Deleted heartbeat file for instant 20260716025337008
2026-07-16 03:08:08,459 ERROR org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor executes action [commits the instant 20260716030800298] error
java.lang.IllegalArgumentException: File hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/timeline/20260716025337008.deltacommit.inflight does not exist!
at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.transitionStateToComplete(ActiveTimelineV2.java:563) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:171) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:162) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:178) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:321) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:276) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:244) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:107) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:74) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:226) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:617) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:602) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$commitInstants$10(StreamWriteOperatorCoordinator.java:557) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_281]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_281]
at java.util.concurrent.ConcurrentSkipListMap$EntrySpliterator.forEachRemaining(ConcurrentSkipListMap.java:3535) ~[?:1.8.0_281]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_281]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_281]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_281]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_281]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_281]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstants(StreamWriteOperatorCoordinator.java:557) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$3(StreamWriteOperatorCoordinator.java:299) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:131) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_281]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_281]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281]
2026-07-16 03:08:08,462 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Trying to recover from a global failure.
org.apache.flink.util.FlinkException: Global failure triggered by OperatorCoordinator for 'bucket_write: yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock' (operator b754eda8ef527d9494a0b43e9bf7abf3).
at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:624) ~[flink-dist-1.18.1.jar:1.18.1]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$start$0(StreamWriteOperatorCoordinator.java:230) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.utils.NonThrownExecutor.handleException(NonThrownExecutor.java:143) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:134) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_281]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_281]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_281]
Caused by: org.apache.hudi.exception.HoodieException: Executor executes action [commits the instant 20260716030800298] error
... 6 more
Caused by: java.lang.IllegalArgumentException: File hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/timeline/20260716025337008.deltacommit.inflight does not exist!
at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.transitionStateToComplete(ActiveTimelineV2.java:563) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:171) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:162) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:178) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:321) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:276) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:244) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:107) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:74) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:226) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:617) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:602) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$commitInstants$10(StreamWriteOperatorCoordinator.java:557) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_281]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_281]
at java.util.concurrent.ConcurrentSkipListMap$EntrySpliterator.forEachRemaining(ConcurrentSkipListMap.java:3535) ~[?:1.8.0_281]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_281]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_281]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_281]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_281]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_281]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstants(StreamWriteOperatorCoordinator.java:557) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$3(StreamWriteOperatorCoordinator.java:299) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:131) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
... 3 more
2026-07-16 03:08:08,463 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - 4 tasks will be restarted to recover from a global failure.
2026-07-16 03:08:08,464 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job insert-into_hoodie_catalog.yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (a9775e03fcf62008b13b248ee8693bb9) switched from state RUNNING to RESTARTING.
2026-07-16 03:08:08,464 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - compact_plan_generate (1/1) (f49a8f6332f3f91794ef2cae46a713ef_2c190601d2e331d8fbe808700bcf6c4e_0_4) switched from RUNNING to CANCELING.
2026-07-16 03:08:08,465 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - compact_task -> Sink: compact_commit (1/1) (f49a8f6332f3f91794ef2cae46a713ef_edd9ec76dbb7cd545d93de35f574c84e_0_4) switched from RUNNING to CANCELING.
2026-07-16 03:08:08,465 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: source_kafka_01[1] -> Calc[2] -> ConstraintEnforcer[3] -> row_data_to_hoodie_record (1/1) (f49a8f6332f3f91794ef2cae46a713ef_cbc357ccb763df2852fee8c4fc7d55f2_0_4) switched from RUNNING to CANCELING.
2026-07-16 03:08:08,465 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing registered reader after failure for subtask 0 (#4) of source Source: source_kafka_01[1].
2026-07-16 03:08:08,466 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - bucket_write: yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (1/1) (f49a8f6332f3f91794ef2cae46a713ef_b754eda8ef527d9494a0b43e9bf7abf3_0_4) switched from RUNNING to CANCELING.
2026-07-16 03:08:08,482 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - compact_plan_generate (1/1) (f49a8f6332f3f91794ef2cae46a713ef_2c190601d2e331d8fbe808700bcf6c4e_0_4) switched from CANCELING to CANCELED.
2026-07-16 03:08:08,483 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - compact_task -> Sink: compact_commit (1/1) (f49a8f6332f3f91794ef2cae46a713ef_edd9ec76dbb7cd545d93de35f574c84e_0_4) switched from CANCELING to CANCELED.
2026-07-16 03:08:08,485 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - bucket_write: yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (1/1) (f49a8f6332f3f91794ef2cae46a713ef_b754eda8ef527d9494a0b43e9bf7abf3_0_4) switched from CANCELING to CANCELED.
2026-07-16 03:08:08,491 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: source_kafka_01[1] -> Calc[2] -> ConstraintEnforcer[3] -> row_data_to_hoodie_record (1/1) (f49a8f6332f3f91794ef2cae46a713ef_cbc357ccb763df2852fee8c4fc7d55f2_0_4) switched from CANCELING to CANCELED.
2026-07-16 03:08:08,492 INFO org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Clearing resource requirements of job a9775e03fcf62008b13b248ee8693bb9
2026-07-16 03:08:08,492 INFO org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedTaskManagerTracker [] - Clear all pending allocations for job a9775e03fcf62008b13b248ee8693bb9.
2026-07-16 03:09:08,467 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job insert-into_hoodie_catalog.yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (a9775e03fcf62008b13b248ee8693bb9) switched from state RESTARTING to RUNNING.
2026-07-16 03:09:08,468 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Restoring job a9775e03fcf62008b13b248ee8693bb9 from Checkpoint 76 @ 1784142480275 for a9775e03fcf62008b13b248ee8693bb9 located at hdfs://rtcluster/user/flink/checkpoints/a9775e03fcf62008b13b248ee8693bb9/chk-76.
2026-07-16 03:09:08,469 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - No master state to restore
2026-07-16 03:09:08,470 INFO org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator [] - Resetting coordinator to checkpoint.
2026-07-16 03:09:08,471 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Closing SourceCoordinator for source Source: source_kafka_01[1].
2026-07-16 03:09:08,473 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: source_kafka_01[1] -> Calc[2] -> ConstraintEnforcer[3] -> row_data_to_hoodie_record (1/1) (f49a8f6332f3f91794ef2cae46a713ef_cbc357ccb763df2852fee8c4fc7d55f2_0_5) switched from CREATED to SCHEDULED.
2026-07-16 03:09:08,473 INFO org.apache.flink.kafka.shaded.org.apache.kafka.common.utils.AppInfoParser [] - App info kafka.admin.client for hudi-11-test-1-ym3_11_mem94-enumerator-admin-client unregistered
2026-07-16 03:09:08,473 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - bucket_write: yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (1/1) (f49a8f6332f3f91794ef2cae46a713ef_b754eda8ef527d9494a0b43e9bf7abf3_0_5) switched from CREATED to SCHEDULED.
2026-07-16 03:09:08,474 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - compact_plan_generate (1/1) (f49a8f6332f3f91794ef2cae46a713ef_2c190601d2e331d8fbe808700bcf6c4e_0_5) switched from CREATED to SCHEDULED.
2026-07-16 03:09:08,474 INFO org.apache.flink.kafka.shaded.org.apache.kafka.common.metrics.Metrics [] - Metrics scheduler closed
2026-07-16 03:09:08,474 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - compact_task -> Sink: compact_commit (1/1) (f49a8f6332f3f91794ef2cae46a713ef_edd9ec76dbb7cd545d93de35f574c84e_0_5) switched from CREATED to SCHEDULED.
2026-07-16 03:09:08,475 INFO org.apache.flink.kafka.shaded.org.apache.kafka.common.metrics.Metrics [] - Closing reporter org.apache.flink.kafka.shaded.org.apache.kafka.common.metrics.JmxReporter
2026-07-16 03:09:08,475 INFO org.apache.flink.kafka.shaded.org.apache.kafka.common.metrics.Metrics [] - Metrics reporters closed
2026-07-16 03:09:08,475 INFO org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Received resource requirements from job a9775e03fcf62008b13b248ee8693bb9: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1}]
2026-07-16 03:09:08,476 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source coordinator for source Source: source_kafka_01[1] closed.
2026-07-16 03:09:08,476 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: source_kafka_01[1] -> Calc[2] -> ConstraintEnforcer[3] -> row_data_to_hoodie_record (1/1) (f49a8f6332f3f91794ef2cae46a713ef_cbc357ccb763df2852fee8c4fc7d55f2_0_5) switched from SCHEDULED to DEPLOYING.
2026-07-16 03:09:08,477 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: source_kafka_01[1] -> Calc[2] -> ConstraintEnforcer[3] -> row_data_to_hoodie_record (1/1) (attempt #5) with attempt id f49a8f6332f3f91794ef2cae46a713ef_cbc357ccb763df2852fee8c4fc7d55f2_0_5 and vertex id cbc357ccb763df2852fee8c4fc7d55f2_0 to container_e05_1732503259026_5944_01_000002 @ b30107u10-72-227.bestpay.bd (dataPort=11906) with allocation id 90793836ff4440d5a292b4c81d20355f
2026-07-16 03:09:08,477 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Restoring SplitEnumerator of source Source: source_kafka_01[1] from checkpoint.
2026-07-16 03:09:08,477 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Starting split enumerator for source Source: source_kafka_01[1].
2026-07-16 03:09:08,477 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - bucket_write: yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (1/1) (f49a8f6332f3f91794ef2cae46a713ef_b754eda8ef527d9494a0b43e9bf7abf3_0_5) switched from SCHEDULED to DEPLOYING.
2026-07-16 03:09:08,478 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying bucket_write: yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (1/1) (attempt #5) with attempt id f49a8f6332f3f91794ef2cae46a713ef_b754eda8ef527d9494a0b43e9bf7abf3_0_5 and vertex id b754eda8ef527d9494a0b43e9bf7abf3_0 to container_e05_1732503259026_5944_01_000002 @ b30107u10-72-227.bestpay.bd (dataPort=11906) with allocation id 90793836ff4440d5a292b4c81d20355f
2026-07-16 03:09:08,479 INFO org.apache.flink.kafka.shaded.org.apache.kafka.clients.admin.AdminClientConfig [] - AdminClientConfig values:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Flink] Metadata table write failure during deltacommit causes permanent job failure with restart loop [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #19301:
URL: https://github.com/apache/hudi/issues/19301#issuecomment-4987912393
the inflight metadata file might got rolled back by the cleaner service since it's heartbeat timeout, did you try 1.1.1 with this patch already: https://github.com/apache/hudi/pull/19237
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Flink] Metadata table write failure during deltacommit causes permanent job failure with restart loop [hudi]
Posted by "YinChunGuang (via GitHub)" <gi...@apache.org>.
YinChunGuang closed issue #19301: [Flink] Metadata table write failure during deltacommit causes permanent job failure with restart loop
URL: https://github.com/apache/hudi/issues/19301
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Flink] Metadata table write failure during deltacommit causes permanent job failure with restart loop [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #19301:
URL: https://github.com/apache/hudi/issues/19301#issuecomment-4988470491
Thanks for confirming that #19237 is already applied. This looks different from the coordinator event-buffer loss fixed there.
In Hudi 1.1.1, the default heartbeat expiry is approximately two minutes, while the first recommit shown here occurs more than five minutes after the original failure. Also, hoodie.clean.manual.batch.enable is not a recognized Apache Hudi configuration, so Flink async cleaning is likely still enabled.
PR #19237 was merged on top of #18904 on master. Applying only #19237 to 1.1.1 does not include #18904’s resilient heartbeat writer or its increase of hoodie.client.heartbeat.tolerable.misses from 2 to 10.
Could you test with #18904 applied as well, or temporarily configure:
```xml
'hoodie.client.heartbeat.tolerable.misses' = '10',
'clean.async.enabled' = 'false'
```
Please also provide logs matching Heartbeat expired, rollback, clean, and the affected instant, plus the corresponding rollback entries from .hoodie/timeline. That will confirm whether the cleaner removed the inflight instant before coordinator recovery.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org