You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "YinChunGuang (via GitHub)" <gi...@apache.org> on 2026/07/16 03:09:29 UTC

[I] [Flink] Metadata table write failure during deltacommit causes permanent job failure with restart loop [hudi]

YinChunGuang opened a new issue, #19301:
URL: https://github.com/apache/hudi/issues/19301

   h2. Summary
   
   When a Flink streaming writer encounters a transient HDFS error during metadata table (MDT) log file write, the entire deltacommit fails. Because the MDT write happens before `saveAsComplete`, the data table's inflight instant is never transitioned to complete. On restart, the coordinator retries from the oldest pending instant, but the inflight file may no longer exist on HDFS, causing a permanent restart failure loop that exhausts all retry attempts and kills the job.
   
   h2. Environment
   
   || Component || Version ||
   | Apache Flink | 1.18.1 |
   | Apache Hudi | 1.1.1 |
   | Hadoop | 3.1.1 |
   | JDK | 1.8 |
   | Table Type | MOR (Merge On Read) |
   | Index Type | BUCKET |
   | Metadata Table | Enabled (default) |
   
   h2. Table Configuration
   
   {code:sql}
   CREATE TABLE `t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock`(
     ...
   )
   PARTITIONED BY (`dt`)
   WITH (
     'connector' = 'hudi',
     'table.type' = 'MERGE_ON_READ',
     'index.type' = 'BUCKET',
     'hoodie.metadata.enable' = 'true',          -- MDT enabled
     'hoodie.cleaner.policy' = 'KEEP_LATEST_COMMITS',
     'hoodie.cleaner.commits.retained' = '10',
     'hoodie.clean.manual.batch.enable' = 'true', -- Auto clean OFF
     'hoodie.archive.manual.batch.enable' = 'true', -- Auto archive OFF
     'compaction.tasks' = '1',
     'compaction.async.enabled' = 'true',
     'compaction.delta_commits' = '20',
     'compaction.trigger.strategy' = 'num_commits',
     'write.tasks' = '8',
     'changelog.enabled' = 'true',
     'write.precombine' = 'true',
     'write.ignore.failed' = 'true'
   );
   {code}
   
   h2. Failure Timeline
   
   h3. Phase 1: MDT Write Failure → Global Job Failure
   
   # A high-priority compaction completes at *02:53:51*
   # The next deltacommit `20260710185353327` starts at *02:53:51*
   # The MDT log file is created successfully:
   {code}
   02:53:51,332 INFO  DFSClient [] - DFSClient_createFile_..._FOR_CREATE [.metadata/.hoodie_mt_log_...]
   {code}
   # 4.4 seconds later, while writing to the MDT log file, `hsync` fails:
   {code}
   02:53:55,728 WARN  DFSClient [] - Caught exception in hsync: Holder DFSClient_-563620450 does not have any open files
   02:53:55,758 WARN  DFSClient [] - DataStreamer Exception: InterruptedException
   02:53:55,760 WARN  DFSClient [] - Abandoning all blocks in the pipeline
   {code}
   # The {{HoodieAppendException}} propagates up through {{BaseHoodieWriteClient.commit()}}:
   {code}
   02:53:55,761 ERROR BaseFlinkCommitActionExecutor [] - Failed to commit deltacommit at 20260710185353327
   {code}
   # The uncaught exception triggers Flink's global failure mechanism:
   {code}
   02:53:55,787 INFO  NonThrownExecutor [] - Executing global failure.
   02:53:55,794 INFO  ExecutionGraph [] - Switching TaskExecutionJobVertex from RUNNING to FAILED
   {code}
   
   h3. Phase 2: Permanent Restart Failure Loop
   
   After restart, the coordinator finds the incomplete instant and retries, but the inflight file was cleaned up or lost during the failed attempt:
   
   {code}
   02:59:15 INFO  StreamWriteOperatorCoordinator [] - Coordinator starts.
   02:59:15 INFO  StreamWriteOperatorCoordinator [] - Found the earliest pending deltacommit instant: 20260710185353327
   02:59:15 INFO  StreamWriteOperatorCoordinator [] - New checkpoint triggers the pending instant commit: 20260710185353327
   02:59:15 ERROR StreamWriteOperatorCoordinator [] - Executor execute action [commitInstants] failed.
   org.apache.hudi.exception.HoodieIOException: Failed to commit deltacommit instant 20260710185353327
     Caused by: org.apache.hudi.exception.HoodieIOException: Cannot find inflight file for ...20260710185353327.deltacommit.inflight
   {code}
   
   This repeats for *10 restart attempts over 30 minutes*. All fail with the same error. At *03:26:41* the job reaches terminal state FAILED.
   
   h2. Log Evidence
   
   h3. 1. MDT log file created (successful)
   
   {code}
   02:53:51,332 INFO  DFSClient [] - DFSClient_createFile_DFSClient_-563620450_1246882867_SYNC_CREATE_FOR_CREATE_ONLY [.metadata/.hoodie_mt_log_20260710185353327_0-1586-02-000003055]
   {code}
   
   h3. 2. hsync fails after 4.4 seconds
   
   {code}
   02:53:55,728 WARN  DFSClient [] - Caught exception in hsync: Holder DFSClient_-563620450 does not have any open files
   java.io.IOException: The file /tmp/.../ .metadata/.hoodie_mt_log_... has been closed and is unavailable for write
   {code}
   
   h3. 3. DataNode pipeline broken
   
   {code}
   02:53:55,758 WARN  DFSClient [] - DataStreamer Exception: InterruptedException
   java.lang.InterruptedException
       at java.lang.Object.wait(Native Method)
       at java.lang.Thread.join(Thread.java:853)
       at org.apache.hudi.org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:16861)
   02:53:55,760 WARN  DFSClient [] - Abandoning all blocks in the pipeline
   {code}
   
   h3. 4. Commit fails
   
   {code}
   02:53:55,761 ERROR BaseFlinkCommitActionExecutor [] - Failed to commit deltacommit at 20260710185353327
   org.apache.hudi.exception.HoodieAppendException: Failed to close writer for metadata log file
     Caused by: java.io.IOException: The file has been closed and is unavailable for write
   {code}
   
   h3. 5. Flink global failure
   
   {code}
   02:53:55,787 INFO  NonThrownExecutor [] - Executing global failure.
   02:53:55,794 INFO  ExecutionGraph [] - Switching from RUNNING to FAILED. Failure Error: org.apache.hudi.exception.HoodieAppendException
   {code}
   
   h3. 6. Restart fails — inflight file missing
   
   {code}
   02:59:15 ERROR StreamWriteOperatorCoordinator [] - Executor execute action [commitInstants] failed.
   Caused by: org.apache.hudi.exception.HoodieIOException: Cannot find inflight file for ...20260710185353327.deltacommit.inflight
   {code}
   
   h3. 7. Terminal state FAILED
   
   {code}
   03:26:41,727 INFO  JobMaster [] - Job job_f6d4153a4696d9b1f29c924dc8704f9b reached terminal state FAILED.
   {code}
   
   h2. Root Cause Analysis
   
   h3. Why the first deltacommit failed
   
   The {{BaseHoodieWriteClient.commit()}} method writes to the metadata table *before* calling {{saveAsComplete}} on the data table timeline. When the MDT write fails (in this case due to an HDFS DataNode pipeline break), {{saveAsComplete}} is never reached, leaving the data table's deltacommit stuck in INFLIGHT state.
   
   Additionally, in {{commitStats()}}, the {{catch}} block only catches {{IOException}}, but {{HoodieAppendException}} extends {{RuntimeException}}, so it bypasses the error handling and propagates directly to Flink's {{NonThrownExecutor}}, triggering a global job failure.
   
   {code:java}
   // BaseHoodieWriteClient.commit() — line ~320
   // ① Write MDT (BEFORE saveAsComplete)
   writeToMetadataTable(skipStreamingWritesToMetadataTable, table, instantTime,
       tableWriteStats.getMetadataTableWriteStats(), metadata);
   // ② Never reached if ① fails
   activeTimeline.saveAsComplete(false,
       table.getMetaClient().createNewInstant(
           HoodieInstant.State.INFLIGHT, commitActionType, instantTime),
       Option.of(metadata), ...);
   {code}
   
   {code:java}
   // BaseHoodieWriteClient.commitStats() — line ~279
   try {
       commit(table, commitActionType, instantTime, metadata,
           tableWriteStats, skipStreamingWritesToMetadataTable);
   } catch (IOException e) {
       // HoodieAppendException extends RuntimeException — NOT caught here!
       throw new HoodieCommitException(...);
   }
   // HoodieAppendException escapes to Flink's NonThrownExecutor → global failure
   {code}
   
   h3. Why every restart attempt fails
   
   The {{StreamWriteOperatorCoordinator}} retries from the oldest pending instant on each restart. However, the inflight file for that instant no longer exists — it was either cleaned up during the failed write or lost due to the pipeline failure. The {{ActiveTimelineV2.transitionStateToComplete()}} method requires the inflight file to exist, so it throws:
   
   {code:java}
   // ActiveTimelineV2.transitionStateToComplete()
   Preconditions.checkArgument(isInflight(inflightInstant),
       "Unable to find instant (%s)", inflightInstant);
   {code}
   
   h3. Why Flink is more affected than Spark
   
   {{SparkRDDWriteClient}} overrides {{writeToMetadataTable()}} with streaming-aware handling. {{HoodieFlinkWriteClient}} does *not* override this method — it uses the base class default which always performs synchronous MDT writes. Furthermore, Flink hardcodes {{skipStreamingWritesToMetadataTable = false}}, meaning MDT writes are never skipped.
   
   h2. Proposed Fixes
   
   h3. Fix 1: Tolerate MDT write failure (BaseHoodieWriteClient.commit)
   
   Wrap the MDT write in a try-catch block so its failure does not block {{saveAsComplete}} on the data table:
   
   {code:java}
   protected void commit(HoodieTable table, String commitActionType, String instantTime,
                         HoodieCommitMetadata metadata, TableWriteStats tableWriteStats,
                         boolean skipStreamingWritesToMetadataTable) throws IOException {
       finalizeWrite(table, instantTime, tableWriteStats.getDataTableWriteStats());
       saveInternalSchema(table, instantTime, metadata);
   
       try {
           writeToMetadataTable(skipStreamingWritesToMetadataTable, table, instantTime,
               tableWriteStats.getMetadataTableWriteStats(), metadata);
       } catch (Exception e) {
           LOG.error("Failed to write to metadata table for instant {}, "
               + "proceeding with data table commit. MDT can be recovered via bootstrap.",
               instantTime, e);
       }
   
       if (isLockRequiredForActiveTimeline(table.getMetaClient())) {
           this.txnManager.beginStateChange(
               Option.of(table.getMetaClient().createNewInstant(
                   HoodieInstant.State.INFLIGHT, commitActionType, instantTime)));
       }
       activeTimeline.saveAsComplete(false,
           table.getMetaClient().createNewInstant(
               HoodieInstant.State.INFLIGHT, commitActionType, instantTime),
           Option.of(metadata), tableWriteStats.getRuntimeStats());
   }
   {code}
   
   h3. Fix 2: Skip missing inflight files on restart (StreamWriteOperatorCoordinator)
   
   When the inflight file is missing, skip the instant instead of failing:
   
   {code:java}
   // StreamWriteOperatorCoordinator.commitInstants()
   if (allInstants.isEmpty()) {
       // no pending commits, skip
       return;
   }
   HoodieInstant instant = allInstants.get(0);
   
   // Skip if inflight file no longer exists (e.g., cleaned up after a previous failed attempt)
   if (!writeClient.getTable().getMetaClient().getActiveTimeline()
           .filter(inflights -> inflights.contains(instant)).isPresent()) {
       LOG.warn("Skipping instant {} — inflight file not found in active timeline", instant);
       return;
   }
   {code}
   
   h3. Fix 3: Catch HoodieAppendException in commitStats
   
   Extend the catch block to include {{RuntimeException}} or specifically {{HoodieAppendException}}:
   
   {code:java}
   try {
       commit(table, commitActionType, instantTime, metadata,
           tableWriteStats, skipStreamingWritesToMetadataTable);
   } catch (IOException | HoodieAppendException e) {
       throw new HoodieCommitException(
           "Failed to commit " + instantTime + " time=" + commitTime, e);
   }
   {code}
   
   h2. Current Workaround
   
   {code:sql}
   ALTER TABLE t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock
   SET TBLPROPERTIES (
     'hoodie.metadata.enable' = 'false'
   );
   {code}
   
   This disables MDT writes entirely, preventing the coupling issue. It is safe for single-writer scenarios using BUCKET index with compaction enabled.
   
   
   **additionLog**
   `2026-07-16 03:02:37,282 INFO  org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2 [] - Completed [==>20260716025337008__deltacommit__INFLIGHT]
   2026-07-16 03:02:37,287 INFO  org.apache.hudi.common.fs.FSUtils                            [] - Removed directory at hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/.hoodie/.temp/20260716025337008
   2026-07-16 03:02:37,288 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Committed 20260716025337008
   2026-07-16 03:02:37,293 INFO  org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2 [] - Marking instant complete [==>20260716025337008__deltacommit__INFLIGHT]
   2026-07-16 03:02:37,294 INFO  org.apache.hudi.client.transaction.TransactionManager        [] - State change ending for action instant Option{val=[==>20260716025337008__deltacommit__INFLIGHT]}
   2026-07-16 03:02:37,295 INFO  org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock, Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@1efbc60b[Write locks = 1, Read locks = 0], Thread meta-event-handle, In-process lock state RELEASING
   2026-07-16 03:02:37,295 INFO  org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock, Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@1efbc60b[Write locks = 0, Read locks = 0], Thread meta-event-handle, In-process lock state RELEASED
   2026-07-16 03:02:37,296 INFO  org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock, Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@1efbc60b[Write locks = 0, Read locks = 0], Thread meta-event-handle, In-process lock state ALREADY_RELEASED
   2026-07-16 03:02:37,297 INFO  org.apache.hudi.client.transaction.lock.LockManager          [] - Released connection created for acquiring lock
   2026-07-16 03:02:37,297 INFO  org.apache.hudi.client.transaction.TransactionManager        [] - State change ended for action instant Option{val=[==>20260716025337008__deltacommit__INFLIGHT]}
   2026-07-16 03:02:37,297 INFO  org.apache.hudi.client.heartbeat.HoodieHeartbeatClient       [] - Stopping heartbeat for instant 20260716025337008
   2026-07-16 03:02:37,298 INFO  org.apache.hudi.client.heartbeat.HoodieHeartbeatClient       [] - Stopped heartbeat for instant 20260716025337008
   2026-07-16 03:02:37,300 INFO  org.apache.hudi.client.heartbeat.HeartbeatUtils              [] - Deleted the heartbeat for instant 20260716025337008
   2026-07-16 03:02:37,300 INFO  org.apache.hudi.client.heartbeat.HoodieHeartbeatClient       [] - Deleted heartbeat file for instant 20260716025337008
   2026-07-16 03:02:37,301 ERROR org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [commits the instant 20260716030218327] error
   java.lang.IllegalArgumentException: File hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/timeline/20260716025337008.deltacommit.inflight does not exist!
   	at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.transitionStateToComplete(ActiveTimelineV2.java:563) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:171) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:162) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:178) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:321) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:276) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:244) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:107) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:74) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:226) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:617) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:602) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$commitInstants$10(StreamWriteOperatorCoordinator.java:557) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_281]
   	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_281]
   	at java.util.concurrent.ConcurrentSkipListMap$EntrySpliterator.forEachRemaining(ConcurrentSkipListMap.java:3535) ~[?:1.8.0_281]
   	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_281]
   	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_281]
   	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_281]
   	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_281]
   	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_281]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstants(StreamWriteOperatorCoordinator.java:557) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$3(StreamWriteOperatorCoordinator.java:299) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:131) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_281]
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_281]
   	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281]
   2026-07-16 03:02:37,304 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Trying to recover from a global failure.
   org.apache.flink.util.FlinkException: Global failure triggered by OperatorCoordinator for 'bucket_write: yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock' (operator b754eda8ef527d9494a0b43e9bf7abf3).
   	at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:624) ~[flink-dist-1.18.1.jar:1.18.1]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$start$0(StreamWriteOperatorCoordinator.java:230) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.utils.NonThrownExecutor.handleException(NonThrownExecutor.java:143) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:134) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_281]
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_281]
   	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_281]
   Caused by: org.apache.hudi.exception.HoodieException: Executor executes action [commits the instant 20260716030218327] error
   	... 6 more
   Caused by: java.lang.IllegalArgumentException: File hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/timeline/20260716025337008.deltacommit.inflight does not exist!
   	at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.transitionStateToComplete(ActiveTimelineV2.java:563) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:171) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:162) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:178) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:321) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:276) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:244) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:107) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:74) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:226) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:617) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:602) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$commitInstants$10(StreamWriteOperatorCoordinator.java:557) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_281]
   	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_281]
   	at java.util.concurrent.ConcurrentSkipListMap$EntrySpliterator.forEachRemaining(ConcurrentSkipListMap.java:3535) ~[?:1.8.0_281]
   	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_281]
   	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_281]
   	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_281]
   	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_281]
   	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_281]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstants(StreamWriteOperatorCoordinator.java:557) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$3(StreamWriteOperatorCoordinator.java:299) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:131) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	... 3 more
   2026-07-16 03:02:37,305 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - 4 tasks will be restarted to recover from a global failure.
   2026-07-16 03:02:37,306 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job insert-into_hoodie_catalog.yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (a9775e03fcf62008b13b248ee8693bb9) switched from state RUNNING to RESTARTING.
   2026-07-16 03:02:37,307 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - compact_plan_generate (1/1) (f49a8f6332f3f91794ef2cae46a713ef_2c190601d2e331d8fbe808700bcf6c4e_0_2) switched from RUNNING to CANCELING.
   2026-07-16 03:02:37,308 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - compact_task -> Sink: compact_commit (1/1) (f49a8f6332f3f91794ef2cae46a713ef_edd9ec76dbb7cd545d93de35f574c84e_0_2) switched from RUNNING to CANCELING.
   2026-07-16 03:02:37,308 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: source_kafka_01[1] -> Calc[2] -> ConstraintEnforcer[3] -> row_data_to_hoodie_record (1/1) (f49a8f6332f3f91794ef2cae46a713ef_cbc357ccb763df2852fee8c4fc7d55f2_0_2) switched from RUNNING to CANCELING.
   2026-07-16 03:02:37,308 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing registered reader after failure for subtask 0 (#2) of source Source: source_kafka_01[1].
   2026-07-16 03:02:37,309 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - bucket_write: yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (1/1) (f49a8f6332f3f91794ef2cae46a713ef_b754eda8ef527d9494a0b43e9bf7abf3_0_2) switched from RUNNING to CANCELING.
   2026-07-16 03:02:37,325 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - compact_plan_generate (1/1) (f49a8f6332f3f91794ef2cae46a713ef_2c190601d2e331d8fbe808700bcf6c4e_0_2) switched from CANCELING to CANCELED.
   2026-07-16 03:02:37,326 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - compact_task -> Sink: compact_commit (1/1) (f49a8f6332f3f91794ef2cae46a713ef_edd9ec76dbb7cd545d93de35f574c84e_0_2) switched from CANCELING to CANCELED.
   2026-07-16 03:02:37,330 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - bucket_write: yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (1/1) (f49a8f6332f3f91794ef2cae46a713ef_b754eda8ef527d9494a0b43e9bf7abf3_0_2) switched from CANCELING to CANCELED.
   2026-07-16 03:02:37,335 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: source_kafka_01[1] -> Calc[2] -> ConstraintEnforcer[3] -> row_data_to_hoodie_record (1/1) (f49a8f6332f3f91794ef2cae46a713ef_cbc357ccb763df2852fee8c4fc7d55f2_0_2) switched from CANCELING to CANCELED.
   2026-07-16 03:02:37,336 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Clearing resource requirements of job a9775e03fcf62008b13b248ee8693bb9
   2026-07-16 03:02:37,337 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedTaskManagerTracker [] - Clear all pending allocations for job a9775e03fcf62008b13b248ee8693bb9.
   2026-07-16 03:03:37,311 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job insert-into_hoodie_catalog.yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (a9775e03fcf62008b13b248ee8693bb9) switched from state RESTARTING to RUNNING.
   2026-07-16 03:03:37,312 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Restoring job a9775e03fcf62008b13b248ee8693bb9 from Checkpoint 74 @ 1784142138303 for a9775e03fcf62008b13b248ee8693bb9 located at hdfs://rtcluster/user/flink/checkpoints/a9775e03fcf62008b13b248ee8693bb9/chk-74.
   2026-07-16 03:03:37,313 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - No master state to restore
   2026-07-16 03:03:37,313 INFO  org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator [] - Resetting coordinator to checkpoint.
   2026-07-16 03:03:37,314 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Closing SourceCoordinator for source Source: source_kafka_01[1].
   2026-07-16 03:03:37,317 INFO  org.apache.flink.kafka.shaded.org.apache.kafka.common.utils.AppInfoParser [] - App info kafka.admin.client for hudi-11-test-1-ym3_11_mem94-enumerator-admin-client unregistered
   2026-07-16 03:03:37,318 INFO  org.apache.flink.kafka.shaded.org.apache.kafka.common.metrics.Metrics [] - Metrics scheduler closed
   2026-07-16 03:03:37,319 INFO  org.apache.flink.kafka.shaded.org.apache.kafka.common.metrics.Metrics [] - Closing reporter org.apache.flink.kafka.shaded.org.apache.kafka.common.metrics.JmxReporter
   2026-07-16 03:03:37,319 INFO  org.apache.flink.kafka.shaded.org.apache.kafka.common.metrics.Metrics [] - Metrics reporters closed
   2026-07-16 03:03:37,320 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source coordinator for source Source: source_kafka_01[1] closed.
   2026-07-16 03:03:37,321 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Restoring SplitEnumerator of source Source: source_kafka_01[1] from checkpoint.
   2026-07-16 03:03:37,322 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Starting split enumerator for source Source: source_kafka_01[1].
   2026-07-16 03:03:37,332 INFO  org.apache.flink.kafka.shaded.org.apache.kafka.clients.admin.AdminClientConfig [] - AdminClientConfig values: 
   	auto.include.jmx.reporter = true
   	bootstrap.servers = [bd-dw-kafka-n1.bestpay.mid:9092, bd-dw-kafka-n2.bestpay.mid:9092, bd-dw-kafka-n3.bestpay.mid:9092, bd-dw-kafka-n4.bestpay.mid:9092, bd-dw-kafka-n5.bestpay.mid:9092]
   	client.dns.lookup = use_all_dns_ips
   	client.id = hudi-11-test-1-ym3_11_mem94-enumerator-admin-client
   	connections.max.idle.ms = 300000
   	default.api.timeout.ms = 60000
   	metadata.max.age.ms = 300000
   	metric.reporters = []
   	metrics.num.samples = 2
   	metrics.recording.level = INFO
   	metrics.sample.window.ms = 30000
   	receive.buffer.bytes = 65536
   	reconnect.backoff.max.ms = 1000
   	reconnect.backoff.ms = 50
   	request.timeout.ms = 30000
   	retries = 2147483647
   	retry.backoff.ms = 100
   	sasl.client.callback.handler.class = null
   	sasl.jaas.config = null
   	sasl.kerberos.kinit.cmd = /usr/bin/kinit
   	sasl.kerberos.min.time.before.relogin = 60000
   	sasl.kerberos.service.name = null
   	sasl.kerberos.ticket.renew.jitter = 0.05
   	sasl.kerberos.ticket.renew.window.factor = 0.8
   	sasl.login.callback.handler.class = null
   	sasl.login.class = null
   	sasl.login.connect.timeout.ms = null
   	sasl.login.read.timeout.ms = null
   	sasl.login.refresh.buffer.seconds = 300
   	sasl.login.refresh.min.period.seconds = 60
   	sasl.login.refresh.window.factor = 0.8
   	sasl.login.refresh.window.jitter = 0.05
   	sasl.login.retry.backoff.max.ms = 10000
   	sasl.login.retry.backoff.ms = 100
   	sasl.mechanism = GSSAPI
   	sasl.oauthbearer.clock.skew.seconds = 30
   	sasl.oauthbearer.expected.audience = null
   	sasl.oauthbearer.expected.issuer = null
   	sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
   	sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
   	sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
   	sasl.oauthbearer.jwks.endpoint.url = null
   	sasl.oauthbearer.scope.claim.name = scope
   	sasl.oauthbearer.sub.claim.name = sub
   	sasl.oauthbearer.token.endpoint.url = null
   	security.protocol = PLAINTEXT
   	security.providers = null
   	send.buffer.bytes = 131072
   	socket.connection.setup.timeout.max.ms = 30000
   	socket.connection.setup.timeout.ms = 10000
   	ssl.cipher.suites = null
   	ssl.enabled.protocols = [TLSv1.2]
   	ssl.endpoint.identification.algorithm = https
   	ssl.engine.factory.class = null
   	ssl.key.password = null
   	ssl.keymanager.algorithm = SunX509
   	ssl.keystore.certificate.chain = null
   	ssl.keystore.key = null
   	ssl.keystore.location = null
   	ssl.keystore.password = null
   	ssl.keystore.type = JKS
   	ssl.protocol = TLSv1.2
   	ssl.provider = null
   	ssl.secure.random.implementation = null
   	ssl.trustmanager.algorithm = PKIX
   	ssl.truststore.certificates = null
   	ssl.truststore.location = null
   	ssl.truststore.password = null
   	ssl.truststore.type = JKS
   
    `


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Flink] Metadata table write failure during deltacommit causes permanent job failure with restart loop [hudi]

Posted by "YinChunGuang (via GitHub)" <gi...@apache.org>.
YinChunGuang commented on issue #19301:
URL: https://github.com/apache/hudi/issues/19301#issuecomment-5001548469

   @danny0405  Sounds cool ,thank you very much ! I merged the pr , and rerun the job. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Flink] Metadata table write failure during deltacommit causes permanent job failure with restart loop [hudi]

Posted by "YinChunGuang (via GitHub)" <gi...@apache.org>.
YinChunGuang commented on issue #19301:
URL: https://github.com/apache/hudi/issues/19301#issuecomment-4987773864

   hoodie.properties
   2026-07-16 03:08:08,235 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating View Manager with storage type MEMORY.
   2026-07-16 03:08:08,235 INFO  org.apache.hudi.keygen.constant.KeyGeneratorType             [] - No key generator type is set properly
   2026-07-16 03:08:08,236 INFO  org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder [] - Building HoodieLogFormat Writer
   2026-07-16 03:08:08,237 INFO  org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder [] - Computing the next log version for files-0000-0 in hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/files
   2026-07-16 03:08:08,237 INFO  org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder [] - Computed the next log version for files-0000-0 in hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/files as 1 with write-token 0-1-0
   2026-07-16 03:08:08,237 INFO  org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder [] - HoodieLogFile on path hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/files/.files-0000-0_20260716025337008.log.1_0-1-0
   2026-07-16 03:08:08,245 INFO  org.apache.hudi.table.marker.DirectWriteMarkers              [] - [direct] Created marker file hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/.hoodie/.temp/20260716025337008/files/.files-0000-0_20260716025337008.log.1_0-1-0.marker.CREATE in 2 ms
   2026-07-16 03:08:08,247 INFO  org.apache.hudi.common.table.log.HoodieLogFormatWriter       [] - Created a new log file: HoodieLogFile{pathStr='hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/files/.files-0000-0_20260716025337008.log.1_0-1-0', fileLen=0}
   2026-07-16 03:08:08,329 INFO  org.apache.hudi.io.HoodieAppendHandle                        [] - AppendHandle for partitionPath files filePath files/.files-0000-0_20260716025337008.log.1_0-1-0, took 93 ms.
   2026-07-16 03:08:08,364 INFO  org.apache.hudi.common.table.HoodieTableConfig               [] - Loading table properties from hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/.hoodie/hoodie.properties
   2026-07-16 03:08:08,365 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Committing 20260716025337008 action deltacommit
   2026-07-16 03:08:08,367 INFO  org.apache.hudi.common.table.HoodieTableConfig               [] - Loading table properties from hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/.hoodie/hoodie.properties
   2026-07-16 03:08:08,368 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating View Manager with storage type MEMORY.
   2026-07-16 03:08:08,368 INFO  org.apache.hudi.common.util.CommitUtils                      [] - Creating  metadata for UPSERT_PREPPED numWriteStats:1 numReplaceFileIds:0
   2026-07-16 03:08:08,369 INFO  org.apache.hudi.common.table.HoodieTableConfig               [] - Loading table properties from hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/.hoodie/hoodie.properties
   2026-07-16 03:08:08,370 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating View Manager with storage type MEMORY.
   2026-07-16 03:08:08,371 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Committing 20260716025337008 action deltacommit
   2026-07-16 03:08:08,381 INFO  org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2 [] - Marking instant complete [==>20260716025337008__deltacommit__INFLIGHT]
   2026-07-16 03:08:08,443 INFO  org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2 [] - Created new file for toInstant: hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/.hoodie/timeline/20260716025337008_20260716030808382.deltacommit
   2026-07-16 03:08:08,444 INFO  org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2 [] - Completed [==>20260716025337008__deltacommit__INFLIGHT]
   2026-07-16 03:08:08,447 INFO  org.apache.hudi.common.fs.FSUtils                            [] - Removed directory at hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/.hoodie/.temp/20260716025337008
   2026-07-16 03:08:08,448 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Committed 20260716025337008
   2026-07-16 03:08:08,454 INFO  org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2 [] - Marking instant complete [==>20260716025337008__deltacommit__INFLIGHT]
   2026-07-16 03:08:08,454 INFO  org.apache.hudi.client.transaction.TransactionManager        [] - State change ending for action instant Option{val=[==>20260716025337008__deltacommit__INFLIGHT]}
   2026-07-16 03:08:08,455 INFO  org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock, Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@1efbc60b[Write locks = 1, Read locks = 0], Thread meta-event-handle, In-process lock state RELEASING
   2026-07-16 03:08:08,455 INFO  org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock, Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@1efbc60b[Write locks = 0, Read locks = 0], Thread meta-event-handle, In-process lock state RELEASED
   2026-07-16 03:08:08,456 INFO  org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock, Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@1efbc60b[Write locks = 0, Read locks = 0], Thread meta-event-handle, In-process lock state ALREADY_RELEASED
   2026-07-16 03:08:08,456 INFO  org.apache.hudi.client.transaction.lock.LockManager          [] - Released connection created for acquiring lock
   2026-07-16 03:08:08,456 INFO  org.apache.hudi.client.transaction.TransactionManager        [] - State change ended for action instant Option{val=[==>20260716025337008__deltacommit__INFLIGHT]}
   2026-07-16 03:08:08,457 INFO  org.apache.hudi.client.heartbeat.HoodieHeartbeatClient       [] - Stopping heartbeat for instant 20260716025337008
   2026-07-16 03:08:08,457 INFO  org.apache.hudi.client.heartbeat.HoodieHeartbeatClient       [] - Stopped heartbeat for instant 20260716025337008
   2026-07-16 03:08:08,458 INFO  org.apache.hudi.client.heartbeat.HeartbeatUtils              [] - Deleted the heartbeat for instant 20260716025337008
   2026-07-16 03:08:08,459 INFO  org.apache.hudi.client.heartbeat.HoodieHeartbeatClient       [] - Deleted heartbeat file for instant 20260716025337008
   2026-07-16 03:08:08,459 ERROR org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [commits the instant 20260716030800298] error
   java.lang.IllegalArgumentException: File hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/timeline/20260716025337008.deltacommit.inflight does not exist!
   	at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.transitionStateToComplete(ActiveTimelineV2.java:563) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:171) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:162) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:178) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:321) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:276) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:244) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:107) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:74) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:226) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:617) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:602) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$commitInstants$10(StreamWriteOperatorCoordinator.java:557) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_281]
   	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_281]
   	at java.util.concurrent.ConcurrentSkipListMap$EntrySpliterator.forEachRemaining(ConcurrentSkipListMap.java:3535) ~[?:1.8.0_281]
   	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_281]
   	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_281]
   	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_281]
   	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_281]
   	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_281]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstants(StreamWriteOperatorCoordinator.java:557) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$3(StreamWriteOperatorCoordinator.java:299) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:131) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_281]
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_281]
   	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281]
   2026-07-16 03:08:08,462 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Trying to recover from a global failure.
   org.apache.flink.util.FlinkException: Global failure triggered by OperatorCoordinator for 'bucket_write: yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock' (operator b754eda8ef527d9494a0b43e9bf7abf3).
   	at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:624) ~[flink-dist-1.18.1.jar:1.18.1]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$start$0(StreamWriteOperatorCoordinator.java:230) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.utils.NonThrownExecutor.handleException(NonThrownExecutor.java:143) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:134) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_281]
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_281]
   	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_281]
   Caused by: org.apache.hudi.exception.HoodieException: Executor executes action [commits the instant 20260716030800298] error
   	... 6 more
   Caused by: java.lang.IllegalArgumentException: File hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/timeline/20260716025337008.deltacommit.inflight does not exist!
   	at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.transitionStateToComplete(ActiveTimelineV2.java:563) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:171) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:162) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:178) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:321) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:276) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:244) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:107) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:74) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:226) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:617) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:602) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$commitInstants$10(StreamWriteOperatorCoordinator.java:557) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_281]
   	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_281]
   	at java.util.concurrent.ConcurrentSkipListMap$EntrySpliterator.forEachRemaining(ConcurrentSkipListMap.java:3535) ~[?:1.8.0_281]
   	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_281]
   	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_281]
   	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_281]
   	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_281]
   	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_281]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstants(StreamWriteOperatorCoordinator.java:557) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$3(StreamWriteOperatorCoordinator.java:299) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:131) ~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
   	... 3 more
   2026-07-16 03:08:08,463 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - 4 tasks will be restarted to recover from a global failure.
   2026-07-16 03:08:08,464 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job insert-into_hoodie_catalog.yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (a9775e03fcf62008b13b248ee8693bb9) switched from state RUNNING to RESTARTING.
   2026-07-16 03:08:08,464 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - compact_plan_generate (1/1) (f49a8f6332f3f91794ef2cae46a713ef_2c190601d2e331d8fbe808700bcf6c4e_0_4) switched from RUNNING to CANCELING.
   2026-07-16 03:08:08,465 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - compact_task -> Sink: compact_commit (1/1) (f49a8f6332f3f91794ef2cae46a713ef_edd9ec76dbb7cd545d93de35f574c84e_0_4) switched from RUNNING to CANCELING.
   2026-07-16 03:08:08,465 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: source_kafka_01[1] -> Calc[2] -> ConstraintEnforcer[3] -> row_data_to_hoodie_record (1/1) (f49a8f6332f3f91794ef2cae46a713ef_cbc357ccb763df2852fee8c4fc7d55f2_0_4) switched from RUNNING to CANCELING.
   2026-07-16 03:08:08,465 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing registered reader after failure for subtask 0 (#4) of source Source: source_kafka_01[1].
   2026-07-16 03:08:08,466 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - bucket_write: yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (1/1) (f49a8f6332f3f91794ef2cae46a713ef_b754eda8ef527d9494a0b43e9bf7abf3_0_4) switched from RUNNING to CANCELING.
   2026-07-16 03:08:08,482 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - compact_plan_generate (1/1) (f49a8f6332f3f91794ef2cae46a713ef_2c190601d2e331d8fbe808700bcf6c4e_0_4) switched from CANCELING to CANCELED.
   2026-07-16 03:08:08,483 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - compact_task -> Sink: compact_commit (1/1) (f49a8f6332f3f91794ef2cae46a713ef_edd9ec76dbb7cd545d93de35f574c84e_0_4) switched from CANCELING to CANCELED.
   2026-07-16 03:08:08,485 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - bucket_write: yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (1/1) (f49a8f6332f3f91794ef2cae46a713ef_b754eda8ef527d9494a0b43e9bf7abf3_0_4) switched from CANCELING to CANCELED.
   2026-07-16 03:08:08,491 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: source_kafka_01[1] -> Calc[2] -> ConstraintEnforcer[3] -> row_data_to_hoodie_record (1/1) (f49a8f6332f3f91794ef2cae46a713ef_cbc357ccb763df2852fee8c4fc7d55f2_0_4) switched from CANCELING to CANCELED.
   2026-07-16 03:08:08,492 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Clearing resource requirements of job a9775e03fcf62008b13b248ee8693bb9
   2026-07-16 03:08:08,492 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedTaskManagerTracker [] - Clear all pending allocations for job a9775e03fcf62008b13b248ee8693bb9.
   2026-07-16 03:09:08,467 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job insert-into_hoodie_catalog.yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (a9775e03fcf62008b13b248ee8693bb9) switched from state RESTARTING to RUNNING.
   2026-07-16 03:09:08,468 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Restoring job a9775e03fcf62008b13b248ee8693bb9 from Checkpoint 76 @ 1784142480275 for a9775e03fcf62008b13b248ee8693bb9 located at hdfs://rtcluster/user/flink/checkpoints/a9775e03fcf62008b13b248ee8693bb9/chk-76.
   2026-07-16 03:09:08,469 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - No master state to restore
   2026-07-16 03:09:08,470 INFO  org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator [] - Resetting coordinator to checkpoint.
   2026-07-16 03:09:08,471 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Closing SourceCoordinator for source Source: source_kafka_01[1].
   2026-07-16 03:09:08,473 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: source_kafka_01[1] -> Calc[2] -> ConstraintEnforcer[3] -> row_data_to_hoodie_record (1/1) (f49a8f6332f3f91794ef2cae46a713ef_cbc357ccb763df2852fee8c4fc7d55f2_0_5) switched from CREATED to SCHEDULED.
   2026-07-16 03:09:08,473 INFO  org.apache.flink.kafka.shaded.org.apache.kafka.common.utils.AppInfoParser [] - App info kafka.admin.client for hudi-11-test-1-ym3_11_mem94-enumerator-admin-client unregistered
   2026-07-16 03:09:08,473 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - bucket_write: yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (1/1) (f49a8f6332f3f91794ef2cae46a713ef_b754eda8ef527d9494a0b43e9bf7abf3_0_5) switched from CREATED to SCHEDULED.
   2026-07-16 03:09:08,474 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - compact_plan_generate (1/1) (f49a8f6332f3f91794ef2cae46a713ef_2c190601d2e331d8fbe808700bcf6c4e_0_5) switched from CREATED to SCHEDULED.
   2026-07-16 03:09:08,474 INFO  org.apache.flink.kafka.shaded.org.apache.kafka.common.metrics.Metrics [] - Metrics scheduler closed
   2026-07-16 03:09:08,474 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - compact_task -> Sink: compact_commit (1/1) (f49a8f6332f3f91794ef2cae46a713ef_edd9ec76dbb7cd545d93de35f574c84e_0_5) switched from CREATED to SCHEDULED.
   2026-07-16 03:09:08,475 INFO  org.apache.flink.kafka.shaded.org.apache.kafka.common.metrics.Metrics [] - Closing reporter org.apache.flink.kafka.shaded.org.apache.kafka.common.metrics.JmxReporter
   2026-07-16 03:09:08,475 INFO  org.apache.flink.kafka.shaded.org.apache.kafka.common.metrics.Metrics [] - Metrics reporters closed
   2026-07-16 03:09:08,475 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Received resource requirements from job a9775e03fcf62008b13b248ee8693bb9: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1}]
   2026-07-16 03:09:08,476 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source coordinator for source Source: source_kafka_01[1] closed.
   2026-07-16 03:09:08,476 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: source_kafka_01[1] -> Calc[2] -> ConstraintEnforcer[3] -> row_data_to_hoodie_record (1/1) (f49a8f6332f3f91794ef2cae46a713ef_cbc357ccb763df2852fee8c4fc7d55f2_0_5) switched from SCHEDULED to DEPLOYING.
   2026-07-16 03:09:08,477 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: source_kafka_01[1] -> Calc[2] -> ConstraintEnforcer[3] -> row_data_to_hoodie_record (1/1) (attempt #5) with attempt id f49a8f6332f3f91794ef2cae46a713ef_cbc357ccb763df2852fee8c4fc7d55f2_0_5 and vertex id cbc357ccb763df2852fee8c4fc7d55f2_0 to container_e05_1732503259026_5944_01_000002 @ b30107u10-72-227.bestpay.bd (dataPort=11906) with allocation id 90793836ff4440d5a292b4c81d20355f
   2026-07-16 03:09:08,477 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Restoring SplitEnumerator of source Source: source_kafka_01[1] from checkpoint.
   2026-07-16 03:09:08,477 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Starting split enumerator for source Source: source_kafka_01[1].
   2026-07-16 03:09:08,477 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - bucket_write: yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (1/1) (f49a8f6332f3f91794ef2cae46a713ef_b754eda8ef527d9494a0b43e9bf7abf3_0_5) switched from SCHEDULED to DEPLOYING.
   2026-07-16 03:09:08,478 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying bucket_write: yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock (1/1) (attempt #5) with attempt id f49a8f6332f3f91794ef2cae46a713ef_b754eda8ef527d9494a0b43e9bf7abf3_0_5 and vertex id b754eda8ef527d9494a0b43e9bf7abf3_0 to container_e05_1732503259026_5944_01_000002 @ b30107u10-72-227.bestpay.bd (dataPort=11906) with allocation id 90793836ff4440d5a292b4c81d20355f
   2026-07-16 03:09:08,479 INFO  org.apache.flink.kafka.shaded.org.apache.kafka.clients.admin.AdminClientConfig [] - AdminClientConfig values: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Flink] Metadata table write failure during deltacommit causes permanent job failure with restart loop [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #19301:
URL: https://github.com/apache/hudi/issues/19301#issuecomment-4987912393

   the inflight metadata file might got rolled back by the cleaner service since it's heartbeat timeout, did you try 1.1.1 with this patch already: https://github.com/apache/hudi/pull/19237


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Flink] Metadata table write failure during deltacommit causes permanent job failure with restart loop [hudi]

Posted by "YinChunGuang (via GitHub)" <gi...@apache.org>.
YinChunGuang closed issue #19301: [Flink] Metadata table write failure during deltacommit causes permanent job failure with restart loop
URL: https://github.com/apache/hudi/issues/19301


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Flink] Metadata table write failure during deltacommit causes permanent job failure with restart loop [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #19301:
URL: https://github.com/apache/hudi/issues/19301#issuecomment-4988470491

   Thanks for confirming that #19237  is already applied. This looks different from the coordinator event-buffer loss fixed there.
   In Hudi 1.1.1, the default heartbeat expiry is approximately two minutes, while the first recommit shown here occurs more than five minutes after the original failure. Also, hoodie.clean.manual.batch.enable is not a recognized Apache Hudi configuration, so Flink async cleaning is likely still enabled.
   
   PR #19237 was merged on top of #18904 on master. Applying only #19237 to 1.1.1 does not include #18904’s resilient heartbeat writer or its increase of hoodie.client.heartbeat.tolerable.misses from 2 to 10.
   Could you test with #18904 applied as well, or temporarily configure:
   
   ```xml
   'hoodie.client.heartbeat.tolerable.misses' = '10',
   'clean.async.enabled' = 'false'
   ```
   
   Please also provide logs matching Heartbeat expired, rollback, clean, and the affected instant, plus the corresponding rollback entries from .hoodie/timeline. That will confirm whether the cleaner removed the inflight instant before coordinator recovery.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org