You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "gejinzh (via GitHub)" <gi...@apache.org> on 2025/02/10 08:34:58 UTC
[I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
gejinzh opened a new issue, #12820:
URL: https://github.com/apache/hudi/issues/12820
**Describe the problem you faced**
The flink task has been running for 17 days; Then an exception occurred
**hudi config**
```
Map<String, String> options= new HashMap<>();
options.put(FlinkOptions.PATH.key(), hudiProperties.getPath());
options.put(FlinkOptions.TABLE_TYPE.key(), HoodieTableType.MERGE_ON_READ.name());
options.put(FlinkOptions.OPERATION.key(), WriteOperationType.INSERT.value());
options.put(FlinkOptions.DATABASE_NAME.key(), hudiProperties.getDatabase());
options.put(FlinkOptions.TABLE_NAME.key(), hudiProperties.getTableName());
options.put(FlinkOptions.WRITE_TASKS.key(), String.valueOf(1));
options.put(FlinkOptions.HIVE_SYNC_ENABLED.key(), String.valueOf(true));
options.put(FlinkOptions.HIVE_SYNC_MODE.key(), HiveSyncMode.HMS.name());
options.put(FlinkOptions.HIVE_SYNC_METASTORE_URIS.key(), hudiProperties.getHiveMetastoreUris());
options.put(FlinkOptions.HIVE_SYNC_DB.key(), hudiProperties.getDatabase());
options.put(FlinkOptions.HIVE_SYNC_CONF_DIR.key(), hudiProperties.getHiveConfDir());
options.put(FlinkOptions.CLUSTERING_SCHEDULE_ENABLED.key(), String.valueOf(true));
options.put(FlinkOptions.CLUSTERING_ASYNC_ENABLED.key(), String.valueOf(true));
options.put(FlinkOptions.CLUSTERING_TASKS.key(), String.valueOf(1));
options.put(FlinkOptions.WRITE_BULK_INSERT_SORT_INPUT.key(), String.valueOf(false));
options.put(FlinkOptions.WRITE_RATE_LIMIT.key(), String.valueOf(500));
```
**Environment Description**
* Hudi version : 0.15.0
* Hadoop version : 3.3.3
* Storage (HDFS/S3/GCS..) : hdfs
* Running on Docker? (yes/no) : no
**Stacktrace**
```
org.apache.hudi.exception.HoodieException: Timeout(121000ms) while waiting for instant initialize
at org.apache.hudi.sink.utils.TimeWait.waitFor(TimeWait.java:57)
at org.apache.hudi.sink.common.AbstractStreamWriteFunction.instantToWrite(AbstractStreamWriteFunction.java:269)
at org.apache.hudi.sink.append.AppendWriteFunction.initWriterHelper(AppendWriteFunction.java:125)
at org.apache.hudi.sink.append.AppendWriteFunction.processElement(AppendWriteFunction.java:84)
at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66)
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:233)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105)
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:546)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:837)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:786)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
at java.lang.Thread.run(Thread.java:750)
```
checkpoint interval: 5minutes
but deltaCommit spend 15minutes or 10 minutes;
```
-rw-rw-r-- 3 yxfcenter hadoop 5455 2025-02-10 05:36 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210052607892.deltacommit
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 05:26 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210052607892.deltacommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 05:26 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210052607892.deltacommit.requested
-rw-rw-r-- 3 yxfcenter hadoop 5455 2025-02-10 05:46 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210053608150.deltacommit
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 05:36 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210053608150.deltacommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 05:36 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210053608150.deltacommit.requested
-rw-rw-r-- 3 yxfcenter hadoop 5455 2025-02-10 05:56 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210054609874.deltacommit
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 05:46 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210054609874.deltacommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 05:46 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210054609874.deltacommit.requested
-rw-rw-r-- 3 yxfcenter hadoop 14324 2025-02-10 06:01 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210055609022.replacecommit
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:01 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210055609022.replacecommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 10382 2025-02-10 05:56 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210055609022.replacecommit.requested
-rw-rw-r-- 3 yxfcenter hadoop 5455 2025-02-10 06:06 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210055609184.deltacommit
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 05:56 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210055609184.deltacommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 05:56 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210055609184.deltacommit.requested
-rw-rw-r-- 3 yxfcenter hadoop 6786 2025-02-10 06:06 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210060609076.clean
-rw-rw-r-- 3 yxfcenter hadoop 7219 2025-02-10 06:06 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210060609076.clean.inflight
-rw-rw-r-- 3 yxfcenter hadoop 7219 2025-02-10 06:06 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210060609076.clean.requested
-rw-rw-r-- 3 yxfcenter hadoop 5455 2025-02-10 06:16 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210060609288.deltacommit
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:06 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210060609288.deltacommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:06 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210060609288.deltacommit.requested
-rw-rw-r-- 3 yxfcenter hadoop 5455 2025-02-10 06:26 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210061609535.deltacommit
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:16 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210061609535.deltacommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:16 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210061609535.deltacommit.requested
-rw-rw-r-- 3 yxfcenter hadoop 5455 2025-02-10 06:36 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210062610189.deltacommit
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:26 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210062610189.deltacommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:26 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210062610189.deltacommit.requested
-rw-rw-r-- 3 yxfcenter hadoop 14326 2025-02-10 06:41 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210063610893.replacecommit
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:41 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210063610893.replacecommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 10382 2025-02-10 06:36 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210063610893.replacecommit.requested
-rw-rw-r-- 3 yxfcenter hadoop 5451 2025-02-10 06:46 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210063611067.deltacommit
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:36 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210063611067.deltacommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:36 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210063611067.deltacommit.requested
-rw-rw-r-- 3 yxfcenter hadoop 6210 2025-02-10 06:46 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210064610874.clean
-rw-rw-r-- 3 yxfcenter hadoop 6555 2025-02-10 06:46 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210064610874.clean.inflight
-rw-rw-r-- 3 yxfcenter hadoop 6555 2025-02-10 06:46 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210064610874.clean.requested
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:46 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210064611475.deltacommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:46 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210064611475.deltacommit.requested
drwxrwxr-x - yxfcenter hadoop 0 2025-01-24 07:20 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/archived
-rw-rw-r-- 3 yxfcenter hadoop 2352 2025-01-23 20:09 /user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/hoodie.properties
```
The entire task is stuck on instant 20250210064611475
what can I do to fix it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on issue #12820:
URL: https://github.com/apache/hudi/issues/12820#issuecomment-2662484348
> SendBootStrap doesnot be executed on this condition; I analyze the logic try to describe what happened in my condition; I change the code on local and solved this problem;
>
> 
>
> 
>
> 
Hi @gejinzh, the timeout exception you reported seems indeed a corner case, would you like create a PR to fix it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on issue #12820:
URL: https://github.com/apache/hudi/issues/12820#issuecomment-2664356189
> I will
👍, you can first create a JIRA to track the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "gejinzh (via GitHub)" <gi...@apache.org>.
gejinzh commented on issue #12820:
URL: https://github.com/apache/hudi/issues/12820#issuecomment-2647699092
```
recommit the inflight instant(AbstractStreamWriteFunction::restoreWriteMetadata)
-> reuse currentInstant, reject bootstrap(handleBootstrapEvent)
-> instantToWrite(AppendWriteFunction::initWriterHelper)
-> error(Timeout(121000ms) while waiting for instant initialize)
-> recommit *******
It seems to be a dead loop
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "gejinzh (via GitHub)" <gi...@apache.org>.
gejinzh commented on issue #12820:
URL: https://github.com/apache/hudi/issues/12820#issuecomment-2662608192
I will
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #12820:
URL: https://github.com/apache/hudi/issues/12820#issuecomment-2658527816
The event restored from state is not bootstrap event, see the last fallback logic:
```java
if (!eventSent) {
sendBootstrapEvent();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "gejinzh (via GitHub)" <gi...@apache.org>.
gejinzh commented on issue #12820:
URL: https://github.com/apache/hudi/issues/12820#issuecomment-2655247495
> > the coordiator reuse the instant but does not unblock the writers; On source code only send the commi
>
> The initInstant will unblock the writers, I want to see the recommit fail log, "Timeout while waiting for instant initialize" is just from the writer side.
InitInstant will not be executed becuase it returned before;
part of jobmanager log on here
[f (1).txt](https://github.com/user-attachments/files/18776208/f.1.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #12820:
URL: https://github.com/apache/hudi/issues/12820#issuecomment-2649688170
Is there any log indicating why the recommit fails?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #12820:
URL: https://github.com/apache/hudi/issues/12820#issuecomment-2652650321
The commit ack event is designed for instant reuse, when a commit is empty, the coordinator can reuse the instant for next commit, but here it should be okay, because in the last step of `handleBootstrapEvent`, `initInstant(instant)` is invoked and unblock the wrtiers, can you show us the detailed logs about the recommit failure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "gejinzh (via GitHub)" <gi...@apache.org>.
gejinzh commented on issue #12820:
URL: https://github.com/apache/hudi/issues/12820#issuecomment-2658368072
> It looks like the condition should be `attemptId >= 0` and the logic in `#handleBootstrapEvent` can be removed.
```
if (context.isRestored()) {
restoreWriteMetadata();
} else {
sendBootstrapEvent();
}
```
```#restoreWriteMetaData``` executed;
```
private void restoreWriteMetadata() throws Exception {
boolean eventSent = false;
HoodieTimeline pendingTimeline = this.metaClient.getActiveTimeline().filterPendingExcludingCompaction();
for (WriteMetadataEvent event : this.writeMetadataState.get()) {
// Must filter out the completed instants in case it is a partial failover,
// the write status should not be accumulated in such case.
if (pendingTimeline.containsInstant(event.getInstantTime())) {
// Reset taskID for event
event.setTaskID(taskID);
// The checkpoint succeed but the meta does not commit,
// re-commit the inflight instant
this.eventGateway.sendEventToCoordinator(event);
LOG.info("Send uncommitted write metadata event to coordinator, task[{}].event: [{}]", taskID, event);
eventSent = true;
}
}
if (!eventSent) {
sendBootstrapEvent();
}
}
```
The write statuses of the event recovered from the checkpoint are empty and have a Inflight instant; so coordinator reused instant but not unblocked write tasks;
I haven't modified the code to solve this problem yet, I will give it a try
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "gejinzh (via GitHub)" <gi...@apache.org>.
gejinzh commented on issue #12820:
URL: https://github.com/apache/hudi/issues/12820#issuecomment-2658698825
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #12820:
URL: https://github.com/apache/hudi/issues/12820#issuecomment-2658180771
@gejinzh Did you make the changes local and validate the change solves the problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "gejinzh (via GitHub)" <gi...@apache.org>.
gejinzh commented on issue #12820:
URL: https://github.com/apache/hudi/issues/12820#issuecomment-2650032653
> Is there any log indicating why the recommit fails?
taskmanager
```
2025-02-10 21:36:48,192 Source: ygs_ods_jt_order_rel -> hoodie_append_write: default_database.ygs_ods_jt_order_rel (1/1)#1 INFO org.apache.hudi.sink.common.AbstractStreamWriteFunction [] - Send uncommitted write metadata event to coordinator, task[0].
```
but jobmanager reuse current pending instant
```
2025-02-10 21:36:48,228 pool-22-thread-1 WARN org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Reuse current pending Instant 20250210210742850 with insert operationType, ignoring empty bootstrap event.
```
Recommit should has log like 'recommit instant **' , but it doesn't;
```
if (this.metaClient.reloadActiveTimeline().filterInflightsAndRequested().containsInstant(this.instant)
&& instant.equals(WriteMetadataEvent.BOOTSTRAP_INSTANT)
&& this.tableState.operationType == WriteOperationType.INSERT) {
```
may be the instant is 'BOOTSTRAP_INSTANT'?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "gejinzh (via GitHub)" <gi...@apache.org>.
gejinzh closed issue #12820: [SUPPORT]Timeout while waiting for instant initialize
URL: https://github.com/apache/hudi/issues/12820
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #12820:
URL: https://github.com/apache/hudi/issues/12820#issuecomment-2658330002
The append write function bootstrap event send is kind of special:
```java
protected void sendBootstrapEvent() {
int attemptId = getRuntimeContext().getAttemptNumber();
if (attemptId > 0) {
// either a partial or global failover, reuses the current inflight instant
if (this.currentInstant != null && !metaClient.getActiveTimeline().filterCompletedInstants().containsInstant(currentInstant)) {
LOG.info("Recover task[{}] for instant [{}] with attemptId [{}]", taskID, this.currentInstant, attemptId);
this.currentInstant = null;
return;
}
// the JM may have also been rebooted, sends the bootstrap event either
}
this.eventGateway.sendEventToCoordinator(WriteMetadataEvent.emptyBootstrap(taskID));
LOG.info("Send bootstrap write metadata event to coordinator, task[{}].", taskID);
}
```
It looks like the condition should be `attemptId >= 0` and the logic in `#handleBootstrapEvent` can be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on issue #12820:
URL: https://github.com/apache/hudi/issues/12820#issuecomment-2662537277
> SendBootStrap doesnot be executed on this condition;
> I analyze the logic try to describe what happened in my condition;
> I change the code on local and solved this problem;
Hi @gejinzh, thks for the detailed description. I think it's indeed a corner bug for INSERT ingestion. Would you like create a PR to fix the problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "gejinzh (via GitHub)" <gi...@apache.org>.
gejinzh commented on issue #12820:
URL: https://github.com/apache/hudi/issues/12820#issuecomment-2661766032
SendBootStrap doesnot be executed on this condition;
I analyze the logic try to describe what happened in my condition;
I change the code on local and fiugre
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #12820:
URL: https://github.com/apache/hudi/issues/12820#issuecomment-2655206143
> the coordiator reuse the instant but does not unblock the writers; On source code only send the commi
The initInstant will unblock the writers, I want to see the recommit fail log, "Timeout while waiting for instant initialize" is just from the writer side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "gejinzh (via GitHub)" <gi...@apache.org>.
gejinzh commented on issue #12820:
URL: https://github.com/apache/hudi/issues/12820#issuecomment-2650111073
The recommit instant has no data; So the code handles it like no data event because write statuses's size is zero; but the 'isConfirming' flag does not be switched;
```
StreamWriteOperatorCoordinator::handleBootstrapEvent
final String instant = Arrays.stream(eventBuffer)
.filter(evt -> evt.getWriteStatuses().size() > 0)
.findFirst().map(WriteMetadataEvent::getInstantTime)
.orElse(WriteMetadataEvent.BOOTSTRAP_INSTANT);
```
Is the good idea to send commit event again on there or just send commit event on there?
like this
```
if (this.metaClient.reloadActiveTimeline().filterInflightsAndRequested().containsInstant(this.instant)
&& instant.equals(WriteMetadataEvent.BOOTSTRAP_INSTANT)
&& this.tableState.operationType == WriteOperationType.INSERT) {
LOG.warn("Reuse current pending Instant {} with {} operationType, "
+ "ignoring empty bootstrap event. writeMetaDataEvent: {}; instant: {}", this.instant, WriteOperationType.INSERT.value(), event, instant);
reset();
sendCommitAckEvents(-1L); // new code
return;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "gejinzh (via GitHub)" <gi...@apache.org>.
gejinzh commented on issue #12820:
URL: https://github.com/apache/hudi/issues/12820#issuecomment-2652748242
> The commit ack event is designed for instant reuse, when a commit is empty, the coordinator can reuse the instant for next commit, but here it should be okay, because in the last step of `handleBootstrapEvent`, `initInstant(instant)` is invoked and unblock the wrtiers, can you show us the detailed logs about the recommit failure?
Hudi source code on 'StreamWriteOperatorCoordinator::handleBootstrapEvent' does not send commit ack event;
I think may be we can add code 'sendCommitAckEvents(-1L)'
Because when a instant is empty and recover from checkpoint, the coordiator reuse the instant but does not unblock the writers; On source code only send the commit ack event on 'notifyCheckpointComplete';
the part log on here not other errors just '[Timeout while waiting for instant initialize]'
[f.txt](https://github.com/user-attachments/files/18763016/f.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "gejinzh (via GitHub)" <gi...@apache.org>.
gejinzh commented on issue #12820:
URL: https://github.com/apache/hudi/issues/12820#issuecomment-2661764824
SendBootStrap doesnot be executed on this condition;
I analyze the logic try to describe what happened in my condition;
I change the code on local and solved this problem;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #12820:
URL: https://github.com/apache/hudi/issues/12820#issuecomment-2660687789
Just modify `attemptId >= 0` and test it, the code is complicated and it does not make too much sense to analyze the logic here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "gejinzh (via GitHub)" <gi...@apache.org>.
gejinzh commented on issue #12820:
URL: https://github.com/apache/hudi/issues/12820#issuecomment-2664758878
jira: https://issues.apache.org/jira/browse/HUDI-9041
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org