You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "gjboomshit (via GitHub)" <gi...@apache.org> on 2024/11/14 10:04:31 UTC
[I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
gjboomshit opened a new issue, #12256:
URL: https://github.com/apache/hudi/issues/12256
I am running a flink job to writing data into huditable; then i get a error;
```2024-11-14 17:15:50,553 Source: ygs_ods_inner_ord_offer_inst_pay_info -> hoodie_append_write: ygs_ods_inner_ord_offer_inst_pay_info (1/1)#1 WARN org.apache.flink.runtime.taskmanager.Task [] - Source: ygs_ods_inner_ord_offer_inst_pay_info -> hoodie_append_write: ygs_ods_inner_ord_offer_inst_pay_info (1/1)#1 (de1741b20c5351bdb59d746b1bacc87f_6dc0226b15c44c9c2e1f9ea1a65fd400_0_1) switched from RUNNING to FAILED with failure cause: java.io.IOException: Failed to deserialize consumer record due to
at org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter.emitRecord(KafkaRecordEmitter.java:56)
at org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter.emitRecord(KafkaRecordEmitter.java:33)
at org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:143)
at org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:389)
at org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:546)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:837)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:786)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException: Could not forward element to next operator
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:99)
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57)
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
at org.apache.flink.streaming.runtime.tasks.SourceOperatorStreamTask$AsyncDataOutputToOutput.emitRecord(SourceOperatorStreamTask.java:313)
at org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:110)
at org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter$SourceOutputWrapper.collect(KafkaRecordEmitter.java:67)
at org.apache.flink.api.common.serialization.DeserializationSchema.deserialize(DeserializationSchema.java:84)
at org.apache.flink.connector.kafka.source.reader.deserializer.KafkaValueOnlyDeserializationSchemaWrapper.deserialize(KafkaValueOnlyDeserializationSchemaWrapper.java:51)
at org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter.emitRecord(KafkaRecordEmitter.java:53)
... 14 more
Caused by: org.apache.hudi.exception.HoodieException: Timeout(61000ms) while waiting for instant initialize
at org.apache.hudi.sink.utils.TimeWait.waitFor(TimeWait.java:57)
at org.apache.hudi.sink.common.AbstractStreamWriteFunction.instantToWrite(AbstractStreamWriteFunction.java:276)
at org.apache.hudi.sink.append.AppendWriteFunction.initWriterHelper(AppendWriteFunction.java:110)
at org.apache.hudi.sink.append.AppendWriteFunction.processElement(AppendWriteFunction.java:84)
at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66)
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:82)
... 22 more
```
have 6 taskmanger with 2048M memery and i have 64 task;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "gjboomshit (via GitHub)" <gi...@apache.org>.
gjboomshit commented on issue #12256:
URL: https://github.com/apache/hudi/issues/12256#issuecomment-2482114699
I solved this problem by adding memory and increasing the interval between checkpoints； But I still haven't figured out what caused the error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #12256:
URL: https://github.com/apache/hudi/issues/12256#issuecomment-2482840695
Because there are two many pipelines in one job, all share the same checkpoint lifecycle, that makes the checkpopint very time-consuming, which is the reason for checkpoint timeout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "gjboomshit (via GitHub)" <gi...@apache.org>.
gjboomshit closed issue #12256: [SUPPORT]Timeout while waiting for instant initialize
URL: https://github.com/apache/hudi/issues/12256
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Timeout while waiting for instant initialize [hudi]
Posted by "gjboomshit (via GitHub)" <gi...@apache.org>.
gjboomshit commented on issue #12256:
URL: https://github.com/apache/hudi/issues/12256#issuecomment-2485180229
Thanks for your answer; the timeout should refer to the time it takes for the current task to complete the snapshotState and wait for the entire checkpoint to complete;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org