You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by "Camellia-lili (via GitHub)" <gi...@apache.org> on 2024/05/16 01:23:55 UTC
[I] flink cdc 2.4.2版本关于报错io.debezium.DebeziumException: Failed to deserialize data of EventHeaderV4 的问题 [flink-cdc]
Camellia-lili opened a new issue, #3322:
URL: https://github.com/apache/flink-cdc/issues/3322
### Please don't check the box below
- [X] I'm aware that bugs and new features should be reported on Apache Jira or Flink mailing list instead of here
### Again, please don't check the box below
- [X] I'm aware that new issues on GitHub will be ignored and automatically closed.
### 请不要勾选以下选项
- [X] 我已知悉缺陷和新功能需要在 Apache Jira 或 Flink 邮件列表中反馈，而不是在这里创建新 issue。
### 也请不要勾选以下选项
- [X] 我已知悉 GitHub 上的新 issue 会被忽略且自动关闭。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] flink cdc 2.4.2版本关于报错io.debezium.DebeziumException: Failed to deserialize data of EventHeaderV4 的问题 [flink-cdc]
Posted by "Camellia-lili (via GitHub)" <gi...@apache.org>.
Camellia-lili commented on issue #3322:
URL: https://github.com/apache/flink-cdc/issues/3322#issuecomment-2113738218
此前我们的同步任务也遇到了 Failed to deserialize data of EventHeaderV4 这个问题，然后看了相关文档说配置debezium的一个参数，connect.keep.alive.interval.ms = 3000，然后这边改了下，重启后taskmanager节点就开始报错挂了
java.lang.RuntimeException: One or more fetchers have encountered exception
at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:225) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:169) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:130) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:156) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:385) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:542) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_341]
Caused by: java.lang.RuntimeException: SplitFetcher thread 0 received unexpected exception while polling the records
at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:150) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:105) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_341]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_341]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_341]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_341]
... 1 more
Caused by: org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:50) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at com.ververica.cdc.connectors.mysql.debezium.task.context.MySqlErrorHandler.setProducerThrowable(MySqlErrorHandler.java:85) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onEventDeserializationFailure(MySqlStreamingChangeEventSource.java:1553) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1064) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
... 1 more
Caused by: io.debezium.DebeziumException: Failed to deserialize data of EventHeaderV4{timestamp=1715762712000, eventType=WRITE_ROWS, serverId=27545817, headerLength=19, dataLength=8069, nextPosition=394608452, flags=0}
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1488) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onEventDeserializationFailure(MySqlStreamingChangeEventSource.java:1553) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1064) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
... 1 more
Caused by: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1715762712000, eventType=WRITE_ROWS, serverId=27545817, headerLength=19, dataLength=8069, nextPosition=394608452, flags=0}
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:341) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:244) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:259) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1051) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
... 1 more
Caused by: com.github.shyiko.mysql.binlog.event.deserialization.MissingTableMapEventException: No TableMapEventData has been found for table id:73858. Usually that means that you have started reading binary log 'within the logical event group' (e.g. from WRITE_ROWS and not proceeding TABLE_MAP
at com.github.shyiko.mysql.binlog.event.deserialization.AbstractRowsEventDataDeserializer.deserializeRow(AbstractRowsEventDataDeserializer.java:109) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.event.deserialization.WriteRowsEventDataDeserializer.deserializeRows(WriteRowsEventDataDeserializer.java:64) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.event.deserialization.WriteRowsEventDataDeserializer.deserialize(WriteRowsEventDataDeserializer.java:56) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.event.deserialization.WriteRowsEventDataDeserializer.deserialize(WriteRowsEventDataDeserializer.java:32) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:335) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:244) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:259) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1051) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932) ~[hlkj-bigdata-1.0-SNAPSHOT.jar:?]
... 1 more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] flink cdc 2.4.2版本关于报错io.debezium.DebeziumException: Failed to deserialize data of EventHeaderV4 的问题 [flink-cdc]
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang closed issue #3322: flink cdc 2.4.2版本关于报错io.debezium.DebeziumException: Failed to deserialize data of EventHeaderV4 的问题
URL: https://github.com/apache/flink-cdc/issues/3322
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org