You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "MOBIN (Jira)" <ji...@apache.org> on 2024/04/09 05:50:00 UTC
[jira] [Created] (FLINK-35058) Encountered change event for table db.table whose schema isn't known to this connector
MOBIN created FLINK-35058:
-----------------------------
Summary: Encountered change event for table db.table whose schema isn't known to this connector
Key: FLINK-35058
URL: https://issues.apache.org/jira/browse/FLINK-35058
Project: Flink
Issue Type: Bug
Components: Flink CDC
Affects Versions: 1.17.1
Reporter: MOBIN
Flink1.17.1
flink-cdc：flink-sql-connector-mysql-cdc-2.4.1.jar
{code:java}
CREATE TABLE `test_cdc_timestamp` (
`id` BIGINT COMMENT '主键id',
....
proctime AS PROCTIME(),
PRIMARY KEY(id) NOT ENFORCED
) WITH (
'connector' = 'mysql-cdc',
'hostname' = 'xxxxx',
'scan.startup.mode' = 'timestamp',
'scan.startup.timestamp-millis' = '1712419200000' ,
'port' = '3306',
'username' = 'xxx',
'password' = 'xxx',
'database-name' = 'xxtablename',
'table-name' = 'xxdatabase',
'scan.incremental.snapshot.enabled' = 'false',
'debezium.snapshot.locking.mode' = 'none',
'server-id' = '5701',
'server-time-zone' = 'Asia/Shanghai',
'debezium.skipped.operations' = 'd'
); {code}
When I use 'scan.startup.mode' = 'latent-offset 'or'initial' to synchronize data normally, when I use 'scan.startup.mode' = 'timestamp', the following error is reported
{code:java}
2024-04-09 11:11:15.619 [debezium-engine] INFO io.debezium.util.Threads - Requested thread factory for connector MySqlConnector, id = mysql_binlog_source named = change-event-source-coordinator
2024-04-09 11:11:15.621 [debezium-engine] INFO io.debezium.util.Threads - Creating thread debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator
2024-04-09 11:11:15.629 [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] INFO io.debezium.pipeline.ChangeEventSourceCoordinator - Metrics registered
2024-04-09 11:11:15.630 [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] INFO io.debezium.pipeline.ChangeEventSourceCoordinator - Context created
2024-04-09 11:11:15.642 [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] INFO io.debezium.connector.mysql.MySqlSnapshotChangeEventSource - No previous offset has been found
2024-04-09 11:11:15.642 [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] INFO io.debezium.connector.mysql.MySqlSnapshotChangeEventSource - According to the connector configuration only schema will be snapshotted
2024-04-09 11:11:15.644 [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] INFO io.debezium.pipeline.ChangeEventSourceCoordinator - Snapshot ended with SnapshotResult [status=SKIPPED, offset=null]
2024-04-09 11:11:15.652 [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] INFO io.debezium.util.Threads - Requested thread factory for connector MySqlConnector, id = mysql_binlog_source named = binlog-client
2024-04-09 11:11:15.656 [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] INFO io.debezium.pipeline.ChangeEventSourceCoordinator - Starting streaming
2024-04-09 11:11:15.682 [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - GTID set purged on server: 0969640a-1d48-11ed-b6cf-28dee561557c:1-27603868993,70958f24-2253-11eb-891d-f875a48ad7b1:1-50323,ec1e6593-2251-11eb-9c18-f875a48ad539:1-25345454762
2024-04-09 11:11:15.682 [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Skip 0 events on streaming start
2024-04-09 11:11:15.682 [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Skip 0 rows on streaming start
2024-04-09 11:11:15.683 [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] INFO io.debezium.util.Threads - Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
2024-04-09 11:11:15.686 [blcxxxx.mysql.com:3306] INFO io.debezium.util.Threads - Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
2024-04-09 11:11:15.700 [blcxxxx.mysql.com:3306] INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Connected to MySQL binlog at xxx.mysql.com:3306, starting at MySqlOffsetContext [sourceInfoSchema=Schema{io.debezium.connector.mysql.Source:STRUCT}, sourceInfo=SourceInfo [currentGtid=null, currentBinlogFilename=, currentBinlogPosition=0, currentRowNumber=0, serverId=0, sourceTime=null, threadId=-1, currentQuery=null, tableIds=[], databaseName=null], snapshotCompleted=false, transactionContext=TransactionContext [currentTransactionId=null, perTableEventCount={}, totalEventCount=0], restartGtidSet=null, currentGtidSet=null, restartBinlogFilename=, restartBinlogPosition=0, restartRowsToSkip=0, restartEventsToSkip=0, currentEventLengthInBytes=0, inTransaction=false, transactionId=null, incrementalSnapshotContext =IncrementalSnapshotContext [windowOpened=false, chunkEndPosition=null, dataCollectionsToSnapshot=[], lastEventKeySent=null, maximumKey=null]]
2024-04-09 11:11:15.700 [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Waiting for keepalive thread to start
2024-04-09 11:11:15.701 [blcxxxx.mysql.com:3306] INFO io.debezium.util.Threads - Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
2024-04-09 11:11:15.701 [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Keepalive thread is running
2024-04-09 11:11:15.818 [blcxxxx.mysql.com:3306] ERROR io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Encountered change event 'Event{header=EventHeaderV4{timestamp=1711939504000, eventType=TABLE_MAP, serverId=431777, headerLength=19, dataLength=81, nextPosition=981190, flags=0}, data=TableMapEventData{tableId=933966, database='xxxdatabase', table='xxxtablename', columnTypes=8, 15, 8, 15, 8, 3, 2, 1, -11, 18, 18, 1, columnMetadata=0, 400, 0, 1024, 0, 0, 0, 0, 4, 0, 0, 0, columnNullability={4, 6, 8, 11}, eventMetadata=null}}' at offset {transaction_id=null, file=mysql-bin.016482, pos=981027, gtids=0969640a-1d48-11ed-b6cf-28dee561557c:27603868994-27603870019, server_id=431777, event=1} for table xxxdatabase.xxxtablename whose schema isn't known to this connector. One possible cause is an incomplete database history topic. Take a new snapshot in this case.
Use the mysqlbinlog tool to view the problematic event: mysqlbinlog --start-position=981090 --stop-position=981190 --verbose mysql-bin.016482
2024-04-09 11:11:15.819 [blcxxxx.mysql.com:3306] ERROR io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Error during binlog processing. Last offset stored = {transaction_id=null, file=mysql-bin.016482, pos=981027, gtids=0969640a-1d48-11ed-b6cf-28dee561557c:27603868994-27603870019, server_id=431777, event=1}, binlog reader near position = mysql-bin.016482/981090
2024-04-09 11:11:15.821 [blcxxxx.mysql.com:3306] ERROR io.debezium.pipeline.ErrorHandler - Producer failure
io.debezium.DebeziumException: Error processing binlog event
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:429)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$25(MySqlStreamingChangeEventSource.java:1095)
at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1246)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1072)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932)
at java.lang.Thread.run(Thread.java:745)
Caused by: io.debezium.DebeziumException: Encountered change event for table xxdatabase.xxtablename whose schema isn't known to this connector
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:753)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:819)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleUpdateTableMetadata(MySqlStreamingChangeEventSource.java:720)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$13(MySqlStreamingChangeEventSource.java:1046)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:408)
... 6 common frames omitted
2024-04-09 11:11:15.821 [blcxxxx.mysql.com:3306] INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Error processing binlog event, and propagating to Kafka Connect so it stops this connector. Future binlog events read before connector is shutdown will be ignored.
2024-04-09 11:11:16.124 [debezium-engine] INFO io.debezium.embedded.EmbeddedEngine - Stopping the task and engine
2024-04-09 11:11:16.124 [debezium-engine] INFO io.debezium.connector.common.BaseSourceTask - Stopping down connector
2024-04-09 11:11:16.216 [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] INFO io.debezium.pipeline.ChangeEventSourceCoordinator - Finished streaming
2024-04-09 11:11:16.216 [blcxxxx.mysql.com:3306] INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Stopped reading binlog after 0 events, last recorded offset: {transaction_id=null, file=mysql-bin.016482, pos=981027, gtids=0969640a-1d48-11ed-b6cf-28dee561557c:27603868994-27603870019, server_id=431777, event=71904}
2024-04-09 11:11:16.218 [pool-10-thread-1] INFO io.debezium.jdbc.JdbcConnection - Connection gracefully closed
2024-04-09 11:11:16.221 [debezium-engine] ERROR com.ververica.cdc.debezium.internal.Handover - Reporting error:
com.ververica.cdc.connectors.shaded.org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:50)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:429)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$25(MySqlStreamingChangeEventSource.java:1095)
at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1246)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1072)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932)
at java.lang.Thread.run(Thread.java:745)
Caused by: io.debezium.DebeziumException: Error processing binlog event
... 7 common frames omitted
Caused by: io.debezium.DebeziumException: Encountered change event for table xxdatabase.xxtablename whose schema isn't known to this connector
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:753)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:819)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleUpdateTableMetadata(MySqlStreamingChangeEventSource.java:720)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$13(MySqlStreamingChangeEventSource.java:1046)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:408)
... 6 common frames omitted
2024-04-09 11:11:16.223 [Source: source_ascribe_all_result_data[1] -> Calc[2] -> Sink: print[3] (1/1)#0] INFO io.debezium.embedded.EmbeddedEngine - Stopping the embedded engine
2024-04-09 11:11:16.227 [Source: source_ascribe_all_result_data[1] -> Calc[2] -> Sink: print[3] (1/1)#0] INFO io.debezium.embedded.EmbeddedEngine - Stopping the embedded engine
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)