You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Flink CDC Issue Import (Jira)" <ji...@apache.org> on 2024/03/20 09:20:00 UTC
[jira] [Created] (FLINK-34764) FlinkCDC is incompatible with MariaDB 10.0.24
Flink CDC Issue Import created FLINK-34764:
----------------------------------------------
Summary: FlinkCDC is incompatible with MariaDB 10.0.24
Key: FLINK-34764
URL: https://issues.apache.org/jira/browse/FLINK-34764
Project: Flink
Issue Type: Bug
Components: Flink CDC
Reporter: Flink CDC Issue Import
FlinkCDC is incompatible with MariaDB 10.0.24.
There will be **exceptions** when **debezium** parses the binlog.
mysqlbinlog fails to read binlog event which inserts **TIME, DATETIME and TIMESTAMP** with wrong number of microseconds.
The low-level temporal format used by TIME, DATETIME and TIMESTAMP is different in MySQL 5.6 and [MariaDB 10.0|https://mariadb.com/kb/en/what-is-mariadb-100/]. (In [MariaDB 10.1|https://mariadb.com/kb/en/what-is-mariadb-101/], the MySQL implementation is used by default - see [mysql56_temporal_format|https://mariadb.com/kb/en/server-system-variables/#mysql56_temporal_format).)
**Environment :**
- Flink version : 1.13.6
- Flink CDC version: 2.2.1
- Database and version: MariaDB 10.0.24
**To Reproduce**
Steps to reproduce the behavior:
1. The test code :
The field type in the database is **Datetime(6)**, and the value is **2016-12-21 17:57:16.000000**.
After analysis, the obtained string is **103946-26-59 06:40:65**.
2. The error :
There will be exceptions when debezium parses the binlog:
`java.io.EOFException: Failed to read next byte from position -1536265298
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:213)
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.readInteger(ByteArrayInputStream.java:52)
at com.github.shyiko.mysql.binlog.event.deserialization.AbstractRowsEventDataDeserializer.deserializeLong(AbstractRowsEventDataDeserializer.java:261)
at com.github.shyiko.mysql.binlog.event.deserialization.AbstractRowsEventDataDeserializer.deserializeCell(AbstractRowsEventDataDeserializer.java:166)
at com.github.shyiko.mysql.binlog.event.deserialization.AbstractRowsEventDataDeserializer.deserializeRow(AbstractRowsEventDataDeserializer.java:145)
at com.github.shyiko.mysql.binlog.event.deserialization.UpdateRowsEventDataDeserializer.deserializeRows(UpdateRowsEventDataDeserializer.java:72)
at com.github.shyiko.mysql.binlog.event.deserialization.UpdateRowsEventDataDeserializer.deserialize(UpdateRowsEventDataDeserializer.java:58)
at com.github.shyiko.mysql.binlog.event.deserialization.UpdateRowsEventDataDeserializer.deserialize(UpdateRowsEventDataDeserializer.java:33)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:329)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:238)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:233)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:949)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:599)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:854)
at java.lang.Thread.run(Thread.java:748)
java.io.EOFException: Failed to read next byte from position -1536265298
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:213)
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.readInteger(ByteArrayInputStream.java:52)
at io.debezium.connector.mysql.RowDeserializers.deserializeVarString(RowDeserializers.java:298)
at io.debezium.connector.mysql.RowDeserializers$UpdateRowsDeserializer.deserializeVarString(RowDeserializers.java:130)
at com.github.shyiko.mysql.binlog.event.deserialization.AbstractRowsEventDataDeserializer.deserializeCell(AbstractRowsEventDataDeserializer.java:194)
at com.github.shyiko.mysql.binlog.event.deserialization.AbstractRowsEventDataDeserializer.deserializeRow(AbstractRowsEventDataDeserializer.java:145)
at com.github.shyiko.mysql.binlog.event.deserialization.UpdateRowsEventDataDeserializer.deserializeRows(UpdateRowsEventDataDeserializer.java:72)
at com.github.shyiko.mysql.binlog.event.deserialization.UpdateRowsEventDataDeserializer.deserialize(UpdateRowsEventDataDeserializer.java:58)
at com.github.shyiko.mysql.binlog.event.deserialization.UpdateRowsEventDataDeserializer.deserialize(UpdateRowsEventDataDeserializer.java:33)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:329)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:238)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:233)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:949)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:599)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:854)
at java.lang.Thread.run(Thread.java:748]
`
<img width="946" alt="image" src="https://user-images.githubusercontent.com/35026750/210321758-45048701-d93c-488f-ac32-2321180e50d9.png">
**Additional Description**
[MariaDB 10.1.2|https://mariadb.com/kb/en/mariadb-1012-release-notes/] introduced the [--mysql56-temporal-format|https://mariadb.com/kb/en/server-system-variables/#mysql56_temporal_format] option, on by default, which allows MariaDB to store DATETMEs using the same low-level format MySQL 5.6 uses. For more information, see [Internal Format|https://mariadb.com/kb/en/datetime/#internal-format], below.
[incompatibilities between MariaDB 10.0 and MySQL 5.6|https://mariadb.com/kb/en/mariadb-vs-mysql-compatibility]
[mysql56_temporal_format|https://mariadb.com/kb/en/server-system-variables/#mysql56_temporal_format]
[DATETIME|https://mariadb.com/kb/en/datetime]
[mysqlbinlog fails to read binlog event which inserts timestamp with wrong number of microseconds|https://jira.mariadb.org/browse/MDEV-9567]
[MariaDB 10.0.25 Datetime 类型无法解析毫秒级|https://github.com/alibaba/canal/issues/385]
---------------- Imported from GitHub ----------------
Url: https://github.com/apache/flink-cdc/issues/1851
Created by: [lollipopcs|https://github.com/lollipopcs]
Labels: bug,
Created at: Tue Jan 03 16:12:57 CST 2023
State: open
--
This message was sent by Atlassian Jira
(v8.20.10#820010)