You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Leonard Xu (Jira)" <ji...@apache.org> on 2020/07/30 06:58:00 UTC
[jira] [Updated] (FLINK-18758) Support debezium data type in
debezium format
[ https://issues.apache.org/jira/browse/FLINK-18758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leonard Xu updated FLINK-18758:
-------------------------------
Description:
Currently debezium json format wrapper the json format to serialize/deserialize the debezuim json format data.
But debezium json format has its own data type which consists of Literal Type and Semantic Type[1], i.g. Date type in debezium is an integer which represents the number of days since epoch rather than a string with 'yyyy-MM-dd' pattern.
{code:java}
{ "schema":{
"fields":[
{
"fields":[
{
"type":"int32", //Literal Type
"optional":false,
"name":"io.debezium.time.Date", //semantic Type
"version":1,
"field":"order_date"
},
{ "type":"int32",
"optional":false,
"field":"quantity"
}
]
}
]
},
"payload":{
"before":null,
"after":{
"order_date":16852, // Literal Value, the number of days since epoch
"quantity":1
},
"op":"c",
"ts_ms":1596081813474
}
} {code}
I think we need obtain the debezuim data type from schema information and then serialize/deserialize the data in payload.
[1][https://debezium.io/documentation/reference/1.2/connectors/mysql.html]
was:
Currently debezium json format wrapper the json format to serialize/deserialize the debezuim json format data.
But debezium json format has its own data type which consists of Literal Type and Semantic Type[1], i.g. Date type in debezium is the an integer which represents the number of days since epoch rather than a string with 'yyyy-MM-dd' pattern.
{code:java}
{ "schema":{ "fields":[ { "fields":[ {
"type":"int32",
"optional":false,
"name":"io.debezium.time.Date",
"version":1,
"field":"order_date"
},
{ "type":"int32",
"optional":false,
"field":"quantity"
}
]
}
]
},
"payload":{
"before":null,
"after":{
"order_date":16852,
"quantity":1
},
"op":"c",
"ts_ms":1596081813474
}
} {code}
I think we need obtain the debezuim data type from schema information and then serialize/deserialize the data in payload.
[1][https://debezium.io/documentation/reference/1.2/connectors/mysql.html]
> Support debezium data type in debezium format
> ----------------------------------------------
>
> Key: FLINK-18758
> URL: https://issues.apache.org/jira/browse/FLINK-18758
> Project: Flink
> Issue Type: Improvement
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
> Reporter: Leonard Xu
> Priority: Major
>
> Currently debezium json format wrapper the json format to serialize/deserialize the debezuim json format data.
> But debezium json format has its own data type which consists of Literal Type and Semantic Type[1], i.g. Date type in debezium is an integer which represents the number of days since epoch rather than a string with 'yyyy-MM-dd' pattern.
> {code:java}
> { "schema":{
> "fields":[
> {
> "fields":[
> {
> "type":"int32", //Literal Type
> "optional":false,
> "name":"io.debezium.time.Date", //semantic Type
> "version":1,
> "field":"order_date"
> },
> { "type":"int32",
> "optional":false,
> "field":"quantity"
> }
> ]
> }
> ]
> },
> "payload":{
> "before":null,
> "after":{
> "order_date":16852, // Literal Value, the number of days since epoch
> "quantity":1
> },
> "op":"c",
> "ts_ms":1596081813474
> }
> } {code}
>
> I think we need obtain the debezuim data type from schema information and then serialize/deserialize the data in payload.
> [1][https://debezium.io/documentation/reference/1.2/connectors/mysql.html]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)