[jira] [Comment Edited] (FLINK-18758) Support debezium data type in debezium format

["Jark Wu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-18758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17167646#comment-17167646 ] 

Jark Wu edited comment on FLINK-18758 at 7/30/20, 5:42 AM:
-----------------------------------------------------------

For this case, I think we can parse integer into date for the json format. Or at least, it can be a configuration which is only enabled in debezium-json. The purpose is that we don't need to parse the schema part, and the schema part might not be included. 


was (Author: jark):
For this case, I think we can parse integer into date for the json format. Or at least, it can be a configuration which is only enabled in debezium-json.

> Support debezium data type in debezium format 
> ----------------------------------------------
>
>                 Key: FLINK-18758
>                 URL: https://issues.apache.org/jira/browse/FLINK-18758
>             Project: Flink
>          Issue Type: Improvement
>          Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>            Reporter: Leonard Xu
>            Priority: Major
>
> Currently debezium json format  wrapper the json format to serialize/deserialize the debezuim json format data.
> But debezium json format has its own data type which consists of Literal Type and Semantic Type[1], i.g. Date type in debezium is the an integer which represents the number of days since epoch rather than a string with 'yyyy-MM-dd' pattern.  
> {code:java}
> {   "schema":{      "fields":[         {            "fields":[               {                 
>                  "type":"int32",
>                   "optional":false,
>                   "name":"io.debezium.time.Date",
>                   "version":1,
>                   "field":"order_date"
>                },
>                {                  "type":"int32",
>                   "optional":false,
>                   "field":"quantity"
>                }
>             ]
>          }
>       ]
>    },
>    "payload":{ 
>       "before":null,
>       "after":{ 
>          "order_date":16852,
>          "quantity":1
>       },
>       "op":"c",
>       "ts_ms":1596081813474
>    }
> } {code}
>  
> I think we need obtain the debezuim data type from schema information and then serialize/deserialize the data in payload.
> [1][https://debezium.io/documentation/reference/1.2/connectors/mysql.html]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)