[jira] [Reopened] (FLINK-16639) Flink SQL Kafka source connector, add the no json format filter params when format.type is json

["Kurt Young (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-16639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kurt Young reopened FLINK-16639:
--------------------------------

> Flink SQL Kafka source connector, add the no json format filter params when format.type is json
> -----------------------------------------------------------------------------------------------
>
>                 Key: FLINK-16639
>                 URL: https://issues.apache.org/jira/browse/FLINK-16639
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / Kafka
>            Reporter: LakeShen
>            Priority: Minor
>             Fix For: 1.10.2
>
>
> In my thought, kafka source connector is the one of most frequently used connector in flink sql.  Flink sql kafka source connector supports the json,csv or other data format. But there is a problem for json format in kafka source connector.  For example, flink sql kafka source ddl l 
>  like this:
> CREATE TABLE team_olap_table (
>     a varchar,
>     b varchar,
>   )
>   with (
>     'connector.type' = 'kafka', 
>     'connector.version' = '0.10', 
>     'connector.topic' = topics', 
>     'connector.properties.0.key' = 'group.id', 
>     'connector.properties.0.value' = 'hello_world', 
>     'connector.properties.1.key' = 'bootstrap.servers', 
>     'connector.properties.1.value' = 'xxx', 
>     'connector.property-version' = '1', 
>     'connector.startup-mode' = 'latest-offset', 
>     'format.type' = 'json', 
>     'format.property-version' = '1', 
>     'format.derive-schema' = 'true', 
>     'update-mode' = 'append'
> );
> If the kafka topic messages are not json format ,just one or two records,the flink sql task will fail-over all the time .
> In order to solve this problem , if flink sql source connector use the json-format, I want to add the 'format.fail-on-not-json-record' param  in flink-json module, if this param is true（default），when read the no-json records, the flink will fail, if this param is false, the flink sql task will filter no-json records,the flink task running normally.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)