[jira] [Commented] (FLINK-33783) Add options to ignore parsing error in Kafka SQL Connector

["Bartosz Mikulski (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-33783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17806001#comment-17806001 ] 

Bartosz Mikulski commented on FLINK-33783:
------------------------------------------

Sure, sounds great [~martijnvisser] !

It would be great if I could create a FLIP, thank you! Any tips on how to write one properly?

How can I open a mailing list discussion, is it from a FLIP page?

> Add options to ignore parsing error in Kafka SQL Connector
> ----------------------------------------------------------
>
>                 Key: FLINK-33783
>                 URL: https://issues.apache.org/jira/browse/FLINK-33783
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / Kafka, Table SQL / API
>            Reporter: Bartosz Mikulski
>            Priority: Major
>
> h1. Current state
> If an unparsable event enters a Flink Kafka Source in SQL then the whole application restarts. For JSON format there is a property that allows to ignore unparsable events. Other formats, like `confluent-avro` etc don't support that.
> h1. Desired state
> We would like to ignore the parsing exception in Kafka Source in SQL regardless of the format used. Additionally, a new metric should be introduced that returns a count of unparsable events seen so far.
> In the future there should be a Dead Letter Queue handling in SQL Sources similar to Kafka Streams: [https://www.confluent.io/blog/kafka-connect-deep-dive-error-handling-dead-letter-queues/].
> For now, the universal ignore with metric would be enough.
> h1. Implementation
> We already have an implementation for this case in the Flink Kafka Connector and we would like to open a pull request for it. However, we created the issue first as per this recommendation [https://flink.apache.org/how-to-contribute/contribute-code/]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)