[jira] [Created] (FLINK-33783) Add options to ignore parsing error in Kafka SQL Connector

["Bartosz Mikulski (Jira)" <ji...@apache.org>]

Bartosz Mikulski created FLINK-33783:
----------------------------------------

             Summary: Add options to ignore parsing error in Kafka SQL Connector
                 Key: FLINK-33783
                 URL: https://issues.apache.org/jira/browse/FLINK-33783
             Project: Flink
          Issue Type: Improvement
          Components: Connectors / Kafka, Table SQL / API
            Reporter: Bartosz Mikulski


h1. Current state

If an unparsable event enters a Flink Kafka Source in SQL then the whole application restarts. For JSON format there is a property that allows to ignore unparsable events. Other formats, like `confluent-avro` etc don't support that.
h1. Desired state

We would like to ignore the parsing exception in Kafka Source in SQL regardless of the format used. Additionally, a new metric should be introduced that returns a count of unparsable events seen so far.

In the future there should be a Dead Letter Queue handling in SQL Sources similar to Kafka Streams: [https://www.confluent.io/blog/kafka-connect-deep-dive-error-handling-dead-letter-queues/].
For now, the universal ignore with metric would be enough.
h1. Implementation

We already have an implementation for this case in the Flink Kafka Connector and we would like to open a pull request for it. However, we created the issue first as per this recommendation [https://flink.apache.org/how-to-contribute/contribute-code/]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)