Handling deserialization exceptions in Kafka consumer

[Sönke Liebau <so...@opencore.com.INVALID>]

Hi all,

I was recently discussing deserialization errors and how they can be
handled in Kafka consumers.

I believe the current code still throws an exception if deserialization
fails which stops consumption unless you seek past that record.

This creates an issue for a JDBC sink connector that uses Avro data for
example, as this is effectively dead in the water when hitting a "poison
pill" record.

I've dug around for a little while and found the following issues on this:
https://issues.apache.org/jira/browse/KAFKA-4740
https://issues.apache.org/jira/browse/KAFKA-5858

https://issues.apache.org/jira/browse/KAFKA-5211 mentions that this
behavior was actually changed for a little while at some time in the past,
but that change was snuck in without a KIP.

There are also a few tickets specifically around Connect, but those refer
more to the wording of the error message than the actual functionality.

--

Is there a "gospel" way of handling this that people have found works for
them?

Or if not, would we want to extend the java consumer to allow configuring
behavior for this case?

At the very least it would be quite easy to add an option to simply ignore
deserialization errors, thinking a bit further, we might also allow
specifying a pluggable error handler that could be used to send malformed
records back to a dead letter queue or something similar.

Best regards,
Sönke