[jira] [Commented] (KAFKA-7499) Extend ProductionExceptionHandler to cover serialization exceptions

["Aniket Singh (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-7499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17287091#comment-17287091 ] 

Aniket Singh commented on KAFKA-7499:
-------------------------------------

Hi [~jbfletch] [~mjsax],

Is this improvement still in progress? I am facing similar issue & have seen many related question on Stackoverflow as well, where askers need this feature very badly.

For me the issue is, I am working on a complex xml to avro (schema xsd -> avsc) conversion and backward compatibility has not been taken care in the pervious versions of xsd schema (7 years old). So, if an old message arrives which doesn't contain mandatory fields, it throws SerializationException and Streams fail.

I do not have control on the incoming source & need to skip similar records or send them to DLQ also keeping Streams Up & running for processing next set of records. Currently, ProductionExceptionHandler or DeserializationExceptionHandler don't cover SerializationException.


Could you please implement this feature soon to help the community?

 

> Extend ProductionExceptionHandler to cover serialization exceptions
> -------------------------------------------------------------------
>
>                 Key: KAFKA-7499
>                 URL: https://issues.apache.org/jira/browse/KAFKA-7499
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Matthias J. Sax
>            Assignee: jbfletch
>            Priority: Major
>              Labels: beginner, kip, newbie
>
> In [KIP-210|https://cwiki.apache.org/confluence/display/KAFKA/KIP-210+-+Provide+for+custom+error+handling++when+Kafka+Streams+fails+to+produce], an exception handler for the write path was introduced. This exception handler covers exception that are raised in the producer callback.
> However, serialization happens before the data is handed to the producer with Kafka Streams itself and the producer uses `byte[]/byte[]` key-value-pair types.
> Thus, we might want to extend the ProductionExceptionHandler to cover serialization exception, too, to skip over corrupted output messages. An example could be a "String" message that contains invalid JSON and should be serialized as JSON.
> KIP-399 (not voted yet; feel free to pick it up): [https://cwiki.apache.org/confluence/display/KAFKA/KIP-399%3A+Extend+ProductionExceptionHandler+to+cover+serialization+exceptions]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)