[jira] [Comment Edited] (KAFKA-6490) JSON SerializationException Stops Connect

["Prasanna Subburaj (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-6490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354908#comment-16354908 ] 

Prasanna Subburaj edited comment on KAFKA-6490 at 2/7/18 3:41 AM:
------------------------------------------------------------------

Thanks for you response [~mjsax]. I think we need a similar feature in connector because now if we get a malformed JSON message the connector will fail and will not process any additional message that are coming after this one. Can you also please add me to the contributors list ?


was (Author: prasanna1433):
Thanks for you response [~mjsax]. I think we need a similar feature in connector because now if we get a malformed JSON message the connector will fail and will not process any additional message that are coming after this one. I can work on KIP for solving this issue if the forks you tagged in agree with what I am saying. Can you also please add me to the contributors list ?

> JSON SerializationException Stops Connect
> -----------------------------------------
>
>                 Key: KAFKA-6490
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6490
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>    Affects Versions: 1.0.0
>            Reporter: William R. Speirs
>            Priority: Major
>         Attachments: KAFKA-6490_v1.patch
>
>
> If you configure KafkaConnect to parse JSON messages, and you send it a non-JSON message, the SerializationException message will bubble up to the top, and stop KafkaConnect. While I understand sending non-JSON to a JSON serializer is a bad idea, I think that a single malformed message stopping all of KafkaConnect is even worse.
> The data exception is thrown here: [https://github.com/apache/kafka/blob/trunk/connect/json/src/main/java/org/apache/kafka/connect/json/JsonConverter.java#L305]
>  
> From the call here: [https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L476]
> This bubbles all the way up to the top, and KafkaConnect simply stops with the message: {{ERROR WorkerSinkTask\{id=elasticsearch-sink-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:172)}}
> Thoughts on adding a {{try/catch}} around the {{for}} loop in WorkerSinkTask's {{convertMessages}} so messages that don't properly parse are logged, but simply ignored? This way KafkaConnect can keep working even when it encounters a message it cannot decode?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)