[jira] [Commented] (FLINK-9155) Provide message context information in DeserializationSchema

["Tzu-Li (Gordon) Tai (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-9155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16444080#comment-16444080 ] 

Tzu-Li (Gordon) Tai commented on FLINK-9155:
--------------------------------------------

I think we can implement both logging and metrics to track this.
For metrics, we should be able to use the user variable functionality to have skip counters that can be grouped by topic / partition. This should allow more prompt alerts of skipped messages, and users should direct to logs for more details on the erroring record.
For logging, the information should contain topic, partition, and offset for debugging.

> Provide message context information in DeserializationSchema
> ------------------------------------------------------------
>
>                 Key: FLINK-9155
>                 URL: https://issues.apache.org/jira/browse/FLINK-9155
>             Project: Flink
>          Issue Type: Improvement
>            Reporter: Alex Smirnov
>            Priority: Minor
>
> There's no way to retrieve more information about corrupted message in the DeserializationSchema class. It is only possible to return null, which is a signal to skip the message, and to throw an exception, which will cause job failure.
> For investigation purposes it would be good to have more information, like:
>  * kafka topic from which the message came from
>  ** in Flink 1.4, it is possible to subscribe using Pattern, so topic name is not always evident
>  * kafka topic offset
> The idea is to write this information into the log file for further analysis. Having topic name and offset allows to retrieve the message and investigate it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)