[jira] [Commented] (KAFKA-10273) Connect Converters should produce actionable error messages

["Randall Hauch (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-10273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17158583#comment-17158583 ] 

Randall Hauch commented on KAFKA-10273:
---------------------------------------

Here are some ideas to maybe start with:
 
For source connectors, we might be able to say something like:
{quote}Error while serializing the key for a source record. Check the {{key.converter}} and {{key.converter.*}} settings in the connector configuration, or in the worker configuration if the connector is inheriting the connector configuration. Underlying converter error: %s{quote}
Ideally these would be specific enough, such as mentioning either the key or value converter (whichever had the error), and to know whether the converter was defined in the connector config or inherited from the worker). This may make mean we can't just catch and wrap an exception with a simple message, and instead we might have to build up or templatize a message.
 
For sink connectors we may be able to offer a bit more (as mentioned above). Another example/idea:
{quote}Error while deserializing the value for record in topic X, partition Y, and at offset Z. Check the {{value.converter}} and {{value.converter.*}} settings in the connector configuration, and ensure that the converter matches the converter/serializer used by the application that produced this record. Underlying converter error: %s{quote}
Users will still have to understand the underlying error (which is why we'd want to include it in the exception message, and to avoid users having to get the stack trace), but the goal of this change is to provide a bit more context (as specific as we can) when they stitch together components (converters and connectors in this case).

> Connect Converters should produce actionable error messages
> -----------------------------------------------------------
>
>                 Key: KAFKA-10273
>                 URL: https://issues.apache.org/jira/browse/KAFKA-10273
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>            Reporter: Greg Harris
>            Priority: Minor
>
> When runtime exceptions are thrown from converters, the current error messages are descriptive, but not actionable. For example:
> {noformat}
> Error encountered in task task-0. Executing stage 'VALUE_CONVERTER' with class 'org.apache.kafka.connect.json.JsonConverter', where consumed record is <snipped>
> Caused by: org.apache.kafka.connect.errors.DataException: Converting byte[] to Kafka Connect data failed due to serialization error:
>  at org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:334)
> <snipped>
> Caused by: org.apache.kafka.common.errors.SerializationException: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('b' (code 98)): was expecting double-quote to start field name
>  at [Source: (byte[])"{brokenjson-:"bar 1"}"; line: 1, column: 3]{noformat}
> This describes the error accurately, explaining when it happened, and what caused the exception to be thrown. However, after receiving this error it's still unclear what remediation step to take next.
> Instead, in addition to the existing descriptive context and instigating error being displayed, these error messages should give users a lead to begin investigating and remediating the error.
> There are a few changes that users may use to remediate exceptions from converters:
>  * Reconfigure the converter (change schemas.enable on the JsonConverter to align with the topic contents)
>  * Use different data (advancing sink/source offsets manually to skip bad records, or changing topics)
>  * Remediate the converter's environment failure (resolving network faults or reconfiguring external systems)
> An error message which suggests manual inspection of the converter configuration would be user-actionable, and could be generically applied across converters without knowledge of specific configurations. This would point users towards the most common remediation step, reconfiguring the converter.
> For sink connectors we can also indicate that bad data in a topic may be skipped in a number of ways, such as changing offsets, ignoring errors, writing to a DLQ, etc. It's unclear which of these is worth suggesting, and it may change if an alternative strategy feature exists in the future. 
> Non-Goals/ Out-of-scope:
> The third type of remediation of the converter's environment is outside the scope of this issue, and requires converter-specific knowledge in order to make a suggestion.
> This change is not attempting to add converter-specific error messaging, and the framework messaging here needs to be converter-agnostic, and as such can only point out general classes of errors and remediation steps, not specific ones. We can programmatically generate lists of relevant configurations, but can't single any out as being the cause. We can say that the data is incompatible with the converter, but can't say why exactly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)