[jira] [Commented] (KAFKA-12391) Add an option to store arbitrary metadata to a SourceRecord

["Chris Egerton (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-12391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17295401#comment-17295401 ] 

Chris Egerton commented on KAFKA-12391:
---------------------------------------

[~lb] I believe this is already possible by a few methods:
 # Add headers to the record (for example, via [this constructor|https://kafka.apache.org/27/javadoc/org/apache/kafka/connect/source/SourceRecord.html#SourceRecord-java.util.Map-java.util.Map-java.lang.String-java.lang.Integer-org.apache.kafka.connect.data.Schema-java.lang.Object-org.apache.kafka.connect.data.Schema-java.lang.Object-java.lang.Long-java.lang.Iterable-]). These will appear in the message stored in Kafka as headers, but won't be part of the message's key or value. If there's a chance users might benefit from this housekeeping metadata, this approach is probably the best.
 # Create a subclass of {{SourceRecord}} and cast to that subclass in your {{SourceTask::commitRecord}} method. This subclass could include your custom {{attributes}} method, for example. This approach would be best if there is absolutely no benefit to downstream users of being able to read this extra metadata.

> Add an option to store arbitrary metadata to a SourceRecord
> -----------------------------------------------------------
>
>                 Key: KAFKA-12391
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12391
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>            Reporter: Luca Burgazzoli
>            Priority: Minor
>
> When writing Source Connectors for Kafka, it may be required to perform some additional house cleaning when an record has been acknowledged by the Kafka broker and as today, it is possible to set up an hook by overriding SourceTask.commitRecord(SourceRecord).
> This works fine in most of the cases but to make it easy for the source connector to perform it's internal house keeping, it would be nice to have an option to set some additional metadata to the SourceRecord without having impacts to the Record sent to the Kafka Broker, something like:
> {code:java}
> class SourceRecord {
>     public SourceRecord(
>         ...,
>         Map<String, ?> attributes) {
>         ...
>         this.attributes = attributes;
>     }
>     Map<String, ?> attributes() { 
>         return attributes;
>     }
> }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)