kafka connect - fetch avro data from the SinkRecord put method

[ravi kumar <nk...@gmail.com>]

Hi ,

I have been working on developing new Kafka connector ( Sink ) which pushes
data to container from the kafka cluster.

I am pushing the data using kafka-avro-console-producer to one of the
topics .

At the time of fetching the records pushed ( using SinkRecord ) at the put
method in the SinkTask. The output looks like below

*org.apache.kafka.connect.data.Struct@ecb4d6ec*
*org.apache.kafka.connect.data.Struct@ecb4d6ef*
*org.apache.kafka.connect.data.Struct@ecb4d6dr*

But the expected output is expected to be a json that is sent from the
kafka-avro-console-producer
.

The output at the  kafka-avro-console-consumer seems fine .

Can I access the  avro message directly from the SinkRecord or is there any
other alternative for fetching the avro messages pushed using
kafka-avro-console-producer

This is the follow up for the question :

https://groups.google.com/forum/#!topic/confluent-platform/xja60s2_TZk


Please let me know

Thanks,
Rk

Re: kafka connect - fetch avro data from the SinkRecord put method

[Ewen Cheslack-Postava <ew...@confluent.io>]

There isn't currently a way to get at the intermediate Avro formatted data
-- the point of Connect's generic data API is to decouple the connector
implementations from the details of (de)serialization. This allows
connectors to work with data written to Kafka in a variety of data formats
without the Connector having to write code for each of those formats. In
fact, the Connect framework itself never even sees the intermediate format.
Converter implementations are responsible for translating from byte[] ->
Connect's data API classes and the framework never sees any intermediate
formats such as Avro's GenericRecord.

Normally you would just use the data returned by the Converter in the data
API (
http://docs.confluent.io/3.0.0/connect/javadocs/org/apache/kafka/connect/data/package-summary.html)
and convert it into whatever format the system you're sending it to
expects. Then it will work whether the data in Kafka is in JSON, Avro, etc.

-Ewen

On Thu, May 26, 2016 at 5:18 AM, ravi kumar <nk...@gmail.com> wrote:

> Hi ,
>
> I have been working on developing new Kafka connector ( Sink ) which pushes
> data to container from the kafka cluster.
>
> I am pushing the data using kafka-avro-console-producer to one of the
> topics .
>
> At the time of fetching the records pushed ( using SinkRecord ) at the put
> method in the SinkTask. The output looks like below
>
> *org.apache.kafka.connect.data.Struct@ecb4d6ec*
> *org.apache.kafka.connect.data.Struct@ecb4d6ef*
> *org.apache.kafka.connect.data.Struct@ecb4d6dr*
>
> But the expected output is expected to be a json that is sent from the
> kafka-avro-console-producer
> .
>
> The output at the  kafka-avro-console-consumer seems fine .
>
> Can I access the  avro message directly from the SinkRecord or is there any
> other alternative for fetching the avro messages pushed using
> kafka-avro-console-producer
>
> This is the follow up for the question :
>
> https://groups.google.com/forum/#!topic/confluent-platform/xja60s2_TZk
>
>
> Please let me know
>
> Thanks,
> Rk
>



-- 
Thanks,
Ewen