Re: Convert log file in Avro json encoding to Avro binary encoding and send into Kafka

[Ewen Cheslack-Postava <ew...@confluent.io>]

Avro JSON encoding is a wire-level format. The AvroConverter accepts Java
runtime data (e.g. primitive types like Strings & Integers, Maps, Arrays,
and Connect Structs).

The component that most closely matches your needs is Confluent's REST
proxy, which supports the Avro JSON encoding when receiving a payload (via
the vnd.kafka.avro.v1+json format). However, since the REST proxy is push,
you'd need another process to send requests to it.

Alternatively, you could build a source connector which pulls Avro
JSON-encoded data and emits generic Connect data. By combining that with
the AvroConverters, you'd end up generating Avro binary-serialized data
into Kafka (albeit via a slightly indirect route).

-Ewen

On Mon, Oct 17, 2016 at 6:14 AM, Takahiro Hozumi <fa...@googlemail.com>
wrote:

> Hi all,
>
> My application will emit log files in avro json encoding so that humans
> easily can read and grep records.
> I need to transfer this logs into Kafka as Avro binary encoding.
> And I want to use confluent schema registry in order to prepare schema
> evolution.
>
> After some research, I think Kafka connect in standalone mode might be the
> easiest solution.
> The avro-converter included in kafka connect interacts with schema registry
> under the hood.
> But I am not sure avro-converter accepts Avro json encoding as input.
> Do I need to create a json-encoding version of the avro-converter?
>
> And I don't find a documentation of Converter interface.
> I think converting job is just byte[] to byte[] job, but the interface
> seems to require internal data object.
>
> http://docs.confluent.io/3.0.1/connect/javadocs/index.html?
> org/apache/kafka/connect/storage/Converter.html
>
> Any advise or guidance would be greatly appreciated!
>
> Thanks,
> --
> Takahiro
>



-- 
Thanks,
Ewen

Re: Convert log file in Avro json encoding to Avro binary encoding and send into Kafka

[Takahiro Hozumi <fa...@googlemail.com>]

Hi Ewen,
Thank you for your advise.

I reached a following compromise:
My application sends records in the Avro Binary encoding to Kafka directly
using Confluent's serializers(io.confluent/kafka-avro-serializer), which
integrated with schema registry, and also does logging in the Avro JSON
encoding to a local file for easy reading.
An advantage of transferring records in log files to Kafka is that the log
files can be used as buffering(nice disaster tolerance).
I think KAFKA-1955[0] will solve this problem in a better way.

[0]: [KAFKA-1955] Explore disk-based buffering in new Kafka Producer - ASF
JIRA
https://issues.apache.org/jira/browse/KAFKA-1955

-Takahiro

2016-11-27 11:00 GMT+09:00 Ewen Cheslack-Postava <ew...@confluent.io>:

> Avro JSON encoding is a wire-level format. The AvroConverter accepts Java
> runtime data (e.g. primitive types like Strings & Integers, Maps, Arrays,
> and Connect Structs).
>
> The component that most closely matches your needs is Confluent's REST
> proxy, which supports the Avro JSON encoding when receiving a payload (via
> the vnd.kafka.avro.v1+json format). However, since the REST proxy is push,
> you'd need another process to send requests to it.
>
> Alternatively, you could build a source connector which pulls Avro
> JSON-encoded data and emits generic Connect data. By combining that with
> the AvroConverters, you'd end up generating Avro binary-serialized data
> into Kafka (albeit via a slightly indirect route).
>
> -Ewen
>
> On Mon, Oct 17, 2016 at 6:14 AM, Takahiro Hozumi <fa...@googlemail.com>
> wrote:
>
> > Hi all,
> >
> > My application will emit log files in avro json encoding so that humans
> > easily can read and grep records.
> > I need to transfer this logs into Kafka as Avro binary encoding.
> > And I want to use confluent schema registry in order to prepare schema
> > evolution.
> >
> > After some research, I think Kafka connect in standalone mode might be
> the
> > easiest solution.
> > The avro-converter included in kafka connect interacts with schema
> registry
> > under the hood.
> > But I am not sure avro-converter accepts Avro json encoding as input.
> > Do I need to create a json-encoding version of the avro-converter?
> >
> > And I don't find a documentation of Converter interface.
> > I think converting job is just byte[] to byte[] job, but the interface
> > seems to require internal data object.
> >
> > http://docs.confluent.io/3.0.1/connect/javadocs/index.html?
> > org/apache/kafka/connect/storage/Converter.html
> >
> > Any advise or guidance would be greatly appreciated!
> >
> > Thanks,
> > --
> > Takahiro
> >
>
>
>
> --
> Thanks,
> Ewen
>