You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Commeau, Gabriel" <ga...@gmail.com> on 2018/02/22 06:18:20 UTC

Avro serialization in streaming context

Hi all, 

I’ve been working for several years now on a streaming data platform, and we’ve been using Avro to serialize the messages that flow through the distributed queue (Kafka/Kinesis). Because the message payload contains just one record or a small batch of Avro records, the serialization mechanisms are slightly different than the typical file-based ones. I wrote a few utility classes that facilitate the serialization and deserialization of records in the data streaming context, and I’m poking the community to see if there’s an appetite for me contributing it back to the main Avro project – as opposed to creating a small independent library. 

The utility allows to convert a record with a single method call:

- from a GenericRecord (and therefore SpecificRecord as well) to binary or json

- from binary or json to a GenericRecord 

- from binary or json to a SpecificRecord 

There’s also a few additional utility methods to generically get an attribute or its schema based on a path, provided as a string array.

So first, I haven’t seen a utility like that, but please correct me if I missed it. Then do you think it’ll be a contribution that you’d welcome?

Thanks!

 

Gabriel


Re: Avro serialization in streaming context

Posted by Niels Basjes <Ni...@basjes.nl>.
HI,

In Avro 1.8.2 we have added a "Message" format specification.
In the Java classes there are now extra methods generated like
toByteBuffer() and fromByteBuffer() that are intended for exactly the
purpose of using Avro as the serialization specification in a streaming
context.
In fact at bol.com (where I work) we already do this in production.

Please have a look at this because I have the feeling this already does a
lot of what you mention.

Niels Basjes


On Thu, Feb 22, 2018 at 7:18 AM, Commeau, Gabriel <gabriel.commeau@gmail.com
> wrote:

> Hi all,
>
> I’ve been working for several years now on a streaming data platform, and
> we’ve been using Avro to serialize the messages that flow through the
> distributed queue (Kafka/Kinesis). Because the message payload contains
> just one record or a small batch of Avro records, the serialization
> mechanisms are slightly different than the typical file-based ones. I wrote
> a few utility classes that facilitate the serialization and deserialization
> of records in the data streaming context, and I’m poking the community to
> see if there’s an appetite for me contributing it back to the main Avro
> project – as opposed to creating a small independent library.
>
> The utility allows to convert a record with a single method call:
>
> - from a GenericRecord (and therefore SpecificRecord as well) to binary or
> json
>
> - from binary or json to a GenericRecord
>
> - from binary or json to a SpecificRecord
>
> There’s also a few additional utility methods to generically get an
> attribute or its schema based on a path, provided as a string array.
>
> So first, I haven’t seen a utility like that, but please correct me if I
> missed it. Then do you think it’ll be a contribution that you’d welcome?
>
> Thanks!
>
>
>
> Gabriel
>
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Avro serialization in streaming context

Posted by "Commeau, Gabriel" <Ga...@comcast.com>.
Great! Will do. 
Thank you!

On 2/22/18, 9:26 AM, "Doug Cutting" <cu...@gmail.com> wrote:

    These sound to me like they'd be useful additions.
    
    Please file an issue in Jira and then issue a Github pull request against
    it.
    
    https://cwiki.apache.org/confluence/display/AVRO/How+To+Contribute
    
    Thanks!
    
    Doug
    
    On Wed, Feb 21, 2018 at 10:18 PM, Commeau, Gabriel <
    gabriel.commeau@gmail.com> wrote:
    
    > Hi all,
    >
    > I’ve been working for several years now on a streaming data platform, and
    > we’ve been using Avro to serialize the messages that flow through the
    > distributed queue (Kafka/Kinesis). Because the message payload contains
    > just one record or a small batch of Avro records, the serialization
    > mechanisms are slightly different than the typical file-based ones. I wrote
    > a few utility classes that facilitate the serialization and deserialization
    > of records in the data streaming context, and I’m poking the community to
    > see if there’s an appetite for me contributing it back to the main Avro
    > project – as opposed to creating a small independent library.
    >
    > The utility allows to convert a record with a single method call:
    >
    > - from a GenericRecord (and therefore SpecificRecord as well) to binary or
    > json
    >
    > - from binary or json to a GenericRecord
    >
    > - from binary or json to a SpecificRecord
    >
    > There’s also a few additional utility methods to generically get an
    > attribute or its schema based on a path, provided as a string array.
    >
    > So first, I haven’t seen a utility like that, but please correct me if I
    > missed it. Then do you think it’ll be a contribution that you’d welcome?
    >
    > Thanks!
    >
    >
    >
    > Gabriel
    >
    >
    


Re: Avro serialization in streaming context

Posted by Doug Cutting <cu...@gmail.com>.
These sound to me like they'd be useful additions.

Please file an issue in Jira and then issue a Github pull request against
it.

https://cwiki.apache.org/confluence/display/AVRO/How+To+Contribute

Thanks!

Doug

On Wed, Feb 21, 2018 at 10:18 PM, Commeau, Gabriel <
gabriel.commeau@gmail.com> wrote:

> Hi all,
>
> I’ve been working for several years now on a streaming data platform, and
> we’ve been using Avro to serialize the messages that flow through the
> distributed queue (Kafka/Kinesis). Because the message payload contains
> just one record or a small batch of Avro records, the serialization
> mechanisms are slightly different than the typical file-based ones. I wrote
> a few utility classes that facilitate the serialization and deserialization
> of records in the data streaming context, and I’m poking the community to
> see if there’s an appetite for me contributing it back to the main Avro
> project – as opposed to creating a small independent library.
>
> The utility allows to convert a record with a single method call:
>
> - from a GenericRecord (and therefore SpecificRecord as well) to binary or
> json
>
> - from binary or json to a GenericRecord
>
> - from binary or json to a SpecificRecord
>
> There’s also a few additional utility methods to generically get an
> attribute or its schema based on a path, provided as a string array.
>
> So first, I haven’t seen a utility like that, but please correct me if I
> missed it. Then do you think it’ll be a contribution that you’d welcome?
>
> Thanks!
>
>
>
> Gabriel
>
>