You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Commeau, Gabriel" <ga...@gmail.com> on 2018/02/22 06:18:20 UTC
Avro serialization in streaming context
Hi all,
I’ve been working for several years now on a streaming data platform, and we’ve been using Avro to serialize the messages that flow through the distributed queue (Kafka/Kinesis). Because the message payload contains just one record or a small batch of Avro records, the serialization mechanisms are slightly different than the typical file-based ones. I wrote a few utility classes that facilitate the serialization and deserialization of records in the data streaming context, and I’m poking the community to see if there’s an appetite for me contributing it back to the main Avro project – as opposed to creating a small independent library.
The utility allows to convert a record with a single method call:
- from a GenericRecord (and therefore SpecificRecord as well) to binary or json
- from binary or json to a GenericRecord
- from binary or json to a SpecificRecord
There’s also a few additional utility methods to generically get an attribute or its schema based on a path, provided as a string array.
So first, I haven’t seen a utility like that, but please correct me if I missed it. Then do you think it’ll be a contribution that you’d welcome?
Thanks!
Gabriel
Re: Avro serialization in streaming context
Posted by Niels Basjes <Ni...@basjes.nl>.
HI,
In Avro 1.8.2 we have added a "Message" format specification.
In the Java classes there are now extra methods generated like
toByteBuffer() and fromByteBuffer() that are intended for exactly the
purpose of using Avro as the serialization specification in a streaming
context.
In fact at bol.com (where I work) we already do this in production.
Please have a look at this because I have the feeling this already does a
lot of what you mention.
Niels Basjes
On Thu, Feb 22, 2018 at 7:18 AM, Commeau, Gabriel <gabriel.commeau@gmail.com
> wrote:
> Hi all,
>
> I’ve been working for several years now on a streaming data platform, and
> we’ve been using Avro to serialize the messages that flow through the
> distributed queue (Kafka/Kinesis). Because the message payload contains
> just one record or a small batch of Avro records, the serialization
> mechanisms are slightly different than the typical file-based ones. I wrote
> a few utility classes that facilitate the serialization and deserialization
> of records in the data streaming context, and I’m poking the community to
> see if there’s an appetite for me contributing it back to the main Avro
> project – as opposed to creating a small independent library.
>
> The utility allows to convert a record with a single method call:
>
> - from a GenericRecord (and therefore SpecificRecord as well) to binary or
> json
>
> - from binary or json to a GenericRecord
>
> - from binary or json to a SpecificRecord
>
> There’s also a few additional utility methods to generically get an
> attribute or its schema based on a path, provided as a string array.
>
> So first, I haven’t seen a utility like that, but please correct me if I
> missed it. Then do you think it’ll be a contribution that you’d welcome?
>
> Thanks!
>
>
>
> Gabriel
>
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: Avro serialization in streaming context
Posted by "Commeau, Gabriel" <Ga...@comcast.com>.
Great! Will do.
Thank you!
On 2/22/18, 9:26 AM, "Doug Cutting" <cu...@gmail.com> wrote:
These sound to me like they'd be useful additions.
Please file an issue in Jira and then issue a Github pull request against
it.
https://cwiki.apache.org/confluence/display/AVRO/How+To+Contribute
Thanks!
Doug
On Wed, Feb 21, 2018 at 10:18 PM, Commeau, Gabriel <
gabriel.commeau@gmail.com> wrote:
> Hi all,
>
> I’ve been working for several years now on a streaming data platform, and
> we’ve been using Avro to serialize the messages that flow through the
> distributed queue (Kafka/Kinesis). Because the message payload contains
> just one record or a small batch of Avro records, the serialization
> mechanisms are slightly different than the typical file-based ones. I wrote
> a few utility classes that facilitate the serialization and deserialization
> of records in the data streaming context, and I’m poking the community to
> see if there’s an appetite for me contributing it back to the main Avro
> project – as opposed to creating a small independent library.
>
> The utility allows to convert a record with a single method call:
>
> - from a GenericRecord (and therefore SpecificRecord as well) to binary or
> json
>
> - from binary or json to a GenericRecord
>
> - from binary or json to a SpecificRecord
>
> There’s also a few additional utility methods to generically get an
> attribute or its schema based on a path, provided as a string array.
>
> So first, I haven’t seen a utility like that, but please correct me if I
> missed it. Then do you think it’ll be a contribution that you’d welcome?
>
> Thanks!
>
>
>
> Gabriel
>
>
Re: Avro serialization in streaming context
Posted by Doug Cutting <cu...@gmail.com>.
These sound to me like they'd be useful additions.
Please file an issue in Jira and then issue a Github pull request against
it.
https://cwiki.apache.org/confluence/display/AVRO/How+To+Contribute
Thanks!
Doug
On Wed, Feb 21, 2018 at 10:18 PM, Commeau, Gabriel <
gabriel.commeau@gmail.com> wrote:
> Hi all,
>
> I’ve been working for several years now on a streaming data platform, and
> we’ve been using Avro to serialize the messages that flow through the
> distributed queue (Kafka/Kinesis). Because the message payload contains
> just one record or a small batch of Avro records, the serialization
> mechanisms are slightly different than the typical file-based ones. I wrote
> a few utility classes that facilitate the serialization and deserialization
> of records in the data streaming context, and I’m poking the community to
> see if there’s an appetite for me contributing it back to the main Avro
> project – as opposed to creating a small independent library.
>
> The utility allows to convert a record with a single method call:
>
> - from a GenericRecord (and therefore SpecificRecord as well) to binary or
> json
>
> - from binary or json to a GenericRecord
>
> - from binary or json to a SpecificRecord
>
> There’s also a few additional utility methods to generically get an
> attribute or its schema based on a path, provided as a string array.
>
> So first, I haven’t seen a utility like that, but please correct me if I
> missed it. Then do you think it’ll be a contribution that you’d welcome?
>
> Thanks!
>
>
>
> Gabriel
>
>