You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Sebastien <le...@gmail.com> on 2017/12/26 18:24:48 UTC
Java API to easily serialize/deserialize Avro-annotated classes
(without codegen)
Hello,
I have a question about the features of the Java library for Avro.
In my application I have a set of "event" classes with Avro annotations
(@AvroName, @AvroIgnore, ...).
Currently I do the following:
1) use ReflectData.get().getSchema(clazz) to get the Schema of the class
2) use the reflection API to get object instances field data to feed into a
GenericRecordBuilder
3) use GenericDatumWriter<GenericRecord> to serialize my data
Note that I deliberately chose not to use code generation.
In my code, what I did for #2 was this:
- iterate over each (public) member property of the class
- find the field name known for avro (i.e., actual field name or name
provided with @AvroName if the annotation is present)
- check if there is a schema field for that name
- get the value of that field
- set the value for that schema field in the generic record builder
I'm wondering if Avro currently provides utilities to easily perform what I
described above?
More precisely, is there an API for easily extracting some sort of Map of
<Schema.Field, Object> where Object would be the value for a specific
field, or even better, an API for directly creating a GenericRecord object
based on an instance of an Avro-annotated class and a Schema?
If not, could it be added to the Java Avro library?
I'd love to be able to do something like new
GenericRecordBuilder(schema).extractValues(someObject).build() and not have
to worry about reflection, caching, etc.
Having the ability to also convert from a GenericRecord object to an
instance of an Avro-annotated class would also be great.
Here's the code I'm currently using:
- convert to generic record:
https://gist.github.com/dsebastien/ab3f2828a84289ad1b7cf29d674c4faf
- get field name:
https://gist.github.com/dsebastien/82bc9b1a7158d69440312c661bbac1a7
kr,
Sébastien D
Re: Java API to easily serialize/deserialize Avro-annotated classes
(without codegen)
Posted by Sebastien <le...@gmail.com>.
Hi Marcus,
No, that isn't what I'm looking for.
To really summarize, I'd like to be able to easily convert any instance of
a class annotated with Avro annotations to a GenericRecord instance
containing all the data.
I've implemented this in my project (see the gists above) and it works, but
I'm wondering if Avro doesn't already provide that feature. If not then I
guess it could be a useful addition to the Java library :)
I'd love to be able to do this:
new GenericRecordBuilder(schema).extractValues(someObject).build()
kr,
Sébastien.
On Fri, Dec 29, 2017 at 1:19 AM, Marcus Simonsen <ms...@gmail.com>
wrote:
> Hi Sebastien, do want to be able to control specific schema for fields
> inside each object to be serialized?
>
> If you want to use a specific schema for an embedded pojo object, then you
> can use the @AvroSchema on embedded fields:
> https://avro.apache.org/docs/1.8.1/api/java/org/apache/
> avro/reflect/AvroSchema.html
>
>
>
>
> On Tue, Dec 26, 2017 at 1:24 PM, Sebastien <le...@gmail.com> wrote:
>
>> Hello,
>>
>> I have a question about the features of the Java library for Avro.
>>
>> In my application I have a set of "event" classes with Avro annotations
>> (@AvroName, @AvroIgnore, ...).
>>
>> Currently I do the following:
>> 1) use ReflectData.get().getSchema(clazz) to get the Schema of the class
>> 2) use the reflection API to get object instances field data to feed into
>> a GenericRecordBuilder
>> 3) use GenericDatumWriter<GenericRecord> to serialize my data
>>
>> Note that I deliberately chose not to use code generation.
>>
>> In my code, what I did for #2 was this:
>> - iterate over each (public) member property of the class
>> - find the field name known for avro (i.e., actual field name or name
>> provided with @AvroName if the annotation is present)
>> - check if there is a schema field for that name
>> - get the value of that field
>> - set the value for that schema field in the generic record builder
>>
>> I'm wondering if Avro currently provides utilities to easily perform what
>> I described above?
>>
>> More precisely, is there an API for easily extracting some sort of Map of
>> <Schema.Field, Object> where Object would be the value for a specific
>> field, or even better, an API for directly creating a GenericRecord object
>> based on an instance of an Avro-annotated class and a Schema?
>>
>> If not, could it be added to the Java Avro library?
>>
>> I'd love to be able to do something like new
>> GenericRecordBuilder(schema).extractValues(someObject).build() and not
>> have to worry about reflection, caching, etc.
>>
>> Having the ability to also convert from a GenericRecord object to an
>> instance of an Avro-annotated class would also be great.
>>
>> Here's the code I'm currently using:
>> - convert to generic record: https://gist.github.co
>> m/dsebastien/ab3f2828a84289ad1b7cf29d674c4faf
>> - get field name: https://gist.github.com/dsebastien/82bc9b1a7158d694403
>> 12c661bbac1a7
>>
>>
>> kr,
>> Sébastien D
>>
>
>
Re: Java API to easily serialize/deserialize Avro-annotated classes
(without codegen)
Posted by Marcus Simonsen <ms...@gmail.com>.
Hi Sebastien, do want to be able to control specific schema for fields
inside each object to be serialized?
If you want to use a specific schema for an embedded pojo object, then you
can use the @AvroSchema on embedded fields:
https://avro.apache.org/docs/1.8.1/api/java/org/apache/avro/reflect/AvroSchema.html
On Tue, Dec 26, 2017 at 1:24 PM, Sebastien <le...@gmail.com> wrote:
> Hello,
>
> I have a question about the features of the Java library for Avro.
>
> In my application I have a set of "event" classes with Avro annotations
> (@AvroName, @AvroIgnore, ...).
>
> Currently I do the following:
> 1) use ReflectData.get().getSchema(clazz) to get the Schema of the class
> 2) use the reflection API to get object instances field data to feed into
> a GenericRecordBuilder
> 3) use GenericDatumWriter<GenericRecord> to serialize my data
>
> Note that I deliberately chose not to use code generation.
>
> In my code, what I did for #2 was this:
> - iterate over each (public) member property of the class
> - find the field name known for avro (i.e., actual field name or name
> provided with @AvroName if the annotation is present)
> - check if there is a schema field for that name
> - get the value of that field
> - set the value for that schema field in the generic record builder
>
> I'm wondering if Avro currently provides utilities to easily perform what
> I described above?
>
> More precisely, is there an API for easily extracting some sort of Map of
> <Schema.Field, Object> where Object would be the value for a specific
> field, or even better, an API for directly creating a GenericRecord object
> based on an instance of an Avro-annotated class and a Schema?
>
> If not, could it be added to the Java Avro library?
>
> I'd love to be able to do something like new GenericRecordBuilder(schema).
> extractValues(someObject).build() and not have to worry about reflection,
> caching, etc.
>
> Having the ability to also convert from a GenericRecord object to an
> instance of an Avro-annotated class would also be great.
>
> Here's the code I'm currently using:
> - convert to generic record: https://gist.github.com/dsebastien/
> ab3f2828a84289ad1b7cf29d674c4faf
> - get field name: https://gist.github.com/dsebastien/
> 82bc9b1a7158d69440312c661bbac1a7
>
>
> kr,
> Sébastien D
>