You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@avro.apache.org by Matan Amir <ma...@apple.com> on 2014/02/26 01:10:35 UTC

Modeling Common Fields?

Hi Avro experts :)

I was searching for an answer in the group already and didn’t find something that I recognized as the answer to this so I apologize if i’m asking something that has been answered frequently.

Let’s say I had a Java library that would send out Avro data over the wire.  However, the library would not own the Avro schemas nor know what the schemas were in advance.  The library only required that the schemas contain two specific fields that it would then set on the way out - like a string field and a timestamp.

What is the best way to model this for Avro?  In protobufs this is modeled with a base schema and “extensions” which third parties would use to include their specific fields.  Is there a way in Avro to get the same result?

Thanks and much appreciated!
Matan

Re: Modeling Common Fields?

Posted by Matan Amir <ma...@apple.com>.

Thanks Doug,

Yep this would work.  It does move the enforcement outside Avro but I guess that’s ok :)

Thanks!
Matan

On Feb 27, 2014, at 9:49, Doug Cutting <cu...@apache.org> wrote:

> This might be handled using GenericData.  You'd specify no reader
> schema when you create your GenericDatumReader so the in-memory data
> is represented using the same schema it was written with.  Then use
> GenericRecord#put & get to access the fields you require.
> 
> http://avro.apache.org/docs/current/api/java/org/apache/avro/generic/GenericRecord.html
> 
> Might that work?
> 
> Doug
> 
> 
> On Tue, Feb 25, 2014 at 4:10 PM, Matan Amir <ma...@apple.com> wrote:
>> Hi Avro experts :)
>> 
>> I was searching for an answer in the group already and didn’t find something that I recognized as the answer to this so I apologize if i’m asking something that has been answered frequently.
>> 
>> Let’s say I had a Java library that would send out Avro data over the wire.  However, the library would not own the Avro schemas nor know what the schemas were in advance.  The library only required that the schemas contain two specific fields that it would then set on the way out - like a string field and a timestamp.
>> 
>> What is the best way to model this for Avro?  In protobufs this is modeled with a base schema and “extensions” which third parties would use to include their specific fields.  Is there a way in Avro to get the same result?
>> 
>> Thanks and much appreciated!
>> Matan

Re: Modeling Common Fields?

Posted by Doug Cutting <cu...@apache.org>.

This might be handled using GenericData.  You'd specify no reader
schema when you create your GenericDatumReader so the in-memory data
is represented using the same schema it was written with.  Then use
GenericRecord#put & get to access the fields you require.

http://avro.apache.org/docs/current/api/java/org/apache/avro/generic/GenericRecord.html

Might that work?

Doug


On Tue, Feb 25, 2014 at 4:10 PM, Matan Amir <ma...@apple.com> wrote:
> Hi Avro experts :)
>
> I was searching for an answer in the group already and didn’t find something that I recognized as the answer to this so I apologize if i’m asking something that has been answered frequently.
>
> Let’s say I had a Java library that would send out Avro data over the wire.  However, the library would not own the Avro schemas nor know what the schemas were in advance.  The library only required that the schemas contain two specific fields that it would then set on the way out - like a string field and a timestamp.
>
> What is the best way to model this for Avro?  In protobufs this is modeled with a base schema and “extensions” which third parties would use to include their specific fields.  Is there a way in Avro to get the same result?
>
> Thanks and much appreciated!
> Matan