Re: embed avro data in an envelop

[Martin Kleppmann <mk...@linkedin.com>]

Two suggested ways of doing this:

1. Create a record type for metadata; add a field to all existing schemas referencing that record type. Depending on your build process, you can probably keep the metadata schema in a separate file that is included into all the other schemas, so you don't need to duplicate it.

2. Create a wrapper type which includes the metadata, and has a "payload" field which is the union of all possible application schemas. In this case, all records in your system are of this wrapper type, and you have to look inside at the payload to see the actual content.

Basically, the choice is between 1. having the metadata inside the application record, or 2. having the application record inside the metadata/wrapper.

Martin

On 30 Apr 2014, at 02:14, Wai Yip Tung <wy...@tungwaiyip.info> wrote:
> I am looking for some avro usage advice. We have created various schema for different applications, say to represent, item id, name, metric, etc. On the other hand our infrastructure group want to include some meta data on all messages. This should include things like timestamp, hostname, etc. This meta data is the same for all application messages.
> 
> One way to do it is to have a meta data schema that has timestamp, hostname and a binary content field for the application data. This way each message need to be decoded twice using two schema.
> 
> Another way is to somehow have a composite schema that include both the meta data and the application specific data. So each message is just decoded once and it automatically include the needed meta data. I wonder if this can be done and if it is a good idea. Have other people considered  similar usage?
> 
> Wai Yip