footer info in avro

[Venkat <vr...@ymail.com>]

Hi All

Does Avro file format support storing some json metadata at the end of the file (footer)? Currently, the header stores schema in json enabling consumers to augment it with application specific info. But I'm not sure if there is a way consumer can store specific metadata at the end of the file.

The use case is that data processing applications can compute statistics during the scan and write the computed statistics at the end of the file before closing the stream. This avoid storing statistics else where. 

If this feature does not exits, is there a plan on adding it sometime in the future?

Thanks a lot. 

Re: footer info in avro

[Doug Cutting <cu...@apache.org>]

There's no plan I know of to add this.

Avro's original file format wrote metadata at the end of the file.
This was changed in Avro 1.3 so that files could always be processed
sequentially, without seeking to the end.

Doug

On Wed, Oct 9, 2013 at 5:08 PM, Venkat <vr...@ymail.com> wrote:
> Hi All
>
> Does Avro file format support storing some json metadata at the end of the
> file (footer)? Currently, the header stores schema in json enabling
> consumers to augment it with application specific info. But I'm not sure if
> there is a way consumer can store specific metadata at the end of the file.
>
> The use case is that data processing applications can compute statistics
> during the scan and write the computed statistics at the end of the file
> before closing the stream. This avoid storing statistics else where.
>
> If this feature does not exits, is there a plan on adding it sometime in the
> future?
>
> Thanks a lot.