Is it possible to write a magic byte in Avro file head?

[Fengyun RAO <ra...@gmail.com>]

I have a cache file using Avro serialization, and I want to add a magic
byte indicating cache version at the beginning of the file.
I find it's easy to serialize, but difficult to deserialize in C#.
First I open a filestream, read my magic byte, and then pass the stream to
the DataFileReader:

var reader = DataFileReader<Dictionary<string,
MyType>>.OpenReader(stream, CACHE_SCHEMA)

but it throws an AvroRuntimeException("Not an Avro data file")

I look into the OpenReader() method:

  // verify magic header
  byte[] magic = new byte[DataFileConstants.Magic.Length];
  inStream.Seek(0, SeekOrigin.Begin);

It will always seek back to the beginning of the FileStream (which includes
my own byte), and thus throws an Exception.

However, in java version, I could use DataFileStream which wouldn't
seek back and it works.

Is there a way to make it work in C# version? I also wonder why there
isn't an equivalent "DataFileStream" class in C#.

Re: Is it possible to write a magic byte in Avro file head?

[Fengyun RAO <ra...@gmail.com>]

Thanks, Doug. The Avro file metadata is exactly what we need!


2014-05-17 7:20 GMT+08:00 Doug Cutting <cu...@apache.org>:

> This incompatibly alters the Avro file format.  Could you perhaps
> instead add this into the Avro file's metadata?
>
> Doug
>
> On Thu, May 15, 2014 at 5:44 AM, Fengyun RAO <ra...@gmail.com> wrote:
> > I have a cache file using Avro serialization, and I want to add a magic
> byte
> > indicating cache version at the beginning of the file.
> > I find it's easy to serialize, but difficult to deserialize in C#.
> > First I open a filestream, read my magic byte, and then pass the stream
> to
> > the DataFileReader:
> >
> > var reader = DataFileReader<Dictionary<string,
> MyType>>.OpenReader(stream,
> > CACHE_SCHEMA)
> >
> > but it throws an AvroRuntimeException("Not an Avro data file")
> >
> > I look into the OpenReader() method:
> >
> >   // verify magic header
> >   byte[] magic = new byte[DataFileConstants.Magic.Length];
> >   inStream.Seek(0, SeekOrigin.Begin);
> >
> > It will always seek back to the beginning of the FileStream (which
> includes
> > my own byte), and thus throws an Exception.
> >
> > However, in java version, I could use DataFileStream which wouldn't seek
> > back and it works.
> >
> > Is there a way to make it work in C# version? I also wonder why there
> isn't
> > an equivalent "DataFileStream" class in C#.
>

Re: Is it possible to write a magic byte in Avro file head?

[Doug Cutting <cu...@apache.org>]

This incompatibly alters the Avro file format.  Could you perhaps
instead add this into the Avro file's metadata?

Doug

On Thu, May 15, 2014 at 5:44 AM, Fengyun RAO <ra...@gmail.com> wrote:
> I have a cache file using Avro serialization, and I want to add a magic byte
> indicating cache version at the beginning of the file.
> I find it's easy to serialize, but difficult to deserialize in C#.
> First I open a filestream, read my magic byte, and then pass the stream to
> the DataFileReader:
>
> var reader = DataFileReader<Dictionary<string, MyType>>.OpenReader(stream,
> CACHE_SCHEMA)
>
> but it throws an AvroRuntimeException("Not an Avro data file")
>
> I look into the OpenReader() method:
>
>   // verify magic header
>   byte[] magic = new byte[DataFileConstants.Magic.Length];
>   inStream.Seek(0, SeekOrigin.Begin);
>
> It will always seek back to the beginning of the FileStream (which includes
> my own byte), and thus throws an Exception.
>
> However, in java version, I could use DataFileStream which wouldn't seek
> back and it works.
>
> Is there a way to make it work in C# version? I also wonder why there isn't
> an equivalent "DataFileStream" class in C#.