Re: Best way to read and write bytes of Avro object

[Philip Zeyliger <ph...@cloudera.com>]

Hi Douglas,

You shoudn't need a DataFileReader/Writer, I believe.  I recently wrote
something very similar as part of
https://issues.apache.org/jira/browse/AVRO-245.

In my case, on the "to bytes" path, I did:

      // Read from JSON
      GenericDatumReader<Object> reader = new
GenericDatumReader<Object>(schema);
      Object datum = reader.read(null, new JsonDecoder(schema, new
ByteArrayInputStream(data.getBytes("utf-8"))));

      // Write to an output stream
      GenericDatumWriter<Object> writer =  new
GenericDatumWriter<Object>(schema);
      writer.write(datum, new BinaryEncoder(out));



BinaryEncoder is probably what you want instead of DataFileWriter.

I used the generic API here, but specific should work in your case.


Cheers,

-- Philip



On Mon, Nov 30, 2009 at 11:42 AM, Douglas Campbell <de...@yahoo.com>wrote:

> Any thoughts on this?
>
>
>
> ----- Original Message ----
> From: Douglas Campbell <de...@yahoo.com>
> To: avro-user@hadoop.apache.org
> Sent: Wed, November 25, 2009 7:31:02 PM
> Subject: Best way to read and write bytes of Avro object
>
> I'm using java 1.6 and avro 1.2.
>
> My use case is to be able to create an avro object, serialize/marshall it
> into a byte array and base64 it, and then un-base64 it,
> desearialize/unmarshall it back into the AvroObject.
>
> Pretty simple right?  This the code I needed to write to do that .  Is
> there a better way or utility classes in Avro java api which do this
> (leaving out the base64 stuff perhaps)?
>
> /** decodes base64'd data into bytes and then re-creates an avro object */
> public AvroModel fromAvroData(String base64) throws IOException
>    {
>        AvroModel am     = new AvroModel();
>        final byte[] avrobytes = new Base64().decode(base64.getBytes());
>        DatumReader dr   = new SpecificDatumReader(am.getSchema());
>        SeekableByteArrayInputStream bin = new
> SeekableByteArrayInputStream(avrobytes);
>        DataFileReader<AvroModel> dfr = new DataFileReader<AvroModel>(bin,
> dr);
>        dfr.next(am);
>        return am;
>    }
>
>    /**
>        Converts an AvroModel to avro base64 encoded string
>    */
>    public String toAvroData(AvroModel m) throws IOException
>    {
>        DatumWriter dw = new SpecificDatumWriter(m.getSchema());
>        ByteArrayOutputStream bout = new ByteArrayOutputStream();
>        DataFileWriter<AvroModel> dfw = new
> DataFileWriter<AvroModel>(m.getSchema(), bout, dw);
>        dfw.append(m);
>        dfw.flush();
>
>        return new String(new Base64().encode(bout.toByteArray()));
>    }
>
> Note that for the "fromAvroData", I had to provide an implementation of
> SeekableInput over a ByteArrayInputStream.
>
> Thanks for the pointers.
>
>
>
>