You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Avinash Dongre <do...@gmail.com> on 2013/06/17 05:15:29 UTC
Re: Sending Apache avro serialized data to thrift server
> Hi,
> I am trying to send avro serialized data to my thrift server. But when I
> try to de-serialized I get exception
> org.apache.avro.AvroRuntimeException: Malformed data. Length is negative:
> -51
>
> If I do the same operation in the same vm process everything works fine.
>
> Here is my small code.
>
> public static GenericRecord createContentNestedObject() throws Exception {
> GenericRecord image1 = new GenericData.Record(IMAGE_SCHEMA);
> image1.put("uri", new Utf8("http://javaone.com/keynote_large.jpg"));
> image1.put("width", 0);
> image1.put("height", 0);
> image1.put("size", 2);
> image1.put("title", new Utf8("Javaone Keynote"));
> return image1;
> }
>
> // Helper Method to serialize the object from avro to bytebuffer
> public static ByteBuffer serialize(GenericRecord content, Schema schema)
> throws Exception {
>
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> DatumWriter<GenericRecord> writer = new
> GenericDatumWriter<GenericRecord>(
> schema);
> // create Binary Encoder
> EncoderFactory ef = new EncoderFactory();
> BinaryEncoder be = ef.binaryEncoder(out, null);
> writer.write(content, be);
> be.flush();
> out.close();
>
> return ByteBuffer.wrap(out.toByteArray());
> }
>
> public static void main(String[] args) throws Exception {
> try {
> GenericRecord rd = createContentNestedObject();
> bf = serialize(rd, IMAGE_SCHEMA);
>
> } catch (Exception e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> }
>
> // Deserialization.
> DatumReader<GenericRecord> reader = new
> GenericDatumReader<GenericRecord>(IMAGE_SCHEMA_1);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> try {
> out.write(bf.array());
> } catch (IOException e1) {
> // TODO Auto-generated catch block
> e1.printStackTrace();
> }
> BinaryDecoder decoder =
> DecoderFactory.get().binaryDecoder(out.toByteArray(), null);
> try {
> GenericRecord result = reader.read(null, decoder);
> System.out.println("RESULT : " + result.toString());
> } catch (IOException e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> }
>
> This code works. When I send the bf to my thrift api , I am getting the
> error,
> my thrift api is simple which accept ByteBuffer and try to deserialize it.
>
> following is the code in the thrift server api
>
> DatumReader<GenericRecord> reader = new
> GenericDatumReader<GenericRecord>(IMAGE_SCHEMA_1);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> try {
> out.write(bf.array());
> } catch (IOException e1) {
> // TODO Auto-generated catch block
> e1.printStackTrace();
> }
> BinaryDecoder decoder =
> DecoderFactory.get().binaryDecoder(out.toByteArray(), null);
> try {
> GenericRecord result = reader.read(null, decoder);
> System.out.println("RESULT : " + result.toString());
> } catch (IOException e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> }
>
>
>
> What is wrong here.
>
>
> Thanks
> Aviansh
>
>
>
Re: Sending Apache avro serialized data to thrift server
Posted by Avinash Dongre <do...@gmail.com>.
Hi Martin,
Following code works for me.
int length = input.limit() - input.position();
byte[] avroSerialized = new byte[length];
System.arraycopy(input.array(), input.position(), avroSerialized, 0,
length);
thanks
Avinash
On Mon, Jun 17, 2013 at 3:53 PM, Avinash Dongre <do...@gmail.com>wrote:
> Thanks Martin,
> With following code exceptions goes away.
>
> DatumReader<GenericRecord> reader = new
> GenericDatumReader<GenericRecord>(schema);
>
> byte[] avroSerialized = new byte[input.limit() - input.position()];
> input.put(avroSerialized);
> BinaryDecoder decoder =
> DecoderFactory.get().binaryDecoder(avroSerialized, null);
>
> try {
> GenericRecord result = reader.read(null, decoder);
> System.out.println("RESULT : " + result.toString());
> } catch (IOException e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> }
>
> But I am not getting any values when I do toString, I will investigate
> further on what is wrong with my code.
>
> Thanks for your quick help.
>
>
> Thanks
> Avinash
>
>
>
> On Mon, Jun 17, 2013 at 11:16 AM, Martin Kleppmann <ma...@rapportive.com>wrote:
>
>> It's hard to tell just by looking at the code -- I suggest you inspect
>> the contents of the array you're decoding in each case, and see if there is
>> any difference.
>>
>> Purely guessing, the ByteBuffer that Thrift gives you may have other data
>> before and after the Avro record, and calling `bf.array()` will give you
>> junk in that case. You may need something more like this (rough outline,
>> please read the API docs):
>>
>> byte[] copy = new byte[bf.limit() - bf.position()];
>> bf.read(copy);
>> DecoderFactory.get().binaryDecoder(copy, null);
>>
>>
>>
>> On 16 June 2013 20:15, Avinash Dongre <do...@gmail.com> wrote:
>>
>>>
>>> Hi,
>>>> I am trying to send avro serialized data to my thrift server. But when
>>>> I try to de-serialized I get exception
>>>> org.apache.avro.AvroRuntimeException: Malformed data. Length is
>>>> negative: -51
>>>>
>>>> If I do the same operation in the same vm process everything works fine.
>>>>
>>>> Here is my small code.
>>>>
>>>> public static GenericRecord createContentNestedObject() throws
>>>> Exception {
>>>> GenericRecord image1 = new GenericData.Record(IMAGE_SCHEMA);
>>>> image1.put("uri", new Utf8("http://javaone.com/keynote_large.jpg
>>>> "));
>>>> image1.put("width", 0);
>>>> image1.put("height", 0);
>>>> image1.put("size", 2);
>>>> image1.put("title", new Utf8("Javaone Keynote"));
>>>> return image1;
>>>> }
>>>>
>>>> // Helper Method to serialize the object from avro to bytebuffer
>>>> public static ByteBuffer serialize(GenericRecord content, Schema
>>>> schema)
>>>> throws Exception {
>>>>
>>>> ByteArrayOutputStream out = new ByteArrayOutputStream();
>>>> DatumWriter<GenericRecord> writer = new
>>>> GenericDatumWriter<GenericRecord>(
>>>> schema);
>>>> // create Binary Encoder
>>>> EncoderFactory ef = new EncoderFactory();
>>>> BinaryEncoder be = ef.binaryEncoder(out, null);
>>>> writer.write(content, be);
>>>> be.flush();
>>>> out.close();
>>>>
>>>> return ByteBuffer.wrap(out.toByteArray());
>>>> }
>>>>
>>>> public static void main(String[] args) throws Exception {
>>>> try {
>>>> GenericRecord rd = createContentNestedObject();
>>>> bf = serialize(rd, IMAGE_SCHEMA);
>>>>
>>>> } catch (Exception e) {
>>>> // TODO Auto-generated catch block
>>>> e.printStackTrace();
>>>> }
>>>>
>>>> // Deserialization.
>>>> DatumReader<GenericRecord> reader = new
>>>> GenericDatumReader<GenericRecord>(IMAGE_SCHEMA_1);
>>>> ByteArrayOutputStream out = new ByteArrayOutputStream();
>>>> try {
>>>> out.write(bf.array());
>>>> } catch (IOException e1) {
>>>> // TODO Auto-generated catch block
>>>> e1.printStackTrace();
>>>> }
>>>> BinaryDecoder decoder =
>>>> DecoderFactory.get().binaryDecoder(out.toByteArray(), null);
>>>> try {
>>>> GenericRecord result = reader.read(null, decoder);
>>>> System.out.println("RESULT : " + result.toString());
>>>> } catch (IOException e) {
>>>> // TODO Auto-generated catch block
>>>> e.printStackTrace();
>>>> }
>>>>
>>>> This code works. When I send the bf to my thrift api , I am getting the
>>>> error,
>>>> my thrift api is simple which accept ByteBuffer and try to deserialize
>>>> it.
>>>>
>>>> following is the code in the thrift server api
>>>>
>>>> DatumReader<GenericRecord> reader = new
>>>> GenericDatumReader<GenericRecord>(IMAGE_SCHEMA_1);
>>>> ByteArrayOutputStream out = new ByteArrayOutputStream();
>>>> try {
>>>> out.write(bf.array());
>>>> } catch (IOException e1) {
>>>> // TODO Auto-generated catch block
>>>> e1.printStackTrace();
>>>> }
>>>> BinaryDecoder decoder =
>>>> DecoderFactory.get().binaryDecoder(out.toByteArray(), null);
>>>> try {
>>>> GenericRecord result = reader.read(null, decoder);
>>>> System.out.println("RESULT : " + result.toString());
>>>> } catch (IOException e) {
>>>> // TODO Auto-generated catch block
>>>> e.printStackTrace();
>>>> }
>>>>
>>>>
>>>>
>>>> What is wrong here.
>>>>
>>>>
>>>> Thanks
>>>> Aviansh
>>>>
>>>>
>>>>
>>>
>>
>
Re: Sending Apache avro serialized data to thrift server
Posted by Avinash Dongre <do...@gmail.com>.
Thanks Martin,
With following code exceptions goes away.
DatumReader<GenericRecord> reader = new
GenericDatumReader<GenericRecord>(schema);
byte[] avroSerialized = new byte[input.limit() - input.position()];
input.put(avroSerialized);
BinaryDecoder decoder =
DecoderFactory.get().binaryDecoder(avroSerialized, null);
try {
GenericRecord result = reader.read(null, decoder);
System.out.println("RESULT : " + result.toString());
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
But I am not getting any values when I do toString, I will investigate
further on what is wrong with my code.
Thanks for your quick help.
Thanks
Avinash
On Mon, Jun 17, 2013 at 11:16 AM, Martin Kleppmann <ma...@rapportive.com>wrote:
> It's hard to tell just by looking at the code -- I suggest you inspect the
> contents of the array you're decoding in each case, and see if there is any
> difference.
>
> Purely guessing, the ByteBuffer that Thrift gives you may have other data
> before and after the Avro record, and calling `bf.array()` will give you
> junk in that case. You may need something more like this (rough outline,
> please read the API docs):
>
> byte[] copy = new byte[bf.limit() - bf.position()];
> bf.read(copy);
> DecoderFactory.get().binaryDecoder(copy, null);
>
>
>
> On 16 June 2013 20:15, Avinash Dongre <do...@gmail.com> wrote:
>
>>
>> Hi,
>>> I am trying to send avro serialized data to my thrift server. But when I
>>> try to de-serialized I get exception
>>> org.apache.avro.AvroRuntimeException: Malformed data. Length is
>>> negative: -51
>>>
>>> If I do the same operation in the same vm process everything works fine.
>>>
>>> Here is my small code.
>>>
>>> public static GenericRecord createContentNestedObject() throws Exception
>>> {
>>> GenericRecord image1 = new GenericData.Record(IMAGE_SCHEMA);
>>> image1.put("uri", new Utf8("http://javaone.com/keynote_large.jpg"));
>>> image1.put("width", 0);
>>> image1.put("height", 0);
>>> image1.put("size", 2);
>>> image1.put("title", new Utf8("Javaone Keynote"));
>>> return image1;
>>> }
>>>
>>> // Helper Method to serialize the object from avro to bytebuffer
>>> public static ByteBuffer serialize(GenericRecord content, Schema
>>> schema)
>>> throws Exception {
>>>
>>> ByteArrayOutputStream out = new ByteArrayOutputStream();
>>> DatumWriter<GenericRecord> writer = new
>>> GenericDatumWriter<GenericRecord>(
>>> schema);
>>> // create Binary Encoder
>>> EncoderFactory ef = new EncoderFactory();
>>> BinaryEncoder be = ef.binaryEncoder(out, null);
>>> writer.write(content, be);
>>> be.flush();
>>> out.close();
>>>
>>> return ByteBuffer.wrap(out.toByteArray());
>>> }
>>>
>>> public static void main(String[] args) throws Exception {
>>> try {
>>> GenericRecord rd = createContentNestedObject();
>>> bf = serialize(rd, IMAGE_SCHEMA);
>>>
>>> } catch (Exception e) {
>>> // TODO Auto-generated catch block
>>> e.printStackTrace();
>>> }
>>>
>>> // Deserialization.
>>> DatumReader<GenericRecord> reader = new
>>> GenericDatumReader<GenericRecord>(IMAGE_SCHEMA_1);
>>> ByteArrayOutputStream out = new ByteArrayOutputStream();
>>> try {
>>> out.write(bf.array());
>>> } catch (IOException e1) {
>>> // TODO Auto-generated catch block
>>> e1.printStackTrace();
>>> }
>>> BinaryDecoder decoder =
>>> DecoderFactory.get().binaryDecoder(out.toByteArray(), null);
>>> try {
>>> GenericRecord result = reader.read(null, decoder);
>>> System.out.println("RESULT : " + result.toString());
>>> } catch (IOException e) {
>>> // TODO Auto-generated catch block
>>> e.printStackTrace();
>>> }
>>>
>>> This code works. When I send the bf to my thrift api , I am getting the
>>> error,
>>> my thrift api is simple which accept ByteBuffer and try to deserialize
>>> it.
>>>
>>> following is the code in the thrift server api
>>>
>>> DatumReader<GenericRecord> reader = new
>>> GenericDatumReader<GenericRecord>(IMAGE_SCHEMA_1);
>>> ByteArrayOutputStream out = new ByteArrayOutputStream();
>>> try {
>>> out.write(bf.array());
>>> } catch (IOException e1) {
>>> // TODO Auto-generated catch block
>>> e1.printStackTrace();
>>> }
>>> BinaryDecoder decoder =
>>> DecoderFactory.get().binaryDecoder(out.toByteArray(), null);
>>> try {
>>> GenericRecord result = reader.read(null, decoder);
>>> System.out.println("RESULT : " + result.toString());
>>> } catch (IOException e) {
>>> // TODO Auto-generated catch block
>>> e.printStackTrace();
>>> }
>>>
>>>
>>>
>>> What is wrong here.
>>>
>>>
>>> Thanks
>>> Aviansh
>>>
>>>
>>>
>>
>
Re: Sending Apache avro serialized data to thrift server
Posted by Martin Kleppmann <ma...@rapportive.com>.
It's hard to tell just by looking at the code -- I suggest you inspect the
contents of the array you're decoding in each case, and see if there is any
difference.
Purely guessing, the ByteBuffer that Thrift gives you may have other data
before and after the Avro record, and calling `bf.array()` will give you
junk in that case. You may need something more like this (rough outline,
please read the API docs):
byte[] copy = new byte[bf.limit() - bf.position()];
bf.read(copy);
DecoderFactory.get().binaryDecoder(copy, null);
On 16 June 2013 20:15, Avinash Dongre <do...@gmail.com> wrote:
>
> Hi,
>> I am trying to send avro serialized data to my thrift server. But when I
>> try to de-serialized I get exception
>> org.apache.avro.AvroRuntimeException: Malformed data. Length is negative:
>> -51
>>
>> If I do the same operation in the same vm process everything works fine.
>>
>> Here is my small code.
>>
>> public static GenericRecord createContentNestedObject() throws Exception {
>> GenericRecord image1 = new GenericData.Record(IMAGE_SCHEMA);
>> image1.put("uri", new Utf8("http://javaone.com/keynote_large.jpg"));
>> image1.put("width", 0);
>> image1.put("height", 0);
>> image1.put("size", 2);
>> image1.put("title", new Utf8("Javaone Keynote"));
>> return image1;
>> }
>>
>> // Helper Method to serialize the object from avro to bytebuffer
>> public static ByteBuffer serialize(GenericRecord content, Schema schema)
>> throws Exception {
>>
>> ByteArrayOutputStream out = new ByteArrayOutputStream();
>> DatumWriter<GenericRecord> writer = new
>> GenericDatumWriter<GenericRecord>(
>> schema);
>> // create Binary Encoder
>> EncoderFactory ef = new EncoderFactory();
>> BinaryEncoder be = ef.binaryEncoder(out, null);
>> writer.write(content, be);
>> be.flush();
>> out.close();
>>
>> return ByteBuffer.wrap(out.toByteArray());
>> }
>>
>> public static void main(String[] args) throws Exception {
>> try {
>> GenericRecord rd = createContentNestedObject();
>> bf = serialize(rd, IMAGE_SCHEMA);
>>
>> } catch (Exception e) {
>> // TODO Auto-generated catch block
>> e.printStackTrace();
>> }
>>
>> // Deserialization.
>> DatumReader<GenericRecord> reader = new
>> GenericDatumReader<GenericRecord>(IMAGE_SCHEMA_1);
>> ByteArrayOutputStream out = new ByteArrayOutputStream();
>> try {
>> out.write(bf.array());
>> } catch (IOException e1) {
>> // TODO Auto-generated catch block
>> e1.printStackTrace();
>> }
>> BinaryDecoder decoder =
>> DecoderFactory.get().binaryDecoder(out.toByteArray(), null);
>> try {
>> GenericRecord result = reader.read(null, decoder);
>> System.out.println("RESULT : " + result.toString());
>> } catch (IOException e) {
>> // TODO Auto-generated catch block
>> e.printStackTrace();
>> }
>>
>> This code works. When I send the bf to my thrift api , I am getting the
>> error,
>> my thrift api is simple which accept ByteBuffer and try to deserialize it.
>>
>> following is the code in the thrift server api
>>
>> DatumReader<GenericRecord> reader = new
>> GenericDatumReader<GenericRecord>(IMAGE_SCHEMA_1);
>> ByteArrayOutputStream out = new ByteArrayOutputStream();
>> try {
>> out.write(bf.array());
>> } catch (IOException e1) {
>> // TODO Auto-generated catch block
>> e1.printStackTrace();
>> }
>> BinaryDecoder decoder =
>> DecoderFactory.get().binaryDecoder(out.toByteArray(), null);
>> try {
>> GenericRecord result = reader.read(null, decoder);
>> System.out.println("RESULT : " + result.toString());
>> } catch (IOException e) {
>> // TODO Auto-generated catch block
>> e.printStackTrace();
>> }
>>
>>
>>
>> What is wrong here.
>>
>>
>> Thanks
>> Aviansh
>>
>>
>>
>