You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Gaurav Nanda <ga...@gmail.com> on 2012/05/10 19:19:48 UTC

How to extract byte array from memoryOutputStream

I want to extract byte array from memoryOutputStream to transfer the
encoded data. How do I achieve that?

Re: How to extract byte array from memoryOutputStream

Posted by Gaurav Nanda <ga...@gmail.com>.
Got it Matt. Thanks !

On Thu, May 17, 2012 at 9:59 PM, Matt Stevenson
<ma...@gmail.com>wrote:

> But what is the suggested way of transferring encoded bytes over the
>> wire. Avro does not seem to expose the encoded bytes anywhere.
>>
>
> I am using a custom method for transporting the encoding bytes, which is
> why I had to get at the bytes.
> Avro has an HTTP transport mechanisms defined, but I haven't used it and I
> don't know if it is fully implemented in the C++ project.
> For typical Avro usage you should not be accessing the raw bytes, which is
> why nobody bothered to expose them.
>
> So even if I am encoding a small dataset, I would end up
>> reading a min "Chunk_size" of memory block.
>>
>
> The "count" variable should be the exact number of bytes in the stream.
> Both for loops are checking "i<count", and "i" is only incremented in the
> inner loop.  Once you reach the count it will break out of both loops.
>
>
> On Wed, May 16, 2012 at 4:45 AM, Gaurav Nanda <ga...@gmail.com> wrote:
>
>> Hi Matt, The approach you are using to read the written bytes would
>> end up reading all the bytes allocated but not the bytes actually
>> written. So even if I am encoding a small dataset, I would end up
>> reading a min "Chunk_size" of memory block.
>>
>> On Mon, May 14, 2012 at 9:03 PM, Gaurav Nanda <ga...@gmail.com>
>> wrote:
>> > Thanks Matt !
>> >
>> > But what is the suggested way of transferring encoded bytes over the
>> > wire. Avro does not seem to expose the encoded bytes anywhere.
>> >
>> > - Gaurav Nanda
>> >
>> > On Sat, May 12, 2012 at 6:29 AM, Matt Stevenson
>> > <ma...@gmail.com> wrote:
>> >> I made a copy of the MemoryOutputStream with a header, so you can get
>> at the
>> >> bytes.
>> >>
>> >> http://branchingworlds.com/avro/OpenMemoryOutputStream.h
>> >> http://branchingworlds.com/avro/OpenMemoryOutputStream.cpp
>> >>
>> >> Add those to your project.
>> >> include "OpenMemoryOutputStream.h" and call openMemoryOutputStream()
>> rather
>> >> than memoryOutputStream().
>> >>
>> >> The memory is stored in chunks.  Here's an example of iterating
>> through it:
>> >>
>> >>> auto_ptr<OpenMemoryOutputStream> os = openMemoryOutputStream();
>> >>> EncoderPtr e = binaryEncoder();
>> >>> e->init(*os);
>> >>>
>> >>> avro::encode(*e, *t);
>> >>> e->flush();
>> >>>
>> >>> int count = os->byteCount();
>> >>> char* data = new char[count];
>> >>> int i=0;
>> >>> for (std::vector<uint8_t*>::const_iterator it = os->data_.begin(); it
>> !=
>> >>> os->data_.end() && i<count; ++it) {
>> >>>     uint8_t* chunk = *it;
>> >>>     int size = os->chunkSize_;
>> >>>     for(int j=0; j<size && i<count; j++, i++){
>> >>>         data[i] = chunk[j];
>> >>>     }
>> >>> }
>> >>
>> >>
>> >> On Thu, May 10, 2012 at 12:19 PM, Gaurav Nanda <ga...@gmail.com>
>> wrote:
>> >>>
>> >>> I want to extract byte array from memoryOutputStream to transfer the
>> >>> encoded data. How do I achieve that?
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Matt Stevenson.
>>
>
>
>
> --
> Matt Stevenson.
>

Re: How to extract byte array from memoryOutputStream

Posted by Matt Stevenson <ma...@gmail.com>.
>
> But what is the suggested way of transferring encoded bytes over the
> wire. Avro does not seem to expose the encoded bytes anywhere.
>

I am using a custom method for transporting the encoding bytes, which is
why I had to get at the bytes.
Avro has an HTTP transport mechanisms defined, but I haven't used it and I
don't know if it is fully implemented in the C++ project.
For typical Avro usage you should not be accessing the raw bytes, which is
why nobody bothered to expose them.

So even if I am encoding a small dataset, I would end up
> reading a min "Chunk_size" of memory block.
>

The "count" variable should be the exact number of bytes in the stream.
Both for loops are checking "i<count", and "i" is only incremented in the
inner loop.  Once you reach the count it will break out of both loops.

On Wed, May 16, 2012 at 4:45 AM, Gaurav Nanda <ga...@gmail.com> wrote:

> Hi Matt, The approach you are using to read the written bytes would
> end up reading all the bytes allocated but not the bytes actually
> written. So even if I am encoding a small dataset, I would end up
> reading a min "Chunk_size" of memory block.
>
> On Mon, May 14, 2012 at 9:03 PM, Gaurav Nanda <ga...@gmail.com> wrote:
> > Thanks Matt !
> >
> > But what is the suggested way of transferring encoded bytes over the
> > wire. Avro does not seem to expose the encoded bytes anywhere.
> >
> > - Gaurav Nanda
> >
> > On Sat, May 12, 2012 at 6:29 AM, Matt Stevenson
> > <ma...@gmail.com> wrote:
> >> I made a copy of the MemoryOutputStream with a header, so you can get
> at the
> >> bytes.
> >>
> >> http://branchingworlds.com/avro/OpenMemoryOutputStream.h
> >> http://branchingworlds.com/avro/OpenMemoryOutputStream.cpp
> >>
> >> Add those to your project.
> >> include "OpenMemoryOutputStream.h" and call openMemoryOutputStream()
> rather
> >> than memoryOutputStream().
> >>
> >> The memory is stored in chunks.  Here's an example of iterating through
> it:
> >>
> >>> auto_ptr<OpenMemoryOutputStream> os = openMemoryOutputStream();
> >>> EncoderPtr e = binaryEncoder();
> >>> e->init(*os);
> >>>
> >>> avro::encode(*e, *t);
> >>> e->flush();
> >>>
> >>> int count = os->byteCount();
> >>> char* data = new char[count];
> >>> int i=0;
> >>> for (std::vector<uint8_t*>::const_iterator it = os->data_.begin(); it
> !=
> >>> os->data_.end() && i<count; ++it) {
> >>>     uint8_t* chunk = *it;
> >>>     int size = os->chunkSize_;
> >>>     for(int j=0; j<size && i<count; j++, i++){
> >>>         data[i] = chunk[j];
> >>>     }
> >>> }
> >>
> >>
> >> On Thu, May 10, 2012 at 12:19 PM, Gaurav Nanda <ga...@gmail.com>
> wrote:
> >>>
> >>> I want to extract byte array from memoryOutputStream to transfer the
> >>> encoded data. How do I achieve that?
> >>
> >>
> >>
> >>
> >> --
> >> Matt Stevenson.
>



-- 
Matt Stevenson.

Re: How to extract byte array from memoryOutputStream

Posted by Gaurav Nanda <ga...@gmail.com>.
Hi Matt, The approach you are using to read the written bytes would
end up reading all the bytes allocated but not the bytes actually
written. So even if I am encoding a small dataset, I would end up
reading a min "Chunk_size" of memory block.

On Mon, May 14, 2012 at 9:03 PM, Gaurav Nanda <ga...@gmail.com> wrote:
> Thanks Matt !
>
> But what is the suggested way of transferring encoded bytes over the
> wire. Avro does not seem to expose the encoded bytes anywhere.
>
> - Gaurav Nanda
>
> On Sat, May 12, 2012 at 6:29 AM, Matt Stevenson
> <ma...@gmail.com> wrote:
>> I made a copy of the MemoryOutputStream with a header, so you can get at the
>> bytes.
>>
>> http://branchingworlds.com/avro/OpenMemoryOutputStream.h
>> http://branchingworlds.com/avro/OpenMemoryOutputStream.cpp
>>
>> Add those to your project.
>> include "OpenMemoryOutputStream.h" and call openMemoryOutputStream() rather
>> than memoryOutputStream().
>>
>> The memory is stored in chunks.  Here's an example of iterating through it:
>>
>>> auto_ptr<OpenMemoryOutputStream> os = openMemoryOutputStream();
>>> EncoderPtr e = binaryEncoder();
>>> e->init(*os);
>>>
>>> avro::encode(*e, *t);
>>> e->flush();
>>>
>>> int count = os->byteCount();
>>> char* data = new char[count];
>>> int i=0;
>>> for (std::vector<uint8_t*>::const_iterator it = os->data_.begin(); it !=
>>> os->data_.end() && i<count; ++it) {
>>>     uint8_t* chunk = *it;
>>>     int size = os->chunkSize_;
>>>     for(int j=0; j<size && i<count; j++, i++){
>>>         data[i] = chunk[j];
>>>     }
>>> }
>>
>>
>> On Thu, May 10, 2012 at 12:19 PM, Gaurav Nanda <ga...@gmail.com> wrote:
>>>
>>> I want to extract byte array from memoryOutputStream to transfer the
>>> encoded data. How do I achieve that?
>>
>>
>>
>>
>> --
>> Matt Stevenson.

Re: How to extract byte array from memoryOutputStream

Posted by Gaurav Nanda <ga...@gmail.com>.
Thanks Matt !

But what is the suggested way of transferring encoded bytes over the
wire. Avro does not seem to expose the encoded bytes anywhere.

- Gaurav Nanda

On Sat, May 12, 2012 at 6:29 AM, Matt Stevenson
<ma...@gmail.com> wrote:
> I made a copy of the MemoryOutputStream with a header, so you can get at the
> bytes.
>
> http://branchingworlds.com/avro/OpenMemoryOutputStream.h
> http://branchingworlds.com/avro/OpenMemoryOutputStream.cpp
>
> Add those to your project.
> include "OpenMemoryOutputStream.h" and call openMemoryOutputStream() rather
> than memoryOutputStream().
>
> The memory is stored in chunks.  Here's an example of iterating through it:
>
>> auto_ptr<OpenMemoryOutputStream> os = openMemoryOutputStream();
>> EncoderPtr e = binaryEncoder();
>> e->init(*os);
>>
>> avro::encode(*e, *t);
>> e->flush();
>>
>> int count = os->byteCount();
>> char* data = new char[count];
>> int i=0;
>> for (std::vector<uint8_t*>::const_iterator it = os->data_.begin(); it !=
>> os->data_.end() && i<count; ++it) {
>>     uint8_t* chunk = *it;
>>     int size = os->chunkSize_;
>>     for(int j=0; j<size && i<count; j++, i++){
>>         data[i] = chunk[j];
>>     }
>> }
>
>
> On Thu, May 10, 2012 at 12:19 PM, Gaurav Nanda <ga...@gmail.com> wrote:
>>
>> I want to extract byte array from memoryOutputStream to transfer the
>> encoded data. How do I achieve that?
>
>
>
>
> --
> Matt Stevenson.

Re: How to extract byte array from memoryOutputStream

Posted by Matt Stevenson <ma...@gmail.com>.
I made a copy of the MemoryOutputStream with a header, so you can get at
the bytes.

http://branchingworlds.com/avro/OpenMemoryOutputStream.h
http://branchingworlds.com/avro/OpenMemoryOutputStream.cpp

Add those to your project.
include "OpenMemoryOutputStream.h" and call openMemoryOutputStream() rather
than memoryOutputStream().

The memory is stored in chunks.  Here's an example of iterating through it:

auto_ptr<OpenMemoryOutputStream> os = openMemoryOutputStream();
> EncoderPtr e = binaryEncoder();
> e->init(*os);
>
> avro::encode(*e, *t);
> e->flush();
>
> int count = os->byteCount();
> char* data = new char[count];
> int i=0;
> for (std::vector<uint8_t*>::const_iterator it = os->data_.begin(); it !=
> os->data_.end() && i<count; ++it) {
>     uint8_t* chunk = *it;
>     int size = os->chunkSize_;
>     for(int j=0; j<size && i<count; j++, i++){
>         data[i] = chunk[j];
>     }
> }
>

On Thu, May 10, 2012 at 12:19 PM, Gaurav Nanda <ga...@gmail.com> wrote:

> I want to extract byte array from memoryOutputStream to transfer the
> encoded data. How do I achieve that?
>



-- 
Matt Stevenson.