Getting access to binary encoded data (C++)

[ellisdp <da...@diegesis.co.uk>]

Having come from the Java world where I can do:

/ByteArrayOutputStream out = new ByteArrayOutputStream();
BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(out, null);
SpecificDatumWriter<lswrapper> writer = new SpecificDatumWriter<>(xx.class);

writer.write(content, encoder);
encoder.flush();
out.close();

return out.toByteArray();/

I am now trying to do the equivalent in C++. The documentation shows:

/std::auto_ptr<avro::OutputStream> out = avro::memoryOutputStream();
avro::EncoderPtr e = avro::binaryEncoder();
e->init(*out);
...
avro::encode(*e, <xx>);/

and I can do this but I don't know how to then get a copy of the encoded
bytes. I would like to get this in a variable of type:

/std::vector<uint8_t>/

so that I can pass this back for use elsewhere in my program, but
avro::OutputStream does not seem to provide any methods to do this.

I have tried instead doing:

/ostringstream os;
auto_ptr<avro::OutputStream> oStream = avro::ostreamOutputStream(os);
/
and then using /os.str()/ to access the data, but I am not sure if this is
right and it does not seem to work.

Seems like it should be simple, but I can't figure it out!




--
View this message in context: http://apache-avro.679487.n3.nabble.com/Getting-access-to-binary-encoded-data-C-tp4028890.html
Sent from the Avro - Users mailing list archive at Nabble.com.

Re: Getting access to binary encoded data (C++)

[ellisdp <da...@diegesis.co.uk>]

venkatnv wrote
> I am facing a similar situation. In case you discovered any working
> solution other than the one you mention, could you please share the same?

No, I did it in the way shown above and this works OK.



--
View this message in context: http://apache-avro.679487.n3.nabble.com/Getting-access-to-binary-encoded-data-C-tp4028890p4030810.html
Sent from the Avro - Users mailing list archive at Nabble.com.

Re: Getting access to binary encoded data (C++)

[ellisdp <da...@diegesis.co.uk>]

OK, I have done the following, based on an example I found on another forum.

/( vector<uint8_t> &bytes ...)

	avro::EncoderPtr encoder = avro::binaryEncoder();

	auto_ptr<avro::OutputStream> out= avro::memoryOutputStream();

	encoder->init(*out);

	avro::encode(*encoder, obj);

	out->flush();

	size_t len = out->byteCount();

	auto_ptr<avro::InputStream> in = avro::memoryInputStream(*out);
	avro::StreamReader* reader = new avro::StreamReader(*in);

	bytes.reserve(len);

	while (reader->hasMore()) {
		uint8_t c = reader->read();
		bytes.push_back(c);
	}/

This works, though seems a little long-winded. Is this OK or is there a
simpler way?

Also, I had an issue with the buffer size used by "memoryOutputStream". This
allocates a default buffer size of 4096 bytes (as documented), and I found
that the length returned by "byteCount", and the number of bytes returned
from the stream reader, were always 4096, even though the size of the object
actually encoded was much smaller (150 bytes).

For our application this is not acceptable since we only want to use the
number of bytes actually required to encode the object. For now I have
changed the code to:

/	auto_ptr<avro::OutputStream> oStream = avro::memoryOutputStream(1);/

so that the memory used by the output stream grows one byte at a time, so
uses no more than the size actually needed to encode the object. This works,
though again seems a little clumsy - is there another way to find out the
size of the actual binary encoded data, as opposed to the buffer size?

Thanks.



--
View this message in context: http://apache-avro.679487.n3.nabble.com/Getting-access-to-binary-encoded-data-C-tp4028890p4028952.html
Sent from the Avro - Users mailing list archive at Nabble.com.