computing the size of an avro encoded message

[Andrew Marlow <ma...@gmail.com>]

Hello everyone,

I have just run into the problem discussed on SO at
https://stackoverflow.com/questions/43738118/size-of-encoded-avro-message-without-encoding-it

I am using avro from C++ and currently have a payload buffer of fixed size
into which I do the encode. I found that for some messages I can get an
exception thrown with the error "EOF found". I worked around the problem by
increasing the buffer size, experimenting with binary chop until I found
something that worked without being too wasteful. But this is tiresome.
There doesn't seem to be a way to find out how many bytes are required
except by trying, catching the exception, and then retrying with a larger
buffer. What can be done please?

I note that in good ol' sprintf one can supply the null pointer for the
target buffer and it will not try to write into it but will return the
number of bytes it would have written. This allows one to allocate a buffer
of exactly the right size. Is there any chance of adding such a facility to
the C++ avro API please?
-- 
Regards,

Andrew Marlow
http://www.andrewpetermarlow.co.uk

Re: computing the size of an avro encoded message

[Martin Grigorov <mg...@apache.org>]

Hi Andrew,

On Thu, May 18, 2023 at 12:12 PM Andrew Marlow <ma...@gmail.com>
wrote:

> Hello everyone,
>
> I have just run into the problem discussed on SO at
>
> https://stackoverflow.com/questions/43738118/size-of-encoded-avro-message-without-encoding-it
>
> I am using avro from C++ and currently have a payload buffer of fixed size
> into which I do the encode. I found that for some messages I can get an
> exception thrown with the error "EOF found". I worked around the problem by
> increasing the buffer size, experimenting with binary chop until I found
> something that worked without being too wasteful. But this is tiresome.
> There doesn't seem to be a way to find out how many bytes are required
> except by trying, catching the exception, and then retrying with a larger
> buffer. What can be done please?
>
> I note that in good ol' sprintf one can supply the null pointer for the
> target buffer and it will not try to write into it but will return the
> number of bytes it would have written. This allows one to allocate a buffer
> of exactly the right size. Is there any chance of adding such a facility to
> the C++ avro API please?
>

I think it is fine to add this kind of functionality to the C++ APIs!
Combined with the "exception catching" one will be able to expand the
buffer with a better precision!



> --
> Regards,
>
> Andrew Marlow
> http://www.andrewpetermarlow.co.uk
>