You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@avro.apache.org by JohnGa <pk...@kubikconcepts.com> on 2020/03/18 05:19:54 UTC

c++ Avro: How to append data to existing file?

Hello,
When I create a new instance of avro::DataFileWriter on a filename, it
always resets the file.  I modified the avro::DataFileWriter instantiation
code to open the file in append mode if the file exists, and also skip
writing the header.  Then I call the write method on the DataFileWriter. 
However, I am unable to read this file.  It raises an exception about the
magic being invalid.  

Can someone please give me a hint on how to modify the C++ avro, to be able
to append data to an existing.

Thanls.



--
Sent from: http://apache-avro.679487.n3.nabble.com/Avro-Users-f679479.html

Re: c++ Avro: How to append data to existing file?

Posted by JohnGa <pk...@kubikconcepts.com>.

Thanks Ahn Le.  I will try what you suggested.



--
Sent from: http://apache-avro.679487.n3.nabble.com/Avro-Users-f679479.html

Re: c++ Avro: How to append data to existing file?

Posted by Andy Le <an...@gmail.com>.

Hi John,

Would you please provide more details about your use cases?

My first guess is: you want to incrementally append to a file?

If yes, I suggest:

- To write data: just use normal file I/O, append your data at the end of the DATA file. You should write the offset & length into an INDEX file
- To read data: read the INDEX file, seek to the offsets and read from the DATA file

Hope this help

On 2020/03/18 05:19:54, JohnGa <pk...@kubikconcepts.com> wrote: 
> Hello,
> When I create a new instance of avro::DataFileWriter on a filename, it
> always resets the file.  I modified the avro::DataFileWriter instantiation
> code to open the file in append mode if the file exists, and also skip
> writing the header.  Then I call the write method on the DataFileWriter. 
> However, I am unable to read this file.  It raises an exception about the
> magic being invalid.  
> 
> Can someone please give me a hint on how to modify the C++ avro, to be able
> to append data to an existing.
> 
> Thanls.
> 
> 
> 
> --
> Sent from: http://apache-avro.679487.n3.nabble.com/Avro-Users-f679479.html
>