You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Vyacheslav Zholudev (Created) (JIRA)" <ji...@apache.org> on 2012/02/22 16:35:49 UTC

[jira] [Created] (AVRO-1035) Add the possibility to append to existing avro files

Add the possibility to append to existing avro files  
------------------------------------------------------

                 Key: AVRO-1035
                 URL: https://issues.apache.org/jira/browse/AVRO-1035
             Project: Avro
          Issue Type: New Feature
            Reporter: Vyacheslav Zholudev


Currently it is not possible to append to avro files that were written and closed. 

Here is a Scott Carey's reply on the mailing list:
{quote}
It is not possible without modifying DataFileWriter. Please open a JIRA
ticket.  

It could not simply append to an OutputStream, since it must either:
* Seek to the start to validate the schemas match and find the sync
marker, or
* Trust that the schemas match and find the sync marker from the last block

DataFileWriter cannot refer to Hadoop classes such as FileSystem, but we
could add something to the mapred module that takes a Path and FileSystem
and returns
something that implemements an interface that DataFileWriter can append
to.  This would be something that is both a
http://avro.apache.org/docs/1.6.2/api/java/org/apache/avro/file/SeekableInp
ut.html
and an OutputStream, or has both an InputStream from the start of the
existing file and an OutputStream at the end.
{quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (AVRO-1035) Add the possibility to append to existing avro files

Posted by "Doug Cutting (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/AVRO-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213874#comment-13213874 ] 

Doug Cutting commented on AVRO-1035:
------------------------------------

Note that append is not reliable in current Hadoop releases.  "Append support" in Hadoop 1.0 just means that flush() works reliably, not that append actually works.  Append should be reliable in 0.23 releases although I doubt it's been well tested there yet.
                
> Add the possibility to append to existing avro files  
> ------------------------------------------------------
>
>                 Key: AVRO-1035
>                 URL: https://issues.apache.org/jira/browse/AVRO-1035
>             Project: Avro
>          Issue Type: New Feature
>            Reporter: Vyacheslav Zholudev
>
> Currently it is not possible to append to avro files that were written and closed. 
> Here is a Scott Carey's reply on the mailing list:
> {quote}
> It is not possible without modifying DataFileWriter. Please open a JIRA
> ticket.  
> It could not simply append to an OutputStream, since it must either:
> * Seek to the start to validate the schemas match and find the sync
> marker, or
> * Trust that the schemas match and find the sync marker from the last block
> DataFileWriter cannot refer to Hadoop classes such as FileSystem, but we
> could add something to the mapred module that takes a Path and FileSystem
> and returns
> something that implemements an interface that DataFileWriter can append
> to.  This would be something that is both a
> http://avro.apache.org/docs/1.6.2/api/java/org/apache/avro/file/SeekableInp
> ut.html
> and an OutputStream, or has both an InputStream from the start of the
> existing file and an OutputStream at the end.
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira