You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Scott Carey (Issue Comment Edited) (JIRA)" <ji...@apache.org> on 2012/01/17 01:35:38 UTC
[jira] [Issue Comment Edited] (AVRO-991) Allow combining multiple
Avro files within a stream. (no files on disk)
[ https://issues.apache.org/jira/browse/AVRO-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187338#comment-13187338 ]
Scott Carey edited comment on AVRO-991 at 1/17/12 12:34 AM:
------------------------------------------------------------
To help out Tom's checksum issue, we could support user-provided sync markers. However, two files with the same logical content can differ in other ways too: User provided metadata, different block sizes, compression codecs, etc.
That brings up another tool -- we could provide a tool that does checksums on the binary content of an avro data file. and ignores the sync markers.
was (Author: scott_carey):
TO help out Tom's md5 issue, we could support user-provided sync markers. However, two files with the same logical content can differ in other ways too: User provided metadata, different block sizes, compression codecs, etc.
That brings up another tool -- we could provide a tool that does checksums on the binary content of an avro data file. and ignores the sync markers.
> Allow combining multiple Avro files within a stream. (no files on disk)
> -----------------------------------------------------------------------
>
> Key: AVRO-991
> URL: https://issues.apache.org/jira/browse/AVRO-991
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.6.1
> Reporter: Frank Grimes
>
> It would be nice to be able to do as follows:
> cat file1.avro file2.avro | java -jar avro-tools.jar streamcombine > combined-file.avro
> or similarly
>
> hadoop dfs -cat hdfs://hadoop/file1.avro hdfs://hadoop/file2.avro | java -jar avro-tools.jar streamcombine | hdfs -put - hdfs://hadoop/combined-file.avro
> See the following thread for details: http://mail-archives.apache.org/mod_mbox/avro-user/201201.mbox/%3cC08F1DE9-97A8-4D28-B0AD-5E4A7F32F028@gmail.com%3e
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira