You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Cheolsoo Park (JIRA)" <ji...@apache.org> on 2013/02/01 15:16:12 UTC

[jira] [Commented] (PIG-3015) Rewrite of AvroStorage

    [ https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13568753#comment-13568753 ] 

Cheolsoo Park commented on PIG-3015:
------------------------------------

I think the patch is very close to being committed. Two main obstacles are:
# Tests do not pass with Hadoop-2.0.x (i.e. ant clean test -Dtestcase=TestAvroStorage -Dhadoopversion=23).
# Documentation is missing.

I will give another shot on debugging #1 when I get more time, but any help would be appreciated!
                
> Rewrite of AvroStorage
> ----------------------
>
>                 Key: PIG-3015
>                 URL: https://issues.apache.org/jira/browse/PIG-3015
>             Project: Pig
>          Issue Type: Improvement
>          Components: piggybank
>            Reporter: Joseph Adler
>            Assignee: Joseph Adler
>         Attachments: bad.avro, good.avro, PIG-3015-2.patch, PIG-3015-3.patch, PIG-3015-4.patch, PIG-3015-5.patch, PIG-3015-6.patch, PIG-3015-7.patch, TestInput.java, Test.java
>
>
> The current AvroStorage implementation has a lot of issues: it requires old versions of Avro, it copies data much more than needed, and it's verbose and complicated. (One pet peeve of mine is that old versions of Avro don't support Snappy compression.)
> I rewrote AvroStorage from scratch to fix these issues. In early tests, the new implementation is significantly faster, and the code is a lot simpler. Rewriting AvroStorage also enabled me to implement support for Trevni (as TrevniStorage).
> I'm opening this ticket to facilitate discussion while I figure out the best way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira