You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@pig.apache.org by Prashant Kommireddi <pr...@gmail.com> on 2011/11/18 06:02:57 UTC

Does Pig support Snappy compression?

I understand Pig supports only Gzip and Bzip compression algorithms. Would
it be fine if I set map output compression (between map and reduce) to
SnappyCodec? I am guessing this should not be a problem.

Also, is there a plan to add support for SnappyCodec (Load snappy
compressed files)?

Thanks,
Prashant

Re: Does Pig support Snappy compression?

Posted by Prashant Kommireddi <pr...@gmail.com>.

I tested it and map output compression works just fine. Thanks Alan.

Quick follow-up question. I am trying to store some files ( which are
currently gzip compressed ) as SequenceFiles (BLOCK compressed using Gzip).
Can you please point me to any documentation that I could use to read/write
such sequence files with Pig? SequenceFileLoader seems to be an option but
I could not find any documentation around it.

-Prashant

On Fri, Nov 18, 2011 at 8:29 AM, Alan Gates <ga...@hortonworks.com> wrote:

>
> On Nov 17, 2011, at 9:02 PM, Prashant Kommireddi wrote:
>
> > I understand Pig supports only Gzip and Bzip compression algorithms.
> Would
> > it be fine if I set map output compression (between map and reduce) to
> > SnappyCodec? I am guessing this should not be a problem.
>
> AFAIK it would be fine, but we haven't tested it.
>
> >
> > Also, is there a plan to add support for SnappyCodec (Load snappy
> > compressed files)?
>
> We would like to do that, but haven't gotten to it yet.  You can watch
> https://issues.apache.org/jira/browse/PIG-2319 for progress.
>
> Alan.
>
> >
> > Thanks,
> > Prashant
>
>

Re: Does Pig support Snappy compression?

Posted by Alan Gates <ga...@hortonworks.com>.

On Nov 17, 2011, at 9:02 PM, Prashant Kommireddi wrote:

> I understand Pig supports only Gzip and Bzip compression algorithms. Would
> it be fine if I set map output compression (between map and reduce) to
> SnappyCodec? I am guessing this should not be a problem.

AFAIK it would be fine, but we haven't tested it.

> 
> Also, is there a plan to add support for SnappyCodec (Load snappy
> compressed files)?

We would like to do that, but haven't gotten to it yet.  You can watch https://issues.apache.org/jira/browse/PIG-2319 for progress.

Alan.

> 
> Thanks,
> Prashant