You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Mohit Anchlia <mo...@gmail.com> on 2012/05/01 02:25:22 UTC
Compressing map only output
Is there a way to compress map only jobs to compress map output that gets
stored on hdfs as part-m-* files? In pig I used :
Would these work form plain map reduce jobs as well?
set output.compression.enabled true;
set output.compression.codec org.apache.hadoop.io.compress.SnappyCodec;
Re: Compressing map only output
Posted by Mohit Anchlia <mo...@gmail.com>.
Thanks a lot for the link!
On Mon, Apr 30, 2012 at 8:22 PM, Harsh J <ha...@cloudera.com> wrote:
> Hey Mohit,
>
> Most of what you need to know for jobs is available at
> http://hadoop.apache.org/common/docs/current/mapred_tutorial.html
>
> A more complete, mostly unseparated list of config params are also
> available at:
> http://hadoop.apache.org/common/docs/current/mapred-default.html
> (core-default.html, hdfs-default.html)
>
> On Tue, May 1, 2012 at 6:36 AM, Mohit Anchlia <mo...@gmail.com>
> wrote:
> > Thanks! When I tried to search for this property I couldn't find it. Is
> > there a page that has complete list of properties and it's usage?
> >
> > On Mon, Apr 30, 2012 at 5:44 PM, Prashant Kommireddi <
> prash1784@gmail.com>wrote:
> >
> >> Yes. These are hadoop properties - using "set" is just a way for Pig to
> set
> >> those properties in your job conf.
> >>
> >>
> >> On Mon, Apr 30, 2012 at 5:25 PM, Mohit Anchlia <mohitanchlia@gmail.com
> >> >wrote:
> >>
> >> > Is there a way to compress map only jobs to compress map output that
> gets
> >> > stored on hdfs as part-m-* files? In pig I used :
> >> >
> >> > Would these work form plain map reduce jobs as well?
> >> >
> >> >
> >> > set output.compression.enabled true;
> >> >
> >> > set output.compression.codec
> org.apache.hadoop.io.compress.SnappyCodec;
> >> >
> >>
>
>
>
> --
> Harsh J
>
Re: Compressing map only output
Posted by Harsh J <ha...@cloudera.com>.
Hey Mohit,
Most of what you need to know for jobs is available at
http://hadoop.apache.org/common/docs/current/mapred_tutorial.html
A more complete, mostly unseparated list of config params are also
available at: http://hadoop.apache.org/common/docs/current/mapred-default.html
(core-default.html, hdfs-default.html)
On Tue, May 1, 2012 at 6:36 AM, Mohit Anchlia <mo...@gmail.com> wrote:
> Thanks! When I tried to search for this property I couldn't find it. Is
> there a page that has complete list of properties and it's usage?
>
> On Mon, Apr 30, 2012 at 5:44 PM, Prashant Kommireddi <pr...@gmail.com>wrote:
>
>> Yes. These are hadoop properties - using "set" is just a way for Pig to set
>> those properties in your job conf.
>>
>>
>> On Mon, Apr 30, 2012 at 5:25 PM, Mohit Anchlia <mohitanchlia@gmail.com
>> >wrote:
>>
>> > Is there a way to compress map only jobs to compress map output that gets
>> > stored on hdfs as part-m-* files? In pig I used :
>> >
>> > Would these work form plain map reduce jobs as well?
>> >
>> >
>> > set output.compression.enabled true;
>> >
>> > set output.compression.codec org.apache.hadoop.io.compress.SnappyCodec;
>> >
>>
--
Harsh J
Re: Compressing map only output
Posted by Mohit Anchlia <mo...@gmail.com>.
Thanks! When I tried to search for this property I couldn't find it. Is
there a page that has complete list of properties and it's usage?
On Mon, Apr 30, 2012 at 5:44 PM, Prashant Kommireddi <pr...@gmail.com>wrote:
> Yes. These are hadoop properties - using "set" is just a way for Pig to set
> those properties in your job conf.
>
>
> On Mon, Apr 30, 2012 at 5:25 PM, Mohit Anchlia <mohitanchlia@gmail.com
> >wrote:
>
> > Is there a way to compress map only jobs to compress map output that gets
> > stored on hdfs as part-m-* files? In pig I used :
> >
> > Would these work form plain map reduce jobs as well?
> >
> >
> > set output.compression.enabled true;
> >
> > set output.compression.codec org.apache.hadoop.io.compress.SnappyCodec;
> >
>
Re: Compressing map only output
Posted by Prashant Kommireddi <pr...@gmail.com>.
Yes. These are hadoop properties - using "set" is just a way for Pig to set
those properties in your job conf.
On Mon, Apr 30, 2012 at 5:25 PM, Mohit Anchlia <mo...@gmail.com>wrote:
> Is there a way to compress map only jobs to compress map output that gets
> stored on hdfs as part-m-* files? In pig I used :
>
> Would these work form plain map reduce jobs as well?
>
>
> set output.compression.enabled true;
>
> set output.compression.codec org.apache.hadoop.io.compress.SnappyCodec;
>