You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by James Kebinger <jk...@gmail.com> on 2012/07/30 20:21:16 UTC
Unable to disable compression of output
Hello, I'm running a pretty simple pig job but despite my best efforts to
disable compression, the output parts are written in in .lzo_deflate form
like
"/user/jkebinger/pigresults3.gz/part-r-00000.lzo_deflate"
I'm launching the job like this JAVA_OPTS=-Dmapred.output.compress=false
PIG_OPTS=-Dmapred.output.compress=false ../pig-0.10.0/bin/pig test.pig
in a vain effort to turn that off.
Can someone please point me in the right direction?
The store statement looks like this:
STORE bothcountedp into '/user/jkebinger/pigresults3.gz' USING
PigStorage();
(the filename there has varied from no extension to .gz with no effect on
how the files inside it are treated)
thanks!
-James
Re: Unable to disable compression of output
Posted by James Kebinger <jk...@gmail.com>.
Turns out I needed to set the option inside my pig script, as in:
SET mapred.output.compress false;
On Mon, Jul 30, 2012 at 2:21 PM, James Kebinger <jk...@gmail.com> wrote:
> Hello, I'm running a pretty simple pig job but despite my best efforts to
> disable compression, the output parts are written in in .lzo_deflate form
> like
>
> "/user/jkebinger/pigresults3.gz/part-r-00000.lzo_deflate"
>
> I'm launching the job like this JAVA_OPTS=-Dmapred.output.compress=false
> PIG_OPTS=-Dmapred.output.compress=false ../pig-0.10.0/bin/pig test.pig
>
> in a vain effort to turn that off.
>
> Can someone please point me in the right direction?
>
>
> The store statement looks like this:
>
> STORE bothcountedp into '/user/jkebinger/pigresults3.gz' USING
> PigStorage();
>
> (the filename there has varied from no extension to .gz with no effect on
> how the files inside it are treated)
>
> thanks!
>
> -James
>
Re: Unable to disable compression of output
Posted by James Kebinger <jk...@gmail.com>.
Yes, whatever I name the output directory, the files inside are compressed
with the lzo codec.
On Mon, Jul 30, 2012 at 2:52 PM, souri datta <so...@gmail.com>wrote:
> James,
> I may not have understood your question fully,but did you try renaming the
> file w/o the '.gz' ? In hadoop/pig, if you have bz2 extn, files get
> compressed.Hence,..
>
> On Mon, Jul 30, 2012 at 11:51 PM, James Kebinger <jkebinger@gmail.com
> >wrote:
>
> > Hello, I'm running a pretty simple pig job but despite my best efforts to
> > disable compression, the output parts are written in in .lzo_deflate form
> > like
> >
> > "/user/jkebinger/pigresults3.gz/part-r-00000.lzo_deflate"
> >
> > I'm launching the job like this JAVA_OPTS=-Dmapred.output.compress=false
> > PIG_OPTS=-Dmapred.output.compress=false ../pig-0.10.0/bin/pig test.pig
> >
> > in a vain effort to turn that off.
> >
> > Can someone please point me in the right direction?
> >
> >
> > The store statement looks like this:
> >
> > STORE bothcountedp into '/user/jkebinger/pigresults3.gz' USING
> > PigStorage();
> >
> > (the filename there has varied from no extension to .gz with no effect on
> > how the files inside it are treated)
> >
> > thanks!
> >
> > -James
> >
>
Re: Unable to disable compression of output
Posted by souri datta <so...@gmail.com>.
James,
I may not have understood your question fully,but did you try renaming the
file w/o the '.gz' ? In hadoop/pig, if you have bz2 extn, files get
compressed.Hence,..
On Mon, Jul 30, 2012 at 11:51 PM, James Kebinger <jk...@gmail.com>wrote:
> Hello, I'm running a pretty simple pig job but despite my best efforts to
> disable compression, the output parts are written in in .lzo_deflate form
> like
>
> "/user/jkebinger/pigresults3.gz/part-r-00000.lzo_deflate"
>
> I'm launching the job like this JAVA_OPTS=-Dmapred.output.compress=false
> PIG_OPTS=-Dmapred.output.compress=false ../pig-0.10.0/bin/pig test.pig
>
> in a vain effort to turn that off.
>
> Can someone please point me in the right direction?
>
>
> The store statement looks like this:
>
> STORE bothcountedp into '/user/jkebinger/pigresults3.gz' USING
> PigStorage();
>
> (the filename there has varied from no extension to .gz with no effect on
> how the files inside it are treated)
>
> thanks!
>
> -James
>