You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Frank Kootte <fr...@gmail.com> on 2012/09/28 13:37:37 UTC
MultipleOutputs side effects
While using MultipleOutputs with CHD3 0.20.2+923.418-1 I get besides the
desired output files also a empty snappy file per reducer for free.
Anybody familiar with this symptom and perhaps knows how to fix it ?
--
Mvrgr. Frank
Re: MultipleOutputs side effects
Posted by Frank Kootte <fr...@gmail.com>.
Thanks guy's - works like a charm !
2012/9/28 Harsh J <ha...@cloudera.com>
> Yes - for new API MultipleOutput, use LazyOutputFormat as job's output
> format, and for old API, use the NullOutputFormat as the jobconf's
> output format.
>
> On Fri, Sep 28, 2012 at 5:14 PM, Hemanth Yamijala
> <yh...@thoughtworks.com> wrote:
> > This came up recently on the forums, IIRC. The answer was to use to
> > something called LazyOutputFormat to suppress the output file ?
> >
> > I think this thread has the discussion:
> >
> >
> http://search-hadoop.com/m/66Thjsj97v1/hadoop+1.0.3+equivalent+of+MultipleTextOutputFormat&subj=RE+hadoop+1+0+3+equivalent+of+MultipleTextOutputFormat
> >
> > Thanks
> > hemanth
> >
> >
> > On Fri, Sep 28, 2012 at 5:07 PM, Frank Kootte <fr...@gmail.com>
> wrote:
> >>
> >> While using MultipleOutputs with CHD3 0.20.2+923.418-1 I get besides the
> >> desired output files also a empty snappy file per reducer for free.
> >> Anybody familiar with this symptom and perhaps knows how to fix it ?
> >>
> >> --
> >> Mvrgr. Frank
> >
> >
>
>
>
> --
> Harsh J
>
--
Mvrgr. Frank
Re: MultipleOutputs side effects
Posted by Frank Kootte <fr...@gmail.com>.
Thanks guy's - works like a charm !
2012/9/28 Harsh J <ha...@cloudera.com>
> Yes - for new API MultipleOutput, use LazyOutputFormat as job's output
> format, and for old API, use the NullOutputFormat as the jobconf's
> output format.
>
> On Fri, Sep 28, 2012 at 5:14 PM, Hemanth Yamijala
> <yh...@thoughtworks.com> wrote:
> > This came up recently on the forums, IIRC. The answer was to use to
> > something called LazyOutputFormat to suppress the output file ?
> >
> > I think this thread has the discussion:
> >
> >
> http://search-hadoop.com/m/66Thjsj97v1/hadoop+1.0.3+equivalent+of+MultipleTextOutputFormat&subj=RE+hadoop+1+0+3+equivalent+of+MultipleTextOutputFormat
> >
> > Thanks
> > hemanth
> >
> >
> > On Fri, Sep 28, 2012 at 5:07 PM, Frank Kootte <fr...@gmail.com>
> wrote:
> >>
> >> While using MultipleOutputs with CHD3 0.20.2+923.418-1 I get besides the
> >> desired output files also a empty snappy file per reducer for free.
> >> Anybody familiar with this symptom and perhaps knows how to fix it ?
> >>
> >> --
> >> Mvrgr. Frank
> >
> >
>
>
>
> --
> Harsh J
>
--
Mvrgr. Frank
Re: MultipleOutputs side effects
Posted by Frank Kootte <fr...@gmail.com>.
Thanks guy's - works like a charm !
2012/9/28 Harsh J <ha...@cloudera.com>
> Yes - for new API MultipleOutput, use LazyOutputFormat as job's output
> format, and for old API, use the NullOutputFormat as the jobconf's
> output format.
>
> On Fri, Sep 28, 2012 at 5:14 PM, Hemanth Yamijala
> <yh...@thoughtworks.com> wrote:
> > This came up recently on the forums, IIRC. The answer was to use to
> > something called LazyOutputFormat to suppress the output file ?
> >
> > I think this thread has the discussion:
> >
> >
> http://search-hadoop.com/m/66Thjsj97v1/hadoop+1.0.3+equivalent+of+MultipleTextOutputFormat&subj=RE+hadoop+1+0+3+equivalent+of+MultipleTextOutputFormat
> >
> > Thanks
> > hemanth
> >
> >
> > On Fri, Sep 28, 2012 at 5:07 PM, Frank Kootte <fr...@gmail.com>
> wrote:
> >>
> >> While using MultipleOutputs with CHD3 0.20.2+923.418-1 I get besides the
> >> desired output files also a empty snappy file per reducer for free.
> >> Anybody familiar with this symptom and perhaps knows how to fix it ?
> >>
> >> --
> >> Mvrgr. Frank
> >
> >
>
>
>
> --
> Harsh J
>
--
Mvrgr. Frank
Re: MultipleOutputs side effects
Posted by Frank Kootte <fr...@gmail.com>.
Thanks guy's - works like a charm !
2012/9/28 Harsh J <ha...@cloudera.com>
> Yes - for new API MultipleOutput, use LazyOutputFormat as job's output
> format, and for old API, use the NullOutputFormat as the jobconf's
> output format.
>
> On Fri, Sep 28, 2012 at 5:14 PM, Hemanth Yamijala
> <yh...@thoughtworks.com> wrote:
> > This came up recently on the forums, IIRC. The answer was to use to
> > something called LazyOutputFormat to suppress the output file ?
> >
> > I think this thread has the discussion:
> >
> >
> http://search-hadoop.com/m/66Thjsj97v1/hadoop+1.0.3+equivalent+of+MultipleTextOutputFormat&subj=RE+hadoop+1+0+3+equivalent+of+MultipleTextOutputFormat
> >
> > Thanks
> > hemanth
> >
> >
> > On Fri, Sep 28, 2012 at 5:07 PM, Frank Kootte <fr...@gmail.com>
> wrote:
> >>
> >> While using MultipleOutputs with CHD3 0.20.2+923.418-1 I get besides the
> >> desired output files also a empty snappy file per reducer for free.
> >> Anybody familiar with this symptom and perhaps knows how to fix it ?
> >>
> >> --
> >> Mvrgr. Frank
> >
> >
>
>
>
> --
> Harsh J
>
--
Mvrgr. Frank
Re: MultipleOutputs side effects
Posted by Harsh J <ha...@cloudera.com>.
Yes - for new API MultipleOutput, use LazyOutputFormat as job's output
format, and for old API, use the NullOutputFormat as the jobconf's
output format.
On Fri, Sep 28, 2012 at 5:14 PM, Hemanth Yamijala
<yh...@thoughtworks.com> wrote:
> This came up recently on the forums, IIRC. The answer was to use to
> something called LazyOutputFormat to suppress the output file ?
>
> I think this thread has the discussion:
>
> http://search-hadoop.com/m/66Thjsj97v1/hadoop+1.0.3+equivalent+of+MultipleTextOutputFormat&subj=RE+hadoop+1+0+3+equivalent+of+MultipleTextOutputFormat
>
> Thanks
> hemanth
>
>
> On Fri, Sep 28, 2012 at 5:07 PM, Frank Kootte <fr...@gmail.com> wrote:
>>
>> While using MultipleOutputs with CHD3 0.20.2+923.418-1 I get besides the
>> desired output files also a empty snappy file per reducer for free.
>> Anybody familiar with this symptom and perhaps knows how to fix it ?
>>
>> --
>> Mvrgr. Frank
>
>
--
Harsh J
Re: MultipleOutputs side effects
Posted by Harsh J <ha...@cloudera.com>.
Yes - for new API MultipleOutput, use LazyOutputFormat as job's output
format, and for old API, use the NullOutputFormat as the jobconf's
output format.
On Fri, Sep 28, 2012 at 5:14 PM, Hemanth Yamijala
<yh...@thoughtworks.com> wrote:
> This came up recently on the forums, IIRC. The answer was to use to
> something called LazyOutputFormat to suppress the output file ?
>
> I think this thread has the discussion:
>
> http://search-hadoop.com/m/66Thjsj97v1/hadoop+1.0.3+equivalent+of+MultipleTextOutputFormat&subj=RE+hadoop+1+0+3+equivalent+of+MultipleTextOutputFormat
>
> Thanks
> hemanth
>
>
> On Fri, Sep 28, 2012 at 5:07 PM, Frank Kootte <fr...@gmail.com> wrote:
>>
>> While using MultipleOutputs with CHD3 0.20.2+923.418-1 I get besides the
>> desired output files also a empty snappy file per reducer for free.
>> Anybody familiar with this symptom and perhaps knows how to fix it ?
>>
>> --
>> Mvrgr. Frank
>
>
--
Harsh J
Re: MultipleOutputs side effects
Posted by Harsh J <ha...@cloudera.com>.
Yes - for new API MultipleOutput, use LazyOutputFormat as job's output
format, and for old API, use the NullOutputFormat as the jobconf's
output format.
On Fri, Sep 28, 2012 at 5:14 PM, Hemanth Yamijala
<yh...@thoughtworks.com> wrote:
> This came up recently on the forums, IIRC. The answer was to use to
> something called LazyOutputFormat to suppress the output file ?
>
> I think this thread has the discussion:
>
> http://search-hadoop.com/m/66Thjsj97v1/hadoop+1.0.3+equivalent+of+MultipleTextOutputFormat&subj=RE+hadoop+1+0+3+equivalent+of+MultipleTextOutputFormat
>
> Thanks
> hemanth
>
>
> On Fri, Sep 28, 2012 at 5:07 PM, Frank Kootte <fr...@gmail.com> wrote:
>>
>> While using MultipleOutputs with CHD3 0.20.2+923.418-1 I get besides the
>> desired output files also a empty snappy file per reducer for free.
>> Anybody familiar with this symptom and perhaps knows how to fix it ?
>>
>> --
>> Mvrgr. Frank
>
>
--
Harsh J
Re: MultipleOutputs side effects
Posted by Harsh J <ha...@cloudera.com>.
Yes - for new API MultipleOutput, use LazyOutputFormat as job's output
format, and for old API, use the NullOutputFormat as the jobconf's
output format.
On Fri, Sep 28, 2012 at 5:14 PM, Hemanth Yamijala
<yh...@thoughtworks.com> wrote:
> This came up recently on the forums, IIRC. The answer was to use to
> something called LazyOutputFormat to suppress the output file ?
>
> I think this thread has the discussion:
>
> http://search-hadoop.com/m/66Thjsj97v1/hadoop+1.0.3+equivalent+of+MultipleTextOutputFormat&subj=RE+hadoop+1+0+3+equivalent+of+MultipleTextOutputFormat
>
> Thanks
> hemanth
>
>
> On Fri, Sep 28, 2012 at 5:07 PM, Frank Kootte <fr...@gmail.com> wrote:
>>
>> While using MultipleOutputs with CHD3 0.20.2+923.418-1 I get besides the
>> desired output files also a empty snappy file per reducer for free.
>> Anybody familiar with this symptom and perhaps knows how to fix it ?
>>
>> --
>> Mvrgr. Frank
>
>
--
Harsh J
Re: MultipleOutputs side effects
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
This came up recently on the forums, IIRC. The answer was to use to
something called LazyOutputFormat to suppress the output file ?
I think this thread has the discussion:
http://search-hadoop.com/m/66Thjsj97v1/hadoop+1.0.3+equivalent+of+MultipleTextOutputFormat&subj=RE+hadoop+1+0+3+equivalent+of+MultipleTextOutputFormat
Thanks
hemanth
On Fri, Sep 28, 2012 at 5:07 PM, Frank Kootte <fr...@gmail.com> wrote:
> While using MultipleOutputs with CHD3 0.20.2+923.418-1 I get besides the
> desired output files also a empty snappy file per reducer for free.
> Anybody familiar with this symptom and perhaps knows how to fix it ?
>
> --
> Mvrgr. Frank
>
Re: MultipleOutputs side effects
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
This came up recently on the forums, IIRC. The answer was to use to
something called LazyOutputFormat to suppress the output file ?
I think this thread has the discussion:
http://search-hadoop.com/m/66Thjsj97v1/hadoop+1.0.3+equivalent+of+MultipleTextOutputFormat&subj=RE+hadoop+1+0+3+equivalent+of+MultipleTextOutputFormat
Thanks
hemanth
On Fri, Sep 28, 2012 at 5:07 PM, Frank Kootte <fr...@gmail.com> wrote:
> While using MultipleOutputs with CHD3 0.20.2+923.418-1 I get besides the
> desired output files also a empty snappy file per reducer for free.
> Anybody familiar with this symptom and perhaps knows how to fix it ?
>
> --
> Mvrgr. Frank
>
Re: MultipleOutputs side effects
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
This came up recently on the forums, IIRC. The answer was to use to
something called LazyOutputFormat to suppress the output file ?
I think this thread has the discussion:
http://search-hadoop.com/m/66Thjsj97v1/hadoop+1.0.3+equivalent+of+MultipleTextOutputFormat&subj=RE+hadoop+1+0+3+equivalent+of+MultipleTextOutputFormat
Thanks
hemanth
On Fri, Sep 28, 2012 at 5:07 PM, Frank Kootte <fr...@gmail.com> wrote:
> While using MultipleOutputs with CHD3 0.20.2+923.418-1 I get besides the
> desired output files also a empty snappy file per reducer for free.
> Anybody familiar with this symptom and perhaps knows how to fix it ?
>
> --
> Mvrgr. Frank
>
Re: MultipleOutputs side effects
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
This came up recently on the forums, IIRC. The answer was to use to
something called LazyOutputFormat to suppress the output file ?
I think this thread has the discussion:
http://search-hadoop.com/m/66Thjsj97v1/hadoop+1.0.3+equivalent+of+MultipleTextOutputFormat&subj=RE+hadoop+1+0+3+equivalent+of+MultipleTextOutputFormat
Thanks
hemanth
On Fri, Sep 28, 2012 at 5:07 PM, Frank Kootte <fr...@gmail.com> wrote:
> While using MultipleOutputs with CHD3 0.20.2+923.418-1 I get besides the
> desired output files also a empty snappy file per reducer for free.
> Anybody familiar with this symptom and perhaps knows how to fix it ?
>
> --
> Mvrgr. Frank
>