You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Chris MacKenzie <st...@chrismackenziephotography.co.uk> on 2014/07/01 13:33:01 UTC
job.setOutputFormatClass(NullOutputFormat.class);
Hi,
What is the anticipated usage of the above with the new api ? Is there
another way to remove the empty part-r files
When using it with MultipleOutputs to remove empty part-r files I have no
output ;O)
Regards,
Chris MacKenzie
<http://www.chrismackenziephotography.co.uk/>
Re: job.setOutputFormatClass(NullOutputFormat.class);
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
Hi Markus And Shahab,
Thanks for getting back to me, I really appreciate it. LazyOutputFormat did
the trick. I tried NUllOutputFormat
(job.setOutputFormatClass(NullOutputFormat.class);) before writing to the
group but was getting an empty folder.
I looked at LazyOutputFormat, in fact, my mos is written from:
http://hadoop.apache.org/docs/r2.3.0/api/org/apache/hadoop/mapreduce/lib/out
put/MultipleOutputs.html
Just couldn¹t see the wood for the trees ;O)
Best,
Chris
Re: job.setOutputFormatClass(NullOutputFormat.class);
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
Hi Markus And Shahab,
Thanks for getting back to me, I really appreciate it. LazyOutputFormat did
the trick. I tried NUllOutputFormat
(job.setOutputFormatClass(NullOutputFormat.class);) before writing to the
group but was getting an empty folder.
I looked at LazyOutputFormat, in fact, my mos is written from:
http://hadoop.apache.org/docs/r2.3.0/api/org/apache/hadoop/mapreduce/lib/out
put/MultipleOutputs.html
Just couldn¹t see the wood for the trees ;O)
Best,
Chris
Re: job.setOutputFormatClass(NullOutputFormat.class);
Posted by Shahab Yunus <sh...@gmail.com>.
To get rid of empty *part files while using MultipleOutputs in the new API,
LazyOutputFormat class' static method should be used to set the output
format.
Details are here at the official Java docs for MultipleOutputs :
http://hadoop.apache.org/docs/r2.3.0/api/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.html
Regards,
Shahab
On Tue, Jul 1, 2014 at 9:08 PM, M. Dale <me...@yahoo.com> wrote:
> NullOutputFormat does not generate any output. Good for jobs where
> counters or some other I/O are your output (for example,
> http://stackoverflow.com/questions/12707726/run-a-hadoop-job-without-output-file
> ).
>
> From Tom White's book it sounds like
> org.apache.hadoop.mapreduce.lib.output.LazyOutputFormat only generates a
> file if at least one key/value pair is written.
>
> Hope this helps,
> Markus
>
>
> On 07/01/2014 07:33 AM, Chris MacKenzie wrote:
>
> Hi,
>
> What is the anticipated usage of the above with the new api ? Is there
> another way to remove the empty part-r files
>
> When using it with MultipleOutputs to remove empty part-r files I have no
> output ;O)
>
>
>
> Regards,
>
> Chris MacKenzie<http://www.chrismackenziephotography.co.uk/> <http://www.chrismackenziephotography.co.uk/>
>
>
>
Re: job.setOutputFormatClass(NullOutputFormat.class);
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
Hi Markus And Shahab,
Thanks for getting back to me, I really appreciate it. LazyOutputFormat did
the trick. I tried NUllOutputFormat
(job.setOutputFormatClass(NullOutputFormat.class);) before writing to the
group but was getting an empty folder.
I looked at LazyOutputFormat, in fact, my mos is written from:
http://hadoop.apache.org/docs/r2.3.0/api/org/apache/hadoop/mapreduce/lib/out
put/MultipleOutputs.html
Just couldn¹t see the wood for the trees ;O)
Best,
Chris
Re: job.setOutputFormatClass(NullOutputFormat.class);
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
Hi Markus And Shahab,
Thanks for getting back to me, I really appreciate it. LazyOutputFormat did
the trick. I tried NUllOutputFormat
(job.setOutputFormatClass(NullOutputFormat.class);) before writing to the
group but was getting an empty folder.
I looked at LazyOutputFormat, in fact, my mos is written from:
http://hadoop.apache.org/docs/r2.3.0/api/org/apache/hadoop/mapreduce/lib/out
put/MultipleOutputs.html
Just couldn¹t see the wood for the trees ;O)
Best,
Chris
Re: job.setOutputFormatClass(NullOutputFormat.class);
Posted by Shahab Yunus <sh...@gmail.com>.
To get rid of empty *part files while using MultipleOutputs in the new API,
LazyOutputFormat class' static method should be used to set the output
format.
Details are here at the official Java docs for MultipleOutputs :
http://hadoop.apache.org/docs/r2.3.0/api/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.html
Regards,
Shahab
On Tue, Jul 1, 2014 at 9:08 PM, M. Dale <me...@yahoo.com> wrote:
> NullOutputFormat does not generate any output. Good for jobs where
> counters or some other I/O are your output (for example,
> http://stackoverflow.com/questions/12707726/run-a-hadoop-job-without-output-file
> ).
>
> From Tom White's book it sounds like
> org.apache.hadoop.mapreduce.lib.output.LazyOutputFormat only generates a
> file if at least one key/value pair is written.
>
> Hope this helps,
> Markus
>
>
> On 07/01/2014 07:33 AM, Chris MacKenzie wrote:
>
> Hi,
>
> What is the anticipated usage of the above with the new api ? Is there
> another way to remove the empty part-r files
>
> When using it with MultipleOutputs to remove empty part-r files I have no
> output ;O)
>
>
>
> Regards,
>
> Chris MacKenzie<http://www.chrismackenziephotography.co.uk/> <http://www.chrismackenziephotography.co.uk/>
>
>
>
Re: job.setOutputFormatClass(NullOutputFormat.class);
Posted by Shahab Yunus <sh...@gmail.com>.
To get rid of empty *part files while using MultipleOutputs in the new API,
LazyOutputFormat class' static method should be used to set the output
format.
Details are here at the official Java docs for MultipleOutputs :
http://hadoop.apache.org/docs/r2.3.0/api/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.html
Regards,
Shahab
On Tue, Jul 1, 2014 at 9:08 PM, M. Dale <me...@yahoo.com> wrote:
> NullOutputFormat does not generate any output. Good for jobs where
> counters or some other I/O are your output (for example,
> http://stackoverflow.com/questions/12707726/run-a-hadoop-job-without-output-file
> ).
>
> From Tom White's book it sounds like
> org.apache.hadoop.mapreduce.lib.output.LazyOutputFormat only generates a
> file if at least one key/value pair is written.
>
> Hope this helps,
> Markus
>
>
> On 07/01/2014 07:33 AM, Chris MacKenzie wrote:
>
> Hi,
>
> What is the anticipated usage of the above with the new api ? Is there
> another way to remove the empty part-r files
>
> When using it with MultipleOutputs to remove empty part-r files I have no
> output ;O)
>
>
>
> Regards,
>
> Chris MacKenzie<http://www.chrismackenziephotography.co.uk/> <http://www.chrismackenziephotography.co.uk/>
>
>
>
Re: job.setOutputFormatClass(NullOutputFormat.class);
Posted by Shahab Yunus <sh...@gmail.com>.
To get rid of empty *part files while using MultipleOutputs in the new API,
LazyOutputFormat class' static method should be used to set the output
format.
Details are here at the official Java docs for MultipleOutputs :
http://hadoop.apache.org/docs/r2.3.0/api/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.html
Regards,
Shahab
On Tue, Jul 1, 2014 at 9:08 PM, M. Dale <me...@yahoo.com> wrote:
> NullOutputFormat does not generate any output. Good for jobs where
> counters or some other I/O are your output (for example,
> http://stackoverflow.com/questions/12707726/run-a-hadoop-job-without-output-file
> ).
>
> From Tom White's book it sounds like
> org.apache.hadoop.mapreduce.lib.output.LazyOutputFormat only generates a
> file if at least one key/value pair is written.
>
> Hope this helps,
> Markus
>
>
> On 07/01/2014 07:33 AM, Chris MacKenzie wrote:
>
> Hi,
>
> What is the anticipated usage of the above with the new api ? Is there
> another way to remove the empty part-r files
>
> When using it with MultipleOutputs to remove empty part-r files I have no
> output ;O)
>
>
>
> Regards,
>
> Chris MacKenzie<http://www.chrismackenziephotography.co.uk/> <http://www.chrismackenziephotography.co.uk/>
>
>
>
Re: job.setOutputFormatClass(NullOutputFormat.class);
Posted by "M. Dale" <me...@yahoo.com>.
NullOutputFormat does not generate any output. Good for jobs where
counters or some other I/O are your output (for example,
http://stackoverflow.com/questions/12707726/run-a-hadoop-job-without-output-file).
From Tom White's book it sounds like
org.apache.hadoop.mapreduce.lib.output.LazyOutputFormat only generates a
file if at least one key/value pair is written.
Hope this helps,
Markus
On 07/01/2014 07:33 AM, Chris MacKenzie wrote:
> Hi,
>
> What is the anticipated usage of the above with the new api ? Is there
> another way to remove the empty part-r files
>
> When using it with MultipleOutputs to remove empty part-r files I have no
> output ;O)
>
>
>
> Regards,
>
> Chris MacKenzie
> <http://www.chrismackenziephotography.co.uk/>
>
>
Re: job.setOutputFormatClass(NullOutputFormat.class);
Posted by "M. Dale" <me...@yahoo.com>.
NullOutputFormat does not generate any output. Good for jobs where
counters or some other I/O are your output (for example,
http://stackoverflow.com/questions/12707726/run-a-hadoop-job-without-output-file).
From Tom White's book it sounds like
org.apache.hadoop.mapreduce.lib.output.LazyOutputFormat only generates a
file if at least one key/value pair is written.
Hope this helps,
Markus
On 07/01/2014 07:33 AM, Chris MacKenzie wrote:
> Hi,
>
> What is the anticipated usage of the above with the new api ? Is there
> another way to remove the empty part-r files
>
> When using it with MultipleOutputs to remove empty part-r files I have no
> output ;O)
>
>
>
> Regards,
>
> Chris MacKenzie
> <http://www.chrismackenziephotography.co.uk/>
>
>
Re: job.setOutputFormatClass(NullOutputFormat.class);
Posted by "M. Dale" <me...@yahoo.com>.
NullOutputFormat does not generate any output. Good for jobs where
counters or some other I/O are your output (for example,
http://stackoverflow.com/questions/12707726/run-a-hadoop-job-without-output-file).
From Tom White's book it sounds like
org.apache.hadoop.mapreduce.lib.output.LazyOutputFormat only generates a
file if at least one key/value pair is written.
Hope this helps,
Markus
On 07/01/2014 07:33 AM, Chris MacKenzie wrote:
> Hi,
>
> What is the anticipated usage of the above with the new api ? Is there
> another way to remove the empty part-r files
>
> When using it with MultipleOutputs to remove empty part-r files I have no
> output ;O)
>
>
>
> Regards,
>
> Chris MacKenzie
> <http://www.chrismackenziephotography.co.uk/>
>
>
Re: job.setOutputFormatClass(NullOutputFormat.class);
Posted by "M. Dale" <me...@yahoo.com>.
NullOutputFormat does not generate any output. Good for jobs where
counters or some other I/O are your output (for example,
http://stackoverflow.com/questions/12707726/run-a-hadoop-job-without-output-file).
From Tom White's book it sounds like
org.apache.hadoop.mapreduce.lib.output.LazyOutputFormat only generates a
file if at least one key/value pair is written.
Hope this helps,
Markus
On 07/01/2014 07:33 AM, Chris MacKenzie wrote:
> Hi,
>
> What is the anticipated usage of the above with the new api ? Is there
> another way to remove the empty part-r files
>
> When using it with MultipleOutputs to remove empty part-r files I have no
> output ;O)
>
>
>
> Regards,
>
> Chris MacKenzie
> <http://www.chrismackenziephotography.co.uk/>
>
>