You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Craig Macdonald <cr...@dcs.gla.ac.uk> on 2009/01/22 19:49:09 UTC
FileOutputFormat.getWorkOutputPath and map-to-reduce-only side-effect
files
Hello Hadoop Core,
I have a very brief question: Our map tasks create side-effect files, in
the directory returned by FileOutputFormat.getWorkOutputPath().
This works fine for the getting the side-effect files that can be
accessed by the reducers.
However, as these map-generated side-effect files are only of use to the
reducers, it would be nice to have them deleted from the output
directory. However, we cant delete them in a reducer.close(), as this
would prevent them being accessible to other reduce tasks (speculative
or otherwise).
Any suggestions, short of deleting them after the job completes?
Craig
Re: FileOutputFormat.getWorkOutputPath and map-to-reduce-only side-effect
files
Posted by Craig Macdonald <cr...@dcs.gla.ac.uk>.
Hi Sharad,
Thanks for the hint - that looks indeed like what I need. We'll be
moving to 0.19 shortly now that Hbase has caught up.
Craig
sharad agarwal wrote:
>
>> Any suggestions, short of deleting them after the job completes?
> which version of hadoop you are using? hadoop 0.19 has an api called
> JobConf.setOutputCommitter see
> http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/mapred/OutputCommitter.html
> via which you can do the clean up work at the end of the job.
> |*
> <http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/mapred/JobConf.html#setOutputCommitter%28java.lang.Class%29>*|
>
>>
>>
>
>
Re: FileOutputFormat.getWorkOutputPath and map-to-reduce-only side-effect
files
Posted by sharad agarwal <sh...@yahoo-inc.com>.
> Any suggestions, short of deleting them after the job completes?
which version of hadoop you are using? hadoop 0.19 has an api called
JobConf.setOutputCommitter see
http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/mapred/OutputCommitter.html
via which you can do the clean up work at the end of the job.
|*
<http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/mapred/JobConf.html#setOutputCommitter%28java.lang.Class%29>*|
>
>