You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Craig Macdonald <cr...@dcs.gla.ac.uk> on 2009/01/22 19:49:09 UTC

FileOutputFormat.getWorkOutputPath and map-to-reduce-only side-effect files

Hello Hadoop Core,

I have a very brief question: Our map tasks create side-effect files, in 
the directory returned by FileOutputFormat.getWorkOutputPath().

This works fine for the getting the side-effect files that can be 
accessed by the reducers.

However, as these map-generated side-effect files are only of use to the 
reducers, it would be nice to have them deleted from the output 
directory. However, we cant delete them in a reducer.close(), as this 
would prevent them being accessible to other reduce tasks (speculative 
or otherwise).

Any suggestions, short of deleting them after the job completes?

Craig

Re: FileOutputFormat.getWorkOutputPath and map-to-reduce-only side-effect files

Posted by Craig Macdonald <cr...@dcs.gla.ac.uk>.
Hi Sharad,

Thanks for the hint - that looks indeed like what I need. We'll be 
moving to 0.19 shortly now that Hbase has caught up.

Craig

sharad agarwal wrote:
>
>> Any suggestions, short of deleting them after the job completes?
> which version of hadoop you are using? hadoop 0.19 has an api called 
> JobConf.setOutputCommitter see 
> http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/mapred/OutputCommitter.html 
> via which you can do the clean up work at the end of the job.
> |* 
> <http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/mapred/JobConf.html#setOutputCommitter%28java.lang.Class%29>*| 
>
>>
>>  
>
>


Re: FileOutputFormat.getWorkOutputPath and map-to-reduce-only side-effect files

Posted by sharad agarwal <sh...@yahoo-inc.com>.
> Any suggestions, short of deleting them after the job completes?
which version of hadoop you are using? hadoop 0.19 has an api called 
JobConf.setOutputCommitter see 
http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/mapred/OutputCommitter.html 
via which you can do the clean up work at the end of the job.
|* 
<http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/mapred/JobConf.html#setOutputCommitter%28java.lang.Class%29>*| 

>
>