Temporary files for mapppers and reducers

[Mark Kerzner <ma...@gmail.com>]

Hi,

what would be the best place to put temporary files for a reducer? I believe
that since reducers each work on its own machine, at its own time, one can
do anything, but I would like a confirmation from the experts.

Thanks,
Mark

Re: Temporary files for mapppers and reducers

[Owen O'Malley <ow...@gmail.com>]

Just use the current working directory. Each task gets a unique  
directory that is erased when the task finished.

-- Owen

On Mar 15, 2009, at 16:08, Mark Kerzner <ma...@gmail.com> wrote:

> Hi,
>
> what would be the best place to put temporary files for a reducer? I  
> believe
> that since reducers each work on its own machine, at its own time,  
> one can
> do anything, but I would like a confirmation from the experts.
>
> Thanks,
> Mark

Re: Temporary files for mapppers and reducers

[jason hadoop <ja...@gmail.com>]

If you use the Java System Property java.io.tmpdir, your reducer will use
the ./tmp directory in the local working directory allocated by the
framework for your task.

If you have a specialty file system for transient data, such as a tmpfs, use
that.



On Sun, Mar 15, 2009 at 4:08 PM, Mark Kerzner <ma...@gmail.com> wrote:

> Hi,
>
> what would be the best place to put temporary files for a reducer? I
> believe
> that since reducers each work on its own machine, at its own time, one can
> do anything, but I would like a confirmation from the experts.
>
> Thanks,
> Mark
>



-- 
Alpha Chapters of my book on Hadoop are available
http://www.apress.com/book/view/9781430219422