You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Eric <er...@gmail.com> on 2010/12/10 10:19:18 UTC
Getting a temporary directory in map jobs
Hi there,
I have a map-reduce job that processes binary files. I'm currently using
/tmp/ as a temporary location to write data to and perform operations like
decompression. If a mapper fails, the temporary filesare left behind on the
nodes.
Is there a way to get a temp location from Hadoop, that is cleaned after the
Map job finishes? I'm sure there must be a better way to do this ;-)
I'm using Hadoop version 0.20.2 (the Cloudera distribution)
Thanks in advance!
Eric
Re: Getting a temporary directory in map jobs
Posted by Harsh J <qw...@gmail.com>.
On Wed, Dec 22, 2010 at 7:14 PM, Eric <er...@gmail.com> wrote:
> Thank you for your suggestion. I need a temp directory, not a single file. I
> successfully used Koji's suggestion to use ./tmp, which is a preexisting
> directory for this purpose.
And whose name is also configurable via "mapred.child.tmp".
--
Harsh J
www.harshj.com
Re: Getting a temporary directory in map jobs
Posted by Eric <er...@gmail.com>.
2010/12/22 Chase Bradford <ch...@gmail.com>
> If you want a tmp file on a task's local host, just use java's
> createTempFile from the File class. It creates a file in java.io.tmp, which
> the task runner sets up in the task's workspace and is cleaned by the TT
> even if the child jvm exits badly.
>
>
Thank you for your suggestion. I need a temp directory, not a single file. I
successfully used Koji's suggestion to use ./tmp, which is a preexisting
directory for this purpose.
Re: Getting a temporary directory in map jobs
Posted by David Rosenstrauch <da...@darose.net>.
On 12/21/2010 09:50 PM, Chase Bradford wrote:
> If you want a tmp file on a task's local host, just use java's
> createTempFile from the File class. It creates a file in
> java.io.tmp, which the task runner sets up in the task's workspace
> and is cleaned by the TT even if the child jvm exits badly.
Mostly correct. But just FYI, the temp file does not automatically get
deleted (see:
http://download.oracle.com/javase/6/docs/api/java/io/File.html#createTempFile%28java.lang.String,%20java.lang.String,%20java.io.File%29)
unless you call deleteOnExit()
(http://download.oracle.com/javase/6/docs/api/java/io/File.html#deleteOnExit%28%29).
DR
Re: Getting a temporary directory in map jobs
Posted by Chase Bradford <ch...@gmail.com>.
If you want a tmp file on a task's local host, just use java's createTempFile from the File class. It creates a file in java.io.tmp, which the task runner sets up in the task's workspace and is cleaned by the TT even if the child jvm exits badly.
Sent from phnoe
On Dec 10, 2010, at 1:19 AM, Eric <er...@gmail.com> wrote:
> Hi there,
>
> I have a map-reduce job that processes binary files. I'm currently using /tmp/ as a temporary location to write data to and perform operations like decompression. If a mapper fails, the temporary filesare left behind on the nodes.
>
> Is there a way to get a temp location from Hadoop, that is cleaned after the Map job finishes? I'm sure there must be a better way to do this ;-)
>
> I'm using Hadoop version 0.20.2 (the Cloudera distribution)
>
> Thanks in advance!
>
> Eric
Re: Getting a temporary directory in map jobs
Posted by Ken Goodhope <ke...@gmail.com>.
FileSystem has a method createTempFile, or something like that. You can
pass it the option to delete the file when the jvm exits. If the jvm exits
abnormally, you may still get some files laying around, but under normal
circumstances they will get cleaned up.
On Fri, Dec 10, 2010 at 8:46 AM, Koji Noguchi <kn...@yahoo-inc.com>wrote:
> Hi Eric.
>
> Try ‘./tmp’ (or current working directory).
>
> Koji
>
>
>
> On 12/10/10 1:19 AM, "Eric" <er...@gmail.com> wrote:
>
> Hi there,
>
> I have a map-reduce job that processes binary files. I'm currently using
> /tmp/ as a temporary location to write data to and perform operations like
> decompression. If a mapper fails, the temporary filesare left behind on the
> nodes.
>
> Is there a way to get a temp location from Hadoop, that is cleaned after
> the Map job finishes? I'm sure there must be a better way to do this ;-)
>
> I'm using Hadoop version 0.20.2 (the Cloudera distribution)
>
> Thanks in advance!
>
> Eric
>
>
Re: Getting a temporary directory in map jobs
Posted by Koji Noguchi <kn...@yahoo-inc.com>.
Hi Eric.
Try './tmp' (or current working directory).
Koji
On 12/10/10 1:19 AM, "Eric" <er...@gmail.com> wrote:
Hi there,
I have a map-reduce job that processes binary files. I'm currently using /tmp/ as a temporary location to write data to and perform operations like decompression. If a mapper fails, the temporary filesare left behind on the nodes.
Is there a way to get a temp location from Hadoop, that is cleaned after the Map job finishes? I'm sure there must be a better way to do this ;-)
I'm using Hadoop version 0.20.2 (the Cloudera distribution)
Thanks in advance!
Eric