mapred.local.dir cleanup

[rakesh kothari <rk...@hotmail.com>]

Hi,

I am seeing lots of leftover directories going back as far as 12 days in the task trackers "mapred.local.dir". These directories are for "M/R task attempts".

How are these directories end up in "mapred.local.dir" as  from my understanding these directories should be in "mapred.local.dir/taskTracker/jobcache/job-Id/" and should be cleaned up once the job finishes (or after some interval) ? How can I enable automatic cleanup of these directories ? 

A big chunk of these leftover directories were created the same day/time when I bounced my hadoop cluster. 

Any pointers is highly appreciated.

Thanks,
-Rakesh
 		 	   		  

RE: mapred.local.dir cleanup

[rakesh kothari <rk...@hotmail.com>]

Any ideas on how "attempt*" directories getting created directly under "mapred.local.dir" ? Pointers to parts of the source code would help too.

Thanks,
-Rakesh

From: rkothari_iit@hotmail.com
To: mapreduce-user@hadoop.apache.org
Subject: mapred.local.dir cleanup
Date: Tue, 18 Jan 2011 17:20:04 -0800








Hi,

I am seeing lots of leftover directories going back as far as 12 days in the task trackers "mapred.local.dir". These directories are for "M/R task attempts".

How are these directories end up in "mapred.local.dir" as  from my understanding these directories should be in "mapred.local.dir/taskTracker/jobcache/job-Id/" and should be cleaned up once the job finishes (or after some interval) ? How can I enable automatic cleanup of these directories ? 

A big chunk of these leftover directories were created the same day/time when I bounced my hadoop cluster. 

Any pointers is highly appreciated.

Thanks,
-Rakesh