how to get rid of attempt_201101170925_****_m_**** directories safely?

[jigar shah <js...@pandora.com>]

Hi,

    we are running a 50 node hadoop cluster and have a problem with these
attempt directories piling up(for eg attempt_201101170925_126956_m_000232_0)
and taking a lot of space. when i restart the tasktracker daemon these
directories get cleaned out and the space usage goes down.

    i understand why these directories are created, but it becomes a pain
when they just hang around indefinitely. its very inconvenient to restart
the tasktracker to get rid of them  and reclaim space.

    anyone knows if there is a setting in the conf somewhere i can set that
will periodically prune these directories or any other way to deal with
this.

    i appreciate any sort of help

 

thanks

Re: how to get rid of attempt_201101170925_****_m_**** directories safely?

[Edward Capriolo <ed...@gmail.com>]

On Thu, Mar 17, 2011 at 1:20 PM, jigar shah <js...@pandora.com> wrote:
> Hi,
>
>    we are running a 50 node hadoop cluster and have a problem with these
> attempt directories piling up(for eg attempt_201101170925_126956_m_000232_0)
> and taking a lot of space. when i restart the tasktracker daemon these
> directories get cleaned out and the space usage goes down.
>
>    i understand why these directories are created, but it becomes a pain
> when they just hang around indefinitely. its very inconvenient to restart
> the tasktracker to get rid of them  and reclaim space.
>
>    anyone knows if there is a setting in the conf somewhere i can set that
> will periodically prune these directories or any other way to deal with
> this.
>
>    i appreciate any sort of help
>
>
>
> thanks
>
>
>

Something you can run from cron.
6 3 * * * hadoop find /disk6/hadoop_root/hdfs_data/hadoop/mapred/local
-maxdepth 1 -name "attempt_*" -ctime +7 -delete