You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Marcus Herou <ma...@tailsweep.com> on 2010/02/10 09:15:38 UTC
Cleaning jobcache manually
Hi.
We run hadoop-0.18.3 and it seems that the jobcache does not get cleaned out
properly.
Would this cron script be to any harm to hadoop ?
# Clean all files which are two or more days old
/usr/bin/find ${JOB_CACHE_PATH} -type f -mtime +2 -exec rm {} \;
Need to start cleaning today so hoping for quick response.
Cheers
//Marcus Herou
--
Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou@tailsweep.com
http://www.tailsweep.com/
Re: Cleaning jobcache manually
Posted by Vibhooti Verma <ve...@gmail.com>.
We faced the same issue and we also use the cron to delete the older
entries.
Please be careful that your mtime for deletion should never be less than the
longest job you can ever have.
On Wed, Feb 10, 2010 at 1:45 PM, Marcus Herou <ma...@tailsweep.com>wrote:
> Hi.
>
> We run hadoop-0.18.3 and it seems that the jobcache does not get cleaned
> out
> properly.
>
> Would this cron script be to any harm to hadoop ?
>
> # Clean all files which are two or more days old
> /usr/bin/find ${JOB_CACHE_PATH} -type f -mtime +2 -exec rm {} \;
>
> Need to start cleaning today so hoping for quick response.
>
> Cheers
>
> //Marcus Herou
>
>
> --
> Marcus Herou CTO and co-founder Tailsweep AB
> +46702561312
> marcus.herou@tailsweep.com
> http://www.tailsweep.com/
>
--
cheers,
Vibhooti
Re: Cleaning jobcache manually
Posted by Marcus Herou <ma...@tailsweep.com>.
OK. Needed to short it down to even 1 day. Scary I know but the cache grows
like crazy.
/M
On Wed, Feb 10, 2010 at 8:19 PM, Allen Wittenauer
<aw...@linkedin.com>wrote:
>
>
>
> On 2/10/10 12:15 AM, "Marcus Herou" <ma...@tailsweep.com> wrote:
> > We run hadoop-0.18.3 and it seems that the jobcache does not get cleaned
> out
> > properly.
> >
> > Would this cron script be to any harm to hadoop ?
> >
> > # Clean all files which are two or more days old
> > /usr/bin/find ${JOB_CACHE_PATH} -type f -mtime +2 -exec rm {} \;
> >
> > Need to start cleaning today so hoping for quick response.
>
> We do it something similar, but wait for it to be 7 days long in case we
> have a particularly long running job.
>
>
--
Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou@tailsweep.com
http://www.tailsweep.com/
Re: Cleaning jobcache manually
Posted by Allen Wittenauer <aw...@linkedin.com>.
On 2/10/10 12:15 AM, "Marcus Herou" <ma...@tailsweep.com> wrote:
> We run hadoop-0.18.3 and it seems that the jobcache does not get cleaned out
> properly.
>
> Would this cron script be to any harm to hadoop ?
>
> # Clean all files which are two or more days old
> /usr/bin/find ${JOB_CACHE_PATH} -type f -mtime +2 -exec rm {} \;
>
> Need to start cleaning today so hoping for quick response.
We do it something similar, but wait for it to be 7 days long in case we
have a particularly long running job.