Re: free disk space

[Annona Keene <an...@yahoo.com>]

I would like to ask the same question. My crawls running last night failed because the disk space had been exhausted with temp files. Why doesn't nutch clean these up when it is done with them? Is there some setting somewhere to remove them that I'm not seeing? If not, is there some way for me to determine which of the temp files are associated with a given crawl? Simply deleting the whole thing when a crawl is done is not an option, as I run multiple crawls concurrently. I can't just blow the temp files away and have my other crawls fail. 

Ann

----- Original Message ----
From: Lyndon Maydwell <ma...@gmail.com>
To: nutch-user@lucene.apache.org
Sent: Monday, September 17, 2007 9:14:49 AM
Subject: Re: free disk space

Thanks Doğacan,

I went ahead and did this anyway after chcking that they weren't being
used, and all was well, but does nutch usually take up this much space
in temp files?

I'm running the crawl on a server that never gets restarted, so I
can't have all the drive space used up like this. I can write a cron
job to regularly remove the files, but this seems a bit haphazard.

Thanks again for your reply,

Lyndon.







       
____________________________________________________________________________________
Got a little couch potato? 
Check out fun summer activities for kids.
http://search.yahoo.com/search?fr=oni_on_mail&p=summer+activities+for+kids&cs=bz 

Re: free disk space

[Tim Gautier <ti...@gmail.com>]

I'm not sure that this will work for you but it might.  Do you have
multiple nutch directories?  If you run each crawl from a separate
nutch directory, you could modify the hadoop-site.xml in each of them
to put the temp files in a different location.  That would allow you
to delete the temp files for just one crawl without needing to know
more than it's base temp directory.  Obviously this wouldn't be
feasible for a large number of crawls running concurrently, but if
you're only doing a couple it just costs you a few megs of disk space.
 I'm not saying it's the best solution, but it's an easy solution. :)

On 10/3/07, Annona Keene <an...@yahoo.com> wrote:
> I would like to ask the same question. My crawls running last night failed because the disk space had been exhausted with temp files. Why doesn't nutch clean these up when it is done with them? Is there some setting somewhere to remove them that I'm not seeing? If not, is there some way for me to determine which of the temp files are associated with a given crawl? Simply deleting the whole thing when a crawl is done is not an option, as I run multiple crawls concurrently. I can't just blow the temp files away and have my other crawls fail.
>
> Ann
>
> ----- Original Message ----
> From: Lyndon Maydwell <ma...@gmail.com>
> To: nutch-user@lucene.apache.org
> Sent: Monday, September 17, 2007 9:14:49 AM
> Subject: Re: free disk space
>
> Thanks Doğacan,
>
> I went ahead and did this anyway after chcking that they weren't being
> used, and all was well, but does nutch usually take up this much space
> in temp files?
>
> I'm running the crawl on a server that never gets restarted, so I
> can't have all the drive space used up like this. I can write a cron
> job to regularly remove the files, but this seems a bit haphazard.
>
> Thanks again for your reply,
>
> Lyndon.
>
>
>
>
>
>
>
>
> ____________________________________________________________________________________
> Got a little couch potato?
> Check out fun summer activities for kids.
> http://search.yahoo.com/search?fr=oni_on_mail&p=summer+activities+for+kids&cs=bz