You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nutch.apache.org by Christophe Noel <ch...@cetic.be> on 2005/03/04 11:26:32 UTC

urgent please help - bug

Hello,

Please all suggestions are welcome !
I need a fresh crawl to show on monday but for a while, I always get 
this error after about one hour of crawling with

bin/nutch crawl urls -dir dir -depth 50 -threads 50

050302 183540 Updating /nutch-0.6/agoria.2mar/db
050302 183540 Updating for  /nutch-0.6/agoria.2mar/segments/20050302183116
050302 183540 Processing document 0
050302 183541 Finishing update
050302 183542 Processing pagesByURL: Sorted 2931 instructions in 0.915  
seconds.
050302 183542 Processing pagesByURL: Sorted 3203.27868852459  
instructions/second
Exception in thread "main" java.io.IOException: already exists:  
/nutch-0.6/agoria.2mar/db/webdb.new/pagesByURL
       at net.nutch.io.MapFile$Writer.<init>(MapFile.java:67)
       at  
net.nutch.db.WebDBWriter$CloseProcessor.closeDown(WebDBWriter.java: 536)
       at net.nutch.db.WebDBWriter.close(WebDBWriter.java:1531)
       at  
net.nutch.tools.UpdateDatabaseTool.close(UpdateDatabaseTool.java:301)
       at  
net.nutch.tools.UpdateDatabaseTool.main(UpdateDatabaseTool.java:351)
       at net.nutch.tools.CrawlTool.main(CrawlTool.java:128)

Thanks.

Christophe.

Banned Hosts

Posted by "yoursoft@freemail.hu" <yo...@freemail.hu>.

Dear Guys!

I inserted some line into the banned-hosts.txt
e.g.:
http://www.somesite.hu

But the crawl get these hosts.

I find the banned-hosts.txt in the conf  files, but not found.

Can you help me, how to banne  hosts  from crawl or from search?

Thanks,
     Ferenc

Re: urgent please help - bug

Posted by Stefan Groschupf <sg...@media-style.com>.

Christophe,
as mentioned I guess you have somehow 2 concurrent processes or the 
last run fails.
Try to find an earlier error message in the log files.
050302 183540 Processing document 0
050302 183541 Finishing update
Looks like there is a general problem.
Try to process the steps the crawl command does manually.
May this simple loop scrip helps.
At least it does more or less the same
http://wiki.apache.org/nutch/useful_20scripts
	
HTH
Stefan

> Hello,
>
> Please all suggestions are welcome !
> I need a fresh crawl to show on monday but for a while, I always get 
> this error after about one hour of crawling with
>
> bin/nutch crawl urls -dir dir -depth 50 -threads 50
>
> 050302 183540 Updating /nutch-0.6/agoria.2mar/db
> 050302 183540 Updating for  
> /nutch-0.6/agoria.2mar/segments/20050302183116
> 050302 183540 Processing document 0
> 050302 183541 Finishing update
> 050302 183542 Processing pagesByURL: Sorted 2931 instructions in 0.915 
>  seconds.
> 050302 183542 Processing pagesByURL: Sorted 3203.27868852459  
> instructions/second
> Exception in thread "main" java.io.IOException: already exists:  
> /nutch-0.6/agoria.2mar/db/webdb.new/pagesByURL
>       at net.nutch.io.MapFile$Writer.<init>(MapFile.java:67)
>       at  
> net.nutch.db.WebDBWriter$CloseProcessor.closeDown(WebDBWriter.java: 
> 536)
>       at net.nutch.db.WebDBWriter.close(WebDBWriter.java:1531)
>       at  
> net.nutch.tools.UpdateDatabaseTool.close(UpdateDatabaseTool.java:301)
>       at  
> net.nutch.tools.UpdateDatabaseTool.main(UpdateDatabaseTool.java:351)
>       at net.nutch.tools.CrawlTool.main(CrawlTool.java:128)
>
> Thanks.
>
> Christophe.
>