Re: nutch-1.0 crawl on distributed Hadoop clusters with "depth=0 - no more URLs to fetch"

[Stefano Cherchi <st...@yahoo.it>]

Hi Xudong,

make sure of the following: 

is hadoop up and running on all cluster machines? (run jps command on each node: are the DataNode and TaskTracker processes running on all slaves? Are the SecondaryNameNode, NameNode and JobTracker processes running on the master node?)

do the slaves are able to communicate with the master and vice-versa? (can you ssh passwordless from each node to each other?)

is Nutch configuration the same across all hadoop slaves?

S

---------------------------------- 
"Anyone proposing to run Windows on servers should be prepared to explain 
what they know about servers that Google, Yahoo, and Amazon don't."
Paul Graham


"A mathematician is a device for turning coffee into theorems."
Paul Erdos (who obviously never met a sysadmin)



----- Messaggio originale -----
> Da: Xudong Du <an...@gmail.com>
> A: nutch-user@lucene.apache.org
> Inviato: Mar 23 marzo 2010, 20:22:35
> Oggetto: nutch-1.0 crawl on distributed Hadoop clusters with "depth=0 - no  more URLs to fetch"
> 
> hi, all.

when crawl running on localhost , everything is ok. but when 
> crawl on
multi-node clusters, it comes to the following 
> situation:
Generator: 0 records selected for fetching, exiting 
> ...
Stopping at depth=0 - no more URLs to fetch.
No URLs to fetch - check 
> your seed list and URL filters.
crawl finished: crawled

any solution 
> to it?
it seems to be a bug for nutch-0.9. however when I was running 
> nutch-1.0,
still came into the same problem.
thank you very 
> much.

-- 
Yours Sincerely
Xudong Du
Department of Computer 
> Science&Technology,
Tsinghua University, Beijing, China, 100084