Simultaneous Nutch Crawls

[Daniel Clark <da...@verizon.net>]

I am running a nutch with hadoop crawl on a two machine cluster.  I am
trying to run 3 crawls simultaneously to the same directory (crawl), but
creating different segments.  I got the following error message from one
crawl.  One crawl completed successfully.  The last crawl is frozen in the
fetcher stage.  What could cause the following error?  Is it possible to
launch multiple crawls to the same directory, but different segments?  What
could cause the crawl to hang in fetcher stage?  Any help would be greatly
appreciated. 

 

Injector: starting
Injector: crawlDb: crawl/crawldb
Injector: urlDir: urls/dmozab
Injector: Converting injected urls to crawl db entries.
Injector: Merging injected urls into crawl db.
Injector: done
Generator: Selecting best-scoring urls due for fetch.
Generator: starting
Generator: segment: crawl/segments/20071004140827
Generator: filtering: true
Generator: Partitioning selected urls by host, for politeness.
Generator: done.
processing segment /user/daclark/crawl/segments/20071004140827
Fetcher: starting
Fetcher: segment: /user/daclark/crawl/segments/20071004140827
Fetcher: done
CrawlDb update: java.io.IOException: lock file crawl/crawldb/.locked already
exists.
        at org.apache.nutch.util.LockUtil.createLockFile(LockUtil.java:44)
        at org.apache.nutch.crawl.CrawlDb.update(CrawlDb.java:68)
        at org.apache.nutch.crawl.CrawlDb.run(CrawlDb.java:199)
        at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:189)
        at org.apache.nutch.crawl.CrawlDb.main(CrawlDb.java:152)

 

~~~~~~~~~~~~~~~~~~~~~

Daniel Clark, President

DAC Systems, Inc.

(703) 403-0340

~~~~~~~~~~~~~~~~~~~~~