You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Christopher Berner <ch...@gmail.com> on 2012/04/20 21:33:02 UTC
Fetch step only runs on one node in Hadoop cluster (Elastic MapReduce)
I've setup a cluster on Elastic MapReduce, and am crawling ~25,000 domains,
with Nutch 1.4. However, it seems that the fetch step only runs on a single
node, even though the rest of the job is processed by all nodes. I read
this thread (
http://www.mail-archive.com/nutch-user@lucene.apache.org/msg11857.html),
which suggested that the "mapred.tasktracker.map.tasks" might be set too
low, but increasing that to 50 had to effect. I also came across the
-numFetchers option, but it seems Crawl.java doesn't support that.
Would very much appreciate any suggestions!
Christopher