You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by Daqing Zhao <de...@gmail.com> on 2005/11/23 14:57:08 UTC

speed of crawl

Hi,

I have set up some crawl jobs on workstation, and I enjoy learning things
using it.  But the speed of crawling I got is only a few links per second.
I saw people mentioning 200 links per second or something.  What are the
things to tweak to increase the through put on a single machine.  I want to
make sure I learn and optimize on a single machine before I go to mapred.

Thanks, Happy Thanksgiving,

Daqing