Fetch Speed Issues

[Matt Zytaruk <ma...@wavefire.com>]

Hi there, I just started working on a search engine based on the nutch 
project, but we are finding that the fetcher is crawling extremely slow. 
I've seen posts talking about people maxing out their 5mb lines with the 
fetcher, but we can't seem to get anymore than about 20k/s or 1.5 
pages/second, which isnt even a smidgen of our capacity, even with 
-threads set to 200 . This is using the mapred branch, by the way.

Are there any settings we might be missing that would cause this 
slowdown? or are there certain network configurations that could be 
causing this?

Also, is the -numFetchers option in 'nutch generate' broken in the 
mapred branch? it worked fine in 0.7, but doesn't seem to do anything in 
0.8-dev.

Thanks a lot for your help.

Matt Zytaruk