You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by Ian Holsman <li...@holsman.net> on 2007/06/20 07:50:49 UTC

how fast can nutch fetch urls ?

I was wondering what kind of reqs/second people are achieving from a 
single node in a hadoop cluster in the fetch phase.

RE: how fast can nutch fetch urls ?

Posted by Robeyns Bart <Ba...@panoptic.biz>.

I'm crawling a static website on the same machine as Nutch (thus eliminating bandwith restrictions).
stats:
 99277 pages, 34 errors, 60.8 pages/s, 20410 kb/s




-----Original Message-----
From: Ian Holsman [mailto:lists@holsman.net]
Sent: Wed 6/20/2007 7:50
To: nutch-user@lucene.apache.org
Subject: how fast can nutch fetch urls ?
 
I was wondering what kind of reqs/second people are achieving from a 
single node in a hadoop cluster in the fetch phase.