You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by Hilkiah Lavinier <hi...@yahoo.com> on 2008/02/11 02:56:22 UTC

nutch/hadoop parameters for optimal performance

Ok guys,

I once again need some advice here.  I have 4 dual proc quad core 1.8 xeon servers, each server has 4gb ram and runs linux.  I am using nutch svn (build #334 i think) and am using hadoop dfs.  I need to know what parameters I can set to get the optimal performance from these servers.  I  have a seed list of about 10,000 urls (ignore external link will be set to true).  My goal is to crawl in the shortest period of time.  Furthermore I intend to run one crawl (depth 5) and thus have one index.  

What advice would you give in terms of this approach and also in terms of nutch/hadoop variables/parameters and their settings.

Regards,
 
Hilkiah G. Lavinier MEng (Hons), ACGI 
6 Winston Lane, 
Goodwill, 
Roseau, Dominica 
Mbl: (767) 275 3382
Hm : (767) 440 3924
Fax: (767) 440 4991
VoIP USA: (646) 432 4487
 
Email: hilkiah@yahoo.com
Email: hilkiah.lavinier@gmail.com
IM: Yahoo hilkiah / MSN hilkiahlavinier@hotmail.com
IM: ICQ #8978201  / AOL hilkiah21






      ____________________________________________________________________________________
Never miss a thing.  Make Yahoo your home page. 
http://www.yahoo.com/r/hs