You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by qi wu <ch...@gmail.com> on 2007/06/28 10:20:26 UTC

too slow for re-parse job ..

Hi,
  Currently,I am running nutch in a single Linux box with 1G memory and one 3GHZ Intel P4 CPU. The hadoop is running in local mode.Now I am trying to reparse html pages fetched. The process is very slow,it require more than 10 days for processing nearly 20M pages. I am wondering whether the  two solutions below can improve the performance ?
1. Increase the memory size ?
2. Run the hadoop in  distributed mode,and use more than map/reduce job in one machine?
Any suggestions about improve the performance are welcome ! Thanks in advance!!  

-chee