You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by Howie Wang <ho...@hotmail.com> on 2005/11/19 01:55:53 UTC

Slow updatedb?

Hi,

Recently my updatedb step has been taking a long time to
complete. There are about 7M links and 1M pages, and they
take up about 4.5GB of space. The update step seems to be
taking about 2 hours to complete.

It looked like it was limiting its memory usage to about 100mb
so I increased the io.sort.mb to 512mb.  After that updatedb
has used up to 250mb of memory, but the overall process
still takes about 2 hours.

Any tips on increasing performance? Is this about the right level
of performance for this DB size? Do I just need a faster disk?

I'm on Nutch 0.7 on Windows XP with 768MB ram, by the way.

Thanks!
Howie