You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by "Goldschmidt, Dave" <dg...@globalspec.com> on 2005/12/05 20:23:11 UTC

Speed of indexing

Hello,

 

I'm currently indexing ~50 segments, each ~2GB in size, for a total of
only ~7,000,000 pages.  From the log output, I see an index rate of ~72
records/second.  Doing the math, this is over 24 hours of time to index
these segments.

 

Does this sound slow?  If so, any suggestions as to how to tune this?
Note I'm using Nutch 0.7.1 on a Linux box with dual CPUs, 2GB of memory
and a 250GB partition to play with.

 

Thanks,

DaveG

Re: Speed of indexing

Posted by Byron Miller <by...@yahoo.com>.

Which plugins do you have enabled? Have you optimized
any of your nutch-site settings yet?

-byron

--- "Goldschmidt, Dave" <dg...@globalspec.com>
wrote:

> Hello,
> 
>  
> 
> I'm currently indexing ~50 segments, each ~2GB in
> size, for a total of
> only ~7,000,000 pages.  From the log output, I see
> an index rate of ~72
> records/second.  Doing the math, this is over 24
> hours of time to index
> these segments.
> 
>  
> 
> Does this sound slow?  If so, any suggestions as to
> how to tune this?
> Note I'm using Nutch 0.7.1 on a Linux box with dual
> CPUs, 2GB of memory
> and a 250GB partition to play with.
> 
>  
> 
> Thanks,
> 
> DaveG
> 
>  
> 
>