Clustering Index job

["R.Mayoran" <ma...@team-lab.com>]

Hi,

I need to index about 100million files.

Is it possible to cluster this job?

Are there any sugestions to increase the speed of indexing?

Thank you in advance.

Mayu.

Re: Clustering Index job

[Stefan Groschupf <sg...@media-style.com>]

Yes, you need to use map reduce on several boxes.
Anyway 100 mio files will also work on powerful box.
There are some configuration values in the nutch-default.xml that can  
improve indexing speed.


Am 28.12.2005 um 09:56 schrieb R.Mayoran:

> Hi,
>
> I need to index about 100million files.
>
> Is it possible to cluster this job?
>
> Are there any sugestions to increase the speed of indexing?
>
> Thank you in advance.
>
> Mayu.
>
>

Re: Clustering Index job

[Byron Miller <by...@yahoo.com>]

Check the list for my earlier discussions. There are
tweaks you can do to enhance the performance if you
have available memory resources.

How large are your segments that you are indexing?
what file system do you use? what OS /JVM are you
building your index on?

-byron

--- "R.Mayoran" <ma...@team-lab.com> wrote:

> Hi,
> 
> I need to index about 100million files.
> 
> Is it possible to cluster this job?
> 
> Are there any sugestions to increase the speed of
> indexing?
> 
> Thank you in advance.
> 
> Mayu.
> 
>