The right strategy for big crawls and indexes

[nutuser2056 <nu...@log1.net>]

Dear All,
 We need to crawl a number of sites which will amount to around 2
million pages and several GB of data. I know this is well within the
ability of Nutch, but I a question about what is the right strategy to
pursue to be able to crawl index with the least amount of bother. 

Obviously I would need a single database create several manageable sized
segments and run the fetcher on each one, and subsequently update the
database, no problem so far. But what would then be the right strategy
to get an index which could be searched? 

Should I :

1) merge all the segments and then index them, or 
2) Should I index each segment individually and then merge the indexes,
keeping the segments separate. Or 
3) Should I index each segment separately, and keep both segments and
indexes separate, and search across multiple indexes (but I have heard
there are issues with the ranking) 

Please let me know your views!!

Thanks a lot!!
Regards,

Re: The right strategy for big crawls and indexes

[Stefan Groschupf <sg...@media-style.com>]

I found it hand to have several segments.
Anyway just try and make experience with your use-case in general there 
is no best practice only different use cases.


HTH
Stefan

Am 09.03.2005 um 19:54 schrieb nutuser2056:

> Dear All,
>  We need to crawl a number of sites which will amount to around 2
> million pages and several GB of data. I know this is well within the
> ability of Nutch, but I a question about what is the right strategy to
> pursue to be able to crawl index with the least amount of bother.
>
> Obviously I would need a single database create several manageable 
> sized
> segments and run the fetcher on each one, and subsequently update the
> database, no problem so far. But what would then be the right strategy
> to get an index which could be searched?
>
> Should I :
>
> 1) merge all the segments and then index them, or
> 2) Should I index each segment individually and then merge the indexes,
> keeping the segments separate. Or
> 3) Should I index each segment separately, and keep both segments and
> indexes separate, and search across multiple indexes (but I have heard
> there are issues with the ranking)
>
> Please let me know your views!!
>
> Thanks a lot!!
> Regards,
>
>
>
>
---------------------------------------------------------------
company:		http://www.media-style.com
forum:		http://www.text-mining.org
blog:			http://www.find23.net

Re: The right strategy for big crawls and indexes

[Luke Baker <lu...@gospelcom.net>]

On 03/09/2005 01:54 PM, nutuser2056 wrote:
[snip]
> Should I :
> 
> 1) merge all the segments and then index them, or 
> 2) Should I index each segment individually and then merge the indexes,
> keeping the segments separate. Or 
> 3) Should I index each segment separately, and keep both segments and
> indexes separate, and search across multiple indexes (but I have heard
> there are issues with the ranking) 

Option #3 is not really that great.  You get better performance with a 
merged index.  Option #1 would be more work with having to merge the 
segments, and I'm not sure that there is a real advantage to doing that 
over option #2.  Option #2 is what most people do.

Luke