New index representation in search results

[Chetan Sahasrabudhe <Ch...@KPITCummins.com>]

 Hello Doug,

    At our nutch setup, whenever we update the index with new index, the
search results are not getting updated accordingly.

Once we restart tomcat the new indices start reflecting in search
results.

I know many of the sites are using nutch in background and I am sure
they don't need to restart the web server to reflect index changes.

Any pointers for avoiding the restart problem ?

Regards
Chetan

-----Original Message-----
From: Doug Cutting [mailto:cutting@nutch.org] 
Sent: Tuesday, January 03, 2006 12:40 AM
To: nutch-user@lucene.apache.org
Subject: Re: Is any one able to successfully run Distributed Crawl?

Pushpesh Kr. Rajwanshi wrote:
> I want to know if anyone is able to successfully run distributed crawl

> on multiple machines involving crawling millions of pages? and how 
> hard is to do that? Do i just have to do some configuration and set up

> or do some implementations also?

I recently performed a four-level deep crawl, starting from urls in
DMOZ, limiting each level to 16M urls.  This ran on 20 machines taking
around 24 hours using about 100Mbit and retrieved around 50M pages.  I
used Nutch unmodified, specifying only a few configuration options.  So,
yes, it is possible.

Doug

Re: New index representation in search results

[Stefan Groschupf <sg...@media-style.com>]

Add a line into you shell script running your cycle that just touch  
the web.xml of the webapp context.
It's not that nice but works.
Stefan

Am 02.01.2006 um 20:13 schrieb Chetan Sahasrabudhe:

>  Hello Doug,
>
>     At our nutch setup, whenever we update the index with new  
> index, the
> search results are not getting updated accordingly.
>
> Once we restart tomcat the new indices start reflecting in search
> results.
>
> I know many of the sites are using nutch in background and I am sure
> they don't need to restart the web server to reflect index changes.
>
> Any pointers for avoiding the restart problem ?
>
> Regards
> Chetan
>
> -----Original Message-----
> From: Doug Cutting [mailto:cutting@nutch.org]
> Sent: Tuesday, January 03, 2006 12:40 AM
> To: nutch-user@lucene.apache.org
> Subject: Re: Is any one able to successfully run Distributed Crawl?
>
> Pushpesh Kr. Rajwanshi wrote:
>> I want to know if anyone is able to successfully run distributed  
>> crawl
>
>> on multiple machines involving crawling millions of pages? and how
>> hard is to do that? Do i just have to do some configuration and  
>> set up
>
>> or do some implementations also?
>
> I recently performed a four-level deep crawl, starting from urls in
> DMOZ, limiting each level to 16M urls.  This ran on 20 machines taking
> around 24 hours using about 100Mbit and retrieved around 50M pages.  I
> used Nutch unmodified, specifying only a few configuration  
> options.  So,
> yes, it is possible.
>
> Doug
>