You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Tyrell Perera <ty...@gmail.com> on 2005/12/16 03:14:16 UTC
Live updating an intranet index
hi,
I am going to implement a Nutch based search for one of our intranet
applications. I created the index using the method described for
intranets in the tutorial.
My question is, How can i update the index live when the application
is online ? At present what i do is recreate the index somewhere else,
take the search app offline and replace the existing one with the new
one.
This is because the 'crawl' fails if i give an exixting index folder.
Am I doing something wrong ? is there any way to update this index
while the app is live ?
Regards,
Tyrell
--
=======================
http://tyrell.perera.blogspot.com
"Simply I learned about her, and ungrudgingly do I share -- her riches
I do not hide away" - The Holy Bible (Book of Wisdom 7:13)
Re: Live updating an intranet index
Posted by Tyrell Perera <ty...@gmail.com>.
Hi,
I found the script below while browsing for a solution, will this help
to keep my index current and my application aware of the updates,
without taking the application offline ?
My plan is to make this script run periodically as a cron job.
----- begin script ------
#!/bin/bash
bin/nutch generate db segments -topN 100
s=`ls -d segments/2* | tail -1`
echo Segment is $s
bin/nutch fetch $s
bin/nutch updatedb db $s
bin/nutch analyze db 2
bin/nutch index $s
bin/nutch dedup segments tmpfile
----- end script --------
Re: Live updating an intranet index
Posted by Bostjan <bg...@siol.net>.
Hi,
Yes, it gives the general idea when using MapReduce. I will think about
suitable counterpart when not using MapReduce, perheps using multiple search
servers (the problem is that for now I only have one computer available).
Thanks
Bostjan
----- Original Message -----
From: "Jack Tang" <hi...@gmail.com>
To: <nu...@lucene.apache.org>; "Bostjan" <bg...@siol.net>
Sent: Friday, December 16, 2005 2:59 PM
Subject: Re: Live updating an intranet index
Hi
Doug proposed the solution before.
http://mail-archives.apache.org/mod_mbox/lucene-nutch-dev/200511.mbox/%3c437244B7.7080702@nutch.org%3e
Hope it helps.
/Jack
On 12/16/05, Bostjan <bg...@siol.net> wrote:
> Hi,
>
> I'm using nutch 0.7.
>
> If I understend correctly, touching the web.xml of my webapp helps my
> webapp
> to reload NutchBean and thus indexes. I tried it and it works fine until I
> don't use merged indexes or even merged segments.
> When I try to use merged indexes my webapp locks indexes (I presume this
> happens as soon as NutchBean instantiates IndexSearcher). This means that
> I
> actually have to take my webapp offline in order to merge indexes. Is
> there
> any way around this other than Tyrell suggested?
>
> Regards,
>
> Bostjan
>
> ----- Original Message -----
> From: "Stefan Groschupf" <sg...@media-style.com>
> To: <nu...@lucene.apache.org>
> Sent: Friday, December 16, 2005 12:09 PM
> Subject: Re: Live updating an intranet index
>
>
> > Actually you need to touch the web.xml of your webapp.
> > Since the nutchbean has not possibilities to notice that the indexed
> > changed or a new index segment is available.
> > I'm sure a solution is coming soon until this you need to redeploy your
> > application or touch the web.xml :-/
> >
> > HTH
> > Stefan
> > Am 16.12.2005 um 03:14 schrieb Tyrell Perera:
> >
> >> hi,
> >>
> >> I am going to implement a Nutch based search for one of our intranet
> >> applications. I created the index using the method described for
> >> intranets in the tutorial.
> >>
> >> My question is, How can i update the index live when the application
> >> is online ? At present what i do is recreate the index somewhere else,
> >> take the search app offline and replace the existing one with the new
> >> one.
> >>
> >> This is because the 'crawl' fails if i give an exixting index folder.
> >>
> >> Am I doing something wrong ? is there any way to update this index
> >> while the app is live ?
> >>
> >> Regards,
> >>
> >> Tyrell
> >> --
> >>
> >> =======================
> >> http://tyrell.perera.blogspot.com
> >>
> >> "Simply I learned about her, and ungrudgingly do I share -- her riches
> >> I do not hide away" - The Holy Bible (Book of Wisdom 7:13)
> >>
> >
> >
>
>
--
Keep Discovering ... ...
http://www.jroller.com/page/jmars
Re: Live updating an intranet index
Posted by Jack Tang <hi...@gmail.com>.
Hi
Doug proposed the solution before.
http://mail-archives.apache.org/mod_mbox/lucene-nutch-dev/200511.mbox/%3c437244B7.7080702@nutch.org%3e
Hope it helps.
/Jack
On 12/16/05, Bostjan <bg...@siol.net> wrote:
> Hi,
>
> I'm using nutch 0.7.
>
> If I understend correctly, touching the web.xml of my webapp helps my webapp
> to reload NutchBean and thus indexes. I tried it and it works fine until I
> don't use merged indexes or even merged segments.
> When I try to use merged indexes my webapp locks indexes (I presume this
> happens as soon as NutchBean instantiates IndexSearcher). This means that I
> actually have to take my webapp offline in order to merge indexes. Is there
> any way around this other than Tyrell suggested?
>
> Regards,
>
> Bostjan
>
> ----- Original Message -----
> From: "Stefan Groschupf" <sg...@media-style.com>
> To: <nu...@lucene.apache.org>
> Sent: Friday, December 16, 2005 12:09 PM
> Subject: Re: Live updating an intranet index
>
>
> > Actually you need to touch the web.xml of your webapp.
> > Since the nutchbean has not possibilities to notice that the indexed
> > changed or a new index segment is available.
> > I'm sure a solution is coming soon until this you need to redeploy your
> > application or touch the web.xml :-/
> >
> > HTH
> > Stefan
> > Am 16.12.2005 um 03:14 schrieb Tyrell Perera:
> >
> >> hi,
> >>
> >> I am going to implement a Nutch based search for one of our intranet
> >> applications. I created the index using the method described for
> >> intranets in the tutorial.
> >>
> >> My question is, How can i update the index live when the application
> >> is online ? At present what i do is recreate the index somewhere else,
> >> take the search app offline and replace the existing one with the new
> >> one.
> >>
> >> This is because the 'crawl' fails if i give an exixting index folder.
> >>
> >> Am I doing something wrong ? is there any way to update this index
> >> while the app is live ?
> >>
> >> Regards,
> >>
> >> Tyrell
> >> --
> >>
> >> =======================
> >> http://tyrell.perera.blogspot.com
> >>
> >> "Simply I learned about her, and ungrudgingly do I share -- her riches
> >> I do not hide away" - The Holy Bible (Book of Wisdom 7:13)
> >>
> >
> >
>
>
--
Keep Discovering ... ...
http://www.jroller.com/page/jmars
Re: Live updating an intranet index
Posted by Bostjan <bg...@siol.net>.
Hi,
I'm using nutch 0.7.
If I understend correctly, touching the web.xml of my webapp helps my webapp
to reload NutchBean and thus indexes. I tried it and it works fine until I
don't use merged indexes or even merged segments.
When I try to use merged indexes my webapp locks indexes (I presume this
happens as soon as NutchBean instantiates IndexSearcher). This means that I
actually have to take my webapp offline in order to merge indexes. Is there
any way around this other than Tyrell suggested?
Regards,
Bostjan
----- Original Message -----
From: "Stefan Groschupf" <sg...@media-style.com>
To: <nu...@lucene.apache.org>
Sent: Friday, December 16, 2005 12:09 PM
Subject: Re: Live updating an intranet index
> Actually you need to touch the web.xml of your webapp.
> Since the nutchbean has not possibilities to notice that the indexed
> changed or a new index segment is available.
> I'm sure a solution is coming soon until this you need to redeploy your
> application or touch the web.xml :-/
>
> HTH
> Stefan
> Am 16.12.2005 um 03:14 schrieb Tyrell Perera:
>
>> hi,
>>
>> I am going to implement a Nutch based search for one of our intranet
>> applications. I created the index using the method described for
>> intranets in the tutorial.
>>
>> My question is, How can i update the index live when the application
>> is online ? At present what i do is recreate the index somewhere else,
>> take the search app offline and replace the existing one with the new
>> one.
>>
>> This is because the 'crawl' fails if i give an exixting index folder.
>>
>> Am I doing something wrong ? is there any way to update this index
>> while the app is live ?
>>
>> Regards,
>>
>> Tyrell
>> --
>>
>> =======================
>> http://tyrell.perera.blogspot.com
>>
>> "Simply I learned about her, and ungrudgingly do I share -- her riches
>> I do not hide away" - The Holy Bible (Book of Wisdom 7:13)
>>
>
>
Re: Live updating an intranet index
Posted by Stefan Groschupf <sg...@media-style.com>.
Actually you need to touch the web.xml of your webapp.
Since the nutchbean has not possibilities to notice that the indexed
changed or a new index segment is available.
I'm sure a solution is coming soon until this you need to redeploy
your application or touch the web.xml :-/
HTH
Stefan
Am 16.12.2005 um 03:14 schrieb Tyrell Perera:
> hi,
>
> I am going to implement a Nutch based search for one of our intranet
> applications. I created the index using the method described for
> intranets in the tutorial.
>
> My question is, How can i update the index live when the application
> is online ? At present what i do is recreate the index somewhere else,
> take the search app offline and replace the existing one with the new
> one.
>
> This is because the 'crawl' fails if i give an exixting index folder.
>
> Am I doing something wrong ? is there any way to update this index
> while the app is live ?
>
> Regards,
>
> Tyrell
> --
>
> =======================
> http://tyrell.perera.blogspot.com
>
> "Simply I learned about her, and ungrudgingly do I share -- her riches
> I do not hide away" - The Holy Bible (Book of Wisdom 7:13)
>