You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Benjamin Higgins <bh...@gmail.com> on 2006/08/02 01:22:19 UTC
Questions about (re)crawling
Hello,
Is it neccesary or desirable to run updatesegs and/or merge for a
single-machine setup that crawls ~1 million pages?
I ask because it appears that the 'crawl' tool, specifically for intranets,
runs these commands, but they aren't included in the whole-web instructions
in the tutorial.
Also, can Nutch still service search queries while it is going through the
whole generate, fetch, updatedb, index, dedup process? At what point do the
new segments become searchable -- right after indexing?
Thanks!
Ben
Re: Questions about (re)crawling
Posted by Benjamin Higgins <bh...@gmail.com>.
On 8/1/06, Matthew Holt <mh...@redhat.com> wrote:
>
>
>
> Benjamin Higgins wrote:
> > Hello,
> >
> > Is it neccesary or desirable to run updatesegs and/or merge for a
> > single-machine setup that crawls ~1 million pages?
> >
>
> Merge you need too i know.. you have to sync the segs with the crawldb
> if that is what you are referring to.
I just mean that the 'manual' instructions leave these steps out, but
searching seems to work fine anyway. Why is that?
Ben
Re: Questions about (re)crawling
Posted by Matthew Holt <mh...@redhat.com>.
If you look at my script.. the way I accomplished this without
restarting tomcat was by touching the web.xml file within the servlet
nutch dir
Benjamin Higgins wrote:
> On 8/1/06, Matthew Holt <mh...@redhat.com> wrote:
>>
>>
>> > Also, can Nutch still service search queries while it is going through
>> > the
>> > whole generate, fetch, updatedb, index, dedup process? At what point
>> > do the
>> > new segments become searchable -- right after indexing?
>> >
>> Yes.. however the new segments only become searchable once the recrawl
>> process is done (indexes are merged i believe) and the tomcat instance
>> reloads the webapp.
>>
>
> Ah - so Tomcat *must* be reloaded? I was hoping to avoid that. Oh well.
>
> Ben
>
Re: Questions about (re)crawling
Posted by Benjamin Higgins <bh...@gmail.com>.
On 8/1/06, Matthew Holt <mh...@redhat.com> wrote:
>
>
> > Also, can Nutch still service search queries while it is going through
> > the
> > whole generate, fetch, updatedb, index, dedup process? At what point
> > do the
> > new segments become searchable -- right after indexing?
> >
> Yes.. however the new segments only become searchable once the recrawl
> process is done (indexes are merged i believe) and the tomcat instance
> reloads the webapp.
>
Ah - so Tomcat *must* be reloaded? I was hoping to avoid that. Oh well.
Ben
Re: Questions about (re)crawling
Posted by Matthew Holt <mh...@redhat.com>.
Benjamin Higgins wrote:
> Hello,
>
> Is it neccesary or desirable to run updatesegs and/or merge for a
> single-machine setup that crawls ~1 million pages?
>
Merge you need too i know.. you have to sync the segs with the crawldb
if that is what you are referring to.
> I ask because it appears that the 'crawl' tool, specifically for
> intranets,
> runs these commands, but they aren't included in the whole-web
> instructions
> in the tutorial.
>
> Also, can Nutch still service search queries while it is going through
> the
> whole generate, fetch, updatedb, index, dedup process? At what point
> do the
> new segments become searchable -- right after indexing?
>
Yes.. however the new segments only become searchable once the recrawl
process is done (indexes are merged i believe) and the tomcat instance
reloads the webapp.
> Thanks!
>
> Ben
>