Growing the index : Merging vs incremental

[sprabhu_PN <sh...@pinakilabs.com>]

Currently we crawl every two days and create a new index and then merge with
earlier index. For one it takes  too long as mergesegs seems to take time
proportional to the size of both indexes combined.  Equally problematic
issue is mergesegs fail a significant portion of the time. Probability
becomes higher with size.Problems exist whether merge is done within Hadoop
or outside.   

Two questions:
(a) Has anybody been successful to do a Nutch merge predictably irrespective
of the size. Any tips.  We are trying to merge upto data for 200K url at a
time.

(b) How can we do incremental indexing, where we add data from latest crawl,
but there is only one index that keeps growing.  I saw lot of older posts
regarding incremental indexing and no clear answers.

Thanks in advance for your help.

Shreekanth 

-- 
View this message in context: http://old.nabble.com/Growing-the-index-%3A-Merging-vs-incremental-tp26228341p26228341.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: Growing the index : Merging vs incremental

[fa...@butterflycluster.net]

hi,

we are in a sort of similar situation;

So would be really happy to hear any suggestions on this.

incremental crawling doesnt seem to really work for us because it seems
the same urls are being crawled over and over (on a daily basis!);

have you tried these settings or similar?

db.fetch.schedule.class = AdaptiveFetchSchedule
db.update.additions.allowed = true
db.ignore.internal.links = false
db.ignore.external.links = true (because we are intranet only)



>
>
> Currently we crawl every two days and create a new index and then merge
> with
> earlier index. For one it takes  too long as mergesegs seems to take time
> proportional to the size of both indexes combined.  Equally problematic
> issue is mergesegs fail a significant portion of the time. Probability
> becomes higher with size.Problems exist whether merge is done within
> Hadoop
> or outside.
>
> Two questions:
> (a) Has anybody been successful to do a Nutch merge predictably
> irrespective
> of the size. Any tips.  We are trying to merge upto data for 200K url at a
> time.
>
> (b) How can we do incremental indexing, where we add data from latest
> crawl,
> but there is only one index that keeps growing.  I saw lot of older posts
> regarding incremental indexing and no clear answers.
>
> Thanks in advance for your help.
>
> Shreekanth
>
> --
> View this message in context:
> http://old.nabble.com/Growing-the-index-%3A-Merging-vs-incremental-tp26228341p26228341.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
>