You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Jed Reynolds <jr...@emediawire.com> on 2005/12/16 21:52:24 UTC
cannot pull up result for fetched document, will a merge help?
I'm doing incremental crawls against my site which has >300,000
documents in it. There are about 200-400 new documents a day. I'm
injecting these new URLs every day. I seem to have lost a day.
I tried reinjecting these URLs and recrawling them, but I still can't
pull up a result.
This is how I'm injecting and updating:
bin/nutch inject db -urlfile $batch >> $INJECTLOG 2>&1
bin/nutch generate db segments $batch >> $GENERATELOG 2>&1
s1=$( ls -ltrd segments/2* \
| awk '{FS=" "} {print $9}' \
| tail -n1 )
bin/nutch fetch segments/$s1 >> $FETCHLOG 2>&1
bin/nutch updatedb db segments/$s1 >> $UPDATELOG 2>&1
bin/nutch index segments/$s1 >> $INDEXLOG 2>&1
bin/nutch dedup segments dedup.tmp >> $DEDUPLOG 2>&1
I can find an URL from the day's url-file ($batch from above):
$ grep -l http://www.site.com/releases/2005/12/site322438.htm \
urls.done/2005-12-*
> urls.done/2005-12-14.0105.01
Then I find the segment(s) it got to, using the date of the segment dir:
$ bin/nutch fetchlist -local -dumpurls \
segments/20051214010517 \
| grep site322438
> Recno 162: http://www.site.com/releases/2005/12/site322438.htm
So I look up the record with fetchlist:
$ bin/nutch fetchlist -local -recno 162 \
segments/20051214010517
> version: 2
> fetch: true
> page: Version: 4
> URL: http://www.site.com/releases/2005/12/site322438.htm
> ID: 7ce9582e9b9d1b3306e3ee59f50fa2da
> Next fetch: Wed Dec 21 01:05:26 PST 2005
> Retries since fetch: 0
> Retry interval: 1 days
> Num outlinks: 0
> Score: 1.0
> NextScore: 1.0
>
> anchors: 0
So presumably its indexed and I should be able to find it (like I can
find other documents by document number), by searching for the phrase
"site322438". But I get no results.
Is this because the segment it's in isn't getting looked at?
Is there a version 1 that's taking precedence? Is this the kind of
behavior I can correct by merging my indexes? Am I on the right track?
TIA
--
Jed Reynolds
System Administrator, PRWeb International, Inc. 360-312-0892