You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Jed Reynolds <jr...@emediawire.com> on 2005/12/16 21:52:24 UTC

cannot pull up result for fetched document, will a merge help?

I'm doing incremental crawls against my site which has >300,000
documents in it. There are about 200-400 new documents a day. I'm
injecting these new URLs every day. I seem to have lost a day.
I tried reinjecting these URLs and recrawling them, but I still can't
pull up a result.

This is how I'm injecting and updating:

    bin/nutch inject db -urlfile $batch >> $INJECTLOG 2>&1
    bin/nutch generate db segments $batch >> $GENERATELOG 2>&1
    s1=$( ls -ltrd segments/2* \
        | awk '{FS=" "} {print $9}' \
        | tail -n1 )
    bin/nutch fetch segments/$s1 >> $FETCHLOG 2>&1
    bin/nutch updatedb db segments/$s1 >> $UPDATELOG  2>&1
    bin/nutch index segments/$s1 >> $INDEXLOG  2>&1
    bin/nutch dedup segments dedup.tmp >> $DEDUPLOG  2>&1

I can find an URL from the day's url-file ($batch from above):

$ grep -l http://www.site.com/releases/2005/12/site322438.htm \
urls.done/2005-12-*
>    urls.done/2005-12-14.0105.01

Then I find the segment(s) it got to, using the date of the segment dir:

$ bin/nutch fetchlist -local -dumpurls \
    segments/20051214010517 \
    | grep site322438
> Recno 162: http://www.site.com/releases/2005/12/site322438.htm

So I look up the record with fetchlist:

$ bin/nutch fetchlist -local -recno 162 \
    segments/20051214010517
> version: 2
> fetch: true
> page: Version: 4
> URL: http://www.site.com/releases/2005/12/site322438.htm
> ID: 7ce9582e9b9d1b3306e3ee59f50fa2da
> Next fetch: Wed Dec 21 01:05:26 PST 2005
> Retries since fetch: 0
> Retry interval: 1 days
> Num outlinks: 0
> Score: 1.0
> NextScore: 1.0
> 
> anchors: 0


So presumably its indexed and I should be able to find it (like I can
find other documents by document number), by searching for the phrase
"site322438". But I get no results.

Is this because the segment it's in isn't getting looked at?
Is there a version 1 that's taking precedence? Is this the kind of
behavior I can correct by merging my indexes? Am I on the right track?

TIA

-- 
Jed Reynolds
System Administrator, PRWeb International, Inc. 360-312-0892