[jira] [Commented] (NUTCH-1113) Merging segments causes URLs to vanish from crawldb/index?

["Sebastian Nagel (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13907160#comment-13907160 ] 

Sebastian Nagel commented on NUTCH-1113:
----------------------------------------

Hi [~markus17], tried test data from NUTCH-1706 with the latest patch (2014-01-22) applied:
{code}
# 4. merge segments
% bin/nutch mergesegs  merged  crawl/segments/20131115203640/   \
    crawl/segments/20131115203847/   \
    crawl/segments/20140217140849/

# 5. index merged segment
% bin/nutch index -Dplugin.includes="index-basic|indexer-dummy" -Ddummy.path=index2014_merged.txt crawl/crawldb  \
    merged/*  -deleteGone

# 6. diff indexes (cf. NUTCH-1706): all documents with status fetch_not_modified in the latest of the merged segments are lost!
% diff index2013.txt index2014_merged.txt 
9d8
< add   http://nutch.apache.org/
26d24
< add   http://tika.apache.org/
{code}

In addtion to fetch_retry, we also need to exclude fetch_notmodified from merging.

> Merging segments causes URLs to vanish from crawldb/index?
> ----------------------------------------------------------
>
>                 Key: NUTCH-1113
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1113
>             Project: Nutch
>          Issue Type: Bug
>    Affects Versions: 1.3
>            Reporter: Edward Drapkin
>            Priority: Blocker
>             Fix For: 1.9
>
>         Attachments: NUTCH-1113-junit.patch, NUTCH-1113-junit.patch, NUTCH-1113-junit.patch, NUTCH-1113-junit.patch, NUTCH-1113-junit.patch, NUTCH-1113-junit.patch, NUTCH-1113-trunk.patch, merged_segment_output.txt, unmerged_segment_output.txt
>
>
> When I run Nutch, I use the following steps:
> nutch inject crawldb/ url.txt
> repeated 3 times:
> nutch generate crawldb/ segments/ -normalize
> nutch fetch `ls -d segments/* | tail -1`
> nutch parse `ls -d segments/* | tail -1`
> nutch update crawldb `ls -d segments/* | tail -1`
> nutch mergesegs merged/ -dir segments/
> nutch invertlinks linkdb/ -dir merged/
> nutch index index/ crawldb/ linkdb/ -dir merged/ (I forward ported the lucene indexing code from Nutch 1.1).
> When I crawl with merging segments, I lose about 20% of the URLs that wind up in the index vs. when I crawl without merging the segments.  Somehow the segment merger causes me to lose ~20% of my crawl database!



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)