You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Edward Drapkin (JIRA)" <ji...@apache.org> on 2011/09/15 22:03:08 UTC
[jira] [Created] (NUTCH-1113) Merging segments causes URLs to
vanish from crawldb/index?
Merging segments causes URLs to vanish from crawldb/index?
----------------------------------------------------------
Key: NUTCH-1113
URL: https://issues.apache.org/jira/browse/NUTCH-1113
Project: Nutch
Issue Type: Bug
Affects Versions: 1.3
Reporter: Edward Drapkin
When I run Nutch, I use the following steps:
nutch inject crawldb/ url.txt
repeated 3 times:
nutch generate crawldb/ segments/ -normalize
nutch fetch `ls -d segments/* | tail -1`
nutch parse `ls -d segments/* | tail -1`
nutch update crawldb `ls -d segments/* | tail -1`
nutch mergesegs merged/ -dir segments/
nutch invertlinks linkdb/ -dir merged/
nutch index index/ crawldb/ linkdb/ -dir merged/ (I forward ported the lucene indexing code from Nutch 1.1).
When I crawl with merging segments, I lose about 20% of the URLs that wind up in the index vs. when I crawl without merging the segments. Somehow the segment merger causes me to lose ~20% of my crawl database!
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1113) Merging segments causes URLs to
vanish from crawldb/index?
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105715#comment-13105715 ]
Markus Jelsma commented on NUTCH-1113:
--------------------------------------
Investigation, debug report; same stuff different words. I'll try to look into it but some more pointers would be helpful. I did once notice that merging already merged segments to an even bigger segment produced strange or zero results; it seems it's the same problem.
> Merging segments causes URLs to vanish from crawldb/index?
> ----------------------------------------------------------
>
> Key: NUTCH-1113
> URL: https://issues.apache.org/jira/browse/NUTCH-1113
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.3
> Reporter: Edward Drapkin
> Fix For: 1.4
>
> Attachments: merged_segment_output.txt, unmerged_segment_output.txt
>
>
> When I run Nutch, I use the following steps:
> nutch inject crawldb/ url.txt
> repeated 3 times:
> nutch generate crawldb/ segments/ -normalize
> nutch fetch `ls -d segments/* | tail -1`
> nutch parse `ls -d segments/* | tail -1`
> nutch update crawldb `ls -d segments/* | tail -1`
> nutch mergesegs merged/ -dir segments/
> nutch invertlinks linkdb/ -dir merged/
> nutch index index/ crawldb/ linkdb/ -dir merged/ (I forward ported the lucene indexing code from Nutch 1.1).
> When I crawl with merging segments, I lose about 20% of the URLs that wind up in the index vs. when I crawl without merging the segments. Somehow the segment merger causes me to lose ~20% of my crawl database!
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1113) Merging segments causes URLs to
vanish from crawldb/index?
Posted by "Edward Drapkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward Drapkin updated NUTCH-1113:
----------------------------------
Attachment: merged_segment_output.txt
unmerged_segment_output.txt
Output for segreader -get for a URL that disappears after merging segments.
> Merging segments causes URLs to vanish from crawldb/index?
> ----------------------------------------------------------
>
> Key: NUTCH-1113
> URL: https://issues.apache.org/jira/browse/NUTCH-1113
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.3
> Reporter: Edward Drapkin
> Attachments: merged_segment_output.txt, unmerged_segment_output.txt
>
>
> When I run Nutch, I use the following steps:
> nutch inject crawldb/ url.txt
> repeated 3 times:
> nutch generate crawldb/ segments/ -normalize
> nutch fetch `ls -d segments/* | tail -1`
> nutch parse `ls -d segments/* | tail -1`
> nutch update crawldb `ls -d segments/* | tail -1`
> nutch mergesegs merged/ -dir segments/
> nutch invertlinks linkdb/ -dir merged/
> nutch index index/ crawldb/ linkdb/ -dir merged/ (I forward ported the lucene indexing code from Nutch 1.1).
> When I crawl with merging segments, I lose about 20% of the URLs that wind up in the index vs. when I crawl without merging the segments. Somehow the segment merger causes me to lose ~20% of my crawl database!
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1113) Merging segments causes URLs to
vanish from crawldb/index?
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105661#comment-13105661 ]
Markus Jelsma commented on NUTCH-1113:
--------------------------------------
Can you rule out the indexer and see what you get when you dump the three segments and the merged segment instead?
> Merging segments causes URLs to vanish from crawldb/index?
> ----------------------------------------------------------
>
> Key: NUTCH-1113
> URL: https://issues.apache.org/jira/browse/NUTCH-1113
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.3
> Reporter: Edward Drapkin
>
> When I run Nutch, I use the following steps:
> nutch inject crawldb/ url.txt
> repeated 3 times:
> nutch generate crawldb/ segments/ -normalize
> nutch fetch `ls -d segments/* | tail -1`
> nutch parse `ls -d segments/* | tail -1`
> nutch update crawldb `ls -d segments/* | tail -1`
> nutch mergesegs merged/ -dir segments/
> nutch invertlinks linkdb/ -dir merged/
> nutch index index/ crawldb/ linkdb/ -dir merged/ (I forward ported the lucene indexing code from Nutch 1.1).
> When I crawl with merging segments, I lose about 20% of the URLs that wind up in the index vs. when I crawl without merging the segments. Somehow the segment merger causes me to lose ~20% of my crawl database!
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1113) Merging segments causes URLs to
vanish from crawldb/index?
Posted by "Edward Drapkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105714#comment-13105714 ]
Edward Drapkin commented on NUTCH-1113:
---------------------------------------
Upon further inspection, it appears that SegmentMerger makes the assumption that all values from the newest segment are preferable, so in this case, there's a crawl_fetch segment piece for this URL in two segments. In the first segment, it's marked Status 33 (fetch success) and in the second segment, it's marked Status 67 (linked), so the status 67 overwrites the status 33 crawl_fetch segment piece. From there, the URL data is excluded (correctly) from the index, because it's not marked as fetch success.
It appears that the URL gets added to later segments with status 67 because it's the location of an HTTP 301 redirect. After the URL is fetched properly the first time, a page link to http://www.wolfram.com/products/mathematica/ gets found, which redirects to the URL in question, causing that entry in the segment data. Because http://www.wolfram.com/mathematica/ has already been fetched, it won't get added with the correct status (33) in future segments.
That is to say it seems that all pages that:
A) Have already been fetched -AND-
B) Are set as the location of a redirect in subsequent iterations through the crawl process
will be "lost" after a segment merge.
> Merging segments causes URLs to vanish from crawldb/index?
> ----------------------------------------------------------
>
> Key: NUTCH-1113
> URL: https://issues.apache.org/jira/browse/NUTCH-1113
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.3
> Reporter: Edward Drapkin
> Fix For: 1.4
>
> Attachments: merged_segment_output.txt, unmerged_segment_output.txt
>
>
> When I run Nutch, I use the following steps:
> nutch inject crawldb/ url.txt
> repeated 3 times:
> nutch generate crawldb/ segments/ -normalize
> nutch fetch `ls -d segments/* | tail -1`
> nutch parse `ls -d segments/* | tail -1`
> nutch update crawldb `ls -d segments/* | tail -1`
> nutch mergesegs merged/ -dir segments/
> nutch invertlinks linkdb/ -dir merged/
> nutch index index/ crawldb/ linkdb/ -dir merged/ (I forward ported the lucene indexing code from Nutch 1.1).
> When I crawl with merging segments, I lose about 20% of the URLs that wind up in the index vs. when I crawl without merging the segments. Somehow the segment merger causes me to lose ~20% of my crawl database!
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1113) Merging segments causes URLs to
vanish from crawldb/index?
Posted by "Markus Jelsma (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1113:
---------------------------------
Fix Version/s: (was: 1.5)
1.6
20120304-push-1.6
> Merging segments causes URLs to vanish from crawldb/index?
> ----------------------------------------------------------
>
> Key: NUTCH-1113
> URL: https://issues.apache.org/jira/browse/NUTCH-1113
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.3
> Reporter: Edward Drapkin
> Fix For: 1.6
>
> Attachments: merged_segment_output.txt, unmerged_segment_output.txt
>
>
> When I run Nutch, I use the following steps:
> nutch inject crawldb/ url.txt
> repeated 3 times:
> nutch generate crawldb/ segments/ -normalize
> nutch fetch `ls -d segments/* | tail -1`
> nutch parse `ls -d segments/* | tail -1`
> nutch update crawldb `ls -d segments/* | tail -1`
> nutch mergesegs merged/ -dir segments/
> nutch invertlinks linkdb/ -dir merged/
> nutch index index/ crawldb/ linkdb/ -dir merged/ (I forward ported the lucene indexing code from Nutch 1.1).
> When I crawl with merging segments, I lose about 20% of the URLs that wind up in the index vs. when I crawl without merging the segments. Somehow the segment merger causes me to lose ~20% of my crawl database!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1113) Merging segments causes URLs to
vanish from crawldb/index?
Posted by "Edward Drapkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105704#comment-13105704 ]
Edward Drapkin commented on NUTCH-1113:
---------------------------------------
I don't have any idea what's causing this problem, so I can't provide a patch. I'll do some more investigation.
What do you mean by debug report?
> Merging segments causes URLs to vanish from crawldb/index?
> ----------------------------------------------------------
>
> Key: NUTCH-1113
> URL: https://issues.apache.org/jira/browse/NUTCH-1113
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.3
> Reporter: Edward Drapkin
> Fix For: 1.4
>
> Attachments: merged_segment_output.txt, unmerged_segment_output.txt
>
>
> When I run Nutch, I use the following steps:
> nutch inject crawldb/ url.txt
> repeated 3 times:
> nutch generate crawldb/ segments/ -normalize
> nutch fetch `ls -d segments/* | tail -1`
> nutch parse `ls -d segments/* | tail -1`
> nutch update crawldb `ls -d segments/* | tail -1`
> nutch mergesegs merged/ -dir segments/
> nutch invertlinks linkdb/ -dir merged/
> nutch index index/ crawldb/ linkdb/ -dir merged/ (I forward ported the lucene indexing code from Nutch 1.1).
> When I crawl with merging segments, I lose about 20% of the URLs that wind up in the index vs. when I crawl without merging the segments. Somehow the segment merger causes me to lose ~20% of my crawl database!
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (NUTCH-1113) Merging segments causes
URLs to vanish from crawldb/index?
Posted by "Edward Drapkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105714#comment-13105714 ]
Edward Drapkin edited comment on NUTCH-1113 at 9/15/11 9:52 PM:
----------------------------------------------------------------
Upon further inspection, it appears that SegmentMerger makes the assumption that all values from the newest segment are preferable, so in this case, there's a crawl_fetch segment piece for this URL in two segments. In the first segment, it's marked Status 33 (fetch success) and in the second segment, it's marked Status 67 (linked), so the status 67 overwrites the status 33 crawl_fetch segment piece. From there, the URL data is excluded (correctly) from the index, because it's not marked as fetch success.
It appears that the URL gets added to later segments with status 67 because it's the location of an HTTP 301 redirect. After the URL is fetched properly the first time, a page link to http://www.wolfram.com/products/mathematica/ gets found, which redirects to the URL in question, causing that entry in the segment data. Because http://www.wolfram.com/mathematica/ has already been fetched, it won't get added with the correct status (33) in future segments.
That is to say it seems that all pages that:
A) Have already been fetched **AND*
B) Are set as the location of a redirect in subsequent iterations through the crawl process
will be "lost" after a segment merge.
was (Author: edwardd):
Upon further inspection, it appears that SegmentMerger makes the assumption that all values from the newest segment are preferable, so in this case, there's a crawl_fetch segment piece for this URL in two segments. In the first segment, it's marked Status 33 (fetch success) and in the second segment, it's marked Status 67 (linked), so the status 67 overwrites the status 33 crawl_fetch segment piece. From there, the URL data is excluded (correctly) from the index, because it's not marked as fetch success.
It appears that the URL gets added to later segments with status 67 because it's the location of an HTTP 301 redirect. After the URL is fetched properly the first time, a page link to http://www.wolfram.com/products/mathematica/ gets found, which redirects to the URL in question, causing that entry in the segment data. Because http://www.wolfram.com/mathematica/ has already been fetched, it won't get added with the correct status (33) in future segments.
That is to say it seems that all pages that:
A) Have already been fetched -AND-
B) Are set as the location of a redirect in subsequent iterations through the crawl process
will be "lost" after a segment merge.
> Merging segments causes URLs to vanish from crawldb/index?
> ----------------------------------------------------------
>
> Key: NUTCH-1113
> URL: https://issues.apache.org/jira/browse/NUTCH-1113
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.3
> Reporter: Edward Drapkin
> Fix For: 1.4
>
> Attachments: merged_segment_output.txt, unmerged_segment_output.txt
>
>
> When I run Nutch, I use the following steps:
> nutch inject crawldb/ url.txt
> repeated 3 times:
> nutch generate crawldb/ segments/ -normalize
> nutch fetch `ls -d segments/* | tail -1`
> nutch parse `ls -d segments/* | tail -1`
> nutch update crawldb `ls -d segments/* | tail -1`
> nutch mergesegs merged/ -dir segments/
> nutch invertlinks linkdb/ -dir merged/
> nutch index index/ crawldb/ linkdb/ -dir merged/ (I forward ported the lucene indexing code from Nutch 1.1).
> When I crawl with merging segments, I lose about 20% of the URLs that wind up in the index vs. when I crawl without merging the segments. Somehow the segment merger causes me to lose ~20% of my crawl database!
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1113) Merging segments causes URLs to
vanish from crawldb/index?
Posted by "Lewis John McGibbney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105972#comment-13105972 ]
Lewis John McGibbney commented on NUTCH-1113:
---------------------------------------------
We have a pretty meaty JUnit test case for the segment merger, which is by far has the longest running time of any of our JUnit test classes. This would also require a rethink to accommodate any changes you have. I would therefore think first about how to re-write the JUnit stuff prior to embarking on the re-engineering of the segment merger.
> Merging segments causes URLs to vanish from crawldb/index?
> ----------------------------------------------------------
>
> Key: NUTCH-1113
> URL: https://issues.apache.org/jira/browse/NUTCH-1113
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.3
> Reporter: Edward Drapkin
> Fix For: 1.4
>
> Attachments: merged_segment_output.txt, unmerged_segment_output.txt
>
>
> When I run Nutch, I use the following steps:
> nutch inject crawldb/ url.txt
> repeated 3 times:
> nutch generate crawldb/ segments/ -normalize
> nutch fetch `ls -d segments/* | tail -1`
> nutch parse `ls -d segments/* | tail -1`
> nutch update crawldb `ls -d segments/* | tail -1`
> nutch mergesegs merged/ -dir segments/
> nutch invertlinks linkdb/ -dir merged/
> nutch index index/ crawldb/ linkdb/ -dir merged/ (I forward ported the lucene indexing code from Nutch 1.1).
> When I crawl with merging segments, I lose about 20% of the URLs that wind up in the index vs. when I crawl without merging the segments. Somehow the segment merger causes me to lose ~20% of my crawl database!
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1113) Merging segments causes URLs to
vanish from crawldb/index?
Posted by "Markus Jelsma (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1113:
---------------------------------
Fix Version/s: (was: 1.4)
1.5
> Merging segments causes URLs to vanish from crawldb/index?
> ----------------------------------------------------------
>
> Key: NUTCH-1113
> URL: https://issues.apache.org/jira/browse/NUTCH-1113
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.3
> Reporter: Edward Drapkin
> Fix For: 1.5
>
> Attachments: merged_segment_output.txt, unmerged_segment_output.txt
>
>
> When I run Nutch, I use the following steps:
> nutch inject crawldb/ url.txt
> repeated 3 times:
> nutch generate crawldb/ segments/ -normalize
> nutch fetch `ls -d segments/* | tail -1`
> nutch parse `ls -d segments/* | tail -1`
> nutch update crawldb `ls -d segments/* | tail -1`
> nutch mergesegs merged/ -dir segments/
> nutch invertlinks linkdb/ -dir merged/
> nutch index index/ crawldb/ linkdb/ -dir merged/ (I forward ported the lucene indexing code from Nutch 1.1).
> When I crawl with merging segments, I lose about 20% of the URLs that wind up in the index vs. when I crawl without merging the segments. Somehow the segment merger causes me to lose ~20% of my crawl database!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1113) Merging segments causes URLs to
vanish from crawldb/index?
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1113:
---------------------------------
Fix Version/s: 1.4
Thanks! It's marked for 1.4 now so it, at least, doesn't slip of the radar. Can you provide a patch or debug report? That would be helpful.
Thanks
> Merging segments causes URLs to vanish from crawldb/index?
> ----------------------------------------------------------
>
> Key: NUTCH-1113
> URL: https://issues.apache.org/jira/browse/NUTCH-1113
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.3
> Reporter: Edward Drapkin
> Fix For: 1.4
>
> Attachments: merged_segment_output.txt, unmerged_segment_output.txt
>
>
> When I run Nutch, I use the following steps:
> nutch inject crawldb/ url.txt
> repeated 3 times:
> nutch generate crawldb/ segments/ -normalize
> nutch fetch `ls -d segments/* | tail -1`
> nutch parse `ls -d segments/* | tail -1`
> nutch update crawldb `ls -d segments/* | tail -1`
> nutch mergesegs merged/ -dir segments/
> nutch invertlinks linkdb/ -dir merged/
> nutch index index/ crawldb/ linkdb/ -dir merged/ (I forward ported the lucene indexing code from Nutch 1.1).
> When I crawl with merging segments, I lose about 20% of the URLs that wind up in the index vs. when I crawl without merging the segments. Somehow the segment merger causes me to lose ~20% of my crawl database!
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1113) Merging segments causes URLs to
vanish from crawldb/index?
Posted by "Sebastian Nagel (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193115#comment-13193115 ]
Sebastian Nagel commented on NUTCH-1113:
----------------------------------------
I had a look at the attached segment dumps: the merged data is by far larger than the unmerged one.
And there are 739 identical linked CrawlDatum objects. Maybe this is an artifact of NUTCH-1252?
{quote}
it seems that all pages that:
A) Have already been fetched *AND
B) Are set as the location of a redirect in subsequent iterations through the crawl process
will be "lost" after a segment merge.
{quote}
I've run into the same situation.
{quote}
SegmentMerger makes the assumption that all values from the newest segment are preferable, so in this case, there's a crawl_fetch segment piece for this URL in two segments. In the first segment, it's marked Status 33 (fetch success) and in the second segment, it's marked Status 67 (linked), so the status 67 overwrites the status 33 crawl_fetch segment piece. From there, the URL data is excluded (correctly) from the index, because it's not marked as fetch success.
{quote}
I think this assumption is ok but it is necessary to preserve more than one (the latest) CrawlDatum:
# at least the latest out of {FETCH_SUCCESS, FETCH_GONE, FETCH_RETRY, FETCH_REDIR*}
# eventually the latest of FETCH_NOTMODIFIED (when re-indexing all segments IndexerMapReduce does not index documents with only a FETCH_NOTMODIFIED)
# possibly all linked CrawlDatums in crawl_fetch of the latest segment (similarily to those in crawl_parse)
> Merging segments causes URLs to vanish from crawldb/index?
> ----------------------------------------------------------
>
> Key: NUTCH-1113
> URL: https://issues.apache.org/jira/browse/NUTCH-1113
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.3
> Reporter: Edward Drapkin
> Fix For: 1.5
>
> Attachments: merged_segment_output.txt, unmerged_segment_output.txt
>
>
> When I run Nutch, I use the following steps:
> nutch inject crawldb/ url.txt
> repeated 3 times:
> nutch generate crawldb/ segments/ -normalize
> nutch fetch `ls -d segments/* | tail -1`
> nutch parse `ls -d segments/* | tail -1`
> nutch update crawldb `ls -d segments/* | tail -1`
> nutch mergesegs merged/ -dir segments/
> nutch invertlinks linkdb/ -dir merged/
> nutch index index/ crawldb/ linkdb/ -dir merged/ (I forward ported the lucene indexing code from Nutch 1.1).
> When I crawl with merging segments, I lose about 20% of the URLs that wind up in the index vs. when I crawl without merging the segments. Somehow the segment merger causes me to lose ~20% of my crawl database!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1113) Merging segments causes URLs to
vanish from crawldb/index?
Posted by "Edward Drapkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105693#comment-13105693 ]
Edward Drapkin commented on NUTCH-1113:
---------------------------------------
Using this command:
nutch readseg -get merged/20110915145111/ "http://www.wolfram.com/mathematica/" -nocontent -noparsetext -noparse
and this command:
readseg -get segments/20110915144153/ "http://www.wolfram.com/mathematica/" -nocontent -noparsetext -noparse
there are some differences...
The most obvious difference is this:
== Unmerged segment:
Crawl Fetch::
Version: 7
Status: 33 (fetch_success)
Fetch time: Thu Sep 15 14:41:57 CDT 2011
Modified time: Wed Dec 31 18:00:00 CST 1969
Retries since fetch: 0
Retry interval: 150000000 seconds (1736 days)
Score: 1.0
Signature: null
Metadata: _ngt_: 1316115711544_pst_: success(1), lastModified=0
== Merged segment:
Crawl Fetch::
Version: 7
Status: 67 (linked)
Fetch time: Thu Sep 15 14:43:30 CDT 2011
Modified time: Wed Dec 31 18:00:00 CST 1969
Retries since fetch: 0
Retry interval: 150000000 seconds (1736 days)
Score: 0.0
Signature: null
Metadata: _ngt_: 1316115784243_pst_: moved(12), lastModified=0: http://www.wolfram.com/mathematica/_repr_: http://www.wolfram.com/mathematica/
I attached the full -get output from both segments for the URL in question (it's a URL I know vanishes after merging segments).
> Merging segments causes URLs to vanish from crawldb/index?
> ----------------------------------------------------------
>
> Key: NUTCH-1113
> URL: https://issues.apache.org/jira/browse/NUTCH-1113
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.3
> Reporter: Edward Drapkin
> Attachments: merged_segment_output.txt, unmerged_segment_output.txt
>
>
> When I run Nutch, I use the following steps:
> nutch inject crawldb/ url.txt
> repeated 3 times:
> nutch generate crawldb/ segments/ -normalize
> nutch fetch `ls -d segments/* | tail -1`
> nutch parse `ls -d segments/* | tail -1`
> nutch update crawldb `ls -d segments/* | tail -1`
> nutch mergesegs merged/ -dir segments/
> nutch invertlinks linkdb/ -dir merged/
> nutch index index/ crawldb/ linkdb/ -dir merged/ (I forward ported the lucene indexing code from Nutch 1.1).
> When I crawl with merging segments, I lose about 20% of the URLs that wind up in the index vs. when I crawl without merging the segments. Somehow the segment merger causes me to lose ~20% of my crawl database!
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1113) Merging segments causes URLs to
vanish from crawldb/index?
Posted by "Edward Drapkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105758#comment-13105758 ]
Edward Drapkin commented on NUTCH-1113:
---------------------------------------
The more I look into this, the more I'm certain that the assumption made my the segment merger (lexicographically sorting the segments ascending by name and assuming the order is the order of priority) is invalid.
I added a quick check to the segment merger (on line 419 of SegmentMerger.java) and changed:
if (lastFname.compareTo(sp.segmentName) < 0) {
to:
if (lastFname.compareTo(sp.segmentName) < 0 && !(val.getStatus() == CrawlDatum.STATUS_LINKED && lastF.getStatus() == CrawlDatum.STATUS_FETCH_SUCCESS)) {
And lo and behold, the problem no longer occurs //for this case//. I'm hesitant to say that this is a fix to this bug, because I think that this is just one case where the reality of the situation is different than SegmentMerger's assumption. I can't say conclusively if there are more situations, but I have a hunch that there are. Polluting the code with incomprehensible logical checks like this isn't the solution, not unless this is the only case. That is to say that I think that SegmentMerger needs to be fundamentally rethought because it seems this basic assumption that it makes invalidates its usefulness. I'd be happy to do the rewrite (as I do think its necessary, the onus is on me to do it), but I'm not sure if I'm quite enough familiar with Nutch's segment format. If someone who is familiar with Nutch's segment format (and the SegmentMerger too, preferably) can get in touch with me and lend me their ear for questions, I'd be happy to do this.
> Merging segments causes URLs to vanish from crawldb/index?
> ----------------------------------------------------------
>
> Key: NUTCH-1113
> URL: https://issues.apache.org/jira/browse/NUTCH-1113
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.3
> Reporter: Edward Drapkin
> Fix For: 1.4
>
> Attachments: merged_segment_output.txt, unmerged_segment_output.txt
>
>
> When I run Nutch, I use the following steps:
> nutch inject crawldb/ url.txt
> repeated 3 times:
> nutch generate crawldb/ segments/ -normalize
> nutch fetch `ls -d segments/* | tail -1`
> nutch parse `ls -d segments/* | tail -1`
> nutch update crawldb `ls -d segments/* | tail -1`
> nutch mergesegs merged/ -dir segments/
> nutch invertlinks linkdb/ -dir merged/
> nutch index index/ crawldb/ linkdb/ -dir merged/ (I forward ported the lucene indexing code from Nutch 1.1).
> When I crawl with merging segments, I lose about 20% of the URLs that wind up in the index vs. when I crawl without merging the segments. Somehow the segment merger causes me to lose ~20% of my crawl database!
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira