You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Mike Alulin <mi...@yahoo.com> on 2006/01/18 04:21:21 UTC
HTTP 404
What is the best way to delete 404 pages from the index files? It seems that the fetcher saves info about 404 pages, however when Nutch builds the segment index it does not include that data. As a result the searcher returns the document link from an older version of the index file, althouth the document does not exist anymore.
SegmentMergeTool does handle this, however it is to slow too use too often.
---------------------------------
Yahoo! Photos
Got holiday prints? See all the ways to get quality prints in your hands ASAP.
Re: HTTP 404 bug?
Posted by Mike Alulin <mi...@yahoo.com>.
OK. I made it work as I wanted to. The following modifications were made in the IndexSegment class:
1. Delete the following lines
if (!fetcherOutput.getProtocolStatus().isSuccess()) {
continue;
}
2. Replace
doc = IndexingFilters.filter(doc, parse, fetcherOutput);
with
if (fetcherOutput.getProtocolStatus().isSuccess())
doc = IndexingFilters.filter(doc, parse, fetcherOutput);
else
doc.add(Field.UnIndexed("url", fetcherOutput.getUrl().toString()));
It would be great if Nutch had a config file option that allows switch to this mode. It is very usefull for incremental crawling, when one knows that documents were updated/added/deleted.
Mike Alulin <mi...@yahoo.com> wrote:
Here is some extra info. Tested with Nutch version 0.71
The IndexSegment class checks the HTTP status of the response and if it is not 200 it does not save any info to the index. As a result the DeleteDuplicates class does not know that the page has been deleted recently and leave it in the index.
Is this how Nutch should work? If yes then how can it remove 404 pages?
Mike Alulin wrote:
What is the best way to delete 404 pages from the index files? It seems that the fetcher saves info about 404 pages, however when Nutch builds the segment index it does not include that data. As a result the searcher returns the document link from an older version of the index file, althouth the document does not exist anymore.
SegmentMergeTool does handle this, however it is to slow too use too often.
---------------------------------
Yahoo! Photos
Got holiday prints? See all the ways to get quality prints in your hands ASAP.
---------------------------------
Yahoo! Photos
Got holiday prints? See all the ways to get quality prints in your hands ASAP.
Re: HTTP 404 bug?
Posted by Mike Alulin <mi...@yahoo.com>.
Here is some extra info. Tested with Nutch version 0.71
The IndexSegment class checks the HTTP status of the response and if it is not 200 it does not save any info to the index. As a result the DeleteDuplicates class does not know that the page has been deleted recently and leave it in the index.
Is this how Nutch should work? If yes then how can it remove 404 pages?
Mike Alulin <mi...@yahoo.com> wrote:
What is the best way to delete 404 pages from the index files? It seems that the fetcher saves info about 404 pages, however when Nutch builds the segment index it does not include that data. As a result the searcher returns the document link from an older version of the index file, althouth the document does not exist anymore.
SegmentMergeTool does handle this, however it is to slow too use too often.
---------------------------------
Yahoo! Photos
Got holiday prints? See all the ways to get quality prints in your hands ASAP.