You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Stefan Groschupf (JIRA)" <ji...@apache.org> on 2006/06/02 18:36:30 UTC

[jira] Closed: (NUTCH-286) Handling common error-pages as 404

     [ http://issues.apache.org/jira/browse/NUTCH-286?page=all ]
     
Stefan Groschupf closed NUTCH-286:
----------------------------------

    Resolution: Won't Fix

I hope everybody agree with the statement: We can not detect http response codes based on responded html content.
Prune the index is a good idea to solve the problem.

> Handling common error-pages as 404
> ----------------------------------
>
>          Key: NUTCH-286
>          URL: http://issues.apache.org/jira/browse/NUTCH-286
>      Project: Nutch
>         Type: Improvement

>     Reporter: Stefan Neufeind

>
> Idea: Some pages from some software-packages/scripts report an "http 200 ok" even though a specific page could not be found. Example I just found  is:
> http://www.deteimmobilien.de/unternehmen/nbjmup;Uipnbt/IfsctuAefufjnnpcjmjfo/ef
> That's a typo3-page explaining in it's standard-layout and wording: "The requested page did not exist or was inaccessible."
> So I had the idea if somebody might create a plugin that could find commonly used formulations for "page does not exist" etc. and turn the page into a 404 before feeding them  into the nutch-index  - although the server responded with status 200 ok.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira