[jira] Commented: (NUTCH-153) TextParser is only supposed to parse plain text, but if given postscript, it can take hours and then fail

["Jerome Charron (JIRA)" <ji...@apache.org>]

    [ http://issues.apache.org/jira/browse/NUTCH-153?page=comments#action_12361887 ] 

Jerome Charron commented on NUTCH-153:
--------------------------------------

What you call the white list is in fact the actual parse-plugins.xml file:
If you remove the default mapping, then unknown file types will not be parsed.
Moreover, in a previous discussion, Andrzej spoke about committing a strings command line like parser that is able to extracts textual content from any file type: It is really the good candidate for the default parser, instead of the text parser.



> TextParser is only supposed to parse plain text, but if given postscript, it can take hours and then fail
> ---------------------------------------------------------------------------------------------------------
>
>          Key: NUTCH-153
>          URL: http://issues.apache.org/jira/browse/NUTCH-153
>      Project: Nutch
>         Type: Bug
>   Components: fetcher
>     Versions: 0.8-dev
>  Environment: all
>     Reporter: Paul Baclace
>  Attachments: TextParser.java.patch
>
> If TextParser is given postscript, it can take hours and then fail.  This can be avoided with careful configuration, but if the server MIME type is wrong and the basename of the URL has no "file extension", then the this parser will take a long time and fail every time.
> Analysis: The real problem is OutlinkExtractor.java as reported with bug NUTCH-150, but the problem cannot be entirely addressed with that patch since the first call to reg expr match() can take a long time, despite quantifier limits.  
> Suggested fix: Reject files with "%!PS-Adobe" in the first 40 characters of the file.
> Actual experience has shown that for safety and fail-safe reasons, it is worth protecting against GIGO directly in TextParse for this case, even though the suggested fix is not a general solution.  (A general solution would be a timeout on match().)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira