You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nutch.apache.org by "Markus Jelsma (Closed) (JIRA)" <ji...@apache.org> on 2011/12/20 12:31:34 UTC

[jira] [Closed] (NUTCH-1043) Add pattern for filtering .js in default url filters

     [ https://issues.apache.org/jira/browse/NUTCH-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Markus Jelsma closed NUTCH-1043.
--------------------------------


Bulk close of resolved issues of 1.4. bulkclose-1.4-20111220
                
> Add pattern for filtering .js in default url filters
> ----------------------------------------------------
>
>                 Key: NUTCH-1043
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1043
>             Project: Nutch
>          Issue Type: Task
>    Affects Versions: 1.4, nutchgora
>            Reporter: Julien Nioche
>            Priority: Minor
>             Fix For: 1.4, nutchgora
>
>         Attachments: NUTCH-1043.patch
>
>
> The Javascript parser is not used by default as it is extremely noisy, however the default URL filters do not filter out URLs ending in .js and the default parser (Tika) can't parse them. In a nutshell we are fetching URLS that we know can't be parsed.
> I suggest that we add a regex to the default URL filters. If people are interested in fetching and parsing .js files they can activate the plugin in their conf and remove the regex in the URL filters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira