You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Markus Jelsma (JIRA)" <ji...@apache.org> on 2014/09/09 14:51:28 UTC
[jira] [Comment Edited] (NUTCH-1838) Host based regex and automaton
filtering
[ https://issues.apache.org/jira/browse/NUTCH-1838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126939#comment-14126939 ]
Markus Jelsma edited comment on NUTCH-1838 at 9/9/14 12:51 PM:
---------------------------------------------------------------
{code}String != String{code} obviously doesn't work, here's a proper patch!
was (Author: markus17):
String != String obviously doesn't work, here's a proper patch!
> Host based regex and automaton filtering
> ----------------------------------------
>
> Key: NUTCH-1838
> URL: https://issues.apache.org/jira/browse/NUTCH-1838
> Project: Nutch
> Issue Type: New Feature
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Attachments: NUTCH-1838.patch, NUTCH-1838.patch
>
>
> Both regex and automaton filter pass all URL's through all rules although this makes little sense if you have a lot of generated rules for many different hosts. This patch allows the users to configure specific rules for a specific host only, making filtering much more efficient.
> Each rule has an optional host field, the filter is applied for rules that have no host and for URL's that match the rule's host name.
> The following line enables host specific rules:
> {code}
> > www.example.org
> {code}
> The following line disables/resets it again:
> {code}
> <
> {code}
> full example:
> {code}
> -some generic filter
> +another generic filter
> > www.example.org
> -rule only applied to URL's of www.example.org
> +another rule only applied to URL's of www.example.org
> > www.apache.org
> -rule only applied to URL's of www.apache.org
> +another rule only applied to URL's of www.apache.org
> <
> -more generic rules
> +and another one
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)