You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Markus Jelsma (JIRA)" <ji...@apache.org> on 2016/02/23 13:46:18 UTC

[jira] [Comment Edited] (NUTCH-2227) RegexParseFilter

    [ https://issues.apache.org/jira/browse/NUTCH-2227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15158808#comment-15158808 ] 

Markus Jelsma edited comment on NUTCH-2227 at 2/23/16 12:45 PM:
----------------------------------------------------------------

Updated patch. It now includes package-info.java. Will commit shortly unless objections.


was (Author: markus17):
Updated patch. It now includes package-info.java. Will commit shortly

> RegexParseFilter
> ----------------
>
>                 Key: NUTCH-2227
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2227
>             Project: Nutch
>          Issue Type: New Feature
>          Components: parser
>    Affects Versions: 1.11
>            Reporter: Markus Jelsma
>            Assignee: Markus Jelsma
>             Fix For: 1.12
>
>         Attachments: NUTCH-2227.patch, NUTCH-2227.patch, NUTCH-2227.patch, NUTCH-2227.patch
>
>
> A parse filter that takes a regex and a field name. If regex matches via matcher.find() on the HTML. The field name is set to true in the CrawlDatum's metadata.
> Combined with the HostDB, it is easy to get a list of hosts that match some regex criteria.
> {code}
> # Example configuration file for parsefilter-regex
> #
> # Parse metadata field <name> is set to true if the HTML matches the regex. The
> # source can either be html or text. If source is html, the regex is applied to
> # the entire HTML tree. If source is text, the regex is applied to the
> # extracted text.
> #
> # format: <name>\t<source>\t<regex>\n
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)