[jira] [Updated] (NUTCH-1614) Plugin to exclude URLs matching regex list from indexing - to enable crawl but do not index

["Riyaz Shaik (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NUTCH-1614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Riyaz Shaik updated NUTCH-1614:
-------------------------------

    Attachment: IndexerJob.java
                RegexUtil.java

> Plugin to exclude URLs matching regex list from indexing - to enable crawl but do not index
> -------------------------------------------------------------------------------------------
>
>                 Key: NUTCH-1614
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1614
>             Project: Nutch
>          Issue Type: Improvement
>          Components: indexer
>    Affects Versions: 2.2.1
>            Reporter: Brian
>            Priority: Minor
>              Labels: plugin
>         Attachments: IndexerJob.java, NUTCH-1614.patch, RegexUtil.java
>
>
> Some pages we need to crawl (such as some main pages and different views of a main page) to get all the other pages, but we don't want to index those pages themselves.  Therefore we cannot use the url filter approach.
> This plugin uses a file containing regex strings (see included sample file).  If one of the regex strings matches with an entire URL, that URL will be excluded form indexing.
> The file to use is specified by the following property in nutch-site.xml:
> <property>
>         <name>indexer.url.filter.exclude.regex.file</name>
>         <value>regex-indexer-exclude-urls.txt</value>
>         <description>
>             Holds the file name containing the regex strings.  Any URL matching one of these strings will be excluded from indexing. 
>             "#" indicates a comment line and will be ignored.
>         </description>
> </property>



--
This message was sent by Atlassian JIRA
(v6.2#6252)