[jira] [Commented] (NUTCH-1300) Indexer to normalize URL's

["Sebastian Nagel (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-1300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225174#comment-13225174 ] 

Sebastian Nagel commented on NUTCH-1300:
----------------------------------------

+1
* effective fix for a serious problem: long running continuous crawls require adjustments of the normalization rules quite often
* tested (with 1.4): costs (time spent for extra normalization) are ok compared to the benefit

Two suggestions:
# Does a URLNormalizer scope "index" make sense? E.g., if only outlinks are normalized and default rules are empty, the scope "index" may use the same rules as scope "outlink".
# Wouldn't commandline options for solrindex be nice? Most other tools (generate, updatedb, invertlinks) have options such as -filter / -norm / -noNorm.
                
> Indexer to normalize URL's
> --------------------------
>
>                 Key: NUTCH-1300
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1300
>             Project: Nutch
>          Issue Type: New Feature
>          Components: indexer
>            Reporter: Markus Jelsma
>            Assignee: Markus Jelsma
>            Priority: Minor
>             Fix For: 1.5
>
>         Attachments: NUTCH-1300-1.5-1.patch
>
>
> Indexers should be able to normalize URL's. This is useful when a new normalizer is applied to the entire CrawlDB. Without it, some or all records in a segment cannot be indexed at all.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira