[jira] [Updated] (NUTCH-1339) Default URL normalization rules to remove page anchors completely

["Markus Jelsma (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NUTCH-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Markus Jelsma updated NUTCH-1339:
---------------------------------

    Fix Version/s: 1.6
    
> Default URL normalization rules to remove page anchors completely
> -----------------------------------------------------------------
>
>                 Key: NUTCH-1339
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1339
>             Project: Nutch
>          Issue Type: Bug
>    Affects Versions: nutchgora, 1.6
>            Reporter: Sebastian Nagel
>             Fix For: 1.6
>
>         Attachments: NUTCH-1339-2.patch, NUTCH-1339.patch
>
>
> The default rules of URLNormalizerRegex remove the anchor up to the first
> occurrence of ? or &. The remaining part of the anchor is kept
> which may cause a large, possibly infinite number of outlinks when the same document
> fetched again and again with different URLs,
> see http://www.mail-archive.com/user%40nutch.apache.org/msg05940.html
> Parameters in inner-page anchors are a common practice in AJAX web sites.
> Currently, crawling AJAX content is not supported (NUTCH-1323).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira