[jira] [Commented] (NUTCH-710) Support for rel="canonical" attribute

["zm (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13528748#comment-13528748 ] 

zm commented on NUTCH-710:
--------------------------

I have implemented non-canonical page detection by means of modifying parse-html plugin. Though I am not that familiar with Nutch architecture and not sure if my implementation is in line with it. What I did is to have utility method boolean isCanonical(Node root, String baseUrl) which returns status of currently parsed html page: true if it is proper page, false if it is non canonical. Then in parse-html plugin's HtmlParser.getParse I call isCanonical and return from the method with ParseStatus

if(!utils.isCanonical(root, baseUrl)
 return ParseStatusUtils.getEmptyParse(9999, "Non canonical page", getConf());

Is this the right way to do it (of cause this needs to be made configurable)? If someone more familiar with Nutch would confirm or suggest more proper way I'd submit a patch. 
                
> Support for rel="canonical" attribute
> -------------------------------------
>
>                 Key: NUTCH-710
>                 URL: https://issues.apache.org/jira/browse/NUTCH-710
>             Project: Nutch
>          Issue Type: New Feature
>    Affects Versions: 1.1
>            Reporter: Frank McCown
>            Priority: Minor
>             Fix For: 1.7
>
>
> There is a the new rel="canonical" attribute which is
> now being supported by Google, Yahoo, and Live:
> http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html
> Adding support for this attribute value will potentially reduce the number of URLs crawled and indexed and reduce duplicate page content.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira