[jira] Updated: (NUTCH-505) Outlink urls should be validated

["Doğacan Güney (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NUTCH-505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doğacan Güney updated NUTCH-505:
--------------------------------

    Attachment: NUTCH-505_draft.patch

Initial draft patch. 

* Uses UrlValidator class from apache commons validator.
* ParseOutputFormat first checks if an outlink is valid. If it is, then it runs normalizers and urlfilters on url.

This patch is tested very lightly, so it probably doesn't work great yet. Comments, reviews, suggestions are welcome.

> Outlink urls should be validated
> --------------------------------
>
>                 Key: NUTCH-505
>                 URL: https://issues.apache.org/jira/browse/NUTCH-505
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Doğacan Güney
>            Priority: Minor
>         Attachments: NUTCH-505_draft.patch
>
>
> See discussion here:
> http://www.nabble.com/fetching-http%3A--www.variety.com-%3C-div%3E%3C-a%3E-tf3961692.html
> Parse plugins may extract garbage urls from pages. We need a url validation system that tests these urls and filters out garbage.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.