[jira] [Commented] (NUTCH-2065) Domain URL filter to support protocols

["Sebastian Nagel (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15068757#comment-15068757 ] 

Sebastian Nagel commented on NUTCH-2065:
----------------------------------------

* in general: wouldn't a URL normalizer be preferable? If URLs of one protocol are suppressed, missing links may get lost. Some documents of a site which uses https mostly may be referenced from few http pages only.
* before domain url filter was agnostic regarding the protocol: shouldn't this behaviour be kept in all cases, i.e., also for ftp? Almost everything now is http or https, but may we should keep the interpretation of "no protocol specified" -> "any protocol allowed".

> Domain URL filter to support protocols
> --------------------------------------
>
>                 Key: NUTCH-2065
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2065
>             Project: Nutch
>          Issue Type: Improvement
>    Affects Versions: 1.10
>            Reporter: Markus Jelsma
>         Attachments: NUTCH-2065.patch, NUTCH-2065.patch
>
>
> The filter allows all protocols for all whitelisted domains, hosts or suffixes but it usually makes little sense to index both http and https URL's of the same domain. This is not unlike the host URL filter, which prevents indexing of duplicate hosts e.g. apache.org and www.apache.org.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)