[jira] [Commented] (NUTCH-2859) urlnormalizer-protocol: allow to normalize domains

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17310695#comment-17310695 ] 

ASF GitHub Bot commented on NUTCH-2859:
---------------------------------------

sebastian-nagel opened a new pull request #576:
URL: https://github.com/apache/nutch/pull/576


   (note: to avoid merge conflicts, this PR includes #575)
   
   In order to address NUTCH-2859 host names starting with `*.` (in the config file `protocols.txt` or in the string rules) are matched as suffixes: `*.example.org` matches `example.org`, `www.example.org`, `www.subdomain.example.org`, etc.
   
   Additional improvements:
   - allow to read config file protocols.txt from hdfs:// or any file system supported by Hadoop - useful if the list of host or domains requiring normalization is large or changes often
   - add Javadoc package documentation
   - document configuration properties in nutch-default.xml
   - reduce the memory footprint by deduplicating protocol strings, so that same protocol values are references to same objects


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> urlnormalizer-protocol: allow to normalize domains
> --------------------------------------------------
>
>                 Key: NUTCH-2859
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2859
>             Project: Nutch
>          Issue Type: Improvement
>          Components: plugin, urlnormalizer
>    Affects Versions: 1.18
>            Reporter: Sebastian Nagel
>            Assignee: Sebastian Nagel
>            Priority: Major
>             Fix For: 1.19
>
>
> The plugin urlnormalizer-protocol normalizes the URL protocol/scheme for a given list of hosts to the desired "normal" protocol (usually one of http or https). It would be handy to allow to specify domain names as well, so that all hosts/subdomains in a given domain are normalized.
> In order to stay backward-compatible this could be done by matching {{*.example.org}} as a pattern for all hosts or subdomains of the domain {{example.org}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)