You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Rod Taylor (JIRA)" <ji...@apache.org> on 2006/04/24 05:38:05 UTC
[jira] Created: (NUTCH-253) Normalize Host during Generate
Normalize Host during Generate
------------------------------
Key: NUTCH-253
URL: http://issues.apache.org/jira/browse/NUTCH-253
Project: Nutch
Type: New Feature
Components: fetcher
Versions: 0.8-dev
Reporter: Rod Taylor
Extend URL Normalizer to allow for normalizion of the Hostname during Generate. By default no rules are applied.
In short, this allows foo.bar.com, bif.baz.bar.com and bar.com to be counted as being the same for generate.max.per.host if an appropriate regex is used.
Add "urlnormalizer-regex" to plugin.includes in nutch-site.xml in order to enable it.
Since several modules now extend the urlnormalizer base we use a "scope" parameter within plugin.xml to allow differentiation between the various urlnormalizer modules to select the right module for Generate.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (NUTCH-253) Normalize Host during Generate
Posted by "Rod Taylor (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-253?page=all ]
Rod Taylor updated NUTCH-253:
-----------------------------
Attachment: nutch_hostnormalize.patch
> Normalize Host during Generate
> ------------------------------
>
> Key: NUTCH-253
> URL: http://issues.apache.org/jira/browse/NUTCH-253
> Project: Nutch
> Type: New Feature
> Components: fetcher
> Versions: 0.8-dev
> Reporter: Rod Taylor
> Attachments: nutch_hostnormalize.patch
>
> Extend URL Normalizer to allow for normalizion of the Hostname during Generate. By default no rules are applied.
> In short, this allows foo.bar.com, bif.baz.bar.com and bar.com to be counted as being the same for generate.max.per.host if an appropriate regex is used.
> Add "urlnormalizer-regex" to plugin.includes in nutch-site.xml in order to enable it.
> Since several modules now extend the urlnormalizer base we use a "scope" parameter within plugin.xml to allow differentiation between the various urlnormalizer modules to select the right module for Generate.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Closed: (NUTCH-253) Normalize Host during Generate
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-253?page=all ]
Andrzej Bialecki closed NUTCH-253.
-----------------------------------
Fix Version/s: 0.9.0
Resolution: Fixed
This issue is fixed as a part of NUTCH-365 in trunk. Changes were too intrusive to be ported to branch-0.8, although the patch in NUTCH-365 should apply more or less cleanly.
> Normalize Host during Generate
> ------------------------------
>
> Key: NUTCH-253
> URL: http://issues.apache.org/jira/browse/NUTCH-253
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher
> Affects Versions: 0.8
> Reporter: Rod Taylor
> Fix For: 0.9.0
>
> Attachments: nutch_hostnormalize.patch
>
>
> Extend URL Normalizer to allow for normalizion of the Hostname during Generate. By default no rules are applied.
> In short, this allows foo.bar.com, bif.baz.bar.com and bar.com to be counted as being the same for generate.max.per.host if an appropriate regex is used.
> Add "urlnormalizer-regex" to plugin.includes in nutch-site.xml in order to enable it.
> Since several modules now extend the urlnormalizer base we use a "scope" parameter within plugin.xml to allow differentiation between the various urlnormalizer modules to select the right module for Generate.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira