You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Markus Jelsma (Created) (JIRA)" <ji...@apache.org> on 2012/03/21 22:06:37 UTC
[jira] [Created] (NUTCH-1319) HostNormalizer
HostNormalizer
--------------
Key: NUTCH-1319
URL: https://issues.apache.org/jira/browse/NUTCH-1319
Project: Nutch
Issue Type: New Feature
Reporter: Markus Jelsma
Assignee: Markus Jelsma
Fix For: 1.5
Nutch would benefit from having a host normalizer. A host normalizer maps a given host to the desired host. A basic example is to map www.apache.org to apache.org. The Apache website is one of many on the internet that has a duplicate website on the same domain just because it allows both www and non-www to return HTTP 200 and proper content.
It is also able to handle wildcards such as *.example.org to example.org if there are multiple sub domains that actually point to the same website.
Large internet crawls tend to get polluted very quickly due to these problems. It also leads to skewed scores in the webgraph as different websites link to different versions of the same duplicate website.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Created] (NUTCH-1319) HostNormalizer
Posted by Mathijs Homminga <ma...@kalooga.com>.
Hi Markus,
Right. I agree that a simple map implementation would help here.
Mathijs
On Mar 21, 2012, at 22:54 , Markus Jelsma wrote:
> Hi Mathijs,
>
> We use this in the fetcher (parse=true) and when updating the CrawlDB and with the free generator. We use it in the fetcher because we follow outlinks and make sure we follow the desired host and in the CrawlDB because there we update records for recently added host normalizer rules.
>
> It is just an URL normalizer like the others but only changes the host part. This is not covered in other standard normalizers. The BasicURLNormalizer cannot do this and the RegexURLNormalizer is far too heavy to take 20MB of expressions and harder to auto-generate. A simple map lookup is very fast.
>
> Cheers,
>
> On Wed, 21 Mar 2012 22:22:54 +0100, Mathijs Homminga <ma...@kalooga.com> wrote:
>> Hi Markus,
>>
>> How (where in the process) do you like to use this normalizer. Isn't
>> this functionality already covered by the URL normalizer(s)?
>>
>> Mathijs Homminga
>>
>> On Mar 21, 2012, at 22:06, "Markus Jelsma (Created) (JIRA)"
>> <ji...@apache.org> wrote:
>>
>>> HostNormalizer
>>> --------------
>>>
>>> Key: NUTCH-1319
>>> URL: https://issues.apache.org/jira/browse/NUTCH-1319
>>> Project: Nutch
>>> Issue Type: New Feature
>>> Reporter: Markus Jelsma
>>> Assignee: Markus Jelsma
>>> Fix For: 1.5
>>>
>>>
>>> Nutch would benefit from having a host normalizer. A host normalizer maps a given host to the desired host. A basic example is to map www.apache.org to apache.org. The Apache website is one of many on the internet that has a duplicate website on the same domain just because it allows both www and non-www to return HTTP 200 and proper content.
>>>
>>> It is also able to handle wildcards such as *.example.org to example.org if there are multiple sub domains that actually point to the same website.
>>>
>>> Large internet crawls tend to get polluted very quickly due to these problems. It also leads to skewed scores in the webgraph as different websites link to different versions of the same duplicate website.
>>>
>>> --
>>> This message is automatically generated by JIRA.
>>> If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
>>> For more information on JIRA, see: http://www.atlassian.com/software/jira
>>>
>>>
>
Re: [jira] [Created] (NUTCH-1319) HostNormalizer
Posted by Markus Jelsma <ma...@openindex.io>.
Hi Mathijs,
We use this in the fetcher (parse=true) and when updating the CrawlDB
and with the free generator. We use it in the fetcher because we follow
outlinks and make sure we follow the desired host and in the CrawlDB
because there we update records for recently added host normalizer
rules.
It is just an URL normalizer like the others but only changes the host
part. This is not covered in other standard normalizers. The
BasicURLNormalizer cannot do this and the RegexURLNormalizer is far too
heavy to take 20MB of expressions and harder to auto-generate. A simple
map lookup is very fast.
Cheers,
On Wed, 21 Mar 2012 22:22:54 +0100, Mathijs Homminga
<ma...@kalooga.com> wrote:
> Hi Markus,
>
> How (where in the process) do you like to use this normalizer. Isn't
> this functionality already covered by the URL normalizer(s)?
>
> Mathijs Homminga
>
> On Mar 21, 2012, at 22:06, "Markus Jelsma (Created) (JIRA)"
> <ji...@apache.org> wrote:
>
>> HostNormalizer
>> --------------
>>
>> Key: NUTCH-1319
>> URL:
>> https://issues.apache.org/jira/browse/NUTCH-1319
>> Project: Nutch
>> Issue Type: New Feature
>> Reporter: Markus Jelsma
>> Assignee: Markus Jelsma
>> Fix For: 1.5
>>
>>
>> Nutch would benefit from having a host normalizer. A host normalizer
>> maps a given host to the desired host. A basic example is to map
>> www.apache.org to apache.org. The Apache website is one of many on the
>> internet that has a duplicate website on the same domain just because
>> it allows both www and non-www to return HTTP 200 and proper content.
>>
>> It is also able to handle wildcards such as *.example.org to
>> example.org if there are multiple sub domains that actually point to
>> the same website.
>>
>> Large internet crawls tend to get polluted very quickly due to these
>> problems. It also leads to skewed scores in the webgraph as different
>> websites link to different versions of the same duplicate website.
>>
>> --
>> This message is automatically generated by JIRA.
>> If you think it was sent incorrectly, please contact your JIRA
>> administrators:
>> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
>> For more information on JIRA, see:
>> http://www.atlassian.com/software/jira
>>
>>
Re: [jira] [Created] (NUTCH-1319) HostNormalizer
Posted by Mathijs Homminga <ma...@kalooga.com>.
Hi Markus,
How (where in the process) do you like to use this normalizer. Isn't this functionality already covered by the URL normalizer(s)?
Mathijs Homminga
On Mar 21, 2012, at 22:06, "Markus Jelsma (Created) (JIRA)" <ji...@apache.org> wrote:
> HostNormalizer
> --------------
>
> Key: NUTCH-1319
> URL: https://issues.apache.org/jira/browse/NUTCH-1319
> Project: Nutch
> Issue Type: New Feature
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
>
> Nutch would benefit from having a host normalizer. A host normalizer maps a given host to the desired host. A basic example is to map www.apache.org to apache.org. The Apache website is one of many on the internet that has a duplicate website on the same domain just because it allows both www and non-www to return HTTP 200 and proper content.
>
> It is also able to handle wildcards such as *.example.org to example.org if there are multiple sub domains that actually point to the same website.
>
> Large internet crawls tend to get polluted very quickly due to these problems. It also leads to skewed scores in the webgraph as different websites link to different versions of the same duplicate website.
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
[jira] [Commented] (NUTCH-1319) HostNormalizer
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13295804#comment-13295804 ]
Hudson commented on NUTCH-1319:
-------------------------------
Integrated in Nutch-trunk #1869 (See [https://builds.apache.org/job/Nutch-trunk/1869/])
NUTCH-1319 HostNormalizer plugin (Revision 1349236)
Result = SUCCESS
markus : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349236
Files :
* /nutch/trunk/CHANGES.txt
* /nutch/trunk/conf/host-urlnormalizer.txt
* /nutch/trunk/src/plugin/urlnormalizer-host
* /nutch/trunk/src/plugin/urlnormalizer-host/build.xml
* /nutch/trunk/src/plugin/urlnormalizer-host/data
* /nutch/trunk/src/plugin/urlnormalizer-host/data/hosts.txt
* /nutch/trunk/src/plugin/urlnormalizer-host/ivy.xml
* /nutch/trunk/src/plugin/urlnormalizer-host/plugin.xml
* /nutch/trunk/src/plugin/urlnormalizer-host/src
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer/host
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer/host/HostURLNormalizer.java
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer/host
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer/host/TestHostURLNormalizer.java
> HostNormalizer
> --------------
>
> Key: NUTCH-1319
> URL: https://issues.apache.org/jira/browse/NUTCH-1319
> Project: Nutch
> Issue Type: New Feature
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.6
>
> Attachments: NUTCH-1319-1.5-1.patch
>
>
> Nutch would benefit from having a host normalizer. A host normalizer maps a given host to the desired host. A basic example is to map www.apache.org to apache.org. The Apache website is one of many on the internet that has a duplicate website on the same domain just because it allows both www and non-www to return HTTP 200 and proper content.
> It is also able to handle wildcards such as *.example.org to example.org if there are multiple sub domains that actually point to the same website.
> Large internet crawls tend to get polluted very quickly due to these problems. It also leads to skewed scores in the webgraph as different websites link to different versions of the same duplicate website.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1319) HostNormalizer
Posted by "Markus Jelsma (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1319:
---------------------------------
Attachment: NUTCH-1319-1.5-1.patch
Patch for 1.5.
> HostNormalizer
> --------------
>
> Key: NUTCH-1319
> URL: https://issues.apache.org/jira/browse/NUTCH-1319
> Project: Nutch
> Issue Type: New Feature
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1319-1.5-1.patch
>
>
> Nutch would benefit from having a host normalizer. A host normalizer maps a given host to the desired host. A basic example is to map www.apache.org to apache.org. The Apache website is one of many on the internet that has a duplicate website on the same domain just because it allows both www and non-www to return HTTP 200 and proper content.
> It is also able to handle wildcards such as *.example.org to example.org if there are multiple sub domains that actually point to the same website.
> Large internet crawls tend to get polluted very quickly due to these problems. It also leads to skewed scores in the webgraph as different websites link to different versions of the same duplicate website.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (NUTCH-1319) HostNormalizer
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma resolved NUTCH-1319.
----------------------------------
Resolution: Fixed
Committed for 1.6 in rev. 1349236.
> HostNormalizer
> --------------
>
> Key: NUTCH-1319
> URL: https://issues.apache.org/jira/browse/NUTCH-1319
> Project: Nutch
> Issue Type: New Feature
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.6
>
> Attachments: NUTCH-1319-1.5-1.patch
>
>
> Nutch would benefit from having a host normalizer. A host normalizer maps a given host to the desired host. A basic example is to map www.apache.org to apache.org. The Apache website is one of many on the internet that has a duplicate website on the same domain just because it allows both www and non-www to return HTTP 200 and proper content.
> It is also able to handle wildcards such as *.example.org to example.org if there are multiple sub domains that actually point to the same website.
> Large internet crawls tend to get polluted very quickly due to these problems. It also leads to skewed scores in the webgraph as different websites link to different versions of the same duplicate website.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1319) HostNormalizer
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293549#comment-13293549 ]
Hudson commented on NUTCH-1319:
-------------------------------
Integrated in nutch-trunk-maven #310 (See [https://builds.apache.org/job/nutch-trunk-maven/310/])
NUTCH-1319 HostNormalizer plugin (Revision 1349236)
Result = SUCCESS
markus :
Files :
* /nutch/trunk/CHANGES.txt
* /nutch/trunk/conf/host-urlnormalizer.txt
* /nutch/trunk/src/plugin/urlnormalizer-host
* /nutch/trunk/src/plugin/urlnormalizer-host/build.xml
* /nutch/trunk/src/plugin/urlnormalizer-host/data
* /nutch/trunk/src/plugin/urlnormalizer-host/data/hosts.txt
* /nutch/trunk/src/plugin/urlnormalizer-host/ivy.xml
* /nutch/trunk/src/plugin/urlnormalizer-host/plugin.xml
* /nutch/trunk/src/plugin/urlnormalizer-host/src
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer/host
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer/host/HostURLNormalizer.java
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer/host
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer/host/TestHostURLNormalizer.java
> HostNormalizer
> --------------
>
> Key: NUTCH-1319
> URL: https://issues.apache.org/jira/browse/NUTCH-1319
> Project: Nutch
> Issue Type: New Feature
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.6
>
> Attachments: NUTCH-1319-1.5-1.patch
>
>
> Nutch would benefit from having a host normalizer. A host normalizer maps a given host to the desired host. A basic example is to map www.apache.org to apache.org. The Apache website is one of many on the internet that has a duplicate website on the same domain just because it allows both www and non-www to return HTTP 200 and proper content.
> It is also able to handle wildcards such as *.example.org to example.org if there are multiple sub domains that actually point to the same website.
> Large internet crawls tend to get polluted very quickly due to these problems. It also leads to skewed scores in the webgraph as different websites link to different versions of the same duplicate website.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1319) HostNormalizer
Posted by "Markus Jelsma (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1319:
---------------------------------
Patch Info: Patch Available
> HostNormalizer
> --------------
>
> Key: NUTCH-1319
> URL: https://issues.apache.org/jira/browse/NUTCH-1319
> Project: Nutch
> Issue Type: New Feature
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1319-1.5-1.patch
>
>
> Nutch would benefit from having a host normalizer. A host normalizer maps a given host to the desired host. A basic example is to map www.apache.org to apache.org. The Apache website is one of many on the internet that has a duplicate website on the same domain just because it allows both www and non-www to return HTTP 200 and proper content.
> It is also able to handle wildcards such as *.example.org to example.org if there are multiple sub domains that actually point to the same website.
> Large internet crawls tend to get polluted very quickly due to these problems. It also leads to skewed scores in the webgraph as different websites link to different versions of the same duplicate website.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1319) HostNormalizer
Posted by "Markus Jelsma (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1319:
---------------------------------
Fix Version/s: (was: 1.5)
1.6
20120304-push-1.6
> HostNormalizer
> --------------
>
> Key: NUTCH-1319
> URL: https://issues.apache.org/jira/browse/NUTCH-1319
> Project: Nutch
> Issue Type: New Feature
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.6
>
> Attachments: NUTCH-1319-1.5-1.patch
>
>
> Nutch would benefit from having a host normalizer. A host normalizer maps a given host to the desired host. A basic example is to map www.apache.org to apache.org. The Apache website is one of many on the internet that has a duplicate website on the same domain just because it allows both www and non-www to return HTTP 200 and proper content.
> It is also able to handle wildcards such as *.example.org to example.org if there are multiple sub domains that actually point to the same website.
> Large internet crawls tend to get polluted very quickly due to these problems. It also leads to skewed scores in the webgraph as different websites link to different versions of the same duplicate website.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1319) HostNormalizer
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401275#comment-13401275 ]
Hudson commented on NUTCH-1319:
-------------------------------
Integrated in nutch-trunk-maven #331 (See [https://builds.apache.org/job/nutch-trunk-maven/331/])
NUTCH-1319 adding test to accomodate HostURLNormalizer (Revision 1353884)
Result = SUCCESS
markus :
Files :
* /nutch/trunk/src/test/org/apache/nutch/net/TestURLNormalizers.java
> HostNormalizer
> --------------
>
> Key: NUTCH-1319
> URL: https://issues.apache.org/jira/browse/NUTCH-1319
> Project: Nutch
> Issue Type: New Feature
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.6
>
> Attachments: NUTCH-1319-1.5-1.patch
>
>
> Nutch would benefit from having a host normalizer. A host normalizer maps a given host to the desired host. A basic example is to map www.apache.org to apache.org. The Apache website is one of many on the internet that has a duplicate website on the same domain just because it allows both www and non-www to return HTTP 200 and proper content.
> It is also able to handle wildcards such as *.example.org to example.org if there are multiple sub domains that actually point to the same website.
> Large internet crawls tend to get polluted very quickly due to these problems. It also leads to skewed scores in the webgraph as different websites link to different versions of the same duplicate website.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1319) HostNormalizer
Posted by "Markus Jelsma (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238315#comment-13238315 ]
Markus Jelsma commented on NUTCH-1319:
--------------------------------------
Objections? I'd like to resolve this issue.
> HostNormalizer
> --------------
>
> Key: NUTCH-1319
> URL: https://issues.apache.org/jira/browse/NUTCH-1319
> Project: Nutch
> Issue Type: New Feature
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1319-1.5-1.patch
>
>
> Nutch would benefit from having a host normalizer. A host normalizer maps a given host to the desired host. A basic example is to map www.apache.org to apache.org. The Apache website is one of many on the internet that has a duplicate website on the same domain just because it allows both www and non-www to return HTTP 200 and proper content.
> It is also able to handle wildcards such as *.example.org to example.org if there are multiple sub domains that actually point to the same website.
> Large internet crawls tend to get polluted very quickly due to these problems. It also leads to skewed scores in the webgraph as different websites link to different versions of the same duplicate website.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira