[jira] [Closed] (NUTCH-363) Fetcher normalizes everything at least twice

["Markus Jelsma (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NUTCH-363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Markus Jelsma closed NUTCH-363.
-------------------------------


Bulk close of resolved issues:
http://www.lucidimagination.com/search/document/2738eeb014805854/clean_up_open_legacy_issues_in_jira

> Fetcher normalizes everything at least twice
> --------------------------------------------
>
>                 Key: NUTCH-363
>                 URL: https://issues.apache.org/jira/browse/NUTCH-363
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 0.8
>         Environment: OS X 10.4.7
>            Reporter: Doug Cook
>            Priority: Minor
>             Fix For: 2.0
>
>
> New links are normalized twice by the fetcher: 
> First in DOMContentUtils.getOutlinks, where the constructor Outlink(url.toString(), linkText.toString().trim(), conf)  normalizes the URL.
> The second time is in ParseOutputFormat.write().
> For some URLs (e.g. those repeated on a page) a given URL may be normalized a number of times, but it is always normalized at least twice.
> For those of us with expensive normalizations, this is probably burning some CPU. 
> I'd gladly fix this, but I'm not yet familiar enough with the code to know if there are some hidden assumptions which rely on this behavior.
> [A related note is that URLs are normalized *before* filtering; this is causing a lot of extra normalization as well. In general, filters may not be safe to run before normalization, but there is likely a class of them which are (filtering out .gif/.jpg etc). Perhaps the notion of a "pre-normalizer filter" would be a useful one?]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira