[jira] [Commented] (NUTCH-690) bug in DomContentUtils.shouldThrowAwayLink?

["Markus Jelsma (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085654#comment-13085654 ] 

Markus Jelsma commented on NUTCH-690:
-------------------------------------

I see these url's as outlinks with 1.4-dev and parse-tika and also using 1.3 and parse-html. Is this still a valid issue?

> bug in DomContentUtils.shouldThrowAwayLink?
> -------------------------------------------
>
>                 Key: NUTCH-690
>                 URL: https://issues.apache.org/jira/browse/NUTCH-690
>             Project: Nutch
>          Issue Type: Bug
>    Affects Versions: 0.9.0
>            Reporter: Peter Sparks
>            Priority: Minor
>
> I found a potential bug in DomContentUtils.shouldThrowAwayLink. It returns true for the 5 links at the top of the home page for www.aksteel.com. Here are the links in the source:
>             <a id="Search" href="/search/default.aspx"></a>
>             <a id="Investor" style="height: 15px;" href="/investor_information/"></a>
>             <a id="Markets" href="/markets_products/"></a>
>             <a id="Production" href="/production_facilities/"></a>
>             <a id="News" href="/news/"></a>
> Perhaps I am just ignorant of what this function is supposed to do but returning true for these 5 links on that site make that site impossible to crawl.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira