[jira] [Commented] (NUTCH-2377) Nutch can't parse relative links

["Sebastian Nagel (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994558#comment-15994558 ] 

Sebastian Nagel commented on NUTCH-2377:
----------------------------------------

Hi [~abhakim1980],
everything looks fine when trying with parsechecker and the recent head of Nutch 2.x:
{noformat}
% $NUTCH_HOME/bin/nutch parsechecker https://www.ouedkniss.com
fetching: https://www.ouedkniss.com
parsing: https://www.ouedkniss.com
contentType: text/html
...
  outlink: toUrl: https://www.ouedkniss.com/?lang=fr anchor: 
  outlink: toUrl: https://www.ouedkniss.com/scripts/jquery-1.11.3.min.js anchor: 
  outlink: toUrl: https://www.ouedkniss.com/scripts/basic.js?v=286 anchor: 
  outlink: toUrl: https://www.ouedkniss.com/images/favicon.ico anchor: 
...
  outlink: toUrl: https://www.ouedkniss.com/loisirs_divertissements/jeu-video/gameboy anchor: GameBoy
  outlink: toUrl: https://www.ouedkniss.com/loisirs_divertissements/chasse-et-peche anchor: Chasse & Peche
{noformat}

447 outlinks, most of them from properly resolved from relative links in the HTML. After increasing the content limit (see property {{http.content.limit}}) the number of found outlinks is 978:
{noformat}
% nutch parsechecker -Dhttp.content.limit=-1 https://www.ouedkniss.com
...
{noformat}
Please, resolve this issue as "Not A Problem" in case this matches your expectations. Thanks!

> Nutch can't parse relative links 
> ---------------------------------
>
>                 Key: NUTCH-2377
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2377
>             Project: Nutch
>          Issue Type: Bug
>          Components: parser
>    Affects Versions: 2.3
>         Environment: centos 7, hbase 0.98
>            Reporter: hakim
>            Priority: Critical
>
> Testing with the following site: https://www.ouedkniss.com, nutch only parse links that does contain the base url. 
> Tried tika as parser, tried to update db.max.outlinks.per.page to -1, tried practically every comments about detecting all the links, doubted urlfilter or regex-normalizer so it was disabled but having the same results. 
> each time I rebuild nutch and test the parser, it gives the same urls count arround 378. 
> Can somebody help out to fix this.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)