You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nutch.apache.org by "Markus Jelsma (JIRA)" <ji...@apache.org> on 2017/10/20 12:45:00 UTC

[jira] [Updated] (NUTCH-2445) Fetcher following outlinks to keep track of already fetched items

     [ https://issues.apache.org/jira/browse/NUTCH-2445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Markus Jelsma updated NUTCH-2445:
---------------------------------
    Attachment: NUTCH-2445.patch

Patch!

> Fetcher following outlinks to keep track of already fetched items
> -----------------------------------------------------------------
>
>                 Key: NUTCH-2445
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2445
>             Project: Nutch
>          Issue Type: Improvement
>    Affects Versions: 1.13
>            Reporter: Markus Jelsma
>            Assignee: Markus Jelsma
>             Fix For: 1.14
>
>         Attachments: NUTCH-2445.patch
>
>
> When fetcher.follow.outlinks.depth is non-zero, fetcher follows outlinks. This patch keeps track of already fetched URL's and thus avoid fetching the same URL twice.
> A Set is used to keep track of them, hashcodes to reduce memory usage. This is not used if fetcher doesn't follow outlinks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)