[jira] [Commented] (NUTCH-3056) Injector to support resolving seed URLs

["Sebastian Nagel (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-3056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880048#comment-17880048 ] 

Sebastian Nagel commented on NUTCH-3056:
----------------------------------------

Hi [~markus17],

thanks! Interesting, I'm currently working on improving the deal better with a large amount of exceptions, see NUTCH-3067. Filtering ahead the URLs whether they are resolvable or not would, of course, limit the problems described in NUTCH-3067. The price is the double fetching of successful URLs.

Few comments about the patch:
- it adds 150 lines to the already comples Injector class
- the scenario how URLs are resolved seems very bound to this specific use-case: a large, diverse one-time crawl of just the seed list
- for other scenarios, e.g., if links are followed and pages are re-fetched after some time, one would keep the redirects and also the failure states to avoid that the fetcher "repeats" the failed resolver requests from Injector.

Given that, what about moving the implementation into a separate class, say InjectorResolver, which inherits from Injector? This would keep the code more maintainable and also adaptable to similar use cases. Of course, some changes in Injector (resp. InjectMapper) itself are required:
- visibility of fields / instance variables
- add a hook to call the resolver (similar to calling filterNormalize(url)) with a default do-nothing resolver

> Initial 1.15 patch.
Applying the path to the current master fails (mostly).

> Injector to support resolving seed URLs
> ---------------------------------------
>
>                 Key: NUTCH-3056
>                 URL: https://issues.apache.org/jira/browse/NUTCH-3056
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Markus Jelsma
>            Assignee: Markus Jelsma
>            Priority: Minor
>             Fix For: 1.21
>
>         Attachments: NUTCH-3056.patch
>
>
> We have a case where clients submit huge uncurated seed files, the host may not longer exist, or redirect via-via to elsewhere, the protocol may be incorrect etc.
> The large crawl itself is not supposed to venture much beyond the seed list, except for regex exceptions listed in {color:#000000}db-ignore-external-exemptions{color}. It is also not allowed to jump to other domains/hosts to control the size of the crawl. This means externally redirecting seeds will not be crawled.
> This ticket will add support for a multi-threaded host/domain/protocol/redirecter/resolver to the injector. Seeds not leading to a non-200 URL will be discarded. Enabling filtering and normalization is highly recommended for handling the redirects.
> If you have a seed file with 10k+ or millions of records, you are highly recommended to split the input file in chunks so that multiple mappers can get to work. Passing a few millions records without resolving through one mapper is no problem, but resolving millions with one mapper, even if threaded, will take many hours.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)