[jira] [Commented] (NUTCH-2455) Speed up the merging of HostDb entries for variable fetch delay

["Sebastian Nagel (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-2455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16247765#comment-16247765 ] 

Sebastian Nagel commented on NUTCH-2455:
----------------------------------------

Hi [~semyon.semyonov@mail.com], thanks for the patch. The solution to cache the entire HostDb in a HashMap is, however, not scalable:
* consider that the HostDb may contain 100 millions of hosts - how much RAM would be required to hold the HashMap in memory?
* when running on a cluster even every parallel task would have to load the HashMap (and hold it in memory): while the CrawlDb is read only once, the HostDb is read many times and every read takes long because it is not parallelized.

A proper solution could be:
- read HostDatums together with CrawlDatums (cf. MultipleInputFormat, depends on NUTCH-2375) as input of the select step
- the map function then emits key-value pairs  <host, score> -> <url,crawldatum,score,...>
-* of course, the HostDatums must be wrapped into the value structure. It's already a custom class (SelectorEntry), so that should be doable
- via partitioning and secondary sorting these arrive in the reduce function:
-* all keys with the same host in one call of the function
-* in the following order: first the HostDatum (just assign an artificially high score), then the CrawlDatum items sorted by decreasing score

Have a look at this modified [Generator variant|https://github.com/commoncrawl/nutch/blob/cc/src/java/org/apache/nutch/crawl/Generator2.java#L395], for how secondary sorting can be implemented.


> Speed up the merging of HostDb entries for variable fetch delay
> ---------------------------------------------------------------
>
>                 Key: NUTCH-2455
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2455
>             Project: Nutch
>          Issue Type: Improvement
>          Components: generator
>    Affects Versions: 1.13
>            Reporter: Markus Jelsma
>         Attachments: NUTCH-2455.patch
>
>
> Citing Sebastian at NUTCH-2420:
> ??The correct solution would be to use <host,score> pairs as keys in the Selector job, with a partitioner and secondary sorting so that all keys with same host end up in the same call of the reducer. If values can also hold a HostDb entry and the sort comparator guarantees that the HostDb entry (entries if partitioned by domain or IP) comes in front of all CrawlDb entries. But that would be a substantial improvement...??



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)