[jira] [Commented] (NUTCH-1687) Pick queue in Round Robin

["Otis Gospodnetic (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-1687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855369#comment-13855369 ] 

Otis Gospodnetic commented on NUTCH-1687:
-----------------------------------------

[~tiennm] - the new class should have ASL at the top.  But.... not sure if Nutch is using anything like Apache Commons or any other libraries that already implement something like this new CircularLinkedList?


> Pick queue in Round Robin
> -------------------------
>
>                 Key: NUTCH-1687
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1687
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>            Reporter: Nguyen Manh Tien
>            Priority: Minor
>             Fix For: 2.3
>
>         Attachments: NUTCH-1687.patch
>
>
> Currently we chose queue to pick url from start of queues list, so queue at the start of list have more change to be pick first, that can cause problem of long tail queue, which only few queue available at the end which have many urls.
> public synchronized FetchItem getFetchItem() {
>       final Iterator<Map.Entry<String, FetchItemQueue>> it =
>         queues.entrySet().iterator(); ==> always reset to find queue from start
>       while (it.hasNext()) {
> ....
> I think it is better to pick queue in round robin, that can make reduce time to find the available queue and make all queue was picked in round robin and if we use TopN during generator there are no long tail queue at the end.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)