[jira] [Created] (NUTCH-1687) Pick queue in Round Robin

["Nguyen Manh Tien (JIRA)" <ji...@apache.org>]

Nguyen Manh Tien created NUTCH-1687:
---------------------------------------

             Summary: Pick queue in Round Robin
                 Key: NUTCH-1687
                 URL: https://issues.apache.org/jira/browse/NUTCH-1687
             Project: Nutch
          Issue Type: Improvement
    Affects Versions: 2.3
            Reporter: Nguyen Manh Tien
            Priority: Minor


Currently we chose queue to pick url from start of queues list, so queue at the start of list have more change to be pick first, that can cause problem of long tail queue, which only few queue available at the end which have many urls.

public synchronized FetchItem getFetchItem() {
      final Iterator<Map.Entry<String, FetchItemQueue>> it =
        queues.entrySet().iterator(); ==> always reset to find queue from start
      while (it.hasNext()) {
....

I think it is better to pick queue in round robin, that can make reduce time to find the available queue and make all queue was picked in round robin and if we use TopN during generator there are no long tail queue at the end.




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)