[jira] [Updated] (YARN-7587) Skip dispatching opportunistic containers to nodes whose queue is already full

["Weiwei Yang (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YARN-7587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Weiwei Yang updated YARN-7587:
------------------------------
    Attachment: YARN-7587.002.patch

bq. We can explain a bit more in comments the different max queue lengths

I have added some more comments, and changed the variable name from term {{maxQueueLength}} to {{queueCapacity}}. Hopefully this helps to avoid confusion.

bq. Rather than performing an extra iteration to filter the list, maybe it might be better to modify the getMetric method in the comparator to return MAX_INT / MIN_INT where appropriate if the isQueueFull is true for the clusternode

I am not sure if that's gonna work. The objective is to exclude some unqualified nodes, if only modify the getMetric method in comparator, it will still return all nodes. But I agree, an extra iteration is not necessary. I have removed the filter code in v2 patch, and replaced that by the check before adding the sorted nodes to the {{retList}}.

bq. we should do the same for de-comissioned nodes as well

Do you mean we should skip dispatching any O containers to the de-commissioned nodes? If so I totally agree with that (I would be surprised if it is not done so). This can be tracked in another JIRA.

Thanks.

> Skip dispatching opportunistic containers to nodes whose queue is already full
> ------------------------------------------------------------------------------
>
>                 Key: YARN-7587
>                 URL: https://issues.apache.org/jira/browse/YARN-7587
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>    Affects Versions: 2.9.0
>            Reporter: Weiwei Yang
>            Assignee: Weiwei Yang
>         Attachments: YARN-7587.001.patch, YARN-7587.002.patch
>
>
> When there is a lot of opportunistic container requests, even if NM queue is full, RM continues to dispatch O containers to that node. Such allocations will be killed immediately by NM causing a waste of resource. Propose to exclude such nodes in the load monitor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org