You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Wangda Tan (JIRA)" <ji...@apache.org> on 2018/12/13 02:27:00 UTC
[jira] [Commented] (YARN-9116) Capacity Scheduler: add the default
maximum-allocation-mb and maximum-allocation-vcores for the queues
[ https://issues.apache.org/jira/browse/YARN-9116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719676#comment-16719676 ]
Wangda Tan commented on YARN-9116:
----------------------------------
[~aihuaxu],
This sounds like a plan, but existing maximum memory, etc. is defined inside yarn-site.xml. And given we will have multiple resource types support like GPU. I suggest to make the default maximum capacity definition use absolute values for all types.
The name of the config could be:
{code:java}
<property>
<name>yarn.scheduler.capacity.default-queue-maximum-capacity</name>
<value>memory=20G,vcores=20,gpu=3</value>
</property>{code}
You can reuse implementation org.apache.hadoop.yarn.submarine.client.cli.CliUtils#createResourceFromString from trunk (and move it to common place if you think it is required).
Thoughts?
+ [~sunilg], [~cheersyang]
> Capacity Scheduler: add the default maximum-allocation-mb and maximum-allocation-vcores for the queues
> ------------------------------------------------------------------------------------------------------
>
> Key: YARN-9116
> URL: https://issues.apache.org/jira/browse/YARN-9116
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
> Affects Versions: 2.7.0
> Reporter: Aihua Xu
> Assignee: Aihua Xu
> Priority: Major
>
> YARN-1582 adds the support of maximum-allocation-mb configuration per queue which is targeting to support larger container features on dedicated queues (larger maximum-allocation-mb/maximum-allocation-vcores for such queue) . While to achieve larger container configuration, we need to increase the global maximum-allocation-mb/maximum-allocation-vcores (e.g. 120G/256) and then override those configurations with desired values on the queues since queue configuration can't be larger than cluster configuration. There are many queues in the system and if we forget to configure such values when adding a new queue, then such queue gets default 120G/256 which typically is not what we want.
> We can come up with a queue-default configuration (set to normal queue configuration like 16G/8), so the leaf queues gets such values by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org