You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Antal Bálint Steinbach (JIRA)" <ji...@apache.org> on 2018/08/30 08:24:00 UTC
[jira] [Comment Edited] (YARN-8468) Limit container sizes per queue
in FairScheduler
[ https://issues.apache.org/jira/browse/YARN-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596174#comment-16596174 ]
Antal Bálint Steinbach edited comment on YARN-8468 at 8/30/18 8:23 AM:
-----------------------------------------------------------------------
Thank you [~leftnoteasy] for checking the patch. I uploaded a new one.
1) The ApplicationMaster gets the maximum allocation value here just like in the other case so the AM can handle it before submitting a request, but I think we still need the normalization/validation of the request just in case the application did something wrong. Furthermore, there was a validation/normalization step before, it was just using only the scheduler level maximums.
2) I reverted the formattings. I run an autoformatting unintentionally on SchedulerUtils.
Please let me know if I misunderstand something.
was (Author: bsteinbach):
Thank you [~leftnoteasy] for checking the patch. I uploaded a new one.
1) The ApplicationMaster gets the maximum allocation value here just like in the other case so the AM can handle it like before submitting a request, but I think we still need the normalization/validation of the request just in case the application did something wrong. Furthermore, there was a validation/normalization step before it was just using only the scheduler level maximums.
2) I reverted the formattings. I run an autoformatting unintentionally on SchedulerUtils.
Please let me know if I misunderstand something.
> Limit container sizes per queue in FairScheduler
> ------------------------------------------------
>
> Key: YARN-8468
> URL: https://issues.apache.org/jira/browse/YARN-8468
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
> Affects Versions: 3.1.0
> Reporter: Antal Bálint Steinbach
> Assignee: Antal Bálint Steinbach
> Priority: Critical
> Attachments: YARN-8468.000.patch, YARN-8468.001.patch, YARN-8468.002.patch, YARN-8468.003.patch, YARN-8468.004.patch, YARN-8468.005.patch, YARN-8468.006.patch, YARN-8468.007.patch, YARN-8468.008.patch
>
>
> When using any scheduler, you can use "yarn.scheduler.maximum-allocation-mb" to limit the overall size of a container. This applies globally to all containers and cannot be limited by queue or and is not scheduler dependent.
>
> The goal of this ticket is to allow this value to be set on a per queue basis.
>
> The use case: User has two pools, one for ad hoc jobs and one for enterprise apps. User wants to limit ad hoc jobs to small containers but allow enterprise apps to request as many resources as needed. Setting yarn.scheduler.maximum-allocation-mb sets a default value for maximum container size for all queues and setting maximum resources per queue with “maxContainerResources” queue config value.
>
> Suggested solution:
>
> All the infrastructure is already in the code. We need to do the following:
> * add the setting to the queue properties for all queue types (parent and leaf), this will cover dynamically created queues.
> * if we set it on the root we override the scheduler setting and we should not allow that.
> * make sure that queue resource cap can not be larger than scheduler max resource cap in the config.
> * implement getMaximumResourceCapability(String queueName) in the FairScheduler
> * implement getMaximumResourceCapability() in both FSParentQueue and FSLeafQueue as follows
> * expose the setting in the queue information in the RM web UI.
> * expose the setting in the metrics etc for the queue.
> * write JUnit tests.
> * update the scheduler documentation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org