[jira] [Created] (YARN-7273) Ability for YARN applications to request lower container size below yarn.scheduler.minimum-allocation-mb

["Scott Brokaw (JIRA)" <ji...@apache.org>]

Scott Brokaw created YARN-7273:
----------------------------------

             Summary: Ability for YARN applications to request lower container size below yarn.scheduler.minimum-allocation-mb
                 Key: YARN-7273
                 URL: https://issues.apache.org/jira/browse/YARN-7273
             Project: Hadoop YARN
          Issue Type: Improvement
            Reporter: Scott Brokaw
            Priority: Minor


Currently, a YARN application must request a container size that is between the yarn.scheduler.minimum-allocation-mb and yarn.scheduler.maximum-allocation-mb else the request has to be raised by the Application to the yarn.scheduler.minimum-allocation-mb

yarn.scheduler.minimum-allocation-mb < Requested Container Size > yarn.scheduler.maximum-allocation-mb

This improvement would either add a new parameter such as yarn.scheduler.default-allocation-mb or treat yarn.scheduler.minimum-allocation-mb instead as a global default size if no container size is requested.  This would allow a particular YARN application to request a smaller amount of resources if it deems it can run with resources below the minimum allocation dictated by YARN.  This allows a significant amount of resources to be saved as otherwise a YARN application that spawns many containers can end up consuming many more resources then it needs.

The same idea can be applied to yarn.scheduler.minimum-allocation-vcores.

The argument can be made that you should just set the minimum values lower to start with, but the idea is that YARN should support a global default value for applications that don't set it, but allow other applications (if they so request) to be granted smaller containers.  The "problem" quickly compounds the more containers an application spins up and with memory being a valuable resource in the cluster, this improvement is looking to maximize the available resources given an application is "willing and able" to run with a smaller size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org