You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by Muthu Ganesh <mu...@gmail.com> on 2015/08/29 18:45:43 UTC
Fractional container resources
Hi,
Why does YARN block an entire resource once a container has been allocated
in the node? A job might use different quantities of resources at different
points of time and allocating a pre-determined set of resources might be
counter-productive. For example, if the resources requested by the job are
1VCORE and 4GB of RAM, the entire 4GB is blocked (other Jobs/processes can
not access this memory) irrespective of whether or not the job uses the
complete 4GB at a given point of time.
This might be a problem with I/O bound tasks wherein the CPU utilization is
close to 1% but 1VCORE is used nevertheless since the hdfs-site.xml
dictates a minimum VCORE of 1. This VCORE could have been given to a CPU
intensive job wherein the 1VCORE allocation actually makes sense.
The YARN-1197 <https://issues.apache.org/jira/browse/YARN-1197> Jira shows
support for dynamic containers but I believe the container sizes are not
increased/decreased fractionally.
Thanks,
Muthu
Re: Fractional container resources
Posted by Karthik Kambatla <ka...@cloudera.com>.
Inline.
On Sat, Aug 29, 2015 at 9:45 AM, Muthu Ganesh <mu...@gmail.com> wrote:
> Hi,
>
> Why does YARN block an entire resource once a container has been allocated
> in the node? A job might use different quantities of resources at different
> points of time and allocating a pre-determined set of resources might be
> counter-productive.
Yes and no, depends on what is more important to the user - utilization or
predictable performance.
Admittedly, Yarn could consider node utilization while scheduling
containers to increase utilization without sacrificing predictable
performance. YARN-1011 and a couple of other JIRAs target this, and could
use contributions. :)
> For example, if the resources requested by the job are
> 1VCORE and 4GB of RAM, the entire 4GB is blocked (other Jobs/processes can
> not access this memory) irrespective of whether or not the job uses the
> complete 4GB at a given point of time.
>
> This might be a problem with I/O bound tasks wherein the CPU utilization is
> close to 1% but 1VCORE is used nevertheless since the hdfs-site.xml
> dictates a minimum VCORE of 1. This VCORE could have been given to a CPU
> intensive job wherein the 1VCORE allocation actually makes sense.
>
If the goal is to get fractional cpus, one could always bump up the number
of virtual cores on a node so each virtual core represents a smaller
fraction of a physical core and use that to schedule.
>
> The YARN-1197 <https://issues.apache.org/jira/browse/YARN-1197> Jira shows
> support for dynamic containers but I believe the container sizes are not
> increased/decreased fractionally.
>
> Thanks,
> Muthu
>