You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by Muthu Ganesh <mu...@gmail.com> on 2015/08/29 18:45:43 UTC

Fractional container resources

Hi,

Why does YARN block an entire resource once a container has been allocated
in the node? A job might use different quantities of resources at different
points of time and allocating a pre-determined set of resources might be
counter-productive. For example, if the resources requested by the job are
1VCORE and 4GB of RAM, the entire 4GB is blocked (other Jobs/processes can
not access this memory) irrespective of whether or not the job uses the
complete 4GB at a given point of time.

This might be a problem with I/O bound tasks wherein the CPU utilization is
close to 1% but 1VCORE is used nevertheless since the hdfs-site.xml
dictates a minimum VCORE of 1. This VCORE could have been given to a CPU
intensive job wherein the 1VCORE allocation actually makes sense.

The YARN-1197 <https://issues.apache.org/jira/browse/YARN-1197> Jira shows
support for dynamic containers but I believe the container sizes are not
increased/decreased fractionally.

Thanks,
Muthu

Re: Fractional container resources

Posted by Karthik Kambatla <ka...@cloudera.com>.
Inline.

On Sat, Aug 29, 2015 at 9:45 AM, Muthu Ganesh <mu...@gmail.com> wrote:

> Hi,
>
> Why does YARN block an entire resource once a container has been allocated
> in the node? A job might use different quantities of resources at different
> points of time and allocating a pre-determined set of resources might be
> counter-productive.


Yes and no, depends on what is more important to the user - utilization or
predictable performance.

Admittedly, Yarn could consider node utilization while scheduling
containers to increase utilization without sacrificing predictable
performance. YARN-1011 and a couple of other JIRAs target this, and could
use contributions. :)


> For example, if the resources requested by the job are
> 1VCORE and 4GB of RAM, the entire 4GB is blocked (other Jobs/processes can
> not access this memory) irrespective of whether or not the job uses the
> complete 4GB at a given point of time.
>
> This might be a problem with I/O bound tasks wherein the CPU utilization is
> close to 1% but 1VCORE is used nevertheless since the hdfs-site.xml
> dictates a minimum VCORE of 1. This VCORE could have been given to a CPU
> intensive job wherein the 1VCORE allocation actually makes sense.
>

If the goal is to get fractional cpus, one could always bump up the number
of virtual cores on a node so each virtual core represents a smaller
fraction of a physical core and use that to schedule.


>
> The YARN-1197 <https://issues.apache.org/jira/browse/YARN-1197> Jira shows
> support for dynamic containers but I believe the container sizes are not
> increased/decreased fractionally.
>
> Thanks,
> Muthu
>