[jira] [Commented] (YARN-2877) Extend YARN to support distributed scheduling

["Sriram Rao (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14217207#comment-14217207 ] 

Sriram Rao commented on YARN-2877:
----------------------------------

The proposal:
 # Extend the NM to support task queueing.  AM's can queue tasks directly at the NM's and the NM's will execute those tasks opportunistically.
 # Extend the type of containers that YARN exposes: 
   #* CONSERVATIVE: This corresponds to containers allocated by YARN today.  
   #* OPTIMISTIC: This corresponds to a new class of containers, which will be queued for execution at the NM.
   This extension allows AM's to control what type of container they are requesting from the RM framework.
 # Extend the NM with a "local RM" (i.e., a local Resource Manager) which uses local policies for deciding when an "OPTIMISTIC container" can be executed.

We are exploring using timed leases for OPTIMISTIC containers to ensure minimum duration of execution.  On the other hand, this mechanism allows NM's to free up resources and thus guarantee predictable start times for CONSERVATIVE containers.

There are additional motivations for the uses of this feature and we will discuss them in follow-up comments.

> Extend YARN to support distributed scheduling
> ---------------------------------------------
>
>                 Key: YARN-2877
>                 URL: https://issues.apache.org/jira/browse/YARN-2877
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, resourcemanager
>            Reporter: Sriram Rao
>
> This is an umbrella JIRA that proposes to extend YARN to support distributed scheduling.  Briefly, some of the motivations for distributed scheduling are the following:
> 1. Improve cluster utilization by opportunistically executing tasks otherwise idle resources on individual machines.
> 2. Reduce allocation latency.  Tasks where the scheduling time dominates (i.e., task execution time is much less compared to the time required for obtaining a container from the RM).
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)