[jira] [Updated] (YARN-2139) [Umbrella] Support for Disk as a Resource in YARN

["Karthik Kambatla (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YARN-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Karthik Kambatla updated YARN-2139:
-----------------------------------
    Attachment: Disk_IO_Isolation_Scheduling_3.pdf

A bunch of us (Jian, Karthik, Ram, Vinod, Wei) met up offline last week. Vinod suggested we follow a three-phase approach - (1) avoid overallocation of disk resources, (2) NMs to provide isolation to disallow a container messing with another, (3) add scheduling support for diverse disk IO requirements between containers. We expect little disagreement on the first two items, and understand consensus on the third item might take a little time. So, we want to proceed with getting the patches in for (1) and (2) while we arbitrate on (3). 

The other primary concern in the discussion and other JIRA comments is the ability to view disk resource in other dimensions - bandwidth, iops etc. 

I have updated the design document accordingly:
# Added a section on the Scope of the work
# Updated the approach to reflect the three main phases involved
# Updated design to plug-in isolation and scheduling along different dimensions

The development would be on a branch - YARN-2139, but would like the patches to go through the same review-commit process as they would on trunk. 

> [Umbrella] Support for Disk as a Resource in YARN 
> --------------------------------------------------
>
>                 Key: YARN-2139
>                 URL: https://issues.apache.org/jira/browse/YARN-2139
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Wei Yan
>         Attachments: Disk_IO_Isolation_Scheduling_3.pdf, Disk_IO_Scheduling_Design_1.pdf, Disk_IO_Scheduling_Design_2.pdf, YARN-2139-prototype-2.patch, YARN-2139-prototype.patch
>
>
> YARN should consider disk as another resource for (1) scheduling tasks on nodes, (2) isolation at runtime, (3) spindle locality. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)