You are viewing a plain text version of this content. The canonical link for it is here.

Posted to yarn-issues@hadoop.apache.org by "He Tianyi (JIRA)" <ji...@apache.org> on 2016/01/30 10:48:40 UTC

[jira] [Commented] (YARN-2139) [Umbrella] Support for Disk as a Resource in YARN

    [ https://issues.apache.org/jira/browse/YARN-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15124824#comment-15124824 ] 

He Tianyi commented on YARN-2139:
---------------------------------

Recently introduced SSD in my cluster for MapReduce shuffle. 
Then there is one issue, if map output gets too large, it cannot be placed on SSD. We have to implement a custom strategy (called SSDFirst) to make best effort to use SSD, but fallbacks to HDD when available space of SSD gets tight. 
This worked in most cases, but it is only a local optimum. To achieve global optimum, scheduler must be aware and management these resources.

> [Umbrella] Support for Disk as a Resource in YARN 
> --------------------------------------------------
>
>                 Key: YARN-2139
>                 URL: https://issues.apache.org/jira/browse/YARN-2139
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Wei Yan
>         Attachments: Disk_IO_Isolation_Scheduling_3.pdf, Disk_IO_Scheduling_Design_1.pdf, Disk_IO_Scheduling_Design_2.pdf, YARN-2139-prototype-2.patch, YARN-2139-prototype.patch
>
>
> YARN should consider disk as another resource for (1) scheduling tasks on nodes, (2) isolation at runtime, (3) spindle locality. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)