You are viewing a plain text version of this content. The canonical link for it is here.

Posted to yarn-issues@hadoop.apache.org by "Carlo Curino (JIRA)" <ji...@apache.org> on 2013/09/25 00:30:07 UTC

[jira] [Commented] (YARN-624) Support gang scheduling in the AM RM protocol

    [ https://issues.apache.org/jira/browse/YARN-624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776847#comment-13776847 ] 

Carlo Curino commented on YARN-624:
-----------------------------------

Hi Guys,

I would like to quantify what is the typical waste of resources while "hoarding" containers towards a gang for Gyraph or Storm. 
Anyone have an intuition/measure of the typical time-delay and container slot-time wasted while hoarding containers, before the 
useful part of the computation starts?  Thanks.. 

                
> Support gang scheduling in the AM RM protocol
> ---------------------------------------------
>
>                 Key: YARN-624
>                 URL: https://issues.apache.org/jira/browse/YARN-624
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: api, scheduler
>    Affects Versions: 2.0.4-alpha
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
>
> Per discussion on YARN-392 and elsewhere, gang scheduling, in which a scheduler runs a set of tasks when they can all be run at the same time, would be a useful feature for YARN schedulers to support.
> Currently, AMs can approximate this by holding on to containers until they get all the ones they need.  However, this lends itself to deadlocks when different AMs are waiting on the same containers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira