[jira] [Commented] (REEF-1048) Allow specifying locality constraints in YARN driver configuration

["Tobin Baker (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/REEF-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15433081#comment-15433081 ] 

Tobin Baker commented on REEF-1048:
-----------------------------------

Note that the YARN API makes this _seem_ possible (by setting the Driver hostname in {{ResourceRequest.setResourceName()}} and passing the {{ResourceRequest}} to {{ApplicationSubmissionContext.setAMContainerResourceRequest()}}), but it doesn't actually work because {{RMAppAttemptImpl}} hardcodes {{ResourceRequest.setResourceName(ResourceRequest.ANY)}} and {{ResourceRequest.setRelaxLocality(true)}}: https://github.com/apache/hadoop/blob/branch-2.7.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java#L960

This is still unchanged in Hadoop 2.8: https://github.com/apache/hadoop/blob/branch-2.8/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java#L1028

despite the comment

bq. TODO: change these fields when we want to support priority/resource-name/relax-locality specification for AM containers allocation.

> Allow specifying locality constraints in YARN driver configuration
> ------------------------------------------------------------------
>
>                 Key: REEF-1048
>                 URL: https://issues.apache.org/jira/browse/REEF-1048
>             Project: REEF
>          Issue Type: Improvement
>          Components: REEF-Runtime-YARN
>            Reporter: Tobin Baker
>            Assignee: Tobin Baker
>            Priority: Minor
>
> In some scenarios it would be useful to specify node/rack locality for the Driver container (YARN ApplicationMaster). One example is a cloud computing cluster where each "worker" instance is provisioned with just enough memory to run a single Task container, but not an additional Driver container. YARN allows you to do this with node labels (presentation: http://www.slideshare.net/Hadoop_Summit/node-labels-in-yarn-49792443), but this isn't exposed in YARN Driver configuration. If we add node/rack parameters to {{org.apache.reef.runtime.yarn.driver. YarnDriverConfiguration}}, we should be able to implement this functionality using node labels.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)