You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Luke Lu (JIRA)" <ji...@apache.org> on 2013/04/01 02:01:15 UTC
[jira] [Commented] (YARN-18) Make locatlity in YARN's container
assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618525#comment-13618525 ]
Luke Lu commented on YARN-18:
-----------------------------
bq. I see a lot of factories and then no changes to actually support node-group..
YARN-18 is explicitly for making the topology pluggable based on previous review requests. So there is little node group specific changes, which are implemented in YARN-19.
bq. we need to abstract the notion of topology in the Scheduler without having to make changes every time we need to add another layer in the topology.
This is a goal of the JIRA. You should be able to add another layer without having to modify scheduler code after this patch is committed. I'm not sure that we could make arbitrary topology pluggable without scheduler change. However, it should work for common class of topologies, e.g., hierarchical network topology.
> Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: YARN-18
> URL: https://issues.apache.org/jira/browse/YARN-18
> Project: Hadoop YARN
> Issue Type: New Feature
> Affects Versions: 2.0.3-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Labels: features
> Attachments: HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, YARN-18-v4.3.patch, YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch
>
>
> There are several classes in YARN’s container assignment and task scheduling algorithms that relate to data locality which were updated to give preference to running a container on other locality besides node-local and rack-local (like nodegroup-local). This propose to make these data structure/algorithms pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class ScheduledRequests was made a package level class to it would be easier to create a subclass, ScheduledRequestsWithNodeGroup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira