You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Junping Du (JIRA)" <ji...@apache.org> on 2013/05/01 17:36:17 UTC
[jira] [Commented] (YARN-18) Make locatlity in YARN's container
assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13646652#comment-13646652 ]
Junping Du commented on YARN-18:
--------------------------------
Thanks Luke for explanation.
Hey Arun ([~acmurthy]), I just attached a doc with implementation details for this patch and YARN-19. Hopefully that will be helpful for your review. Your thought to abstract notion of topology logic in scheduler make great sense to me. We already did this for each scheduler, but you mean that need to abstract common part for all schedulers which is a little complicated but still doable. Can we do this code refactoring work in a separated jira and I am glad to work on it? For a new NodeGroup Scheduler, I think it may not be necessary as it address different topologies rather than different algorithm to isolate/prioritise job. So even under topology with NodeGroup layer, different user still need different scheduler like: Fair, Capacity. etc. Thoughts?
> Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: YARN-18
> URL: https://issues.apache.org/jira/browse/YARN-18
> Project: Hadoop YARN
> Issue Type: New Feature
> Affects Versions: 2.0.3-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Labels: features
> Attachments: HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, MAPREDUCE-4309-v7.patch, Pluggable topologies with NodeGroup for YARN.pdf, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, YARN-18-v4.3.patch, YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch, YARN-18-v6.patch
>
>
> There are several classes in YARN’s container assignment and task scheduling algorithms that relate to data locality which were updated to give preference to running a container on other locality besides node-local and rack-local (like nodegroup-local). This propose to make these data structure/algorithms pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class ScheduledRequests was made a package level class to it would be easier to create a subclass, ScheduledRequestsWithNodeGroup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira