[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests

["Steve Loughran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14082252#comment-14082252 ] 

Steve Loughran commented on YARN-796:
-------------------------------------

I'm not going to comment on the current architecture; I think I need to understand the proposals better to understand what is being proposed for the first iteration. And iterations are, as others propose, a good way to do it.

FWIW, the SLIDER-81 case is about allowing us to allocate bits of a YARN cluster explicitly to groups, having queues select labels should suffice there. Although you can get exclusive use of a node by asking for all its resources, that does not guarantee that a node will be free for your team (ignoring premption)

There's also a possible need in future: label-based block placement. Can I label a set of nodes "hbase-production" and be confident that >1 node in that set will have a copy of the hbase data blocks? I don't think its timely to address that today, but having the means to do so would be useful in future. That argues for having the HDFS layer being able to see/receive the same label data.

h3. the patch

This is a major patch -- it always hurts me to see how much coding we need to do to work with protobuf, as that's a major portion of the diff.

# too much duplication of {{"-showLabels"}} and {{"-refreshLabels"}} strings in the code. These should be made constants somewhere.
# why is {{ getClusterNodeLabels() }} catching {{YarnException}} and rethrowing as an IOE? Can't it just be added to the signature?
# version of {{net.java.dev.eval}} dependency must go into hadoop-project POM.
# Could you use SLF4J for the logging in new classes...we're slowly moving towards that everywhere
# Label manager should just be a service in its own right. If you do want to wrap it in {{LabelManagementService}}
then can you (a) justify this and (b) match the full lifecycle
# I don't want yet another text file format for configuration. the label config should either be hadoop XML or some JSON syntax. Why? Helps other tools parse and generate.


h3. tests
# Tests assume that {{"/tmp/labelFile"}} is writeable; they should use {{"./target/labelFile"}}} or something else under {{./target}}
# use {{assertEquals}} in service state tests too
# why the sleep in setup? that adds 6 seconds/test method
# {{equalsIgnoreCase}} mustn't be used, go {{.toLower(LOCALE_EN).equals()}} for i18n.
# there's a lot of testing that could be factored into commonality (probes for configs files, assertContains on labels). This will simplify the tests
# we'll need tests that the schedulers work with labels, obviously.

> Allow for (admin) labels on nodes and resource-requests
> -------------------------------------------------------
>
>                 Key: YARN-796
>                 URL: https://issues.apache.org/jira/browse/YARN-796
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>    Affects Versions: 2.4.1
>            Reporter: Arun C Murthy
>            Assignee: Wangda Tan
>         Attachments: LabelBasedScheduling.pdf, Node-labels-Requirements-Design-doc-V1.pdf, YARN-796.patch, YARN-796.patch4
>
>
> It will be useful for admins to specify labels for nodes. Examples of labels are OS, processor architecture etc.
> We should expose these labels and allow applications to specify labels on resource-requests.
> Obviously we need to support admin operations on adding/removing node labels.



--
This message was sent by Atlassian JIRA
(v6.2#6252)