You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by Lei Guo <le...@huawei.com> on 2015/06/25 21:34:03 UTC
Node Label, partition or an attribute of node?
The definition of label in YARN-796 is pretty clear,
* Node Label label that describes a node. Node can have multiple labels
* Label expression logical combination of labels (using && and, || or, ! not)
With this definition (without considering the implementation), a node label could be treated as a Boolean type attribute of a node. Assuming the administrator define label1, label2, label3 in the system, and associate node1 with "label1, label2", this means the value of attribute label1 and label2 for node1 is true, while the value of attribute label3 will be false for node1.
In current Yarn implementation, seems the resource partitioning has been treated as the primary use case for node label, and some design/implementation of node label mainly consider on the partitioning case, even the multiple label support has been disabled in YARN-2694. To cover the workload resource request use case, YARN-3409 was proposed while the description/example of constraint node label in YARN-3409 seems more like a String type attribute for a node instead of Boolean type anymore. I got confused about node label.
>From my view, node label is one attribute of a node, and it could be used for different scheduling scenarios. No matter resource partitioning or job scheduling constraints, a node label is just a label on a node. For partitioning, we can have one object calling partition or resource group representing the group of nodes with certain label; for workload scheduling constraints purpose, we may want to extend the current application level constraint to be container level constraint, for one application, the master may request one type of resource while the slave request another type of resource.
Comments?
Lei Guo
Senior Architect, Huawei Canada Research Centre
RE: Node Label, partition or an attribute of node?
Posted by Lei Guo <le...@huawei.com>.
I fully understand the challenges about customized/user defined resource and related enforcement for consumable resources. Maybe we can extend the current resource modeling a little bit on non-consumable resource first, just like the node labels, admin can define it via CLI without major protocol change. For how to use the resource, leave it to scheduling plug-in. For example, capacity scheduler can use node's label for partitioning; while fair scheduler may use label for job constraints.
Lei
-----Original Message-----
From: Wangda Tan [mailto:wheeleast@gmail.com]
Sent: Thursday, June 25, 2015 4:46 PM
To: yarn-dev@hadoop.apache.org
Subject: Re: Node Label, partition or an attribute of node?
Hi Lei,
This is a very interesting topic, we have LOTS OF discussions with many
folks for resource vs. label.
The major concerns of not accepting user-provided resource type to resource
vectors are:
- It may simply doesn't work, dominate resource fairness cannot deal with
nodes with different set of resource vectors (some nodes have GPU and some
not).
- You cannot enforce it. For example, a node has 8 GPUs, and you allocate
to 5 processes, one gets 5 and the other one gets 3. But from YARN, you
cannot ensure nobody uses more than allocated.
- Adding new vectors needs lots of refactoring in protocol, scheduler, etc.
in YARN which is not figured out yet.
Currently, YARN doesn't have plan in short term accept user-provided
resource type to resource vector. But adding controllable resources like
IO/bandwidth, etc is on the road map.
So in the short term, for such non controllable resource, you can put them
to node label, you can enforce resource sharing via partition, or just a
binding requirement via attribute.
Thoughts?
WAngda
On Thu, Jun 25, 2015 at 1:27 PM, Lei Guo <le...@huawei.com> wrote:
> Hi, Wangda,
>
> Actually, it's the two types of node label causing the confusion for me.
> For a scheduling system, it's to find the best match between "resource" and
> "workload" via different "policies". Both node partition and node
> constraints are scheduling policies. The node label should be part of
> resource (as an attribute), and be used in two different policies. If we
> position the node label this way, there is only one type of node label.
> This is related to how we define/abstract "resource" in Yarn. Currently,
> only CPU/memory (extending with disk/network bandwidth) considered as
> resource for a node. As you mentioned in Summit slides, GPU could also be a
> resource. These are all consumable resources/attributes provided by node
> while node label is non-consumable resource/attribute provided by a node.
>
> Lei
>
> -----Original Message-----
> From: Wangda Tan [mailto:wheeleast@gmail.com]
> Sent: Thursday, June 25, 2015 3:51 PM
> To: yarn-dev@hadoop.apache.org
> Subject: Re: Node Label, partition or an attribute of node?
>
> Hi Lei,
> To clarify, node label has two types, one is node partition, which is
> supported already. Another is node constraint (you can also call it node
> attribute), which is still under design. Both of them are string typed.
>
> And node label (including partition and attribute), can be either
> per-application level and per-container level, this is supported by API
> already.
>
> And you can take a look at my summit slides:
> http://www.slideshare.net/Hadoop_Summit/node-labels-in-yarn-49792443.
> Which
> may help you better understand partition / constraint and relationship
> between them.
>
> Thanks,
> Wangda
>
> On Thu, Jun 25, 2015 at 12:34 PM, Lei Guo <le...@huawei.com> wrote:
>
> > The definition of label in YARN-796 is pretty clear,
> >
> > * Node Label label that describes a node. Node can have multiple
> > labels
> >
> > * Label expression logical combination of labels (using && and,
> > || or, ! not)
> > With this definition (without considering the implementation), a node
> > label could be treated as a Boolean type attribute of a node. Assuming
> the
> > administrator define label1, label2, label3 in the system, and associate
> > node1 with "label1, label2", this means the value of attribute label1 and
> > label2 for node1 is true, while the value of attribute label3 will be
> false
> > for node1.
> >
> > In current Yarn implementation, seems the resource partitioning has been
> > treated as the primary use case for node label, and some
> > design/implementation of node label mainly consider on the partitioning
> > case, even the multiple label support has been disabled in YARN-2694. To
> > cover the workload resource request use case, YARN-3409 was proposed
> while
> > the description/example of constraint node label in YARN-3409 seems more
> > like a String type attribute for a node instead of Boolean type anymore.
> I
> > got confused about node label.
> >
> > From my view, node label is one attribute of a node, and it could be used
> > for different scheduling scenarios. No matter resource partitioning or
> job
> > scheduling constraints, a node label is just a label on a node. For
> > partitioning, we can have one object calling partition or resource group
> > representing the group of nodes with certain label; for workload
> scheduling
> > constraints purpose, we may want to extend the current application level
> > constraint to be container level constraint, for one application, the
> > master may request one type of resource while the slave request another
> > type of resource.
> >
> > Comments?
> >
> > Lei Guo
> > Senior Architect, Huawei Canada Research Centre
> >
> >
>
Re: Node Label, partition or an attribute of node?
Posted by Wangda Tan <wh...@gmail.com>.
Hi Lei,
This is a very interesting topic, we have LOTS OF discussions with many
folks for resource vs. label.
The major concerns of not accepting user-provided resource type to resource
vectors are:
- It may simply doesn't work, dominate resource fairness cannot deal with
nodes with different set of resource vectors (some nodes have GPU and some
not).
- You cannot enforce it. For example, a node has 8 GPUs, and you allocate
to 5 processes, one gets 5 and the other one gets 3. But from YARN, you
cannot ensure nobody uses more than allocated.
- Adding new vectors needs lots of refactoring in protocol, scheduler, etc.
in YARN which is not figured out yet.
Currently, YARN doesn't have plan in short term accept user-provided
resource type to resource vector. But adding controllable resources like
IO/bandwidth, etc is on the road map.
So in the short term, for such non controllable resource, you can put them
to node label, you can enforce resource sharing via partition, or just a
binding requirement via attribute.
Thoughts?
WAngda
On Thu, Jun 25, 2015 at 1:27 PM, Lei Guo <le...@huawei.com> wrote:
> Hi, Wangda,
>
> Actually, it's the two types of node label causing the confusion for me.
> For a scheduling system, it's to find the best match between "resource" and
> "workload" via different "policies". Both node partition and node
> constraints are scheduling policies. The node label should be part of
> resource (as an attribute), and be used in two different policies. If we
> position the node label this way, there is only one type of node label.
> This is related to how we define/abstract "resource" in Yarn. Currently,
> only CPU/memory (extending with disk/network bandwidth) considered as
> resource for a node. As you mentioned in Summit slides, GPU could also be a
> resource. These are all consumable resources/attributes provided by node
> while node label is non-consumable resource/attribute provided by a node.
>
> Lei
>
> -----Original Message-----
> From: Wangda Tan [mailto:wheeleast@gmail.com]
> Sent: Thursday, June 25, 2015 3:51 PM
> To: yarn-dev@hadoop.apache.org
> Subject: Re: Node Label, partition or an attribute of node?
>
> Hi Lei,
> To clarify, node label has two types, one is node partition, which is
> supported already. Another is node constraint (you can also call it node
> attribute), which is still under design. Both of them are string typed.
>
> And node label (including partition and attribute), can be either
> per-application level and per-container level, this is supported by API
> already.
>
> And you can take a look at my summit slides:
> http://www.slideshare.net/Hadoop_Summit/node-labels-in-yarn-49792443.
> Which
> may help you better understand partition / constraint and relationship
> between them.
>
> Thanks,
> Wangda
>
> On Thu, Jun 25, 2015 at 12:34 PM, Lei Guo <le...@huawei.com> wrote:
>
> > The definition of label in YARN-796 is pretty clear,
> >
> > * Node Label label that describes a node. Node can have multiple
> > labels
> >
> > * Label expression logical combination of labels (using && and,
> > || or, ! not)
> > With this definition (without considering the implementation), a node
> > label could be treated as a Boolean type attribute of a node. Assuming
> the
> > administrator define label1, label2, label3 in the system, and associate
> > node1 with "label1, label2", this means the value of attribute label1 and
> > label2 for node1 is true, while the value of attribute label3 will be
> false
> > for node1.
> >
> > In current Yarn implementation, seems the resource partitioning has been
> > treated as the primary use case for node label, and some
> > design/implementation of node label mainly consider on the partitioning
> > case, even the multiple label support has been disabled in YARN-2694. To
> > cover the workload resource request use case, YARN-3409 was proposed
> while
> > the description/example of constraint node label in YARN-3409 seems more
> > like a String type attribute for a node instead of Boolean type anymore.
> I
> > got confused about node label.
> >
> > From my view, node label is one attribute of a node, and it could be used
> > for different scheduling scenarios. No matter resource partitioning or
> job
> > scheduling constraints, a node label is just a label on a node. For
> > partitioning, we can have one object calling partition or resource group
> > representing the group of nodes with certain label; for workload
> scheduling
> > constraints purpose, we may want to extend the current application level
> > constraint to be container level constraint, for one application, the
> > master may request one type of resource while the slave request another
> > type of resource.
> >
> > Comments?
> >
> > Lei Guo
> > Senior Architect, Huawei Canada Research Centre
> >
> >
>
RE: Node Label, partition or an attribute of node?
Posted by Lei Guo <le...@huawei.com>.
Hi, Wangda,
Actually, it's the two types of node label causing the confusion for me. For a scheduling system, it's to find the best match between "resource" and "workload" via different "policies". Both node partition and node constraints are scheduling policies. The node label should be part of resource (as an attribute), and be used in two different policies. If we position the node label this way, there is only one type of node label. This is related to how we define/abstract "resource" in Yarn. Currently, only CPU/memory (extending with disk/network bandwidth) considered as resource for a node. As you mentioned in Summit slides, GPU could also be a resource. These are all consumable resources/attributes provided by node while node label is non-consumable resource/attribute provided by a node.
Lei
-----Original Message-----
From: Wangda Tan [mailto:wheeleast@gmail.com]
Sent: Thursday, June 25, 2015 3:51 PM
To: yarn-dev@hadoop.apache.org
Subject: Re: Node Label, partition or an attribute of node?
Hi Lei,
To clarify, node label has two types, one is node partition, which is
supported already. Another is node constraint (you can also call it node
attribute), which is still under design. Both of them are string typed.
And node label (including partition and attribute), can be either
per-application level and per-container level, this is supported by API
already.
And you can take a look at my summit slides:
http://www.slideshare.net/Hadoop_Summit/node-labels-in-yarn-49792443. Which
may help you better understand partition / constraint and relationship
between them.
Thanks,
Wangda
On Thu, Jun 25, 2015 at 12:34 PM, Lei Guo <le...@huawei.com> wrote:
> The definition of label in YARN-796 is pretty clear,
>
> * Node Label label that describes a node. Node can have multiple
> labels
>
> * Label expression logical combination of labels (using && and,
> || or, ! not)
> With this definition (without considering the implementation), a node
> label could be treated as a Boolean type attribute of a node. Assuming the
> administrator define label1, label2, label3 in the system, and associate
> node1 with "label1, label2", this means the value of attribute label1 and
> label2 for node1 is true, while the value of attribute label3 will be false
> for node1.
>
> In current Yarn implementation, seems the resource partitioning has been
> treated as the primary use case for node label, and some
> design/implementation of node label mainly consider on the partitioning
> case, even the multiple label support has been disabled in YARN-2694. To
> cover the workload resource request use case, YARN-3409 was proposed while
> the description/example of constraint node label in YARN-3409 seems more
> like a String type attribute for a node instead of Boolean type anymore. I
> got confused about node label.
>
> From my view, node label is one attribute of a node, and it could be used
> for different scheduling scenarios. No matter resource partitioning or job
> scheduling constraints, a node label is just a label on a node. For
> partitioning, we can have one object calling partition or resource group
> representing the group of nodes with certain label; for workload scheduling
> constraints purpose, we may want to extend the current application level
> constraint to be container level constraint, for one application, the
> master may request one type of resource while the slave request another
> type of resource.
>
> Comments?
>
> Lei Guo
> Senior Architect, Huawei Canada Research Centre
>
>
Re: Node Label, partition or an attribute of node?
Posted by Wangda Tan <wh...@gmail.com>.
Hi Lei,
To clarify, node label has two types, one is node partition, which is
supported already. Another is node constraint (you can also call it node
attribute), which is still under design. Both of them are string typed.
And node label (including partition and attribute), can be either
per-application level and per-container level, this is supported by API
already.
And you can take a look at my summit slides:
http://www.slideshare.net/Hadoop_Summit/node-labels-in-yarn-49792443. Which
may help you better understand partition / constraint and relationship
between them.
Thanks,
Wangda
On Thu, Jun 25, 2015 at 12:34 PM, Lei Guo <le...@huawei.com> wrote:
> The definition of label in YARN-796 is pretty clear,
>
> * Node Label label that describes a node. Node can have multiple
> labels
>
> * Label expression logical combination of labels (using && and,
> || or, ! not)
> With this definition (without considering the implementation), a node
> label could be treated as a Boolean type attribute of a node. Assuming the
> administrator define label1, label2, label3 in the system, and associate
> node1 with "label1, label2", this means the value of attribute label1 and
> label2 for node1 is true, while the value of attribute label3 will be false
> for node1.
>
> In current Yarn implementation, seems the resource partitioning has been
> treated as the primary use case for node label, and some
> design/implementation of node label mainly consider on the partitioning
> case, even the multiple label support has been disabled in YARN-2694. To
> cover the workload resource request use case, YARN-3409 was proposed while
> the description/example of constraint node label in YARN-3409 seems more
> like a String type attribute for a node instead of Boolean type anymore. I
> got confused about node label.
>
> From my view, node label is one attribute of a node, and it could be used
> for different scheduling scenarios. No matter resource partitioning or job
> scheduling constraints, a node label is just a label on a node. For
> partitioning, we can have one object calling partition or resource group
> representing the group of nodes with certain label; for workload scheduling
> constraints purpose, we may want to extend the current application level
> constraint to be container level constraint, for one application, the
> master may request one type of resource while the slave request another
> type of resource.
>
> Comments?
>
> Lei Guo
> Senior Architect, Huawei Canada Research Centre
>
>