You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by David Fox <da...@gmail.com> on 2015/09/29 17:29:51 UTC
YARN Fair Scheduler - maxResources limit for parent and child queues
with inheritance
Hello,
I would like to know how it works with Fair Scheduler in YARN-hadoop. I'm
trying to configure parent queue with MaxResorces in allocations file, e.i.
fair-scheduler.xml. Then I want to create child queues. As documentation
about fair scheduler says "Queues can be arranged in a hierarchy to divide
resources and configured with weights to share the cluster in specific
proportions"
http://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/FairScheduler.html
I suppose if I configure parent queue with Max Resources, the child queues
will be limited to max of the parent resources with fair shares. Is my
undersatnding of queue hierarchy mechanism right?
Example to be more concrete:
{code}
<?xml version="1.0"?>
<allocations>
<queue name="someParentQueue" type="parent">
<maxResources>111000 mb,100 vcores, 100 disks</maxResources>
</queue>
<queuePlacementPolicy>
<rule name="nestedUserQueue" create="true">
<rule name="specified" create="false"/>
</rule>
<rule name="user" create="true"/>
</queuePlacementPolicy>
</allocations>
{code}
The queues someParentQueue.user1 has different Max Resources in web UI. Is
it issue of web UI or intended behavoior of Fair Scheduler?
Thanks
Re: YARN Fair Scheduler - maxResources limit for parent and child
queues with inheritance
Posted by Ashwin Sai Shankar <as...@netflix.com>.
Yes this is a issue with webui. maxResources would be enforced at
*someParentQueue,
*so childQueues of it will get atmost ${maxResources} depending
on usage of its siblings.
On Tue, Sep 29, 2015 at 8:29 AM, David Fox <da...@gmail.com> wrote:
> Hello,
>
> I would like to know how it works with Fair Scheduler in YARN-hadoop. I'm
> trying to configure parent queue with MaxResorces in allocations file, e.i.
> fair-scheduler.xml. Then I want to create child queues. As documentation
> about fair scheduler says "Queues can be arranged in a hierarchy to divide
> resources and configured with weights to share the cluster in specific
> proportions"
> http://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/FairScheduler.html
> I suppose if I configure parent queue with Max Resources, the child queues
> will be limited to max of the parent resources with fair shares. Is my
> undersatnding of queue hierarchy mechanism right?
>
> Example to be more concrete:
> {code}
> <?xml version="1.0"?>
> <allocations>
> <queue name="someParentQueue" type="parent">
> <maxResources>111000 mb,100 vcores, 100 disks</maxResources>
> </queue>
> <queuePlacementPolicy>
> <rule name="nestedUserQueue" create="true">
> <rule name="specified" create="false"/>
> </rule>
> <rule name="user" create="true"/>
> </queuePlacementPolicy>
> </allocations>
> {code}
>
> The queues someParentQueue.user1 has different Max Resources in web UI. Is
> it issue of web UI or intended behavoior of Fair Scheduler?
>
> Thanks
>
Re: YARN Fair Scheduler - maxResources limit for parent and child
queues with inheritance
Posted by Ashwin Sai Shankar <as...@netflix.com>.
Yes this is a issue with webui. maxResources would be enforced at
*someParentQueue,
*so childQueues of it will get atmost ${maxResources} depending
on usage of its siblings.
On Tue, Sep 29, 2015 at 8:29 AM, David Fox <da...@gmail.com> wrote:
> Hello,
>
> I would like to know how it works with Fair Scheduler in YARN-hadoop. I'm
> trying to configure parent queue with MaxResorces in allocations file, e.i.
> fair-scheduler.xml. Then I want to create child queues. As documentation
> about fair scheduler says "Queues can be arranged in a hierarchy to divide
> resources and configured with weights to share the cluster in specific
> proportions"
> http://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/FairScheduler.html
> I suppose if I configure parent queue with Max Resources, the child queues
> will be limited to max of the parent resources with fair shares. Is my
> undersatnding of queue hierarchy mechanism right?
>
> Example to be more concrete:
> {code}
> <?xml version="1.0"?>
> <allocations>
> <queue name="someParentQueue" type="parent">
> <maxResources>111000 mb,100 vcores, 100 disks</maxResources>
> </queue>
> <queuePlacementPolicy>
> <rule name="nestedUserQueue" create="true">
> <rule name="specified" create="false"/>
> </rule>
> <rule name="user" create="true"/>
> </queuePlacementPolicy>
> </allocations>
> {code}
>
> The queues someParentQueue.user1 has different Max Resources in web UI. Is
> it issue of web UI or intended behavoior of Fair Scheduler?
>
> Thanks
>
Re: YARN Fair Scheduler - maxResources limit for parent and child
queues with inheritance
Posted by Ashwin Sai Shankar <as...@netflix.com>.
Yes this is a issue with webui. maxResources would be enforced at
*someParentQueue,
*so childQueues of it will get atmost ${maxResources} depending
on usage of its siblings.
On Tue, Sep 29, 2015 at 8:29 AM, David Fox <da...@gmail.com> wrote:
> Hello,
>
> I would like to know how it works with Fair Scheduler in YARN-hadoop. I'm
> trying to configure parent queue with MaxResorces in allocations file, e.i.
> fair-scheduler.xml. Then I want to create child queues. As documentation
> about fair scheduler says "Queues can be arranged in a hierarchy to divide
> resources and configured with weights to share the cluster in specific
> proportions"
> http://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/FairScheduler.html
> I suppose if I configure parent queue with Max Resources, the child queues
> will be limited to max of the parent resources with fair shares. Is my
> undersatnding of queue hierarchy mechanism right?
>
> Example to be more concrete:
> {code}
> <?xml version="1.0"?>
> <allocations>
> <queue name="someParentQueue" type="parent">
> <maxResources>111000 mb,100 vcores, 100 disks</maxResources>
> </queue>
> <queuePlacementPolicy>
> <rule name="nestedUserQueue" create="true">
> <rule name="specified" create="false"/>
> </rule>
> <rule name="user" create="true"/>
> </queuePlacementPolicy>
> </allocations>
> {code}
>
> The queues someParentQueue.user1 has different Max Resources in web UI. Is
> it issue of web UI or intended behavoior of Fair Scheduler?
>
> Thanks
>
Re: YARN Fair Scheduler - maxResources limit for parent and child
queues with inheritance
Posted by Ashwin Sai Shankar <as...@netflix.com>.
Yes this is a issue with webui. maxResources would be enforced at
*someParentQueue,
*so childQueues of it will get atmost ${maxResources} depending
on usage of its siblings.
On Tue, Sep 29, 2015 at 8:29 AM, David Fox <da...@gmail.com> wrote:
> Hello,
>
> I would like to know how it works with Fair Scheduler in YARN-hadoop. I'm
> trying to configure parent queue with MaxResorces in allocations file, e.i.
> fair-scheduler.xml. Then I want to create child queues. As documentation
> about fair scheduler says "Queues can be arranged in a hierarchy to divide
> resources and configured with weights to share the cluster in specific
> proportions"
> http://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/FairScheduler.html
> I suppose if I configure parent queue with Max Resources, the child queues
> will be limited to max of the parent resources with fair shares. Is my
> undersatnding of queue hierarchy mechanism right?
>
> Example to be more concrete:
> {code}
> <?xml version="1.0"?>
> <allocations>
> <queue name="someParentQueue" type="parent">
> <maxResources>111000 mb,100 vcores, 100 disks</maxResources>
> </queue>
> <queuePlacementPolicy>
> <rule name="nestedUserQueue" create="true">
> <rule name="specified" create="false"/>
> </rule>
> <rule name="user" create="true"/>
> </queuePlacementPolicy>
> </allocations>
> {code}
>
> The queues someParentQueue.user1 has different Max Resources in web UI. Is
> it issue of web UI or intended behavoior of Fair Scheduler?
>
> Thanks
>