You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Marcin Tustin <mt...@handybook.com> on 2016/03/02 15:39:28 UTC
Yarn with CapacityScheduler will only schedule two applications -
open to consultants today
Hi All,
We're hitting this issue. If you're a consultant with capacity today (2
March 2016 EST in New York) please feel free to contact me on or off list.
In terms of stack, we're using Yarn 2.7.1.2.3 from the latest Hortonworks
distribution. It's possible we're hitting this bug:
https://issues.apache.org/jira/browse/YARN-3216
The behaviour we're seeing is that even though we have 12 machines in our
yarn cluster, yarn will only schedule two applications. The breakdown of
machines is:
10 machines with the data label
2 with the yarn label
We have three queues: interactive, noninteractive, and default.
We're expecting to split capacity in the data label 20%/80% and default
100% of yarn label.
We have the following capacity scheduler config (Key=value format taken
from ambari):
yarn.scheduler.capacity.maximum-am-resource-percent=100
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.queue-mappings-override.enable=true
yarn.scheduler.capacity.root.accessible-node-labels=*
yarn.scheduler.capacity.root.accessible-node-labels.data.capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.data.maximum-capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.yarn.capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.yarn.maximum-capacity=100
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default-node-label-expression=data
yarn.scheduler.capacity.root.default.accessible-node-labels=yarn
yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.capacity=100
yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.maximum-capacity=100
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=50
yarn.scheduler.capacity.root.default.default-node-label-expression=yarn
yarn.scheduler.capacity.root.default.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.default.maximum-capacity=100
yarn.scheduler.capacity.root.default.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.default.ordering-policy=fair
yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=100
yarn.scheduler.capacity.root.interactive.accessible-node-labels=data
yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.capacity=20
yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.maximum-capacity=100
yarn.scheduler.capacity.root.interactive.acl_administer_queue=*
yarn.scheduler.capacity.root.interactive.acl_submit_applications=*
yarn.scheduler.capacity.root.interactive.capacity=10
yarn.scheduler.capacity.root.interactive.maximum-am-resource-percent=50
yarn.scheduler.capacity.root.interactive.maximum-applications=2000
yarn.scheduler.capacity.root.interactive.maximum-capacity=100
yarn.scheduler.capacity.root.interactive.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.interactive.ordering-policy=fifo
yarn.scheduler.capacity.root.interactive.state=RUNNING
yarn.scheduler.capacity.root.interactive.user-limit-factor=100
yarn.scheduler.capacity.root.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.maximum-capacity=100
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels=data
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.capacity=80
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-capacity=80
yarn.scheduler.capacity.root.noninteractive.acl_submit_applications=*
yarn.scheduler.capacity.root.noninteractive.capacity=40
yarn.scheduler.capacity.root.noninteractive.default-node-label-expression=data
yarn.scheduler.capacity.root.noninteractive.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.noninteractive.maximum-applications=8000
yarn.scheduler.capacity.root.noninteractive.maximum-capacity=100
yarn.scheduler.capacity.root.noninteractive.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.noninteractive.ordering-policy=fair
yarn.scheduler.capacity.root.noninteractive.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.noninteractive.state=RUNNING
yarn.scheduler.capacity.root.noninteractive.user-limit-factor=100
yarn.scheduler.capacity.root.queues=default,interactive,noninteractive
yarn.scheduler.capacity.root.user-limit-factor=40
Also yarn.resourcemanager.scheduler.class is
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.
Any suggestions gratefully received.
Marcin
--
Want to work at Handy? Check out our culture deck and open roles
<http://www.handy.com/careers>
Latest news <http://www.handy.com/press> at Handy
Handy just raised $50m
<http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/> led
by Fidelity
Re: Yarn with CapacityScheduler will only schedule two applications -
open to consultants today
Posted by Marcin Tustin <mt...@handybook.com>.
Hi Naga,
You're quite right. Our fix was balance out the resources in each label
more, in order to have the bug bite us less. Beyond that I would direct any
readers to your text and my text below.
Marcin
On Wednesday, March 2, 2016, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Marcin,
>
> *we're seeing is that even though we have 12 machines in our yarn cluster,
> yarn will only schedule two applications. *
> If this is the problem you are facing with the below configuration then
> its mostly YARN-3216 <https://issues.apache.org/jira/browse/YARN-3216>
> itself.
> And i think its not yet out in any released hadoop version yet (not sure
> about HDP) also not aware of the version 2.7.1.2.3 (may be HDP version
> number)
> IIUC you would have tried submitting in 2 diff partitions "data" and
> "yarn" hence 2 if not for a given partition only one would have run (if the
> am resource is less than the mimium size CS allows atleast 1 am to run)
>
> Given that you have already identified the issue, what more are you
> expecting ?
>
> Regards,
> + Naga
> ------------------------------
> *From:* Marcin Tustin [mtustin@handybook.com
> <javascript:_e(%7B%7D,'cvml','mtustin@handybook.com');>]
> *Sent:* Wednesday, March 02, 2016 20:09
> *To:* user@hadoop.apache.org
> <javascript:_e(%7B%7D,'cvml','user@hadoop.apache.org');>
> *Subject:* Yarn with CapacityScheduler will only schedule two
> applications - open to consultants today
>
> Hi All,
>
> We're hitting this issue. If you're a consultant with capacity today (2
> March 2016 EST in New York) please feel free to contact me on or off list.
>
> In terms of stack, we're using Yarn 2.7.1.2.3 from the latest Hortonworks
> distribution. It's possible we're hitting this bug:
> https://issues.apache.org/jira/browse/YARN-3216
>
> The behaviour we're seeing is that even though we have 12 machines in our
> yarn cluster, yarn will only schedule two applications. The breakdown of
> machines is:
> 10 machines with the data label
> 2 with the yarn label
>
> We have three queues: interactive, noninteractive, and default.
> We're expecting to split capacity in the data label 20%/80% and default
> 100% of yarn label.
>
> We have the following capacity scheduler config (Key=value format taken
> from ambari):
>
> yarn.scheduler.capacity.maximum-am-resource-percent=100
> yarn.scheduler.capacity.maximum-applications=10000
> yarn.scheduler.capacity.node-locality-delay=40
> yarn.scheduler.capacity.queue-mappings-override.enable=true
> yarn.scheduler.capacity.root.accessible-node-labels=*
> yarn.scheduler.capacity.root.accessible-node-labels.data.capacity=100
>
> yarn.scheduler.capacity.root.accessible-node-labels.data.maximum-capacity=100
> yarn.scheduler.capacity.root.accessible-node-labels.yarn.capacity=100
>
> yarn.scheduler.capacity.root.accessible-node-labels.yarn.maximum-capacity=100
> yarn.scheduler.capacity.root.acl_administer_queue=*
> yarn.scheduler.capacity.root.capacity=100
> yarn.scheduler.capacity.root.default-node-label-expression=data
> yarn.scheduler.capacity.root.default.accessible-node-labels=yarn
>
> yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.capacity=100
>
> yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.maximum-capacity=100
> yarn.scheduler.capacity.root.default.acl_submit_applications=*
> yarn.scheduler.capacity.root.default.capacity=50
> yarn.scheduler.capacity.root.default.default-node-label-expression=yarn
> yarn.scheduler.capacity.root.default.maximum-am-resource-percent=80
> yarn.scheduler.capacity.root.default.maximum-capacity=100
> yarn.scheduler.capacity.root.default.minimum-user-limit-percent=100
> yarn.scheduler.capacity.root.default.ordering-policy=fair
>
> yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
> yarn.scheduler.capacity.root.default.state=RUNNING
> yarn.scheduler.capacity.root.default.user-limit-factor=100
> yarn.scheduler.capacity.root.interactive.accessible-node-labels=data
>
> yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.capacity=20
>
> yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.maximum-capacity=100
> yarn.scheduler.capacity.root.interactive.acl_administer_queue=*
> yarn.scheduler.capacity.root.interactive.acl_submit_applications=*
> yarn.scheduler.capacity.root.interactive.capacity=10
> yarn.scheduler.capacity.root.interactive.maximum-am-resource-percent=50
> yarn.scheduler.capacity.root.interactive.maximum-applications=2000
> yarn.scheduler.capacity.root.interactive.maximum-capacity=100
> yarn.scheduler.capacity.root.interactive.minimum-user-limit-percent=100
> yarn.scheduler.capacity.root.interactive.ordering-policy=fifo
> yarn.scheduler.capacity.root.interactive.state=RUNNING
> yarn.scheduler.capacity.root.interactive.user-limit-factor=100
> yarn.scheduler.capacity.root.maximum-am-resource-percent=80
> yarn.scheduler.capacity.root.maximum-capacity=100
> yarn.scheduler.capacity.root.noninteractive.accessible-node-labels=data
>
> yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.capacity=80
>
> yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-am-resource-percent=80
>
> yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-capacity=80
> yarn.scheduler.capacity.root.noninteractive.acl_submit_applications=*
> yarn.scheduler.capacity.root.noninteractive.capacity=40
>
> yarn.scheduler.capacity.root.noninteractive.default-node-label-expression=data
> yarn.scheduler.capacity.root.noninteractive.maximum-am-resource-percent=80
> yarn.scheduler.capacity.root.noninteractive.maximum-applications=8000
> yarn.scheduler.capacity.root.noninteractive.maximum-capacity=100
> yarn.scheduler.capacity.root.noninteractive.minimum-user-limit-percent=100
> yarn.scheduler.capacity.root.noninteractive.ordering-policy=fair
>
> yarn.scheduler.capacity.root.noninteractive.ordering-policy.fair.enable-size-based-weight=false
> yarn.scheduler.capacity.root.noninteractive.state=RUNNING
> yarn.scheduler.capacity.root.noninteractive.user-limit-factor=100
> yarn.scheduler.capacity.root.queues=default,interactive,noninteractive
> yarn.scheduler.capacity.root.user-limit-factor=40
>
>
> Also yarn.resourcemanager.scheduler.class is
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.
>
> Any suggestions gratefully received.
>
> Marcin
>
> Want to work at Handy? Check out our culture deck and open roles
> <http://www.handy.com/careers>
> Latest news <http://www.handy.com/press> at Handy
> Handy just raised $50m
> <http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/> led
> by Fidelity
>
>
--
Want to work at Handy? Check out our culture deck and open roles
<http://www.handy.com/careers>
Latest news <http://www.handy.com/press> at Handy
Handy just raised $50m
<http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/> led
by Fidelity
Re: Yarn with CapacityScheduler will only schedule two applications -
open to consultants today
Posted by Marcin Tustin <mt...@handybook.com>.
Hi Naga,
You're quite right. Our fix was balance out the resources in each label
more, in order to have the bug bite us less. Beyond that I would direct any
readers to your text and my text below.
Marcin
On Wednesday, March 2, 2016, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Marcin,
>
> *we're seeing is that even though we have 12 machines in our yarn cluster,
> yarn will only schedule two applications. *
> If this is the problem you are facing with the below configuration then
> its mostly YARN-3216 <https://issues.apache.org/jira/browse/YARN-3216>
> itself.
> And i think its not yet out in any released hadoop version yet (not sure
> about HDP) also not aware of the version 2.7.1.2.3 (may be HDP version
> number)
> IIUC you would have tried submitting in 2 diff partitions "data" and
> "yarn" hence 2 if not for a given partition only one would have run (if the
> am resource is less than the mimium size CS allows atleast 1 am to run)
>
> Given that you have already identified the issue, what more are you
> expecting ?
>
> Regards,
> + Naga
> ------------------------------
> *From:* Marcin Tustin [mtustin@handybook.com
> <javascript:_e(%7B%7D,'cvml','mtustin@handybook.com');>]
> *Sent:* Wednesday, March 02, 2016 20:09
> *To:* user@hadoop.apache.org
> <javascript:_e(%7B%7D,'cvml','user@hadoop.apache.org');>
> *Subject:* Yarn with CapacityScheduler will only schedule two
> applications - open to consultants today
>
> Hi All,
>
> We're hitting this issue. If you're a consultant with capacity today (2
> March 2016 EST in New York) please feel free to contact me on or off list.
>
> In terms of stack, we're using Yarn 2.7.1.2.3 from the latest Hortonworks
> distribution. It's possible we're hitting this bug:
> https://issues.apache.org/jira/browse/YARN-3216
>
> The behaviour we're seeing is that even though we have 12 machines in our
> yarn cluster, yarn will only schedule two applications. The breakdown of
> machines is:
> 10 machines with the data label
> 2 with the yarn label
>
> We have three queues: interactive, noninteractive, and default.
> We're expecting to split capacity in the data label 20%/80% and default
> 100% of yarn label.
>
> We have the following capacity scheduler config (Key=value format taken
> from ambari):
>
> yarn.scheduler.capacity.maximum-am-resource-percent=100
> yarn.scheduler.capacity.maximum-applications=10000
> yarn.scheduler.capacity.node-locality-delay=40
> yarn.scheduler.capacity.queue-mappings-override.enable=true
> yarn.scheduler.capacity.root.accessible-node-labels=*
> yarn.scheduler.capacity.root.accessible-node-labels.data.capacity=100
>
> yarn.scheduler.capacity.root.accessible-node-labels.data.maximum-capacity=100
> yarn.scheduler.capacity.root.accessible-node-labels.yarn.capacity=100
>
> yarn.scheduler.capacity.root.accessible-node-labels.yarn.maximum-capacity=100
> yarn.scheduler.capacity.root.acl_administer_queue=*
> yarn.scheduler.capacity.root.capacity=100
> yarn.scheduler.capacity.root.default-node-label-expression=data
> yarn.scheduler.capacity.root.default.accessible-node-labels=yarn
>
> yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.capacity=100
>
> yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.maximum-capacity=100
> yarn.scheduler.capacity.root.default.acl_submit_applications=*
> yarn.scheduler.capacity.root.default.capacity=50
> yarn.scheduler.capacity.root.default.default-node-label-expression=yarn
> yarn.scheduler.capacity.root.default.maximum-am-resource-percent=80
> yarn.scheduler.capacity.root.default.maximum-capacity=100
> yarn.scheduler.capacity.root.default.minimum-user-limit-percent=100
> yarn.scheduler.capacity.root.default.ordering-policy=fair
>
> yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
> yarn.scheduler.capacity.root.default.state=RUNNING
> yarn.scheduler.capacity.root.default.user-limit-factor=100
> yarn.scheduler.capacity.root.interactive.accessible-node-labels=data
>
> yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.capacity=20
>
> yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.maximum-capacity=100
> yarn.scheduler.capacity.root.interactive.acl_administer_queue=*
> yarn.scheduler.capacity.root.interactive.acl_submit_applications=*
> yarn.scheduler.capacity.root.interactive.capacity=10
> yarn.scheduler.capacity.root.interactive.maximum-am-resource-percent=50
> yarn.scheduler.capacity.root.interactive.maximum-applications=2000
> yarn.scheduler.capacity.root.interactive.maximum-capacity=100
> yarn.scheduler.capacity.root.interactive.minimum-user-limit-percent=100
> yarn.scheduler.capacity.root.interactive.ordering-policy=fifo
> yarn.scheduler.capacity.root.interactive.state=RUNNING
> yarn.scheduler.capacity.root.interactive.user-limit-factor=100
> yarn.scheduler.capacity.root.maximum-am-resource-percent=80
> yarn.scheduler.capacity.root.maximum-capacity=100
> yarn.scheduler.capacity.root.noninteractive.accessible-node-labels=data
>
> yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.capacity=80
>
> yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-am-resource-percent=80
>
> yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-capacity=80
> yarn.scheduler.capacity.root.noninteractive.acl_submit_applications=*
> yarn.scheduler.capacity.root.noninteractive.capacity=40
>
> yarn.scheduler.capacity.root.noninteractive.default-node-label-expression=data
> yarn.scheduler.capacity.root.noninteractive.maximum-am-resource-percent=80
> yarn.scheduler.capacity.root.noninteractive.maximum-applications=8000
> yarn.scheduler.capacity.root.noninteractive.maximum-capacity=100
> yarn.scheduler.capacity.root.noninteractive.minimum-user-limit-percent=100
> yarn.scheduler.capacity.root.noninteractive.ordering-policy=fair
>
> yarn.scheduler.capacity.root.noninteractive.ordering-policy.fair.enable-size-based-weight=false
> yarn.scheduler.capacity.root.noninteractive.state=RUNNING
> yarn.scheduler.capacity.root.noninteractive.user-limit-factor=100
> yarn.scheduler.capacity.root.queues=default,interactive,noninteractive
> yarn.scheduler.capacity.root.user-limit-factor=40
>
>
> Also yarn.resourcemanager.scheduler.class is
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.
>
> Any suggestions gratefully received.
>
> Marcin
>
> Want to work at Handy? Check out our culture deck and open roles
> <http://www.handy.com/careers>
> Latest news <http://www.handy.com/press> at Handy
> Handy just raised $50m
> <http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/> led
> by Fidelity
>
>
--
Want to work at Handy? Check out our culture deck and open roles
<http://www.handy.com/careers>
Latest news <http://www.handy.com/press> at Handy
Handy just raised $50m
<http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/> led
by Fidelity
Re: Yarn with CapacityScheduler will only schedule two applications -
open to consultants today
Posted by Marcin Tustin <mt...@handybook.com>.
Hi Naga,
You're quite right. Our fix was balance out the resources in each label
more, in order to have the bug bite us less. Beyond that I would direct any
readers to your text and my text below.
Marcin
On Wednesday, March 2, 2016, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Marcin,
>
> *we're seeing is that even though we have 12 machines in our yarn cluster,
> yarn will only schedule two applications. *
> If this is the problem you are facing with the below configuration then
> its mostly YARN-3216 <https://issues.apache.org/jira/browse/YARN-3216>
> itself.
> And i think its not yet out in any released hadoop version yet (not sure
> about HDP) also not aware of the version 2.7.1.2.3 (may be HDP version
> number)
> IIUC you would have tried submitting in 2 diff partitions "data" and
> "yarn" hence 2 if not for a given partition only one would have run (if the
> am resource is less than the mimium size CS allows atleast 1 am to run)
>
> Given that you have already identified the issue, what more are you
> expecting ?
>
> Regards,
> + Naga
> ------------------------------
> *From:* Marcin Tustin [mtustin@handybook.com
> <javascript:_e(%7B%7D,'cvml','mtustin@handybook.com');>]
> *Sent:* Wednesday, March 02, 2016 20:09
> *To:* user@hadoop.apache.org
> <javascript:_e(%7B%7D,'cvml','user@hadoop.apache.org');>
> *Subject:* Yarn with CapacityScheduler will only schedule two
> applications - open to consultants today
>
> Hi All,
>
> We're hitting this issue. If you're a consultant with capacity today (2
> March 2016 EST in New York) please feel free to contact me on or off list.
>
> In terms of stack, we're using Yarn 2.7.1.2.3 from the latest Hortonworks
> distribution. It's possible we're hitting this bug:
> https://issues.apache.org/jira/browse/YARN-3216
>
> The behaviour we're seeing is that even though we have 12 machines in our
> yarn cluster, yarn will only schedule two applications. The breakdown of
> machines is:
> 10 machines with the data label
> 2 with the yarn label
>
> We have three queues: interactive, noninteractive, and default.
> We're expecting to split capacity in the data label 20%/80% and default
> 100% of yarn label.
>
> We have the following capacity scheduler config (Key=value format taken
> from ambari):
>
> yarn.scheduler.capacity.maximum-am-resource-percent=100
> yarn.scheduler.capacity.maximum-applications=10000
> yarn.scheduler.capacity.node-locality-delay=40
> yarn.scheduler.capacity.queue-mappings-override.enable=true
> yarn.scheduler.capacity.root.accessible-node-labels=*
> yarn.scheduler.capacity.root.accessible-node-labels.data.capacity=100
>
> yarn.scheduler.capacity.root.accessible-node-labels.data.maximum-capacity=100
> yarn.scheduler.capacity.root.accessible-node-labels.yarn.capacity=100
>
> yarn.scheduler.capacity.root.accessible-node-labels.yarn.maximum-capacity=100
> yarn.scheduler.capacity.root.acl_administer_queue=*
> yarn.scheduler.capacity.root.capacity=100
> yarn.scheduler.capacity.root.default-node-label-expression=data
> yarn.scheduler.capacity.root.default.accessible-node-labels=yarn
>
> yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.capacity=100
>
> yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.maximum-capacity=100
> yarn.scheduler.capacity.root.default.acl_submit_applications=*
> yarn.scheduler.capacity.root.default.capacity=50
> yarn.scheduler.capacity.root.default.default-node-label-expression=yarn
> yarn.scheduler.capacity.root.default.maximum-am-resource-percent=80
> yarn.scheduler.capacity.root.default.maximum-capacity=100
> yarn.scheduler.capacity.root.default.minimum-user-limit-percent=100
> yarn.scheduler.capacity.root.default.ordering-policy=fair
>
> yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
> yarn.scheduler.capacity.root.default.state=RUNNING
> yarn.scheduler.capacity.root.default.user-limit-factor=100
> yarn.scheduler.capacity.root.interactive.accessible-node-labels=data
>
> yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.capacity=20
>
> yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.maximum-capacity=100
> yarn.scheduler.capacity.root.interactive.acl_administer_queue=*
> yarn.scheduler.capacity.root.interactive.acl_submit_applications=*
> yarn.scheduler.capacity.root.interactive.capacity=10
> yarn.scheduler.capacity.root.interactive.maximum-am-resource-percent=50
> yarn.scheduler.capacity.root.interactive.maximum-applications=2000
> yarn.scheduler.capacity.root.interactive.maximum-capacity=100
> yarn.scheduler.capacity.root.interactive.minimum-user-limit-percent=100
> yarn.scheduler.capacity.root.interactive.ordering-policy=fifo
> yarn.scheduler.capacity.root.interactive.state=RUNNING
> yarn.scheduler.capacity.root.interactive.user-limit-factor=100
> yarn.scheduler.capacity.root.maximum-am-resource-percent=80
> yarn.scheduler.capacity.root.maximum-capacity=100
> yarn.scheduler.capacity.root.noninteractive.accessible-node-labels=data
>
> yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.capacity=80
>
> yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-am-resource-percent=80
>
> yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-capacity=80
> yarn.scheduler.capacity.root.noninteractive.acl_submit_applications=*
> yarn.scheduler.capacity.root.noninteractive.capacity=40
>
> yarn.scheduler.capacity.root.noninteractive.default-node-label-expression=data
> yarn.scheduler.capacity.root.noninteractive.maximum-am-resource-percent=80
> yarn.scheduler.capacity.root.noninteractive.maximum-applications=8000
> yarn.scheduler.capacity.root.noninteractive.maximum-capacity=100
> yarn.scheduler.capacity.root.noninteractive.minimum-user-limit-percent=100
> yarn.scheduler.capacity.root.noninteractive.ordering-policy=fair
>
> yarn.scheduler.capacity.root.noninteractive.ordering-policy.fair.enable-size-based-weight=false
> yarn.scheduler.capacity.root.noninteractive.state=RUNNING
> yarn.scheduler.capacity.root.noninteractive.user-limit-factor=100
> yarn.scheduler.capacity.root.queues=default,interactive,noninteractive
> yarn.scheduler.capacity.root.user-limit-factor=40
>
>
> Also yarn.resourcemanager.scheduler.class is
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.
>
> Any suggestions gratefully received.
>
> Marcin
>
> Want to work at Handy? Check out our culture deck and open roles
> <http://www.handy.com/careers>
> Latest news <http://www.handy.com/press> at Handy
> Handy just raised $50m
> <http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/> led
> by Fidelity
>
>
--
Want to work at Handy? Check out our culture deck and open roles
<http://www.handy.com/careers>
Latest news <http://www.handy.com/press> at Handy
Handy just raised $50m
<http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/> led
by Fidelity
Re: Yarn with CapacityScheduler will only schedule two applications -
open to consultants today
Posted by Marcin Tustin <mt...@handybook.com>.
Hi Naga,
You're quite right. Our fix was balance out the resources in each label
more, in order to have the bug bite us less. Beyond that I would direct any
readers to your text and my text below.
Marcin
On Wednesday, March 2, 2016, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Marcin,
>
> *we're seeing is that even though we have 12 machines in our yarn cluster,
> yarn will only schedule two applications. *
> If this is the problem you are facing with the below configuration then
> its mostly YARN-3216 <https://issues.apache.org/jira/browse/YARN-3216>
> itself.
> And i think its not yet out in any released hadoop version yet (not sure
> about HDP) also not aware of the version 2.7.1.2.3 (may be HDP version
> number)
> IIUC you would have tried submitting in 2 diff partitions "data" and
> "yarn" hence 2 if not for a given partition only one would have run (if the
> am resource is less than the mimium size CS allows atleast 1 am to run)
>
> Given that you have already identified the issue, what more are you
> expecting ?
>
> Regards,
> + Naga
> ------------------------------
> *From:* Marcin Tustin [mtustin@handybook.com
> <javascript:_e(%7B%7D,'cvml','mtustin@handybook.com');>]
> *Sent:* Wednesday, March 02, 2016 20:09
> *To:* user@hadoop.apache.org
> <javascript:_e(%7B%7D,'cvml','user@hadoop.apache.org');>
> *Subject:* Yarn with CapacityScheduler will only schedule two
> applications - open to consultants today
>
> Hi All,
>
> We're hitting this issue. If you're a consultant with capacity today (2
> March 2016 EST in New York) please feel free to contact me on or off list.
>
> In terms of stack, we're using Yarn 2.7.1.2.3 from the latest Hortonworks
> distribution. It's possible we're hitting this bug:
> https://issues.apache.org/jira/browse/YARN-3216
>
> The behaviour we're seeing is that even though we have 12 machines in our
> yarn cluster, yarn will only schedule two applications. The breakdown of
> machines is:
> 10 machines with the data label
> 2 with the yarn label
>
> We have three queues: interactive, noninteractive, and default.
> We're expecting to split capacity in the data label 20%/80% and default
> 100% of yarn label.
>
> We have the following capacity scheduler config (Key=value format taken
> from ambari):
>
> yarn.scheduler.capacity.maximum-am-resource-percent=100
> yarn.scheduler.capacity.maximum-applications=10000
> yarn.scheduler.capacity.node-locality-delay=40
> yarn.scheduler.capacity.queue-mappings-override.enable=true
> yarn.scheduler.capacity.root.accessible-node-labels=*
> yarn.scheduler.capacity.root.accessible-node-labels.data.capacity=100
>
> yarn.scheduler.capacity.root.accessible-node-labels.data.maximum-capacity=100
> yarn.scheduler.capacity.root.accessible-node-labels.yarn.capacity=100
>
> yarn.scheduler.capacity.root.accessible-node-labels.yarn.maximum-capacity=100
> yarn.scheduler.capacity.root.acl_administer_queue=*
> yarn.scheduler.capacity.root.capacity=100
> yarn.scheduler.capacity.root.default-node-label-expression=data
> yarn.scheduler.capacity.root.default.accessible-node-labels=yarn
>
> yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.capacity=100
>
> yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.maximum-capacity=100
> yarn.scheduler.capacity.root.default.acl_submit_applications=*
> yarn.scheduler.capacity.root.default.capacity=50
> yarn.scheduler.capacity.root.default.default-node-label-expression=yarn
> yarn.scheduler.capacity.root.default.maximum-am-resource-percent=80
> yarn.scheduler.capacity.root.default.maximum-capacity=100
> yarn.scheduler.capacity.root.default.minimum-user-limit-percent=100
> yarn.scheduler.capacity.root.default.ordering-policy=fair
>
> yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
> yarn.scheduler.capacity.root.default.state=RUNNING
> yarn.scheduler.capacity.root.default.user-limit-factor=100
> yarn.scheduler.capacity.root.interactive.accessible-node-labels=data
>
> yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.capacity=20
>
> yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.maximum-capacity=100
> yarn.scheduler.capacity.root.interactive.acl_administer_queue=*
> yarn.scheduler.capacity.root.interactive.acl_submit_applications=*
> yarn.scheduler.capacity.root.interactive.capacity=10
> yarn.scheduler.capacity.root.interactive.maximum-am-resource-percent=50
> yarn.scheduler.capacity.root.interactive.maximum-applications=2000
> yarn.scheduler.capacity.root.interactive.maximum-capacity=100
> yarn.scheduler.capacity.root.interactive.minimum-user-limit-percent=100
> yarn.scheduler.capacity.root.interactive.ordering-policy=fifo
> yarn.scheduler.capacity.root.interactive.state=RUNNING
> yarn.scheduler.capacity.root.interactive.user-limit-factor=100
> yarn.scheduler.capacity.root.maximum-am-resource-percent=80
> yarn.scheduler.capacity.root.maximum-capacity=100
> yarn.scheduler.capacity.root.noninteractive.accessible-node-labels=data
>
> yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.capacity=80
>
> yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-am-resource-percent=80
>
> yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-capacity=80
> yarn.scheduler.capacity.root.noninteractive.acl_submit_applications=*
> yarn.scheduler.capacity.root.noninteractive.capacity=40
>
> yarn.scheduler.capacity.root.noninteractive.default-node-label-expression=data
> yarn.scheduler.capacity.root.noninteractive.maximum-am-resource-percent=80
> yarn.scheduler.capacity.root.noninteractive.maximum-applications=8000
> yarn.scheduler.capacity.root.noninteractive.maximum-capacity=100
> yarn.scheduler.capacity.root.noninteractive.minimum-user-limit-percent=100
> yarn.scheduler.capacity.root.noninteractive.ordering-policy=fair
>
> yarn.scheduler.capacity.root.noninteractive.ordering-policy.fair.enable-size-based-weight=false
> yarn.scheduler.capacity.root.noninteractive.state=RUNNING
> yarn.scheduler.capacity.root.noninteractive.user-limit-factor=100
> yarn.scheduler.capacity.root.queues=default,interactive,noninteractive
> yarn.scheduler.capacity.root.user-limit-factor=40
>
>
> Also yarn.resourcemanager.scheduler.class is
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.
>
> Any suggestions gratefully received.
>
> Marcin
>
> Want to work at Handy? Check out our culture deck and open roles
> <http://www.handy.com/careers>
> Latest news <http://www.handy.com/press> at Handy
> Handy just raised $50m
> <http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/> led
> by Fidelity
>
>
--
Want to work at Handy? Check out our culture deck and open roles
<http://www.handy.com/careers>
Latest news <http://www.handy.com/press> at Handy
Handy just raised $50m
<http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/> led
by Fidelity
RE: Yarn with CapacityScheduler will only schedule two applications
- open to consultants today
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Marcin,
we're seeing is that even though we have 12 machines in our yarn cluster, yarn will only schedule two applications.
If this is the problem you are facing with the below configuration then its mostly YARN-3216<https://issues.apache.org/jira/browse/YARN-3216> itself.
And i think its not yet out in any released hadoop version yet (not sure about HDP) also not aware of the version 2.7.1.2.3 (may be HDP version number)
IIUC you would have tried submitting in 2 diff partitions "data" and "yarn" hence 2 if not for a given partition only one would have run (if the am resource is less than the mimium size CS allows atleast 1 am to run)
Given that you have already identified the issue, what more are you expecting ?
Regards,
+ Naga
________________________________
From: Marcin Tustin [mtustin@handybook.com]
Sent: Wednesday, March 02, 2016 20:09
To: user@hadoop.apache.org
Subject: Yarn with CapacityScheduler will only schedule two applications - open to consultants today
Hi All,
We're hitting this issue. If you're a consultant with capacity today (2 March 2016 EST in New York) please feel free to contact me on or off list.
In terms of stack, we're using Yarn 2.7.1.2.3 from the latest Hortonworks distribution. It's possible we're hitting this bug: https://issues.apache.org/jira/browse/YARN-3216
The behaviour we're seeing is that even though we have 12 machines in our yarn cluster, yarn will only schedule two applications. The breakdown of machines is:
10 machines with the data label
2 with the yarn label
We have three queues: interactive, noninteractive, and default.
We're expecting to split capacity in the data label 20%/80% and default 100% of yarn label.
We have the following capacity scheduler config (Key=value format taken from ambari):
yarn.scheduler.capacity.maximum-am-resource-percent=100
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.queue-mappings-override.enable=true
yarn.scheduler.capacity.root.accessible-node-labels=*
yarn.scheduler.capacity.root.accessible-node-labels.data.capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.data.maximum-capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.yarn.capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.yarn.maximum-capacity=100
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default-node-label-expression=data
yarn.scheduler.capacity.root.default.accessible-node-labels=yarn
yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.capacity=100
yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.maximum-capacity=100
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=50
yarn.scheduler.capacity.root.default.default-node-label-expression=yarn
yarn.scheduler.capacity.root.default.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.default.maximum-capacity=100
yarn.scheduler.capacity.root.default.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.default.ordering-policy=fair
yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=100
yarn.scheduler.capacity.root.interactive.accessible-node-labels=data
yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.capacity=20
yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.maximum-capacity=100
yarn.scheduler.capacity.root.interactive.acl_administer_queue=*
yarn.scheduler.capacity.root.interactive.acl_submit_applications=*
yarn.scheduler.capacity.root.interactive.capacity=10
yarn.scheduler.capacity.root.interactive.maximum-am-resource-percent=50
yarn.scheduler.capacity.root.interactive.maximum-applications=2000
yarn.scheduler.capacity.root.interactive.maximum-capacity=100
yarn.scheduler.capacity.root.interactive.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.interactive.ordering-policy=fifo
yarn.scheduler.capacity.root.interactive.state=RUNNING
yarn.scheduler.capacity.root.interactive.user-limit-factor=100
yarn.scheduler.capacity.root.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.maximum-capacity=100
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels=data
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.capacity=80
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-capacity=80
yarn.scheduler.capacity.root.noninteractive.acl_submit_applications=*
yarn.scheduler.capacity.root.noninteractive.capacity=40
yarn.scheduler.capacity.root.noninteractive.default-node-label-expression=data
yarn.scheduler.capacity.root.noninteractive.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.noninteractive.maximum-applications=8000
yarn.scheduler.capacity.root.noninteractive.maximum-capacity=100
yarn.scheduler.capacity.root.noninteractive.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.noninteractive.ordering-policy=fair
yarn.scheduler.capacity.root.noninteractive.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.noninteractive.state=RUNNING
yarn.scheduler.capacity.root.noninteractive.user-limit-factor=100
yarn.scheduler.capacity.root.queues=default,interactive,noninteractive
yarn.scheduler.capacity.root.user-limit-factor=40
Also yarn.resourcemanager.scheduler.class is org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.
Any suggestions gratefully received.
Marcin
Want to work at Handy? Check out our culture deck and open roles<http://www.handy.com/careers>
Latest news<http://www.handy.com/press> at Handy
Handy just raised $50m<http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/> led by Fidelity
[http://marketing-email-assets.handybook.com/smalllogo.png]
RE: Yarn with CapacityScheduler will only schedule two applications
- open to consultants today
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Marcin,
we're seeing is that even though we have 12 machines in our yarn cluster, yarn will only schedule two applications.
If this is the problem you are facing with the below configuration then its mostly YARN-3216<https://issues.apache.org/jira/browse/YARN-3216> itself.
And i think its not yet out in any released hadoop version yet (not sure about HDP) also not aware of the version 2.7.1.2.3 (may be HDP version number)
IIUC you would have tried submitting in 2 diff partitions "data" and "yarn" hence 2 if not for a given partition only one would have run (if the am resource is less than the mimium size CS allows atleast 1 am to run)
Given that you have already identified the issue, what more are you expecting ?
Regards,
+ Naga
________________________________
From: Marcin Tustin [mtustin@handybook.com]
Sent: Wednesday, March 02, 2016 20:09
To: user@hadoop.apache.org
Subject: Yarn with CapacityScheduler will only schedule two applications - open to consultants today
Hi All,
We're hitting this issue. If you're a consultant with capacity today (2 March 2016 EST in New York) please feel free to contact me on or off list.
In terms of stack, we're using Yarn 2.7.1.2.3 from the latest Hortonworks distribution. It's possible we're hitting this bug: https://issues.apache.org/jira/browse/YARN-3216
The behaviour we're seeing is that even though we have 12 machines in our yarn cluster, yarn will only schedule two applications. The breakdown of machines is:
10 machines with the data label
2 with the yarn label
We have three queues: interactive, noninteractive, and default.
We're expecting to split capacity in the data label 20%/80% and default 100% of yarn label.
We have the following capacity scheduler config (Key=value format taken from ambari):
yarn.scheduler.capacity.maximum-am-resource-percent=100
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.queue-mappings-override.enable=true
yarn.scheduler.capacity.root.accessible-node-labels=*
yarn.scheduler.capacity.root.accessible-node-labels.data.capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.data.maximum-capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.yarn.capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.yarn.maximum-capacity=100
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default-node-label-expression=data
yarn.scheduler.capacity.root.default.accessible-node-labels=yarn
yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.capacity=100
yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.maximum-capacity=100
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=50
yarn.scheduler.capacity.root.default.default-node-label-expression=yarn
yarn.scheduler.capacity.root.default.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.default.maximum-capacity=100
yarn.scheduler.capacity.root.default.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.default.ordering-policy=fair
yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=100
yarn.scheduler.capacity.root.interactive.accessible-node-labels=data
yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.capacity=20
yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.maximum-capacity=100
yarn.scheduler.capacity.root.interactive.acl_administer_queue=*
yarn.scheduler.capacity.root.interactive.acl_submit_applications=*
yarn.scheduler.capacity.root.interactive.capacity=10
yarn.scheduler.capacity.root.interactive.maximum-am-resource-percent=50
yarn.scheduler.capacity.root.interactive.maximum-applications=2000
yarn.scheduler.capacity.root.interactive.maximum-capacity=100
yarn.scheduler.capacity.root.interactive.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.interactive.ordering-policy=fifo
yarn.scheduler.capacity.root.interactive.state=RUNNING
yarn.scheduler.capacity.root.interactive.user-limit-factor=100
yarn.scheduler.capacity.root.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.maximum-capacity=100
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels=data
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.capacity=80
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-capacity=80
yarn.scheduler.capacity.root.noninteractive.acl_submit_applications=*
yarn.scheduler.capacity.root.noninteractive.capacity=40
yarn.scheduler.capacity.root.noninteractive.default-node-label-expression=data
yarn.scheduler.capacity.root.noninteractive.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.noninteractive.maximum-applications=8000
yarn.scheduler.capacity.root.noninteractive.maximum-capacity=100
yarn.scheduler.capacity.root.noninteractive.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.noninteractive.ordering-policy=fair
yarn.scheduler.capacity.root.noninteractive.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.noninteractive.state=RUNNING
yarn.scheduler.capacity.root.noninteractive.user-limit-factor=100
yarn.scheduler.capacity.root.queues=default,interactive,noninteractive
yarn.scheduler.capacity.root.user-limit-factor=40
Also yarn.resourcemanager.scheduler.class is org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.
Any suggestions gratefully received.
Marcin
Want to work at Handy? Check out our culture deck and open roles<http://www.handy.com/careers>
Latest news<http://www.handy.com/press> at Handy
Handy just raised $50m<http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/> led by Fidelity
[http://marketing-email-assets.handybook.com/smalllogo.png]
RE: Yarn with CapacityScheduler will only schedule two applications
- open to consultants today
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Marcin,
we're seeing is that even though we have 12 machines in our yarn cluster, yarn will only schedule two applications.
If this is the problem you are facing with the below configuration then its mostly YARN-3216<https://issues.apache.org/jira/browse/YARN-3216> itself.
And i think its not yet out in any released hadoop version yet (not sure about HDP) also not aware of the version 2.7.1.2.3 (may be HDP version number)
IIUC you would have tried submitting in 2 diff partitions "data" and "yarn" hence 2 if not for a given partition only one would have run (if the am resource is less than the mimium size CS allows atleast 1 am to run)
Given that you have already identified the issue, what more are you expecting ?
Regards,
+ Naga
________________________________
From: Marcin Tustin [mtustin@handybook.com]
Sent: Wednesday, March 02, 2016 20:09
To: user@hadoop.apache.org
Subject: Yarn with CapacityScheduler will only schedule two applications - open to consultants today
Hi All,
We're hitting this issue. If you're a consultant with capacity today (2 March 2016 EST in New York) please feel free to contact me on or off list.
In terms of stack, we're using Yarn 2.7.1.2.3 from the latest Hortonworks distribution. It's possible we're hitting this bug: https://issues.apache.org/jira/browse/YARN-3216
The behaviour we're seeing is that even though we have 12 machines in our yarn cluster, yarn will only schedule two applications. The breakdown of machines is:
10 machines with the data label
2 with the yarn label
We have three queues: interactive, noninteractive, and default.
We're expecting to split capacity in the data label 20%/80% and default 100% of yarn label.
We have the following capacity scheduler config (Key=value format taken from ambari):
yarn.scheduler.capacity.maximum-am-resource-percent=100
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.queue-mappings-override.enable=true
yarn.scheduler.capacity.root.accessible-node-labels=*
yarn.scheduler.capacity.root.accessible-node-labels.data.capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.data.maximum-capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.yarn.capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.yarn.maximum-capacity=100
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default-node-label-expression=data
yarn.scheduler.capacity.root.default.accessible-node-labels=yarn
yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.capacity=100
yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.maximum-capacity=100
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=50
yarn.scheduler.capacity.root.default.default-node-label-expression=yarn
yarn.scheduler.capacity.root.default.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.default.maximum-capacity=100
yarn.scheduler.capacity.root.default.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.default.ordering-policy=fair
yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=100
yarn.scheduler.capacity.root.interactive.accessible-node-labels=data
yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.capacity=20
yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.maximum-capacity=100
yarn.scheduler.capacity.root.interactive.acl_administer_queue=*
yarn.scheduler.capacity.root.interactive.acl_submit_applications=*
yarn.scheduler.capacity.root.interactive.capacity=10
yarn.scheduler.capacity.root.interactive.maximum-am-resource-percent=50
yarn.scheduler.capacity.root.interactive.maximum-applications=2000
yarn.scheduler.capacity.root.interactive.maximum-capacity=100
yarn.scheduler.capacity.root.interactive.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.interactive.ordering-policy=fifo
yarn.scheduler.capacity.root.interactive.state=RUNNING
yarn.scheduler.capacity.root.interactive.user-limit-factor=100
yarn.scheduler.capacity.root.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.maximum-capacity=100
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels=data
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.capacity=80
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-capacity=80
yarn.scheduler.capacity.root.noninteractive.acl_submit_applications=*
yarn.scheduler.capacity.root.noninteractive.capacity=40
yarn.scheduler.capacity.root.noninteractive.default-node-label-expression=data
yarn.scheduler.capacity.root.noninteractive.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.noninteractive.maximum-applications=8000
yarn.scheduler.capacity.root.noninteractive.maximum-capacity=100
yarn.scheduler.capacity.root.noninteractive.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.noninteractive.ordering-policy=fair
yarn.scheduler.capacity.root.noninteractive.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.noninteractive.state=RUNNING
yarn.scheduler.capacity.root.noninteractive.user-limit-factor=100
yarn.scheduler.capacity.root.queues=default,interactive,noninteractive
yarn.scheduler.capacity.root.user-limit-factor=40
Also yarn.resourcemanager.scheduler.class is org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.
Any suggestions gratefully received.
Marcin
Want to work at Handy? Check out our culture deck and open roles<http://www.handy.com/careers>
Latest news<http://www.handy.com/press> at Handy
Handy just raised $50m<http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/> led by Fidelity
[http://marketing-email-assets.handybook.com/smalllogo.png]
RE: Yarn with CapacityScheduler will only schedule two applications
- open to consultants today
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Marcin,
we're seeing is that even though we have 12 machines in our yarn cluster, yarn will only schedule two applications.
If this is the problem you are facing with the below configuration then its mostly YARN-3216<https://issues.apache.org/jira/browse/YARN-3216> itself.
And i think its not yet out in any released hadoop version yet (not sure about HDP) also not aware of the version 2.7.1.2.3 (may be HDP version number)
IIUC you would have tried submitting in 2 diff partitions "data" and "yarn" hence 2 if not for a given partition only one would have run (if the am resource is less than the mimium size CS allows atleast 1 am to run)
Given that you have already identified the issue, what more are you expecting ?
Regards,
+ Naga
________________________________
From: Marcin Tustin [mtustin@handybook.com]
Sent: Wednesday, March 02, 2016 20:09
To: user@hadoop.apache.org
Subject: Yarn with CapacityScheduler will only schedule two applications - open to consultants today
Hi All,
We're hitting this issue. If you're a consultant with capacity today (2 March 2016 EST in New York) please feel free to contact me on or off list.
In terms of stack, we're using Yarn 2.7.1.2.3 from the latest Hortonworks distribution. It's possible we're hitting this bug: https://issues.apache.org/jira/browse/YARN-3216
The behaviour we're seeing is that even though we have 12 machines in our yarn cluster, yarn will only schedule two applications. The breakdown of machines is:
10 machines with the data label
2 with the yarn label
We have three queues: interactive, noninteractive, and default.
We're expecting to split capacity in the data label 20%/80% and default 100% of yarn label.
We have the following capacity scheduler config (Key=value format taken from ambari):
yarn.scheduler.capacity.maximum-am-resource-percent=100
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.queue-mappings-override.enable=true
yarn.scheduler.capacity.root.accessible-node-labels=*
yarn.scheduler.capacity.root.accessible-node-labels.data.capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.data.maximum-capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.yarn.capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.yarn.maximum-capacity=100
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default-node-label-expression=data
yarn.scheduler.capacity.root.default.accessible-node-labels=yarn
yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.capacity=100
yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.maximum-capacity=100
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=50
yarn.scheduler.capacity.root.default.default-node-label-expression=yarn
yarn.scheduler.capacity.root.default.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.default.maximum-capacity=100
yarn.scheduler.capacity.root.default.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.default.ordering-policy=fair
yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=100
yarn.scheduler.capacity.root.interactive.accessible-node-labels=data
yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.capacity=20
yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.maximum-capacity=100
yarn.scheduler.capacity.root.interactive.acl_administer_queue=*
yarn.scheduler.capacity.root.interactive.acl_submit_applications=*
yarn.scheduler.capacity.root.interactive.capacity=10
yarn.scheduler.capacity.root.interactive.maximum-am-resource-percent=50
yarn.scheduler.capacity.root.interactive.maximum-applications=2000
yarn.scheduler.capacity.root.interactive.maximum-capacity=100
yarn.scheduler.capacity.root.interactive.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.interactive.ordering-policy=fifo
yarn.scheduler.capacity.root.interactive.state=RUNNING
yarn.scheduler.capacity.root.interactive.user-limit-factor=100
yarn.scheduler.capacity.root.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.maximum-capacity=100
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels=data
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.capacity=80
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-capacity=80
yarn.scheduler.capacity.root.noninteractive.acl_submit_applications=*
yarn.scheduler.capacity.root.noninteractive.capacity=40
yarn.scheduler.capacity.root.noninteractive.default-node-label-expression=data
yarn.scheduler.capacity.root.noninteractive.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.noninteractive.maximum-applications=8000
yarn.scheduler.capacity.root.noninteractive.maximum-capacity=100
yarn.scheduler.capacity.root.noninteractive.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.noninteractive.ordering-policy=fair
yarn.scheduler.capacity.root.noninteractive.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.noninteractive.state=RUNNING
yarn.scheduler.capacity.root.noninteractive.user-limit-factor=100
yarn.scheduler.capacity.root.queues=default,interactive,noninteractive
yarn.scheduler.capacity.root.user-limit-factor=40
Also yarn.resourcemanager.scheduler.class is org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.
Any suggestions gratefully received.
Marcin
Want to work at Handy? Check out our culture deck and open roles<http://www.handy.com/careers>
Latest news<http://www.handy.com/press> at Handy
Handy just raised $50m<http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/> led by Fidelity
[http://marketing-email-assets.handybook.com/smalllogo.png]