Yarn underutilisation ,

[Navdeep Agarwal <na...@helpshift.com>]

Hi , I have cluster of around 30 nodes , where each node is configured to 32 gb memory , with each mapper and reducer to use 4gb memory , in one of the queue(backfill_q) , queue is always underutilized. A simple mapreduce job can max use 44 containers instead of 70 available and this applies to sum of all jobs containers currently running in the queue , means to say yarn can only provide 44 out of 70 containers . Below is our scheduling properties. Any help in which config do I need to look , or some messing I have done with config would be highly appreciated.

Some more info 
* These are pig jobs
* at a time some 2000 containers are requested
Let me know if more info is required .

Thanks in advance.

yarn.scheduler.capacity.maximum-am-resource-percent=0.25
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator
yarn.scheduler.capacity.queue-mappings-override.enable=false
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.maximum-capacity=100
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.queues=air_q,analytics_q,backfill_q,campaigns_q,common_csv_q,default,faq_q,sdk_q
yarn.scheduler.capacity.root.accessible-node-labels=campaigns_nl,air_nl,backfill_nl,sdk_nl,common_csv_nl
yarn.scheduler.capacity.root.accessible-node-labels.campaigns_nl.capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.campaigns_nl.maximum-capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.air_nl.capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.air_nl.maximum-capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.backfill_nl.capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.backfill_nl.maximum-capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.sdk_nl.capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.sdk_nl.maximum-capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.common_csv_nl.capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.common_csv_nl.maximum-capacity=100
yarn.scheduler.capacity.root.air_q.acl_administer_queue=*
yarn.scheduler.capacity.root.air_q.acl_submit_applications=*
yarn.scheduler.capacity.root.air_q.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.air_q.maximum-capacity=1
yarn.scheduler.capacity.root.air_q.user-limit-factor=1
yarn.scheduler.capacity.root.air_q.state=RUNNING
yarn.scheduler.capacity.root.air_q.capacity=1
yarn.scheduler.capacity.root.air_q.default-node-label-expression=air_nl
yarn.scheduler.capacity.root.air_q.ordering-policy=fifo
yarn.scheduler.capacity.root.air_q.accessible-node-labels=air_nl
yarn.scheduler.capacity.root.air_q.accessible-node-labels.air_nl.capacity=100
yarn.scheduler.capacity.root.air_q.accessible-node-labels.air_nl.maximum-capacity=100
yarn.scheduler.capacity.root.analytics_q.acl_administer_queue=*
yarn.scheduler.capacity.root.analytics_q.acl_submit_applications=*
yarn.scheduler.capacity.root.analytics_q.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.analytics_q.maximum-capacity=100
yarn.scheduler.capacity.root.analytics_q.user-limit-factor=1
yarn.scheduler.capacity.root.analytics_q.state=RUNNING
yarn.scheduler.capacity.root.analytics_q.capacity=0
yarn.scheduler.capacity.root.analytics_q.ordering-policy=fifo
yarn.scheduler.capacity.root.backfill_q.acl_administer_queue=*
yarn.scheduler.capacity.root.backfill_q.acl_submit_applications=*
yarn.scheduler.capacity.root.backfill_q.minimum-user-limit-percent=10
yarn.scheduler.capacity.root.backfill_q.maximum-capacity=10
yarn.scheduler.capacity.root.backfill_q.user-limit-factor=10
yarn.scheduler.capacity.root.backfill_q.state=RUNNING
yarn.scheduler.capacity.root.backfill_q.capacity=10
yarn.scheduler.capacity.root.backfill_q.ordering-policy=fair
yarn.scheduler.capacity.root.backfill_q.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.backfill_q.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.root.backfill_q.accessible-node-labels=
yarn.scheduler.capacity.root.campaigns_q.acl_administer_queue=*
yarn.scheduler.capacity.root.campaigns_q.acl_submit_applications=*
yarn.scheduler.capacity.root.campaigns_q.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.campaigns_q.maximum-capacity=100
yarn.scheduler.capacity.root.campaigns_q.user-limit-factor=3
yarn.scheduler.capacity.root.campaigns_q.state=RUNNING
yarn.scheduler.capacity.root.campaigns_q.capacity=15
yarn.scheduler.capacity.root.campaigns_q.default-node-label-expression=campaigns_nl
yarn.scheduler.capacity.root.campaigns_q.ordering-policy=fifo
yarn.scheduler.capacity.root.campaigns_q.accessible-node-labels=campaigns_nl
yarn.scheduler.capacity.root.campaigns_q.accessible-node-labels.campaigns_nl.capacity=100
yarn.scheduler.capacity.root.campaigns_q.accessible-node-labels.campaigns_nl.maximum-capacity=100
yarn.scheduler.capacity.root.common_csv_q.acl_administer_queue=*
yarn.scheduler.capacity.root.common_csv_q.acl_submit_applications=*
yarn.scheduler.capacity.root.common_csv_q.minimum-user-limit-percent=25
yarn.scheduler.capacity.root.common_csv_q.maximum-capacity=100
yarn.scheduler.capacity.root.common_csv_q.user-limit-factor=4
yarn.scheduler.capacity.root.common_csv_q.state=RUNNING
yarn.scheduler.capacity.root.common_csv_q.capacity=25
yarn.scheduler.capacity.root.common_csv_q.default-node-label-expression=common_csv_nl
yarn.scheduler.capacity.root.common_csv_q.ordering-policy=fifo
yarn.scheduler.capacity.root.common_csv_q.accessible-node-labels=common_csv_nl
yarn.scheduler.capacity.root.common_csv_q.accessible-node-labels.common_csv_nl.capacity=100
yarn.scheduler.capacity.root.common_csv_q.accessible-node-labels.common_csv_nl.maximum-capacity=100
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.maximum-capacity=100
yarn.scheduler.capacity.root.default.user-limit-factor=1
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.capacity=47
yarn.scheduler.capacity.root.default.ordering-policy=fifo
yarn.scheduler.capacity.root.default.maximum-am-resource-percent=0.1
yarn.scheduler.capacity.root.faq_q.acl_administer_queue=*
yarn.scheduler.capacity.root.faq_q.acl_submit_applications=*
yarn.scheduler.capacity.root.faq_q.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.faq_q.maximum-capacity=1
yarn.scheduler.capacity.root.faq_q.user-limit-factor=1
yarn.scheduler.capacity.root.faq_q.state=RUNNING
yarn.scheduler.capacity.root.faq_q.capacity=1
yarn.scheduler.capacity.root.faq_q.default-node-label-expression=sdk_nl
yarn.scheduler.capacity.root.faq_q.ordering-policy=fifo
yarn.scheduler.capacity.root.faq_q.accessible-node-labels=sdk_nl
yarn.scheduler.capacity.root.faq_q.accessible-node-labels.sdk_nl.capacity=25
yarn.scheduler.capacity.root.faq_q.accessible-node-labels.sdk_nl.maximum-capacity=100
yarn.scheduler.capacity.root.sdk_q.acl_administer_queue=*
yarn.scheduler.capacity.root.sdk_q.acl_submit_applications=*
yarn.scheduler.capacity.root.sdk_q.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.sdk_q.maximum-capacity=1
yarn.scheduler.capacity.root.sdk_q.user-limit-factor=1
yarn.scheduler.capacity.root.sdk_q.state=RUNNING
yarn.scheduler.capacity.root.sdk_q.capacity=1
yarn.scheduler.capacity.root.sdk_q.default-node-label-expression=sdk_nl
yarn.scheduler.capacity.root.sdk_q.ordering-policy=fifo
yarn.scheduler.capacity.root.sdk_q.accessible-node-labels=sdk_nl
yarn.scheduler.capacity.root.sdk_q.accessible-node-labels.sdk_nl.capacity=75
yarn.scheduler.capacity.root.sdk_q.accessible-node-labels.sdk_nl.maximum-capacity=100


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org