You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Brahma Reddy Battula (JIRA)" <ji...@apache.org> on 2016/02/08 06:18:39 UTC
[jira] [Commented] (YARN-4678) Cluster used capacity is > 100 when
container reserved
[ https://issues.apache.org/jira/browse/YARN-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136594#comment-15136594 ]
Brahma Reddy Battula commented on YARN-4678:
--------------------------------------------
*As reserved memory is include in the used resources. queue capacity > 100* ..Please let me know any thoughts on this..
2016-02-22 01:15:09,359 | INFO | ResourceManager Event Processor | completedContainer queue=root usedCapacity=0.9583333 absoluteUsedCapacity=0.9583333 *{color:red}used=<memory:23552, vCores:7> cluster=<memory:24576, vCores:24>{color}* | ParentQueue.java:631
2016-02-22 01:15:09,359 | INFO | ResourceManager Event Processor | Re-sorting completed queue: root.QueueA stats: QueueA: capacity=0.4, absoluteCapacity=0.4, usedResources=<memory:17920, vCores:5>, usedCapacity=1.8229909, absoluteUsedCapacity=0.7291667, numApps=1, numContainers=5 | ParentQueue.java:648
2016-02-22 01:15:09,359 | INFO | ResourceManager Event Processor | Application attempt appattempt_1456061422894_0003_000001 released container container_e05_1456061422894_0003_01_004504 on *{color:blue}node: host: vm-3:26009{color}* #containers=1 available=<memory:6656, vCores:7> used=<memory:1536, vCores:1> with event: FINISHED | CapacityScheduler.java:1452
2016-02-22 01:15:09,359 | INFO | ResourceManager Event Processor | Trying to fulfill reservation for application application_1456061422894_0003 on node: vm-3:26009 | CapacityScheduler.java:1110
2016-02-22 01:15:09,359 | INFO | ResourceManager Event Processor | Application application_1456061422894_0003 *{color:blue}unreserved on node host: vm-3:26009{color}* #containers=1 available=<memory:6656, vCores:7> used=<memory:1536, vCores:1>, currently has 0 at priority 20; currentReservation <memory:0, vCores:0> on node-label= | FiCaSchedulerApp.java:229
2016-02-22 01:15:09,359 | INFO | ResourceManager Event Processor | container_e05_1456061422894_0003_01_004505 Container Transitioned from NEW to ALLOCATED | RMContainerImpl.java:419
2016-02-22 01:15:09,359 | INFO | ResourceManager Event Processor | Assigned container container_e05_1456061422894_0003_01_004505 of capacity <memory:4096, vCores:1> on *{color:blue}host vm-3:26009{color}* , which has 2 containers, <memory:5632, vCores:2> used and <memory:2560, vCores:6> available after allocation | SchedulerNode.java:154
2016-02-22 01:15:09,359 | INFO | ResourceManager Event Processor | assignedContainer application attempt=appattempt_1456061422894_0003_000001 container=Container: [ContainerId: container_e05_1456061422894_0003_01_004505, NodeId: vm-3:26009, NodeHttpAddress: vm-3:26010, Resource: <memory:4096, vCores:1>, Priority: 20, Token: null, ] queue=QueueA: capacity=0.4, absoluteCapacity=0.4, usedResources=<memory:17920, vCores:5>, usedCapacity=1.8229909, absoluteUsedCapacity=0.7291667, numApps=1, numContainers=5 clusterResource=<memory:24576, vCores:24> | LeafQueue.java:1731
2016-02-22 01:15:09,360 | INFO | ResourceManager Event Processor | container_e05_1456061422894_0003_01_004508 Container Transitioned from *{color:blue}NEW to RESERVED{color}* | RMContainerImpl.java:419
2016-02-22 01:15:09,360 | INFO | ResourceManager Event Processor | *Reserved container application=application_1456061422894_0003 resource=<memory:4096, vCores:1>* queue=QueueA: capacity=0.4, absoluteCapacity=0.4, usedResources=<memory:17920, vCores:5>, usedCapacity=1.8229909, absoluteUsedCapacity=0.7291667, numApps=1, numContainers=5 usedCapacity=1.8229909 absoluteUsedCapacity=0.7291667 used=<memory:17920, vCores:5> cluster=<memory:24576, vCores:24> | LeafQueue.java:1764
2016-02-22 01:15:09,360 | INFO | ResourceManager Event Processor | Re-sorting assigned queue: root.QueueA stats: QueueA: capacity=0.4, absoluteCapacity=0.4, usedResources=<memory:22016, vCores:6>, usedCapacity=2.2396743, absoluteUsedCapacity=0.8958333, numApps=1, numContainers=6 | ParentQueue.java:585
2016-02-22 01:15:09,360 | INFO | ResourceManager Event Processor | *assignedContainer queue=root {color:red}usedCapacity=1.125 absoluteUsedCapacity=1.125
used=<memory:27648, vCores:8>
cluster=<memory:24576, vCores:24>{color}* | ParentQueue.java:465
> Cluster used capacity is > 100 when container reserved
> -------------------------------------------------------
>
> Key: YARN-4678
> URL: https://issues.apache.org/jira/browse/YARN-4678
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Brahma Reddy Battula
> Assignee: Brahma Reddy Battula
>
> *Scenario:*
> * Start cluster with Three NM's each having 8GB (cluster memory:24GB).
> * Configure queues with elasticity and userlimitfactor=10.
> * disable pre-emption.
> * run two job with different priority in different queue at the same time
> ** yarn jar hadoop-mapreduce-examples-2.7.2.jar pi -Dyarn.app.priority=LOW -Dmapreduce.job.queuename=QueueA -Dmapreduce.map.memory.mb=4096 -Dyarn.app.mapreduce.am.resource.mb=1536 -Dmapreduce.job.reduce.slowstart.completedmaps=1.0 10 1000000000000
> ** yarn jar hadoop-mapreduce-examples-2.7.2.jar pi -Dyarn.app.priority=HIGH -Dmapreduce.job.queuename=QueueB -Dmapreduce.map.memory.mb=4096 -Dyarn.app.mapreduce.am.resource.mb=1536 3 1000000000000
> * observe the cluster capacity which was used in RM web UI
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)