CapacityScheduler ResourceReservation problem.

[Pablo Casares Crespo <pc...@redborder.com>]

Hi all,

I have a problem with CapacityScheduler container reservations. 

My cluster has the following specs:

Hadoop version: 2.7.1


*Queues: 

yarn.scheduler.capacity.root.default.capacity	20
yarn.scheduler.capacity.root.samza.capacity	40
yarn.scheduler.capacity.root.loaders.capacity  40
	yarn.scheduler.capacity.root.loaders.fast.capacity  60
	yarn.scheduler.capacity.root.loaders.slow.capacity  40


*Number Nodemanagers: 3

**Nodemanagers:

—>rbcx5qyowm23.redborder.cluster

Total Vmem allocated for Containers	53.89 GB
Vmem enforcement enabled	true
Total Pmem allocated for Container	5.39 GB



—>rbbxvxmciv15.redborder.cluster

Total Vmem allocated for Containers	125.45 GB
Vmem enforcement enabled	true
Total Pmem allocated for Container	12.54 GB



—>rbbxdz032p71.redborder.cluster

Total Vmem allocated for Containers	79.11 GB
Vmem enforcement enabled	true
Total Pmem allocated for Container	7.91 GB

**Total cluster memory: 25.84GB


This is my problem:

I start the ResourceManager and all the Nodemanagers(3). At this point I have 25.84GB free at the cluster as I described above. Now I submit 4 Samza applications. These applications are correctly uploaded and they start running. Samza applications are applications that never ends so they are indefinitely running. That’s normal. So at this point I have:

10GB Used
15.84GB Free.

Detailed for each node:

rbcx5qyowm23.redborder.cluster 	Used: 2GB	Free: 3.39GB


rbbxvxmciv15.redborder.cluster	Used: 3GB	Free: 9.54GB


rbbxdz032p71.redborder.cluster	Used: 5GB	Free: 2.91GB


At this point I upload an oozie application. Oozie have a workflow. Oozie submit the oozielauncher application which is the responsible to upload the applications defined at the workflow. So at this point I have 4 samza applications, 1 oozie launcher and 1 application uploaded by oozie launcher (this application uploaded by oozie launcher has an AM and a container uploaded by this AM). My problem appear at this point. 

On Hadoop Scheduler webpage I see:

4 Samza applications Running.
1 Oozie launcher  Running
1 PigLatin application (submitted by oozielauncher) (this application never ends and it should!). 

**When I inspect the PigLatin application:

YarnApplicationState:	RUNNING: AM has registered with RM and started running.

The AM container is allocated and running.
It has made an “Off Switch Request” requesting a Container of 2GB Ram. 

At this point the cluster is:

rbcx5qyowm23.redborder.cluster 	Used: 4GB	Free: 1.39GB


rbbxvxmciv15.redborder.cluster	Used: 3GB	Free: 9.54GB


rbbxdz032p71.redborder.cluster	Used: 7GB	Free: 933MB


The container requested by AM is never allocated although there is 9.54GB free at rbbxvxmciv15.redborder.cluster.
Both the oozie launcher and the application are submitted on “yarn.scheduler.capacity.root.loaders.fast” and it have enough free space to run the container.

So I started to see the logs. I enabled CapacityScheduler and ResourceManager DEBUG log level. 
Thats what I have found on ResourceManager logs:


**AM is correctly running as I saw at ResourceManager web

2016-03-14_10:37:32.01418 16/03/14 10:37:32 INFO resourcemanager.ApplicationMasterService: AM registration appattempt_1457951280710_0006_000001
2016-03-14_10:37:32.01423 16/03/14 10:37:32 INFO resourcemanager.RMAuditLogger: USER=oozie	IP=10.0.107.16	OPERATION=Register App Master	TARGET=ApplicationMasterService	RESULT=SUCCESS	APPID=application_1457951280710_0006	APPATTEMPTID=appattempt_1457951280710_0006_000001
2016-03-14_10:37:32.01426 16/03/14 10:37:32 INFO attempt.RMAppAttemptImpl: appattempt_1457951280710_0006_000001 State change from LAUNCHED to RUNNING
2016-03-14_10:37:32.01427 16/03/14 10:37:32 INFO rmapp.RMAppImpl: application_1457951280710_0006 State change from ACCEPTED to RUNNING


** It reserves a Container at rbcx5qyowm23.redborder.cluster (I remember that at this state this node has 1.39GB free)


2016-03-14_10:37:33.65925 16/03/14 10:37:33 INFO rmcontainer.RMContainerImpl: container_1457951280710_0006_01_000002 Container Transitioned from NEW to RESERVED
2016-03-14_10:37:33.65934 16/03/14 10:37:33 INFO capacity.LeafQueue: Reserved container  application=application_1457951280710_0006 resource=<memory:2048, vCores:1> queue=fast: capacity=0.6, absoluteCapacity=0.24000001, usedResources=<memory:4096, vCores:3>, usedCapacity=0.6449378, absoluteUsedCapacity=0.15477045, numApps=2, numContainers=3 usedCapacity=0.6449378 absoluteUsedCapacity=0.15477045 used=<memory:4096, vCores:3> cluster=<memory:26465, vCores:45>
2016-03-14_10:37:33.65936 16/03/14 10:37:33 INFO capacity.ParentQueue: Re-sorting assigned queue: root.loaders.fast stats: fast: capacity=0.6, absoluteCapacity=0.24000001, usedResources=<memory:6144, vCores:4>, usedCapacity=0.9674067, absoluteUsedCapacity=0.23215568, numApps=2, numContainers=4
2016-03-14_10:37:33.65937 16/03/14 10:37:33 INFO capacity.ParentQueue: assignedContainer queue=loaders usedCapacity=0.5803892 absoluteUsedCapacity=0.23215568 used=<memory:6144, vCores:4> cluster=<memory:26465, vCores:45>
2016-03-14_10:37:33.65939 16/03/14 10:37:33 INFO capacity.ParentQueue: Re-sorting assigned queue: root.loaders stats: loaders: numChildQueue= 2, capacity=0.4, absoluteCapacity=0.4, usedResources=<memory:6144, vCores:4>usedCapacity=0.5803892, numApps=2, numContainers=4
2016-03-14_10:37:33.65940 16/03/14 10:37:33 INFO capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.6190818 absoluteUsedCapacity=0.6190818 used=<memory:16384, vCores:12> cluster=<memory:26465, vCores:45>


**And now, It’s indefinitely logging:

2016-03-14_10:37:35.66182 16/03/14 10:37:35 INFO capacity.CapacityScheduler: Trying to fulfill reservation for application application_1457951280710_0006 on node: rbcx5qyowm23.redborder.cluster:45454
2016-03-14_10:37:35.66228 16/03/14 10:37:35 INFO capacity.LeafQueue: Reserved container  application=application_1457951280710_0006 resource=<memory:2048, vCores:1> queue=fast: capacity=0.6, absoluteCapacity=0.24000001, usedResources=<memory:6144, vCores:4>, usedCapacity=0.9674067, absoluteUsedCapacity=0.23215568, numApps=2, numContainers=4 usedCapacity=0.9674067 absoluteUsedCapacity=0.23215568 used=<memory:6144, vCores:4> cluster=<memory:26465, vCores:45>
2016-03-14_10:37:35.66235 16/03/14 10:37:35 INFO capacity.CapacityScheduler: Skipping scheduling since node rbcx5qyowm23.redborder.cluster:45454 is reserved by application appattempt_1457951280710_0006_000001

**Its always trying to allocate the reserved container at rbcx5qyowm23.redborder.cluster but it doesn’t try to allocate it at rbbxvxmciv15.redborder.cluster (I remember that at this point this node have 9.54GB free). So, as it is trying to allocate the container at rbcx5qyowm23.redborder.cluster and rbcx5qyowm23.redborder.cluster is running Samza applications which never end, the reservation is stucked at this point. Always trying to allocate the container in a node that is running applications which will never free the resources. If I kill the oozie yarn application it, at some submissions, runs ok, but when this situation happen again (reservation at rbbxvxmciv15.redborder.cluster) it have a deadlock again.

I have inspected the CapacityScheduler code and I saw that there is a “reservationsContinueLooking” variable that enable looking on other nodes although a reservation is made for a node. 

At the log appear that this variable is true:


2016-03-14_10:28:00.66642 16/03/14 10:28:00 INFO capacity.LeafQueue: Initializing fast
2016-03-14_10:28:00.66643 capacity = 0.6 [= (float) configuredCapacity / 100 ]
2016-03-14_10:28:00.66644 asboluteCapacity = 0.24000001 [= parentAbsoluteCapacity * capacity ]
2016-03-14_10:28:00.66645 maxCapacity = 0.6 [= configuredMaxCapacity ]
2016-03-14_10:28:00.66645 absoluteMaxCapacity = 0.24000001 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
2016-03-14_10:28:00.66646 userLimit = 100 [= configuredUserLimit ]
2016-03-14_10:28:00.66648 userLimitFactor = 1.0 [= configuredUserLimitFactor ]
2016-03-14_10:28:00.66648 maxApplications = 2400 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
2016-03-14_10:28:00.66649 maxApplicationsPerUser = 2400 [= (int)(maxApplications * (userLimit / 100.0f) * userLimitFactor) ]
2016-03-14_10:28:00.66650 usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory * absoluteCapacity)]
2016-03-14_10:28:00.66650 absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
2016-03-14_10:28:00.66651 maxAMResourcePerQueuePercent = 1.0 [= configuredMaximumAMResourcePercent ]
2016-03-14_10:28:00.66651 minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory) / maximumAllocationMemory ]
2016-03-14_10:28:00.66653 maximumAllocation = <memory:8192, vCores:32> [= configuredMaxAllocation ]
2016-03-14_10:28:00.66653 numContainers = 0 [= currentNumContainers ]
2016-03-14_10:28:00.66654 state = RUNNING [= configuredState ]
2016-03-14_10:28:00.66654 acls = ADMINISTER_QUEUE: SUBMIT_APP:  [= configuredAcls ]
2016-03-14_10:28:00.66654 nodeLocalityDelay = 10
2016-03-14_10:28:00.66655 labels=*,
2016-03-14_10:28:00.66655 nodeLocalityDelay = 10
2016-03-14_10:28:00.66656 reservationsContinueLooking = true
2016-03-14_10:28:00.66656 preemptionDisabled = true 


(As i see, “reservationsContinueLooking” is true by default at fast queue) 

At the code I notice:

// if reservation continous looking enabled, check to see if could we
// potentially use this node instead of a reserved node if the application
// has reserved containers.
// TODO, now only consider reservation cases when the node has no label
if (this.reservationsContinueLooking
    && label.equals(RMNodeLabelsManager.NO_LABEL)

But my nodes have not labels. 




Thank you in advance.
Regards.


Pablo Casares Crespo
Big Data Developer


pcasares@redborder.com
+34 667735764 | +34 955 601 160

SAN FRANCISCO - SEVILLE - MADRID
This email, including attachments, is intended exclusively for its addressee. It contains information that is CONFIDENTIAL whose disclosure is prohibited by law and may be covered by legal privilege. If you have received this email in error, please notify the sender and delete it from your system.
Este correo electrónico, incluidos sus anexos, se dirige exclusivamente a su destinatario. Contiene información CONFIDENCIAL cuya divulgación está prohibida por la ley o puede estar sometida a secreto profesional. Si ha recibido este mensaje por error, le rogamos nos lo comunique inmediatamente y proceda a su destrucción.