You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Hamza Zafar <11...@seecs.edu.pk> on 2014/11/21 20:15:14 UTC
Resource Manager's container allocation behavior !
My Hadoop Cluster has 52GB memory , 56 virtual cores
Scenario: I submit an application to a default queue while there is no
other application running on the cluster. I create a request for 32
containers with same priority, 512MB memory and 1 virtual core . In the
first allocate call I receive 0 containers from RM, in further allocate
calls I start receiving containers. I keep on sending allocate calls until
all the containers have been allocated.
Why the application is not provided with all requested containers in first
allocate call?
I changed the configuration property
"yarn.resourcemanager.nodemanagers.heartbeat-interval-ms" from 1000ms to
100ms .Now at 100ms heatbeat interval the container allocation time has
reduced, but still the AM has to make the same number of allocate calls as
it was done before when the heartbeat interval was 1000ms.
RE: Resource Manager's container allocation behavior !
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi Hamza Zafar
I would like to let you know first that ApplicationMasterProtocol# allocate() has not only for requesting container but also doubles up as a heartbeat to let the ResourceManager know that the ApplicationMaster is alive
So basically your ApplicationMaster should be keep sending heartbeat to RM via allocate() call.
Container allocation will happen when NodeMager sends heartbeats to RM. This is the reason for you allocation time reduced when you decrease heartbet-interval-ms.
Why the application is not provided with all requested containers in first allocate call?
>> For the first call, RM updates request but allocation will happen when NM heartbeat to RM. So for 2nd call , containers will be received by AM.
Thanks & Regards
Rohith Sharma K S
From: Hamza Zafar [mailto:11bscshzafar@seecs.edu.pk]
Sent: 22 November 2014 00:45
To: user@hadoop.apache.org
Subject: Resource Manager's container allocation behavior !
My Hadoop Cluster has 52GB memory , 56 virtual cores
Scenario: I submit an application to a default queue while there is no other application running on the cluster. I create a request for 32 containers with same priority, 512MB memory and 1 virtual core . In the first allocate call I receive 0 containers from RM, in further allocate calls I start receiving containers. I keep on sending allocate calls until all the containers have been allocated.
Why the application is not provided with all requested containers in first allocate call?
I changed the configuration property "yarn.resourcemanager.nodemanagers.heartbeat-interval-ms" from 1000ms to 100ms .Now at 100ms heatbeat interval the container allocation time has reduced, but still the AM has to make the same number of allocate calls as it was done before when the heartbeat interval was 1000ms.
RE: Resource Manager's container allocation behavior !
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi Hamza Zafar
I would like to let you know first that ApplicationMasterProtocol# allocate() has not only for requesting container but also doubles up as a heartbeat to let the ResourceManager know that the ApplicationMaster is alive
So basically your ApplicationMaster should be keep sending heartbeat to RM via allocate() call.
Container allocation will happen when NodeMager sends heartbeats to RM. This is the reason for you allocation time reduced when you decrease heartbet-interval-ms.
Why the application is not provided with all requested containers in first allocate call?
>> For the first call, RM updates request but allocation will happen when NM heartbeat to RM. So for 2nd call , containers will be received by AM.
Thanks & Regards
Rohith Sharma K S
From: Hamza Zafar [mailto:11bscshzafar@seecs.edu.pk]
Sent: 22 November 2014 00:45
To: user@hadoop.apache.org
Subject: Resource Manager's container allocation behavior !
My Hadoop Cluster has 52GB memory , 56 virtual cores
Scenario: I submit an application to a default queue while there is no other application running on the cluster. I create a request for 32 containers with same priority, 512MB memory and 1 virtual core . In the first allocate call I receive 0 containers from RM, in further allocate calls I start receiving containers. I keep on sending allocate calls until all the containers have been allocated.
Why the application is not provided with all requested containers in first allocate call?
I changed the configuration property "yarn.resourcemanager.nodemanagers.heartbeat-interval-ms" from 1000ms to 100ms .Now at 100ms heatbeat interval the container allocation time has reduced, but still the AM has to make the same number of allocate calls as it was done before when the heartbeat interval was 1000ms.
RE: Resource Manager's container allocation behavior !
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi Hamza Zafar
I would like to let you know first that ApplicationMasterProtocol# allocate() has not only for requesting container but also doubles up as a heartbeat to let the ResourceManager know that the ApplicationMaster is alive
So basically your ApplicationMaster should be keep sending heartbeat to RM via allocate() call.
Container allocation will happen when NodeMager sends heartbeats to RM. This is the reason for you allocation time reduced when you decrease heartbet-interval-ms.
Why the application is not provided with all requested containers in first allocate call?
>> For the first call, RM updates request but allocation will happen when NM heartbeat to RM. So for 2nd call , containers will be received by AM.
Thanks & Regards
Rohith Sharma K S
From: Hamza Zafar [mailto:11bscshzafar@seecs.edu.pk]
Sent: 22 November 2014 00:45
To: user@hadoop.apache.org
Subject: Resource Manager's container allocation behavior !
My Hadoop Cluster has 52GB memory , 56 virtual cores
Scenario: I submit an application to a default queue while there is no other application running on the cluster. I create a request for 32 containers with same priority, 512MB memory and 1 virtual core . In the first allocate call I receive 0 containers from RM, in further allocate calls I start receiving containers. I keep on sending allocate calls until all the containers have been allocated.
Why the application is not provided with all requested containers in first allocate call?
I changed the configuration property "yarn.resourcemanager.nodemanagers.heartbeat-interval-ms" from 1000ms to 100ms .Now at 100ms heatbeat interval the container allocation time has reduced, but still the AM has to make the same number of allocate calls as it was done before when the heartbeat interval was 1000ms.
RE: Resource Manager's container allocation behavior !
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi Hamza Zafar
I would like to let you know first that ApplicationMasterProtocol# allocate() has not only for requesting container but also doubles up as a heartbeat to let the ResourceManager know that the ApplicationMaster is alive
So basically your ApplicationMaster should be keep sending heartbeat to RM via allocate() call.
Container allocation will happen when NodeMager sends heartbeats to RM. This is the reason for you allocation time reduced when you decrease heartbet-interval-ms.
Why the application is not provided with all requested containers in first allocate call?
>> For the first call, RM updates request but allocation will happen when NM heartbeat to RM. So for 2nd call , containers will be received by AM.
Thanks & Regards
Rohith Sharma K S
From: Hamza Zafar [mailto:11bscshzafar@seecs.edu.pk]
Sent: 22 November 2014 00:45
To: user@hadoop.apache.org
Subject: Resource Manager's container allocation behavior !
My Hadoop Cluster has 52GB memory , 56 virtual cores
Scenario: I submit an application to a default queue while there is no other application running on the cluster. I create a request for 32 containers with same priority, 512MB memory and 1 virtual core . In the first allocate call I receive 0 containers from RM, in further allocate calls I start receiving containers. I keep on sending allocate calls until all the containers have been allocated.
Why the application is not provided with all requested containers in first allocate call?
I changed the configuration property "yarn.resourcemanager.nodemanagers.heartbeat-interval-ms" from 1000ms to 100ms .Now at 100ms heatbeat interval the container allocation time has reduced, but still the AM has to make the same number of allocate calls as it was done before when the heartbeat interval was 1000ms.