You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by "Fabio C." <an...@gmail.com> on 2015/02/24 11:41:23 UTC

Can RM ignore heartbeats?

Hi everyone,
I have a question about the ResourceManager behavior:
when the ResourceManager allocates a container, it takes some time before
the NMToken is sent and then received by the ApplicationMaster.
During this time, it is possible to receive another heartbeat from the AM,
equal to the last one (since the AM is not aware of the allocated
resources).
Is there any policy in YARN that makes the RM aware of this and ignore this
last heartbeat?
I ask this because I would expect way more superfluous containers
allocated, in comparison to the ones I can see from the logs.

Thanks in advance

Fabio

Re: Can RM ignore heartbeats?

Posted by Xuan Gong <xg...@hortonworks.com>.
Hey, Fabio:
     In RM side, it will keep tracking on resource request status, such as how many containers that AM has requested, how many containers that RM has assigned to AM, how many containers that is pending, etc.
     In AM side, it is user's code. It should keep tracking on the similar resource request status, too. If AM keeps asking the resource, eventually it will hit the queue limit/user limit, and the resources will not be allocated.
     Also, the allocate is a blocking call. It will always get something back (could be nothing, some of the request resources, all request resources). AM should use this information to update the resource request status.

Thanks

Xuan Gong

From: "Fabio C." <an...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, February 24, 2015 at 2:41 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Can RM ignore heartbeats?

Hi everyone,
I have a question about the ResourceManager behavior:
when the ResourceManager allocates a container, it takes some time before the NMToken is sent and then received by the ApplicationMaster.
During this time, it is possible to receive another heartbeat from the AM, equal to the last one (since the AM is not aware of the allocated resources).
Is there any policy in YARN that makes the RM aware of this and ignore this last heartbeat?
I ask this because I would expect way more superfluous containers allocated, in comparison to the ones I can see from the logs.

Thanks in advance

Fabio

Re: Can RM ignore heartbeats?

Posted by Xuan Gong <xg...@hortonworks.com>.
Hey, Fabio:
     In RM side, it will keep tracking on resource request status, such as how many containers that AM has requested, how many containers that RM has assigned to AM, how many containers that is pending, etc.
     In AM side, it is user's code. It should keep tracking on the similar resource request status, too. If AM keeps asking the resource, eventually it will hit the queue limit/user limit, and the resources will not be allocated.
     Also, the allocate is a blocking call. It will always get something back (could be nothing, some of the request resources, all request resources). AM should use this information to update the resource request status.

Thanks

Xuan Gong

From: "Fabio C." <an...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, February 24, 2015 at 2:41 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Can RM ignore heartbeats?

Hi everyone,
I have a question about the ResourceManager behavior:
when the ResourceManager allocates a container, it takes some time before the NMToken is sent and then received by the ApplicationMaster.
During this time, it is possible to receive another heartbeat from the AM, equal to the last one (since the AM is not aware of the allocated resources).
Is there any policy in YARN that makes the RM aware of this and ignore this last heartbeat?
I ask this because I would expect way more superfluous containers allocated, in comparison to the ones I can see from the logs.

Thanks in advance

Fabio

Re: Can RM ignore heartbeats?

Posted by Xuan Gong <xg...@hortonworks.com>.
Hey, Fabio:
     In RM side, it will keep tracking on resource request status, such as how many containers that AM has requested, how many containers that RM has assigned to AM, how many containers that is pending, etc.
     In AM side, it is user's code. It should keep tracking on the similar resource request status, too. If AM keeps asking the resource, eventually it will hit the queue limit/user limit, and the resources will not be allocated.
     Also, the allocate is a blocking call. It will always get something back (could be nothing, some of the request resources, all request resources). AM should use this information to update the resource request status.

Thanks

Xuan Gong

From: "Fabio C." <an...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, February 24, 2015 at 2:41 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Can RM ignore heartbeats?

Hi everyone,
I have a question about the ResourceManager behavior:
when the ResourceManager allocates a container, it takes some time before the NMToken is sent and then received by the ApplicationMaster.
During this time, it is possible to receive another heartbeat from the AM, equal to the last one (since the AM is not aware of the allocated resources).
Is there any policy in YARN that makes the RM aware of this and ignore this last heartbeat?
I ask this because I would expect way more superfluous containers allocated, in comparison to the ones I can see from the logs.

Thanks in advance

Fabio

Re: Can RM ignore heartbeats?

Posted by Xuan Gong <xg...@hortonworks.com>.
Hey, Fabio:
     In RM side, it will keep tracking on resource request status, such as how many containers that AM has requested, how many containers that RM has assigned to AM, how many containers that is pending, etc.
     In AM side, it is user's code. It should keep tracking on the similar resource request status, too. If AM keeps asking the resource, eventually it will hit the queue limit/user limit, and the resources will not be allocated.
     Also, the allocate is a blocking call. It will always get something back (could be nothing, some of the request resources, all request resources). AM should use this information to update the resource request status.

Thanks

Xuan Gong

From: "Fabio C." <an...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, February 24, 2015 at 2:41 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Can RM ignore heartbeats?

Hi everyone,
I have a question about the ResourceManager behavior:
when the ResourceManager allocates a container, it takes some time before the NMToken is sent and then received by the ApplicationMaster.
During this time, it is possible to receive another heartbeat from the AM, equal to the last one (since the AM is not aware of the allocated resources).
Is there any policy in YARN that makes the RM aware of this and ignore this last heartbeat?
I ask this because I would expect way more superfluous containers allocated, in comparison to the ones I can see from the logs.

Thanks in advance

Fabio