You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Rahul Bhattacharjee <ra...@gmail.com> on 2013/06/07 17:54:57 UTC
Resource manager question - Yarn
Hello,
I have a basic question related to RM of Yarn. Why is the allocate
container request to RM doesn't always return with containers. The call can
actually block and return when containers are available.
Scenario :
I launcher AM.
The AM requests for 5 containers.
The response returns without any containers.
I send the request for containers , again and again and the RM returns
containers (sometimes less than the number that I have asked for).
Is this something to do with the heartbeat as the allocate call must return
within a predictable time period?
Thanks,
Rahul
Re: Resource manager question - Yarn
Posted by Arun C Murthy <ac...@hortonworks.com>.
The reason is that you don't want a socket to the RM machine to be held up - we expect to have thousands of AMs running concurrently. Essentially a blocking call could result in a DoS attack on the RM.
Eventually, YARN will move to async RPC and won't have this quirk.
Arun
On Jun 7, 2013, at 8:54 AM, Rahul Bhattacharjee wrote:
> Hello,
>
> I have a basic question related to RM of Yarn. Why is the allocate container request to RM doesn't always return with containers. The call can actually block and return when containers are available.
>
> Scenario :
>
> I launcher AM.
> The AM requests for 5 containers.
> The response returns without any containers.
>
> I send the request for containers , again and again and the RM returns containers (sometimes less than the number that I have asked for).
>
> Is this something to do with the heartbeat as the allocate call must return within a predictable time period?
>
> Thanks,
> Rahul
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: Resource manager question - Yarn
Posted by Arun C Murthy <ac...@hortonworks.com>.
The reason is that you don't want a socket to the RM machine to be held up - we expect to have thousands of AMs running concurrently. Essentially a blocking call could result in a DoS attack on the RM.
Eventually, YARN will move to async RPC and won't have this quirk.
Arun
On Jun 7, 2013, at 8:54 AM, Rahul Bhattacharjee wrote:
> Hello,
>
> I have a basic question related to RM of Yarn. Why is the allocate container request to RM doesn't always return with containers. The call can actually block and return when containers are available.
>
> Scenario :
>
> I launcher AM.
> The AM requests for 5 containers.
> The response returns without any containers.
>
> I send the request for containers , again and again and the RM returns containers (sometimes less than the number that I have asked for).
>
> Is this something to do with the heartbeat as the allocate call must return within a predictable time period?
>
> Thanks,
> Rahul
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: Resource manager question - Yarn
Posted by Arun C Murthy <ac...@hortonworks.com>.
The reason is that you don't want a socket to the RM machine to be held up - we expect to have thousands of AMs running concurrently. Essentially a blocking call could result in a DoS attack on the RM.
Eventually, YARN will move to async RPC and won't have this quirk.
Arun
On Jun 7, 2013, at 8:54 AM, Rahul Bhattacharjee wrote:
> Hello,
>
> I have a basic question related to RM of Yarn. Why is the allocate container request to RM doesn't always return with containers. The call can actually block and return when containers are available.
>
> Scenario :
>
> I launcher AM.
> The AM requests for 5 containers.
> The response returns without any containers.
>
> I send the request for containers , again and again and the RM returns containers (sometimes less than the number that I have asked for).
>
> Is this something to do with the heartbeat as the allocate call must return within a predictable time period?
>
> Thanks,
> Rahul
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: Resource manager question - Yarn
Posted by Arun C Murthy <ac...@hortonworks.com>.
The reason is that you don't want a socket to the RM machine to be held up - we expect to have thousands of AMs running concurrently. Essentially a blocking call could result in a DoS attack on the RM.
Eventually, YARN will move to async RPC and won't have this quirk.
Arun
On Jun 7, 2013, at 8:54 AM, Rahul Bhattacharjee wrote:
> Hello,
>
> I have a basic question related to RM of Yarn. Why is the allocate container request to RM doesn't always return with containers. The call can actually block and return when containers are available.
>
> Scenario :
>
> I launcher AM.
> The AM requests for 5 containers.
> The response returns without any containers.
>
> I send the request for containers , again and again and the RM returns containers (sometimes less than the number that I have asked for).
>
> Is this something to do with the heartbeat as the allocate call must return within a predictable time period?
>
> Thanks,
> Rahul
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/