You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Debraj Manna <su...@gmail.com> on 2020/03/23 10:22:21 UTC
High Availability for an application running on yarn
Can someone let me know what is the recommended way to ensure high
availability for a yarn application?
For example, let say I have 3 physical nodes. The yarn java application
needs 3 containers and each physical node has enough resources to hold 3
yarn containers. What is the recommended way to ensure that all the
containers do not land upon the same physical node thereby becoming SPOF?
Re: Re: High Availability for an application running on yarn
Posted by Malcolm McFarland <mm...@cavulus.com>.
I think you can use the reservation system in Hadoop 2.8.x as well:
https://hadoop.apache.org/docs/r2.8.5/hadoop-yarn/hadoop-yarn-site/ReservationSystem.html
Cheers,
Malcolm McFarland
Cavulus
This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
unauthorized or improper disclosure, copying, distribution, or use of the
contents of this message is prohibited. The information contained in this
message is intended only for the personal and confidential use of the
recipient(s) named above. If you have received this message in error,
please notify the sender immediately and delete the original message.
On Mon, Mar 23, 2020 at 8:14 AM xyf <xy...@163.com> wrote:
> You can use Placements Constraints(
> https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html)
> or set
> yarn.scheduler.capacity.per-node-heartbeat.maximum-container-assignments to
> 1 if you RM config with capacityscheduler.
>
>
>
>
>
>
> At 2020-03-23 22:58:59, "Debraj Manna" <su...@gmail.com> wrote:
>
> The same has been posted in StackOverflow
> <https://stackoverflow.com/questions/60815863/high-availability-for-a-java-yarn-application> but
> did not get any response there also. Anyone any thoughts on this ?
>
> On Mon, Mar 23, 2020 at 3:52 PM Debraj Manna <su...@gmail.com>
> wrote:
>
>> Can someone let me know what is the recommended way to ensure high
>> availability for a yarn application?
>>
>> For example, let say I have 3 physical nodes. The yarn java application
>> needs 3 containers and each physical node has enough resources to hold 3
>> yarn containers. What is the recommended way to ensure that all the
>> containers do not land upon the same physical node thereby becoming SPOF?
>>
>
>
>
>
Re:Re: High Availability for an application running on yarn
Posted by xyf <xy...@163.com>.
You can use Placements Constraints(https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html) or set yarn.scheduler.capacity.per-node-heartbeat.maximum-container-assignments to 1 if you RM config with capacityscheduler.
At 2020-03-23 22:58:59, "Debraj Manna" <su...@gmail.com> wrote:
The same has been posted in StackOverflow but did not get any response there also. Anyone any thoughts on this ?
On Mon, Mar 23, 2020 at 3:52 PM Debraj Manna <su...@gmail.com> wrote:
Can someone let me know what is the recommended way to ensure high availability for a yarn application?
For example, let say I have 3 physical nodes. The yarn java application needs 3 containers and each physical node has enough resources to hold 3 yarn containers. What is the recommended way to ensure that all the containers do not land upon the same physical node thereby becoming SPOF?
Re: High Availability for an application running on yarn
Posted by Debraj Manna <su...@gmail.com>.
The same has been posted in StackOverflow
<https://stackoverflow.com/questions/60815863/high-availability-for-a-java-yarn-application>
but
did not get any response there also. Anyone any thoughts on this ?
On Mon, Mar 23, 2020 at 3:52 PM Debraj Manna <su...@gmail.com>
wrote:
> Can someone let me know what is the recommended way to ensure high
> availability for a yarn application?
>
> For example, let say I have 3 physical nodes. The yarn java application
> needs 3 containers and each physical node has enough resources to hold 3
> yarn containers. What is the recommended way to ensure that all the
> containers do not land upon the same physical node thereby becoming SPOF?
>