[jira] [Commented] (YARN-9548) [Umbrella] Make YARN work well in elastic cloud environments

["Junping Du (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-9548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906068#comment-16906068 ] 

Junping Du commented on YARN-9548:
----------------------------------

+1. 
I am quite interesting on autoscaling part. For horizontal scaling, we can leverage graceful decommission (YARN-914) to decommission/recommission nodes based on metrics monitoring. For vertical scaling, we can leverage dynamic resource allocation (YARN-291) to have a min/max resource setting on each node and update according to resource profiling of each node.

> [Umbrella] Make YARN work well in elastic cloud environments
> ------------------------------------------------------------
>
>                 Key: YARN-9548
>                 URL: https://issues.apache.org/jira/browse/YARN-9548
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Vinod Kumar Vavilapalli
>            Priority: Major
>
> YARN works well in static environments but there isn't fundamentally broken in YARN to stop us from making it work well in dynamic environments like cloud (public or private) as well.
> There are few areas where we need to invest though
>  # Autoscaling
>  -- cluster level: add/remove nodes intelligently based on metrics and/or admin plugins
>  -- node level: scale nodes up/down vertically?
>  # Smarter scheduling
>     -- to pack containers as opposed to spreading them around to account for nodes going away
>     -- to account for speculative nodes like spot instances
>  # Handling nodes going away better
>     -- by decommissioning sanely
>     -- dealing with auxiliary services data
>  # And any installation helpers in this dynamic world - scripts, operators etc.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org