[jira] [Commented] (YARN-5261) Lease/Reclaim Extension to Yarn

["Yu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365914#comment-15365914 ] 

Yu commented on YARN-5261:
--------------------------

Yarn-5261 main purpose is: lease the unused resource on some nodes which already having some important services (outside of Yarn), and make the lease/reclaim controllable by the nodes' owners (landlords).
It also uses zookeeper to make the lease/reclaim operation recoverable even there's RM restarting or failover, so it's very stable.

I checked the issues you mentioned, and I got:
	* Yarn-5215:
		It has similar purpose as my issue: leverage the unused resources on some nodes.
		But, it's another approach, and can get different result.

		It assumes all nodes are schedulable by Yarn, and changes the unallocated resource by reducing the external utilization on each node dynamically.  
		My issue (Yarn-5261) is different:
		1. don't assume all nodes are schedulable (though Yarn NM is running on all nodes).  The nodes are schedulable only when the landlord leases them.  So, it's more controllable.
		2. the total schedulable resource is estimated by the landlord and set when node is leased, which is steadier than changing after calculating the utilization.  And, landlord can plan how to use the resources better, not rely on automatic calculating.
		3. when the schedulable resource is over used, the containers will be killed to make sure obeying the landlord.


	* Yarn-1011:
		It's more like a Yarn scheduler enhancement for containers, but don't consider the external utilization at all.
		It's not proper for the scenario that running yarn on nodes with other services.
	
	* Yarn-5202:
		It's about resource overcommit, not the same purpose.

So, they look similar, but are different.  Users can use them based on their requirements.

> Lease/Reclaim Extension to Yarn
> -------------------------------
>
>                 Key: YARN-5261
>                 URL: https://issues.apache.org/jira/browse/YARN-5261
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: scheduler
>            Reporter: Yu
>         Attachments: YARN-5261-1.patch, Yarn-5261.pdf
>
>
> In some clusters outside of Yarn, the machines resources are not fully utilized, e.g., resource usage is quite low at night.  
> To better utilize the resources while keep the existing SLA of the cluster, Lease/Reclaim Extension to Yarn is introduced.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org