[jira] [Commented] (YARN-2986) (Umbrella) Support hierarchical and unified scheduler configuration

["Wangda Tan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329917#comment-14329917 ] 

Wangda Tan commented on YARN-2986:
----------------------------------

An update:

After an offline discussion with [~vinodkv] and [~jianhe], now proposed configuration file looks like:
{code}
<scheduler>
  <type>capacity</type>
  <maximum-applications>9999</maximum-applications>
  <queue-mappings></queue-mappings>
  <queue-mappings-override-enable></queue-mappings-override-enable>
  <maximum-am-resource-percent>0.3</maximum-am-resource-percent> 
  
  <policy-properties>
    <resource-calculator>
    	org.apache.hadoop.yarn.util.resource.DominantResourceCalculator
  	</resource-calculator>
  </policy-properties>
  
  <queue name="root">
  	<queues>
	    <queue name="default">
	      <state>RUNNING</state>
	      <acl_submit_applications>*</acl_submit_applications>
	      <acl_administer_queue>*</acl_administer_queue>
	      <accessible-node-labels>x</accessible-node-labels>
	      
	      <policy-properties>
	        <user-limit-factor>2</user-limit-factor>
	        <capacity>50</capacity>
	        <maximum-capacity>90</maximum-capacity>
	        <node-locality-delay>30</node-locality-delay>
	        <node-labels>
		        <node-label name="x">
			      	<capacity>20</capacity>
			      	<maximum-capacity>50</maximum-capacity>
		        </node-label>
	        </node-labels>
	      </policy-properties>
	    </queue>
    </queues>
  </queue>
</scheduler>
{code}

One highlight of this proposal and previous proposal is: this contains a "policy-properties" for each configuration node, which means a scheduler-specific configurations, like capacity in CapacityScheduler and minShare in FairScheduler, etc. ("policy" here means different kinds of scheduling method).

For other common options (not belongs to a specific scheduler implementation), should be placed outside of "policy-properties".

*Please feel free to share your thoughts about this proposal :).*

To move this forward, I filed several sub ticket, YARN-3233 is targeted to solve the configuration file (for common scheduler and capacity scheduler) definition and parsing, I will upload a patch right now. YARN-3234 is to solve Capacity Scheduler integration with the new config file.

> (Umbrella) Support hierarchical and unified scheduler configuration
> -------------------------------------------------------------------
>
>                 Key: YARN-2986
>                 URL: https://issues.apache.org/jira/browse/YARN-2986
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Wangda Tan
>         Attachments: YARN-2986.1.patch
>
>
> Today's scheduler configuration is fragmented and non-intuitive, and needs to be improved. Details in comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)