[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue

["Sunil G (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985160#comment-13985160 ] 

Sunil G commented on YARN-1963:
-------------------------------

We have done few analysis and implemented support for application priority.
I wish to share the thoughts here, kindly check the same.

Design thoughts:
1. Configuration Part
We planned to use some existing priority configuration as given below. These are used to set a Job priority.
a.	JobConf.getJobPriority() and Job.setPriority(JobPriority priority) 
b.	We can also use configuration "mapreduce.job.priority".

The values for priority can be VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW

2. Scheduler Side
If the Capacity Scheduler queue has multiple applications(Jobs) to run with different priorities, CS will allocate containers for the highest priority application and then for next priority and so on.
When multiple queues are configured with different capacities, this priority will work internal to the each queue.

For this, we planned to add a priority comparison check in the below data structure.
	Comparator<FiCaSchedulerApp> applicationComparator

We added a priority check here in compare() of applicationComparator while selecting applications. Updated design here will be like,
1.	Check for priority first. If there, return highest priority job.
2.	Continue existing logic such as App ID comparison and TimeStamp comparison.
	
With these changes, we can make highest priority job will get preference in a queue.

NB: In addition to this, we added a preemption module also to get High priority jobs resources fast by preempting lower priority ones.

I wish to upload a patch if this approach is fine.

> Support priorities across applications within the same queue 
> -------------------------------------------------------------
>
>                 Key: YARN-1963
>                 URL: https://issues.apache.org/jira/browse/YARN-1963
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: api, resourcemanager
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>
> It will be very useful to support priorities among applications within the same queue, particularly in production scenarios. It allows for finer-grained controls without having to force admins to create a multitude of queues, plus allows existing applications to continue using existing queues which are usually part of institutional memory.



--
This message was sent by Atlassian JIRA
(v6.2#6252)