[jira] [Commented] (YARN-3054) Preempt policy in FairScheduler may cause mapreduce job never finish

["Wei Yan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277174#comment-14277174 ] 

Wei Yan commented on YARN-3054:
-------------------------------

Hi, [~peng.zhang]. Firstly, FairScheduler will check whether the usage is over fairness.
{code}
  private boolean preemptContainerPreCheck() {
    return parent.getPolicy().checkIfUsageOverFairShare(getResourceUsage(),
        getFairShare());
  }
{code}

bq. Mapreduce jobs can get additional resources when others are idle.
I'm not sure what your "idle" meaning here. But in YARN, one queue can take the over-fairshare resource, if the resources are not used by other queues. And in FairScheduler, each queue has "steady" fairshare and "dynamic" fairshare. For example, if we have two queues (Q1 and Q2), both with weight 1. So Q1's steady share is 50%, and Q2 is also 50%. Assume only Q1 has jobs and no job submitted to Q2, Q1's dynamic fairness is 100% and Q2 is 0. The dynamic fairshare calculation only considers "active" queues.

bq. Mapreduce jobs for one user in one queue can still progress with its min share when others preempt resources back.
As I said above, each queue is guaranted with minshare and fairshare. That means, some jobs can still move on. We cannot assign a minshare to each job. Otherwise, the job with multiple concurrent jobs may take over the cluster.

> Preempt policy in FairScheduler may cause mapreduce job never finish
> --------------------------------------------------------------------
>
>                 Key: YARN-3054
>                 URL: https://issues.apache.org/jira/browse/YARN-3054
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: fairscheduler
>    Affects Versions: 2.6.0
>            Reporter: Peng Zhang
>
> Preemption policy is related with schedule policy now. Using comparator of schedule policy to find preemption candidate cannot guarantee a subset of containers never be preempted. And this may cause tasks to be preempted periodically before they finish. So job cannot make any progress. 
> I think preemption in YARN should got below assurance:
> 1. Mapreduce jobs can get additional resources when others are idle;
> 2. Mapreduce jobs for one user in one queue can still progress with its min share when others preempt resources back.
> Maybe always preempt the latest app and container can get this? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)