[jira] [Commented] (YARN-4781) Support intra-queue preemption for fairness ordering policy.

["Anup Tiwari (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16253273#comment-16253273 ] 

Anup Tiwari commented on YARN-4781:
-----------------------------------

Hi [~sunilg]/ [~leftnoteasy],

I think i am also getting above issue in our scenario like one big query is taking all resources of a queue lets say Q1. And when i am launching another query in Q1, almost always it is hanging in ACCEPTED state and sometime in RUNNING state but just got 1 container but not progressing. We are using Apache Hadoop Distribution version 2.8.0 and Apache Hive 2.1.1.

After reading this JIRA, as i understood we are not yet supporting "fair" ordering policy within same queue.
But i am bit confused in what exactly "yarn.scheduler.capacity.<queue-path>.ordering-policy" property does when set to "fair".
As i can read on net above property used to specify ordering policies on a per-queue basis, set the following property to fifo or fair. The default setting is fifo.
Can you or anyone explain how capacity scheduler works when orderin-policy set to "fair" with example?

I have also gone through https://issues.apache.org/jira/browse/YARN-3463 but didn't understood.

[~milesc], Can you bit explain your solution? Because i think we would end creating too many queues and also how each queue uses all resources of cluster when nothing is running? And also as far as i know, if we have to remove existing queues then it can't be done on runtime, it requires yarn restart. Let me know if i am missing anything.

Thanks in advance.

> Support intra-queue preemption for fairness ordering policy.
> ------------------------------------------------------------
>
>                 Key: YARN-4781
>                 URL: https://issues.apache.org/jira/browse/YARN-4781
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>
> We introduced fairness queue policy since YARN-3319, which will let large applications make progresses and not starve small applications. However, if a large application takes the queue’s resources, and containers of the large app has long lifespan, small applications could still wait for resources for long time and SLAs cannot be guaranteed.
> Instead of wait for application release resources on their own, we need to preempt resources of queue with fairness policy enabled.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org