[jira] Commented: (HADOOP-4513) Capacity scheduler should initialize tasks asynchronously

["Vivek Ratan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-4513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644943#action_12644943 ] 

Vivek Ratan commented on HADOOP-4513:
-------------------------------------

We should have a thread (call it the init thread) within the Capacity Scheduler that initializes jobs (i.e., calls initTasks() on jobs even when the job is not ready to run, as per the scheduler). The following features are desirable: 
* We want pre-initialized jobs in all the queues, since the Scheduler can possibly look at all queues between consecutive runs of this thread. 
* Since initTasks() may take a fair bit of time for some jobs, we don't want to block on this call as much as possible. 
* Remember that the init thread is really predicting which jobs the scheduler will run in the next many heartbeats. It's hard to make this prediction as the decision of which job gets a free slot depends on the state of the running jobs at that very instant (to compute user quotas, for example). But there are some constraints that the thread can consider: 
** the waiting queue is sorted, and the scheduler considers jobs in that sort order. So the init thread should pre-initialize jobs in the same order. 
** User quotas are tricky. Suppose a queue supports N concurrent users. The init thread should really pre-initialize jobs from these N users only, as described earlier. But this list of N users can change often, depending on which jobs from which users finish. So, to be safe, the init thread should pre-initialize jobs from, say N+2 users, to be safe. 
* Since we're keeping the run-queue sorted (HADOOP-4471), and that jobs are moved from the wait queue to the run queue once initTasks is called and the setup task is complete, the Scheduler only needs to look at jobs in its run queue. If it doesn't find any job that can accept a task, it moves on to the next queue. 
* If the init thread runs periodically, there may be a brief startup delay. Suppose the thread runs every 5 seconds. Suppose a job is submitted to an empty queue just after the thread runs. That job will not be initialized till the init thread runs next (a max delay of 5 secs). This is not too bad, given that there will be a delay in running the setup task of that job too. Worst case, the Scheduler can always cause the init thread to run as soon as a job is submitted to an empty queue, but this may be an overkill. 

> Capacity scheduler should initialize tasks asynchronously
> ---------------------------------------------------------
>
>                 Key: HADOOP-4513
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4513
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/capacity-sched
>    Affects Versions: 0.19.0
>            Reporter: Hemanth Yamijala
>            Assignee: Sreekanth Ramakrishnan
>
> Currently, the capacity scheduler initializes tasks on demand, as opposed to the eager initialization technique used by the default scheduler. This is done in order to save JT memory footprint. However, the initialization is done in the {{assignTasks}} API which is not a good idea as task initialization could be a time consuming operation. This JIRA is to move out the initialization outside the {{assignTasks}} API and do it asynchronously.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.