[jira] Commented: (HADOOP-4035) Modify the capacity scheduler (HADOOP-3445) to schedule tasks based on memory requirements and task trackers free memory

["Hemanth Yamijala (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12632716#action_12632716 ] 

Hemanth Yamijala commented on HADOOP-4035:
------------------------------------------

Here's a summary of the approach that is implemented in the patch, so we can have a discussion around it.

There are 2 requirements we are trying to address:
- When a task is assigned to be run on the tasktracker, the scheduler must ensure that the task's job has a memory requirement that is matched by the free memory on the tasktracker.
- A user whose job requests for higher resources than usual would decrease the free memory on the tasktracker more than other jobs would. Therefore the user must be 'charged' the additional usage so that he would hit his limits and capacities sooner.

To handle the first requirement we obviously have to consider what happens when the job in the front of the scheduler's list does not match the tasktracker's memory availability. We considered the following choices:
- Move on to consider the next job (this can cause starvation ?)
- Block, and do not look at any other job, subject of course to other limits etc.
- Some variant of the first option where we move on to consider the next job some configured number of times, and then block.
Since it is not very clear what would be the right approach, we decided in favor of option 2, which had the advantage that the system behavior is very, very predictable. Of course, this is open for discussion.

In order to solve the second requirement, we need to match tasks to slots in the scheduler. One simplifying assumption made to achieve this is mentioned in the comments above. We assume that cluster wide, the default memory per slot is a fixed value, and specified in configuration. Note that this does not preclude heterogenity of cluster nodes. By tweaking the maximum memory on the tasktracker and the number of slots, it is possible to have the same value for all nodes in a cluster.

Using this configuration variable, it is possible to map tasks to slots as follows:
slots for a job = tasks for a job * (ceil(memory requested by job / configured memory per slot)). All computation of limits and capacities uses this formula for computing slots used / required etc.

Clearly, this issue requires discussion and consensus. It likely will not make 0.19. But we are hoping we can reach a consensus on the approach.

> Modify the capacity scheduler (HADOOP-3445) to schedule tasks based on memory requirements and task trackers free memory
> ------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4035
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4035
>             Project: Hadoop Core
>          Issue Type: Bug
>    Affects Versions: 0.19.0
>            Reporter: Hemanth Yamijala
>            Assignee: Vinod K V
>             Fix For: 0.19.0
>
>         Attachments: 4035.1.patch, HADOOP-4035-20080918.1.txt
>
>
> HADOOP-3759 introduced configuration variables that can be used to specify memory requirements for jobs, and also modified the tasktrackers to report their free memory. The capacity scheduler in HADOOP-3445 should schedule tasks based on these parameters. A task that is scheduled on a TT that uses more than the default amount of memory per slot can be viewed as effectively using more than one slot, as it would decrease the amount of free memory on the TT by more than the default amount while it runs. The scheduler should make the used capacity account for this additional usage while enforcing limits, etc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.