[jira] Commented: (HADOOP-3759) Provide ability to run memory intensive jobs without affecting other running tasks on the nodes

["Hemanth Yamijala (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12619474#action_12619474 ] 

Hemanth Yamijala commented on HADOOP-3759:
------------------------------------------

After some internal discussions, we are proposing the following behavior with respect to default values of the configuration options MAX_MEM and MAX_MEM_PER_TASK. In the description below, MAX_MEM is specified for a tasktracker, and MAX_MEM_PER_TASK is specified in a job's configuration.

- We should provide an option to turn off the behavior described in the above comment. We can do this by defaulting the configuration option values to something invalid, like -1.
- This gives rise to the following 4 conditions, and the proposed behavior of the system for each:
-- Both MAX_MEM  and MAX_MEM_PER_TASK are set to invalid values: This will result in the current Map/Reduce system behavior. Essentially, it will turn off the fixes in this JIRA and in HADOOP-3581, thereby providing backwards compatibility.
-- Both MAX_MEM and MAX_MEM_PER_TASK are specified as valid values: This will result in the behavior described on this JIRA and also in HADOOP-3581. In short, it will result in the tasks of the job being scheduled on TaskTrackers with sufficient free memory, and tasks which exceed the memory limits will be killed.
-- MAX_MEM is set to invalid value but MAX_MEM_PER_TASK is specified for a job: MAX_MEM_PER_TASK is ignored. Since MAX_MEM is an administrative option, if an administrator has disabled it , no memory specific scheduling can be turned on by users.
-- MAX_MEM is specified, but MAX_MEM_PER_TASK is set to an invalid value: A task of this job will run on this TT only if the free memory is at least equal to MAX_MEM / number of slots on the TT. For e.g. consider a TT with 8 GB as MAX_MEM and 2 slots. Consider two jobs: one with a 6 GB MAX_MEM_PER_TASK and the other which has no valid value specified. If a task of the first job is running on this TT, there is only 2 GB that can be guaranteed for any other job. Since MAX_MEM / number of slots is 4 GB, no task of the second job would run on this TT until the high memory task completes. In other words, we are defining that if a job has not specified any MAX_MEM_PER_TASK, it is guaranteed at least a MAX_MEM / number of slots amount of memory by the system. This seems to be fairly deterministic compared to scheduling tasks of the second job and binding their MAX_MEM_PER_TASK to some arbitrary amount of free memory.

As the above choice means that the RAM intensive jobs can prevent other jobs from running, there must be some way to counter-balance this effect. One way could be for schedulers like HADOOP-3445 to artifically consider the RAM intensive job as using more slots than actual, and thereby subjecting them to limits faster.

Comments ?

> Provide ability to run memory intensive jobs without affecting other running tasks on the nodes
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-3759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3759
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>             Fix For: 0.19.0
>
>         Attachments: HADOOP-3759.patch
>
>
> In HADOOP-3581, we are discussing how to prevent memory intensive tasks from affecting Hadoop daemons and other tasks running on a node. A related requirement is that users be provided an ability to run jobs which are memory intensive. The system must provide enough knobs to allow such jobs to be run while still maintaining the requirements of HADOOP-3581.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.