[jira] Commented: (HADOOP-3759) Provide ability to run memory intensive jobs without affecting other running tasks on the nodes

["Hemanth Yamijala (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613596#action_12613596 ] 

Hemanth Yamijala commented on HADOOP-3759:
------------------------------------------

Initial proposal:

- HADOOP-3581 proposes a maximum amount of virtual memory, say MAX_MEM,  that all tasks (and their descendants) on that tasktracker would use.
- By default, we can translate this to a per task memory limit which is = MAX_MEM / number of slots
- To allow high memory jobs to run, we define a new configuration variable that users can set to specify the maximum memory they expect their tasks to take.
- In each heartbeat, the tasktracker computes the amount of free memory (using HADOOP-3581's fix) and reports that to the jobtracker. This is similar to the approach followed in HADOOP-657 for disk space.
- The jobtracker schedules a new task on this tasktracker, only if it's job's memory limit per task is less than the free memory.
- If a job's task takes more memory than what is specified (either default or explicitly configured), per HADOOP-3581, it will be killed.

Comments ?

> Provide ability to run memory intensive jobs without affecting other running tasks on the nodes
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-3759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3759
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>             Fix For: 0.19.0
>
>
> In HADOOP-3581, we are discussing how to prevent memory intensive tasks from affecting Hadoop daemons and other tasks running on a node. A related requirement is that users be provided an ability to run jobs which are memory intensive. The system must provide enough knobs to allow such jobs to be run while still maintaining the requirements of HADOOP-3581.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.