[jira] Commented: (HADOOP-4439) Cleanup memory related resource management

["Hemanth Yamijala (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-4439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640465#action_12640465 ] 

Hemanth Yamijala commented on HADOOP-4439:
------------------------------------------

The following changes are identified:

- We remove the concept of having a default memory per task on the TT, introduced in HADOOP-3759 as the max memory per TT / number of slots. The problem with this model is that in a heterogenous cluster, different TTs could give different default memory per task values for the same job, which is confusing.
- Instead, we introduce a default memory per task configuration variable that is expected to be controlled by the cluster admin. This is the value that will be used for a job which does not specify any memory requirements. The advantage with this model is that it eases configuration, and makes the default value consistent for the users.
- If a job has not specified any memory requirements, this variable would be set to the job's configuration, maybe via the {{Task}} object.
- We modify the algorithm of protecting RAM limits introduced in HADOOP-3581 to use the configured memory per task instead of the default memory per task.
- We remove the reporting of the default memory per task, introduced in HADOOP-3759, done via the {{TaskTrackerStatus.ResourceStatus}}. Instead we report the total memory available on the TT instead of the default memory per task.
- When HADOOP-4053 is fixed, the above values would be used to schedule tasks.
- However until HADOOP-4053 is fixed, these configuration parameters and the corresponding {{JobConf}} variables should not be exposed in any public API or documentation, as they could confuse users. This can be turned on after Hadoop 0.19

> Cleanup memory related resource management
> ------------------------------------------
>
>                 Key: HADOOP-4439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4439
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.19.0
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>            Priority: Blocker
>             Fix For: 0.19.0
>
>
> HADOOP-3759 and HADOOP-3581 introduced memory based resource management. This JIRA is to cleanup certain aspects of the two issues that came up while doing HADOOP-4035, which is filed to support memory based scheduling 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.