You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Vinod K V (JIRA)" <ji...@apache.org> on 2009/05/21 07:54:46 UTC
[jira] Created: (HADOOP-5881) Simplify configuration related to
task-memory-monitoring and memory-based scheduling
Simplify configuration related to task-memory-monitoring and memory-based scheduling
------------------------------------------------------------------------------------
Key: HADOOP-5881
URL: https://issues.apache.org/jira/browse/HADOOP-5881
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Reporter: Vinod K V
The configuration we have now is pretty complicated. Besides everything else, the mechanism of not specifying parameters separately for maps and reduces leads to problems like HADOOP-5811. This JIRA should address simplifying things and at the same time solving these problems.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5881) Simplify configuration related to
task-memory-monitoring and memory-based scheduling
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713425#action_12713425 ]
Hemanth Yamijala commented on HADOOP-5881:
------------------------------------------
I had a chat with Owen regarding the objections raised.
bq. I'm concerned that this went in without an opportunity for review. The patch was uploaded at 5:49am and committed at 6:36am.
I apologize for the rushed commit and take responsibility. However, as I explained above I had reviewed the patch for a significant time. The first version which I reviewed is not on JIRA. In retrospect, I think that was a mistake, and will avoid repeating it in future. Sorry !
bq. I'm also concerned that this incompatibly changes the config variables in the 0.20 branch. What is the rationale for pushing these configuration changes back to 0.20? If they are required, they need a compatability story.
There was a misunderstanding on this point. It seemed like in an internal discussion we agreed to this approach. However, we are reverting stand after rethinking. I opened HADOOP-5919 to discuss and provide a backwards compatibility story for this feature. Let's keep the discussion there.
Based on these explanations, can I resolve this bug ?
> Simplify configuration related to task-memory-monitoring and memory-based scheduling
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5881
> URL: https://issues.apache.org/jira/browse/HADOOP-5881
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Vinod K V
> Assignee: Vinod K V
> Fix For: 0.20.1
>
> Attachments: HADOOP-5881-20090526-branch-20.1.txt, HADOOP-5881-20090526.1.txt
>
>
> The configuration we have now is pretty complicated. Besides everything else, the mechanism of not specifying parameters separately for maps and reduces leads to problems like HADOOP-5811. This JIRA should address simplifying things and at the same time solving these problems.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5881) Simplify configuration related to
task-memory-monitoring and memory-based scheduling
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715495#action_12715495 ]
Hemanth Yamijala commented on HADOOP-5881:
------------------------------------------
bq. It would be useful if the description of the above memory management parameters is added in mapred-default.xml. Also mapred.tasktracker.vmem.reserved, mapred.task.default.maxvmem and mapred.task.limit.maxvmem has to be removed from the xml.
+1 regarding documentation. I filed HADOOP-5957 to track this. But we are planning to introduce backwards compatible options, so removing the configuration parameters from xml will not be necessary. Please see HADOOP-5919.
> Simplify configuration related to task-memory-monitoring and memory-based scheduling
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5881
> URL: https://issues.apache.org/jira/browse/HADOOP-5881
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Vinod K V
> Assignee: Vinod K V
> Fix For: 0.20.1
>
> Attachments: HADOOP-5881-20090526-branch-20.1.txt, HADOOP-5881-20090526.1.txt
>
>
> The configuration we have now is pretty complicated. Besides everything else, the mechanism of not specifying parameters separately for maps and reduces leads to problems like HADOOP-5811. This JIRA should address simplifying things and at the same time solving these problems.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5881) Simplify configuration related to
task-memory-monitoring and memory-based scheduling
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718619#action_12718619 ]
Hudson commented on HADOOP-5881:
--------------------------------
Integrated in Hadoop-trunk #863 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/863/])
> Simplify configuration related to task-memory-monitoring and memory-based scheduling
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5881
> URL: https://issues.apache.org/jira/browse/HADOOP-5881
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Vinod K V
> Assignee: Vinod K V
> Fix For: 0.20.1
>
> Attachments: HADOOP-5881-20090526-branch-20.1.txt, HADOOP-5881-20090526.1.txt
>
>
> The configuration we have now is pretty complicated. Besides everything else, the mechanism of not specifying parameters separately for maps and reduces leads to problems like HADOOP-5811. This JIRA should address simplifying things and at the same time solving these problems.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5881) Simplify configuration related to
task-memory-monitoring and memory-based scheduling
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V updated HADOOP-5881:
------------------------------
Status: Patch Available (was: Open)
> Simplify configuration related to task-memory-monitoring and memory-based scheduling
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5881
> URL: https://issues.apache.org/jira/browse/HADOOP-5881
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Vinod K V
> Attachments: HADOOP-5881-20090526.1.txt
>
>
> The configuration we have now is pretty complicated. Besides everything else, the mechanism of not specifying parameters separately for maps and reduces leads to problems like HADOOP-5811. This JIRA should address simplifying things and at the same time solving these problems.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5881) Simplify configuration related to
task-memory-monitoring and memory-based scheduling
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713004#action_12713004 ]
Hemanth Yamijala commented on HADOOP-5881:
------------------------------------------
I've been reviewing this patch offline and already had given comments that are incorporated into the new patch. Based on that check, I am a +1 for the code changes. Will commit this.
> Simplify configuration related to task-memory-monitoring and memory-based scheduling
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5881
> URL: https://issues.apache.org/jira/browse/HADOOP-5881
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Vinod K V
> Attachments: HADOOP-5881-20090526-branch-20.1.txt, HADOOP-5881-20090526.1.txt
>
>
> The configuration we have now is pretty complicated. Besides everything else, the mechanism of not specifying parameters separately for maps and reduces leads to problems like HADOOP-5811. This JIRA should address simplifying things and at the same time solving these problems.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HADOOP-5881) Simplify configuration related to
task-memory-monitoring and memory-based scheduling
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley reopened HADOOP-5881:
-----------------------------------
I'm concerned that this went in without an opportunity for review. The patch was uploaded at 5:49am and committed at 6:36am.
I'm also concerned that this incompatibly changes the config variables in the 0.20 branch. What is the rationale for pushing these configuration changes back to 0.20? If they are required, they need a compatability story.
> Simplify configuration related to task-memory-monitoring and memory-based scheduling
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5881
> URL: https://issues.apache.org/jira/browse/HADOOP-5881
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Vinod K V
> Assignee: Vinod K V
> Fix For: 0.20.1
>
> Attachments: HADOOP-5881-20090526-branch-20.1.txt, HADOOP-5881-20090526.1.txt
>
>
> The configuration we have now is pretty complicated. Besides everything else, the mechanism of not specifying parameters separately for maps and reduces leads to problems like HADOOP-5811. This JIRA should address simplifying things and at the same time solving these problems.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-5881) Simplify configuration related to
task-memory-monitoring and memory-based scheduling
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley resolved HADOOP-5881.
-----------------------------------
Resolution: Fixed
Yes, thanks for addressing my concerns Hemanth.
> Simplify configuration related to task-memory-monitoring and memory-based scheduling
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5881
> URL: https://issues.apache.org/jira/browse/HADOOP-5881
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Vinod K V
> Assignee: Vinod K V
> Fix For: 0.20.1
>
> Attachments: HADOOP-5881-20090526-branch-20.1.txt, HADOOP-5881-20090526.1.txt
>
>
> The configuration we have now is pretty complicated. Besides everything else, the mechanism of not specifying parameters separately for maps and reduces leads to problems like HADOOP-5811. This JIRA should address simplifying things and at the same time solving these problems.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5881) Simplify configuration related to
task-memory-monitoring and memory-based scheduling
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712991#action_12712991 ]
Vinod K V commented on HADOOP-5881:
-----------------------------------
The uploaded patch (for trunk) passed ant test-patch, run-mapred-tests and test-contrib, leaving aside the usual suspects for failure - TestReduceFetch, TestStreamingBadRecords, TestStreamingExitStatus, TestQueueCapacities and TestJobHistory(which succeeds on a clean workspace but not when run along with all the other tests).
> Simplify configuration related to task-memory-monitoring and memory-based scheduling
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5881
> URL: https://issues.apache.org/jira/browse/HADOOP-5881
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Vinod K V
> Attachments: HADOOP-5881-20090526.1.txt
>
>
> The configuration we have now is pretty complicated. Besides everything else, the mechanism of not specifying parameters separately for maps and reduces leads to problems like HADOOP-5811. This JIRA should address simplifying things and at the same time solving these problems.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5881) Simplify configuration related to
task-memory-monitoring and memory-based scheduling
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712919#action_12712919 ]
Vinod K V commented on HADOOP-5881:
-----------------------------------
Here's the proposal:
|| *Parameter* || *Type* || *Meaning* ||
| mapred.cluster.map.memory.mb | set by admin, cluster-wide | Cluster definition of memory per map slot. The maximum amount of memory, in MB, each map task on a tasktracker can consume.|
| mapred.cluster.reduce.memory.mb |set by admin, cluster-wide | Cluster definition of memory per reduce slot. The maximum amount of memory, in MB, each reduce task on a tasktracker can consume. |
| mapred.job.map.memory.mb | set by user, per-job | Job requirement for map tasks. The maximum amount of memory each map task of a job can consume, in MB. |
| mapred.job.reduce.memory.mb | set by user, per-job | job requirement for reduce tasks. The maximum amount of memory each reduce task of a job can consume, in MB. |
|mapred.cluster.max.map.memory.mb | set by admin, cluster-wide | Max limit on jobs. The maximum value that can be specified by a user via mapred.job.map.memory.mb, in MB. A job that asks for more than this number will be failed at submission itself.|
|mapred.cluster.max.reduce.memory.mb | set by admin, cluster-wide | Max limit on jobs. The maximum value that can be specified by a user via mapred.job.reduce.memory.mb, in MB. A job that asks for more than this number will be failed at submission itself.|
> Simplify configuration related to task-memory-monitoring and memory-based scheduling
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5881
> URL: https://issues.apache.org/jira/browse/HADOOP-5881
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Vinod K V
>
> The configuration we have now is pretty complicated. Besides everything else, the mechanism of not specifying parameters separately for maps and reduces leads to problems like HADOOP-5811. This JIRA should address simplifying things and at the same time solving these problems.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5881) Simplify configuration related to
task-memory-monitoring and memory-based scheduling
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V updated HADOOP-5881:
------------------------------
Attachment: HADOOP-5881-20090526-branch-20.1.txt
Patch for branch-0.20.
> Simplify configuration related to task-memory-monitoring and memory-based scheduling
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5881
> URL: https://issues.apache.org/jira/browse/HADOOP-5881
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Vinod K V
> Attachments: HADOOP-5881-20090526-branch-20.1.txt, HADOOP-5881-20090526.1.txt
>
>
> The configuration we have now is pretty complicated. Besides everything else, the mechanism of not specifying parameters separately for maps and reduces leads to problems like HADOOP-5811. This JIRA should address simplifying things and at the same time solving these problems.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5881) Simplify configuration related to
task-memory-monitoring and memory-based scheduling
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712751#action_12712751 ]
Vinod K V commented on HADOOP-5881:
-----------------------------------
Some of the problems that stood out and the corresponding solutions, after some discussions with Eric, Arun and Hemanth:
- The current configuration system doesn't distinguish memory usage of maps and reduces. In general, reduces are more memory intensive than maps. Also, because of this lack of distinguishing, we use the total memory available on the TT as a shared resource across slot types. This lead to problems mentioned in HADOOP-5811. The solution is to divide memory resource between map slots and reduce slots. In presence of high memory jobs, map slots of these jobs will use memory of other map slots and don't take over memory from reduce slots. And vice versa. To reflect the differences in general usage between map slots and reduce slots, we may want to specify more map slots on a node than the number of reduce slots.
- We now have separate configuration for specifying default values for the job's memory configuration. This is unnecessary and can better be handled by using layers of configuration. As Arun suggests, default values can be provided by cluster admins to the users via the configuration files distributed to clients.
- With the current configuration, we have 1) total memory available calculated on a TT, 2) a `reserved` memory on TT for system usage. Because of this mechanism, if a TT has lower memory overall, then we assign lower memory to a single slot. As per the discussions, this seems like a wrong idea. To paraphrase Eric - "A slot is a slot, is a slot. TT will just be configured with the number of slots (map & reduce). " In essential, if a TT has lower memory, the correct scheme is to decrease the number of slots and not the memory per slot. Memory allotted for slot should be more or less same across all the TTs in the cluster.
- We are distinguishing virtual memory used by processes with the physical memory. This seems necessary when considering streaming/pipes tasks. However, "in Java, once the VM hits swap, performance degrades fast, we want to configure the limits based on the physical memory on the machine (not including swap), to avoid thrashing". With this in view, there doesn't seem to any need for distinguishing vmem from physical memory w.r.t configuration. Depending on a site's requirements, the configuration items can reflect whether we want tasks to go beyond physical memory or not.
> Simplify configuration related to task-memory-monitoring and memory-based scheduling
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5881
> URL: https://issues.apache.org/jira/browse/HADOOP-5881
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Vinod K V
>
> The configuration we have now is pretty complicated. Besides everything else, the mechanism of not specifying parameters separately for maps and reduces leads to problems like HADOOP-5811. This JIRA should address simplifying things and at the same time solving these problems.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5881) Simplify configuration related to
task-memory-monitoring and memory-based scheduling
Posted by "Ramya R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715488#action_12715488 ]
Ramya R commented on HADOOP-5881:
---------------------------------
It would be useful if the description of the above memory management parameters is added in mapred-default.xml. Also mapred.tasktracker.vmem.reserved, mapred.task.default.maxvmem and mapred.task.limit.maxvmem has to be removed from the xml.
> Simplify configuration related to task-memory-monitoring and memory-based scheduling
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5881
> URL: https://issues.apache.org/jira/browse/HADOOP-5881
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Vinod K V
> Assignee: Vinod K V
> Fix For: 0.20.1
>
> Attachments: HADOOP-5881-20090526-branch-20.1.txt, HADOOP-5881-20090526.1.txt
>
>
> The configuration we have now is pretty complicated. Besides everything else, the mechanism of not specifying parameters separately for maps and reduces leads to problems like HADOOP-5811. This JIRA should address simplifying things and at the same time solving these problems.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5881) Simplify configuration related to
task-memory-monitoring and memory-based scheduling
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-5881:
-------------------------------------
Resolution: Fixed
Fix Version/s: 0.20.1
Assignee: Vinod K V
Hadoop Flags: [Incompatible change, Reviewed]
Status: Resolved (was: Patch Available)
I just committed this to trunk and branch 0.20. Thanks, Vinod !
> Simplify configuration related to task-memory-monitoring and memory-based scheduling
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5881
> URL: https://issues.apache.org/jira/browse/HADOOP-5881
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Vinod K V
> Assignee: Vinod K V
> Fix For: 0.20.1
>
> Attachments: HADOOP-5881-20090526-branch-20.1.txt, HADOOP-5881-20090526.1.txt
>
>
> The configuration we have now is pretty complicated. Besides everything else, the mechanism of not specifying parameters separately for maps and reduces leads to problems like HADOOP-5811. This JIRA should address simplifying things and at the same time solving these problems.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5881) Simplify configuration related to
task-memory-monitoring and memory-based scheduling
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V updated HADOOP-5881:
------------------------------
Attachment: HADOOP-5881-20090526.1.txt
Patch implementing the proposal.
> Simplify configuration related to task-memory-monitoring and memory-based scheduling
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5881
> URL: https://issues.apache.org/jira/browse/HADOOP-5881
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Vinod K V
> Attachments: HADOOP-5881-20090526.1.txt
>
>
> The configuration we have now is pretty complicated. Besides everything else, the mechanism of not specifying parameters separately for maps and reduces leads to problems like HADOOP-5811. This JIRA should address simplifying things and at the same time solving these problems.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.