You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Arun C Murthy (JIRA)" <ji...@apache.org> on 2007/10/26 13:31:54 UTC
[jira] Issue Comment Edited: (HADOOP-1274) Configuring different
number of mappers and reducers per TaskTracker
[ https://issues.apache.org/jira/browse/HADOOP-1274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12537926 ]
acmurthy edited comment on HADOOP-1274 at 10/26/07 4:30 AM:
-----------------------------------------------------------------
Comments:
1. Conversion to short here is unnecessary, probably dangerous:
{noformat}
- metricsRecord.setMetric("taskSlots", (short)maxCurrentTasks);
+ metricsRecord.setMetric("mapTaskSlots", (short)maxCurrentMapTasks);
+ metricsRecord.setMetric("reduceTaskSlots",
+ (short)maxCurrentReduceTasks);
{noformat}
2. Please deprecate {{mapred.tasktracker.tasks.maximum}} for this release, as per existing norms. We cannot remove it right-away.
{noformat}
- maxCurrentTasks = conf.getInt("mapred.tasktracker.tasks.maximum", 2);
+ maxCurrentMapTasks = conf.getInt(
+ "mapred.tasktracker.map.tasks.maximum", 2);
+ maxCurrentReduceTasks = conf.getInt(
+ "mapred.tasktracker.reduce.tasks.maximum", 1);
{noformat}
{{mapred.tasktracker.tasks.maximum}} should be superceded by {{mapred.tasktracker.map.tasks.maximum}} and {{mapred.tasktracker.reduce.tasks.maximum}} for hadoop-0.16.0.
3. Should we consider having different {{JobTracker.PAD_FRACTION}} for maps and reduces? Clearly the no. of padded slots for reduces should be higher since reduce-failures are more expensive...
was (Author: acmurthy):
Comments:
1. Conversion to short here is unnecessary, probably dangerous:
{noformat}
- metricsRecord.setMetric("taskSlots", (short)maxCurrentTasks);
+ metricsRecord.setMetric("mapTaskSlots", (short)maxCurrentMapTasks);
+ metricsRecord.setMetric("reduceTaskSlots",
+ (short)maxCurrentReduceTasks);
{noformat}
2. Please deprecate {{mapred.tasktracker.tasks.maximum}} for this release, as per existing norms. We cannot remove it right-away. {{mapred.tasktracker.tasks.maximum}} should be superceded by {{mapred.tasktracker.map.tasks.maximum} and {{mapred.tasktracker.reduce.tasks.maximum}} for 0.16.0.
{noformat}
- maxCurrentTasks = conf.getInt("mapred.tasktracker.tasks.maximum", 2);
+ maxCurrentMapTasks = conf.getInt(
+ "mapred.tasktracker.map.tasks.maximum", 2);
+ maxCurrentReduceTasks = conf.getInt(
+ "mapred.tasktracker.reduce.tasks.maximum", 1);
{noformat}
3. Should we consider having different {{JobTracker.PAD_FRACTION}} for maps and reduces? Clearly the no. of padded slots for reduces should be higher since reduce-failures are more expensive...
> Configuring different number of mappers and reducers per TaskTracker
> --------------------------------------------------------------------
>
> Key: HADOOP-1274
> URL: https://issues.apache.org/jira/browse/HADOOP-1274
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Reporter: Koji Noguchi
> Assignee: Amareshwari Sri Ramadasu
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: patch-1274.txt
>
>
> Depending on the application, it sometimes make sense to have more mappers than reducers assigned to each node.
> (I'm assuming user either has a dedicated cluster or use HOD.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.