You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2006/10/31 01:26:16 UTC
[jira] Created: (HADOOP-657) Free temporary space should be
modelled better
Free temporary space should be modelled better
----------------------------------------------
Key: HADOOP-657
URL: http://issues.apache.org/jira/browse/HADOOP-657
Project: Hadoop
Issue Type: Improvement
Components: mapred
Affects Versions: 0.7.2
Reporter: Owen O'Malley
Assigned To: Owen O'Malley
Currently, there is a configurable size that must be free for a task tracker to accept a new task. However, that isn't a very good model of what the task is likely to take. I'd like to propose:
Map tasks: totalInputSize * conf.getFloat("map.output.growth.factor", 1.0) / numMaps
Reduce tasks: totalInputSize * 2 * conf.getFloat("map.output.growth.factor", 1.0) / numReduces
where totalInputSize is the size of all the maps inputs for the given job.
To start a new task,
newTaskAllocation + (sum over running tasks of (1.0 - done) * allocation) >=
free disk * conf.getFloat("mapred.max.scratch.allocation", 0.90);
So in English, we will model the expected sizes of tasks and only task tasks that should leave us a 10% margin. With:
map.output.growth.factor -- the relative size of the transient data relative to the map inputs
mapred.max.scratch.allocation -- the maximum amount of our disk we want to allocate to tasks.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-657) Free temporary space should be
modelled better
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-657?page=comments#action_12450121 ]
Owen O'Malley commented on HADOOP-657:
--------------------------------------
option #2 with putting the "unallocated" disk space into the TaskTrackerStatus should work well.
> Free temporary space should be modelled better
> ----------------------------------------------
>
> Key: HADOOP-657
> URL: http://issues.apache.org/jira/browse/HADOOP-657
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.7.2
> Reporter: Owen O'Malley
> Assigned To: Arun C Murthy
>
> Currently, there is a configurable size that must be free for a task tracker to accept a new task. However, that isn't a very good model of what the task is likely to take. I'd like to propose:
> Map tasks: totalInputSize * conf.getFloat("map.output.growth.factor", 1.0) / numMaps
> Reduce tasks: totalInputSize * 2 * conf.getFloat("map.output.growth.factor", 1.0) / numReduces
> where totalInputSize is the size of all the maps inputs for the given job.
> To start a new task,
> newTaskAllocation + (sum over running tasks of (1.0 - done) * allocation) >=
> free disk * conf.getFloat("mapred.max.scratch.allocation", 0.90);
> So in English, we will model the expected sizes of tasks and only task tasks that should leave us a 10% margin. With:
> map.output.growth.factor -- the relative size of the transient data relative to the map inputs
> mapred.max.scratch.allocation -- the maximum amount of our disk we want to allocate to tasks.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-657) Free temporary space should be
modelled better
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-657?page=comments#action_12454714 ]
Arun C Murthy commented on HADOOP-657:
--------------------------------------
It also necessiates a 'Writable' MetricsRecordImpl (for RPC) and some apis for 'reading' the metrics i.e. getMetric/getTag apis which the JobTracker can use to retrieve information.
> Free temporary space should be modelled better
> ----------------------------------------------
>
> Key: HADOOP-657
> URL: http://issues.apache.org/jira/browse/HADOOP-657
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.7.2
> Reporter: Owen O'Malley
> Assigned To: Arun C Murthy
>
> Currently, there is a configurable size that must be free for a task tracker to accept a new task. However, that isn't a very good model of what the task is likely to take. I'd like to propose:
> Map tasks: totalInputSize * conf.getFloat("map.output.growth.factor", 1.0) / numMaps
> Reduce tasks: totalInputSize * 2 * conf.getFloat("map.output.growth.factor", 1.0) / numReduces
> where totalInputSize is the size of all the maps inputs for the given job.
> To start a new task,
> newTaskAllocation + (sum over running tasks of (1.0 - done) * allocation) >=
> free disk * conf.getFloat("mapred.max.scratch.allocation", 0.90);
> So in English, we will model the expected sizes of tasks and only task tasks that should leave us a 10% margin. With:
> map.output.growth.factor -- the relative size of the transient data relative to the map inputs
> mapred.max.scratch.allocation -- the maximum amount of our disk we want to allocate to tasks.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-657) Free temporary space should be
modelled better
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-657?page=comments#action_12450471 ]
Owen O'Malley commented on HADOOP-657:
--------------------------------------
Actually, I'd rather have the unallocated disk space in the heartbeat, because when HADOOP-639 is implemented, the JobTracker should be given fresh information to decide which tasks should be launched there.
> Free temporary space should be modelled better
> ----------------------------------------------
>
> Key: HADOOP-657
> URL: http://issues.apache.org/jira/browse/HADOOP-657
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.7.2
> Reporter: Owen O'Malley
> Assigned To: Arun C Murthy
>
> Currently, there is a configurable size that must be free for a task tracker to accept a new task. However, that isn't a very good model of what the task is likely to take. I'd like to propose:
> Map tasks: totalInputSize * conf.getFloat("map.output.growth.factor", 1.0) / numMaps
> Reduce tasks: totalInputSize * 2 * conf.getFloat("map.output.growth.factor", 1.0) / numReduces
> where totalInputSize is the size of all the maps inputs for the given job.
> To start a new task,
> newTaskAllocation + (sum over running tasks of (1.0 - done) * allocation) >=
> free disk * conf.getFloat("mapred.max.scratch.allocation", 0.90);
> So in English, we will model the expected sizes of tasks and only task tasks that should leave us a 10% margin. With:
> map.output.growth.factor -- the relative size of the transient data relative to the map inputs
> mapred.max.scratch.allocation -- the maximum amount of our disk we want to allocate to tasks.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-657) Free temporary space should be
modelled better
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-657?page=comments#action_12448519 ]
Arun C Murthy commented on HADOOP-657:
--------------------------------------
Current flow relevant to this discussion:
TaskTracker.offerService() -> TaskTracker.checkForNewTasks() -> if (TaskTracker.enoughFreeSpace()) then poll/startNewTask
We could put the above checks (infact we can do better by checking if we have assigned fileSplit's size * conf.getFloat("map.output.growth.factor", 1.0)) in TaskTracker.enoughFreeSpace()...
... alternatively we could make '(sum over running tasks of (1.0 - done) * allocation)' part of TaskTrackerStatus i.e. a 'availableDiskSpace' member, check to ensure that 'sufficient' free space is available on the tasktracker before assigning it the task itself in JobInProgress.findNewTask - this ensures that a task isn't allocated in the first place to a tasktracker if it can't handle it.
What do you guys think? Am I missing out on something which prevents option #2 from working?
> Free temporary space should be modelled better
> ----------------------------------------------
>
> Key: HADOOP-657
> URL: http://issues.apache.org/jira/browse/HADOOP-657
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.7.2
> Reporter: Owen O'Malley
> Assigned To: Arun C Murthy
>
> Currently, there is a configurable size that must be free for a task tracker to accept a new task. However, that isn't a very good model of what the task is likely to take. I'd like to propose:
> Map tasks: totalInputSize * conf.getFloat("map.output.growth.factor", 1.0) / numMaps
> Reduce tasks: totalInputSize * 2 * conf.getFloat("map.output.growth.factor", 1.0) / numReduces
> where totalInputSize is the size of all the maps inputs for the given job.
> To start a new task,
> newTaskAllocation + (sum over running tasks of (1.0 - done) * allocation) >=
> free disk * conf.getFloat("mapred.max.scratch.allocation", 0.90);
> So in English, we will model the expected sizes of tasks and only task tasks that should leave us a 10% margin. With:
> map.output.growth.factor -- the relative size of the transient data relative to the map inputs
> mapred.max.scratch.allocation -- the maximum amount of our disk we want to allocate to tasks.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-657) Free temporary space should be
modelled better
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-657?page=comments#action_12450470 ]
Arun C Murthy commented on HADOOP-657:
--------------------------------------
I see the value in Doug's suggestion... for e.g. at some point in the future we might also put in metrics like CPU load, VM stats etc. and this would let the JobTracker make 'smarter' decisions about which task to assign to which TaskTrackers i.e. CPU-bound tasks to IO-laden TTs and vice-versa.
I do agree that it might be a very futuristic scenario, but the point is to keep the infrastructure robust when we can...
> Free temporary space should be modelled better
> ----------------------------------------------
>
> Key: HADOOP-657
> URL: http://issues.apache.org/jira/browse/HADOOP-657
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.7.2
> Reporter: Owen O'Malley
> Assigned To: Arun C Murthy
>
> Currently, there is a configurable size that must be free for a task tracker to accept a new task. However, that isn't a very good model of what the task is likely to take. I'd like to propose:
> Map tasks: totalInputSize * conf.getFloat("map.output.growth.factor", 1.0) / numMaps
> Reduce tasks: totalInputSize * 2 * conf.getFloat("map.output.growth.factor", 1.0) / numReduces
> where totalInputSize is the size of all the maps inputs for the given job.
> To start a new task,
> newTaskAllocation + (sum over running tasks of (1.0 - done) * allocation) >=
> free disk * conf.getFloat("mapred.max.scratch.allocation", 0.90);
> So in English, we will model the expected sizes of tasks and only task tasks that should leave us a 10% margin. With:
> map.output.growth.factor -- the relative size of the transient data relative to the map inputs
> mapred.max.scratch.allocation -- the maximum amount of our disk we want to allocate to tasks.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-657) Free temporary space should be
modelled better
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-657?page=comments#action_12450141 ]
Doug Cutting commented on HADOOP-657:
-------------------------------------
> include the 'unallocated disk space' in the global and per-host jsps
Yes, I agree this is another metric that should be displayed in the central web ui. It should be reported through the metrics API, as discussed in HADOOP-481 and HADOOP-481. The MapReduce framework should implement a MetricsContext, and system (and perhaps user) code can use this to route statisitics to central locations. I don't think we want to fix that as a part of this issue, but I also don't think we should hack in an alternate mechanism just for this statistic.
> Free temporary space should be modelled better
> ----------------------------------------------
>
> Key: HADOOP-657
> URL: http://issues.apache.org/jira/browse/HADOOP-657
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.7.2
> Reporter: Owen O'Malley
> Assigned To: Arun C Murthy
>
> Currently, there is a configurable size that must be free for a task tracker to accept a new task. However, that isn't a very good model of what the task is likely to take. I'd like to propose:
> Map tasks: totalInputSize * conf.getFloat("map.output.growth.factor", 1.0) / numMaps
> Reduce tasks: totalInputSize * 2 * conf.getFloat("map.output.growth.factor", 1.0) / numReduces
> where totalInputSize is the size of all the maps inputs for the given job.
> To start a new task,
> newTaskAllocation + (sum over running tasks of (1.0 - done) * allocation) >=
> free disk * conf.getFloat("mapred.max.scratch.allocation", 0.90);
> So in English, we will model the expected sizes of tasks and only task tasks that should leave us a 10% margin. With:
> map.output.growth.factor -- the relative size of the transient data relative to the map inputs
> mapred.max.scratch.allocation -- the maximum amount of our disk we want to allocate to tasks.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (HADOOP-657) Free temporary space should be
modelled better
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-657?page=all ]
Arun C Murthy reassigned HADOOP-657:
------------------------------------
Assignee: Arun C Murthy (was: Owen O'Malley)
> Free temporary space should be modelled better
> ----------------------------------------------
>
> Key: HADOOP-657
> URL: http://issues.apache.org/jira/browse/HADOOP-657
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.7.2
> Reporter: Owen O'Malley
> Assigned To: Arun C Murthy
>
> Currently, there is a configurable size that must be free for a task tracker to accept a new task. However, that isn't a very good model of what the task is likely to take. I'd like to propose:
> Map tasks: totalInputSize * conf.getFloat("map.output.growth.factor", 1.0) / numMaps
> Reduce tasks: totalInputSize * 2 * conf.getFloat("map.output.growth.factor", 1.0) / numReduces
> where totalInputSize is the size of all the maps inputs for the given job.
> To start a new task,
> newTaskAllocation + (sum over running tasks of (1.0 - done) * allocation) >=
> free disk * conf.getFloat("mapred.max.scratch.allocation", 0.90);
> So in English, we will model the expected sizes of tasks and only task tasks that should leave us a 10% margin. With:
> map.output.growth.factor -- the relative size of the transient data relative to the map inputs
> mapred.max.scratch.allocation -- the maximum amount of our disk we want to allocate to tasks.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-657) Free temporary space should be
modelled better
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-657?page=comments#action_12454061 ]
Arun C Murthy commented on HADOOP-657:
--------------------------------------
Taking things forward, looks like both Owen/Doug agree that we need to send metrics through the heartbeat...
... given this we are looking at implementing a MapReduceMetricsContext which sends over heartbeat (with metrics) via RPC to the JT and TaskTracker.offerService() becoming a callback for the timer. Does that make sense or do folks prefer something different?
> Free temporary space should be modelled better
> ----------------------------------------------
>
> Key: HADOOP-657
> URL: http://issues.apache.org/jira/browse/HADOOP-657
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.7.2
> Reporter: Owen O'Malley
> Assigned To: Arun C Murthy
>
> Currently, there is a configurable size that must be free for a task tracker to accept a new task. However, that isn't a very good model of what the task is likely to take. I'd like to propose:
> Map tasks: totalInputSize * conf.getFloat("map.output.growth.factor", 1.0) / numMaps
> Reduce tasks: totalInputSize * 2 * conf.getFloat("map.output.growth.factor", 1.0) / numReduces
> where totalInputSize is the size of all the maps inputs for the given job.
> To start a new task,
> newTaskAllocation + (sum over running tasks of (1.0 - done) * allocation) >=
> free disk * conf.getFloat("mapred.max.scratch.allocation", 0.90);
> So in English, we will model the expected sizes of tasks and only task tasks that should leave us a 10% margin. With:
> map.output.growth.factor -- the relative size of the transient data relative to the map inputs
> mapred.max.scratch.allocation -- the maximum amount of our disk we want to allocate to tasks.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-657) Free temporary space should be
modelled better
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-657?page=comments#action_12450138 ]
Arun C Murthy commented on HADOOP-657:
--------------------------------------
It should be very useful to include the 'unallocated disk space' in the global and per-host jsps so as to provide an easy way for operator to diagnose if and when tasks can't be allocated due to lack of diskspace... I think it should be a part of this same bug.
> Free temporary space should be modelled better
> ----------------------------------------------
>
> Key: HADOOP-657
> URL: http://issues.apache.org/jira/browse/HADOOP-657
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.7.2
> Reporter: Owen O'Malley
> Assigned To: Arun C Murthy
>
> Currently, there is a configurable size that must be free for a task tracker to accept a new task. However, that isn't a very good model of what the task is likely to take. I'd like to propose:
> Map tasks: totalInputSize * conf.getFloat("map.output.growth.factor", 1.0) / numMaps
> Reduce tasks: totalInputSize * 2 * conf.getFloat("map.output.growth.factor", 1.0) / numReduces
> where totalInputSize is the size of all the maps inputs for the given job.
> To start a new task,
> newTaskAllocation + (sum over running tasks of (1.0 - done) * allocation) >=
> free disk * conf.getFloat("mapred.max.scratch.allocation", 0.90);
> So in English, we will model the expected sizes of tasks and only task tasks that should leave us a 10% margin. With:
> map.output.growth.factor -- the relative size of the transient data relative to the map inputs
> mapred.max.scratch.allocation -- the maximum amount of our disk we want to allocate to tasks.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-657) Free temporary space should be
modelled better
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-657?page=comments#action_12450472 ]
Doug Cutting commented on HADOOP-657:
-------------------------------------
> I'd rather have the unallocated disk space in the heartbeat [ ...]
I agree, but I think we should use a general mechanism to route metrics to the jobtracker through heartbeats, rather than hack things in one-by-one.
> Free temporary space should be modelled better
> ----------------------------------------------
>
> Key: HADOOP-657
> URL: http://issues.apache.org/jira/browse/HADOOP-657
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.7.2
> Reporter: Owen O'Malley
> Assigned To: Arun C Murthy
>
> Currently, there is a configurable size that must be free for a task tracker to accept a new task. However, that isn't a very good model of what the task is likely to take. I'd like to propose:
> Map tasks: totalInputSize * conf.getFloat("map.output.growth.factor", 1.0) / numMaps
> Reduce tasks: totalInputSize * 2 * conf.getFloat("map.output.growth.factor", 1.0) / numReduces
> where totalInputSize is the size of all the maps inputs for the given job.
> To start a new task,
> newTaskAllocation + (sum over running tasks of (1.0 - done) * allocation) >=
> free disk * conf.getFloat("mapred.max.scratch.allocation", 0.90);
> So in English, we will model the expected sizes of tasks and only task tasks that should leave us a 10% margin. With:
> map.output.growth.factor -- the relative size of the transient data relative to the map inputs
> mapred.max.scratch.allocation -- the maximum amount of our disk we want to allocate to tasks.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-657) Free temporary space should be
modelled better
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-657?page=comments#action_12448251 ]
Arun C Murthy commented on HADOOP-657:
--------------------------------------
Looks like Owen's got it nailed pretty good... anyone wants to shout on this one?
What do you guys think is a reasonable default for "map.output.growth.factor" ? 1.0? Do we need more leeway?
> Free temporary space should be modelled better
> ----------------------------------------------
>
> Key: HADOOP-657
> URL: http://issues.apache.org/jira/browse/HADOOP-657
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.7.2
> Reporter: Owen O'Malley
> Assigned To: Arun C Murthy
>
> Currently, there is a configurable size that must be free for a task tracker to accept a new task. However, that isn't a very good model of what the task is likely to take. I'd like to propose:
> Map tasks: totalInputSize * conf.getFloat("map.output.growth.factor", 1.0) / numMaps
> Reduce tasks: totalInputSize * 2 * conf.getFloat("map.output.growth.factor", 1.0) / numReduces
> where totalInputSize is the size of all the maps inputs for the given job.
> To start a new task,
> newTaskAllocation + (sum over running tasks of (1.0 - done) * allocation) >=
> free disk * conf.getFloat("mapred.max.scratch.allocation", 0.90);
> So in English, we will model the expected sizes of tasks and only task tasks that should leave us a 10% margin. With:
> map.output.growth.factor -- the relative size of the transient data relative to the map inputs
> mapred.max.scratch.allocation -- the maximum amount of our disk we want to allocate to tasks.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-657) Free temporary space should be
modelled better
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-657?page=comments#action_12450130 ]
Doug Cutting commented on HADOOP-657:
-------------------------------------
We should also log something when tasks are not accepted due to space limitations. At present I think nothing is displayed in this case.
> Free temporary space should be modelled better
> ----------------------------------------------
>
> Key: HADOOP-657
> URL: http://issues.apache.org/jira/browse/HADOOP-657
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.7.2
> Reporter: Owen O'Malley
> Assigned To: Arun C Murthy
>
> Currently, there is a configurable size that must be free for a task tracker to accept a new task. However, that isn't a very good model of what the task is likely to take. I'd like to propose:
> Map tasks: totalInputSize * conf.getFloat("map.output.growth.factor", 1.0) / numMaps
> Reduce tasks: totalInputSize * 2 * conf.getFloat("map.output.growth.factor", 1.0) / numReduces
> where totalInputSize is the size of all the maps inputs for the given job.
> To start a new task,
> newTaskAllocation + (sum over running tasks of (1.0 - done) * allocation) >=
> free disk * conf.getFloat("mapred.max.scratch.allocation", 0.90);
> So in English, we will model the expected sizes of tasks and only task tasks that should leave us a 10% margin. With:
> map.output.growth.factor -- the relative size of the transient data relative to the map inputs
> mapred.max.scratch.allocation -- the maximum amount of our disk we want to allocate to tasks.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira