You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Ben Clay <rb...@ncsu.edu> on 2011/09/29 14:52:57 UTC
modify a specific TaskTracker's Map slots on the fly - FairScheduler?
I need to modify the number of Map slots on a TaskTracker dynamically over
the course of a job without restarting the TaskTracker process. If the
number of allowed slots is lowered, the current Map tasks should be allowed
to finish, while new tasks should be prevented until the count falls below
the limit.
Conceptually, a custom scheduler should work for this, and I have a modified
FairScheduler working which allows me to "turn off" TaskTrackers,
disallowing new Map task assignments. To do so, the target TaskTracker's
hostname is placed in a refreshable config file, and canAssignMap() always
returns false for that hostname.
The problem is that when I want to raise / lower the Map slots to some value
in between 0 and mapred.tasktracker.map.tasks.maximum, I need to know the
currently-held number of Map tasks. However, I can't get accurate
information about the current number of slots occupied. The following
function calls all appear to return "stale" info:
TaskTrackerStatus.countOccupiedMapSlots()
TaskTrackerStatus.getAvailableMapSlots()
TaskTrackerStatus.countMapTasks()
I've concluded these are stale because I can see multiple quick-succession
calls to canAssignMap() yield the same value for these counter functions,
even though new tasks have been assigned. I thought about keeping track of
the number of assigned tasks within canAssignMap() itself, but there is
unfortunately no way to tell when tasks have been completed, making this
moot.
Is there another approach that would work in this situation? It doesn't
have to be via the FairScheduler. OR, is there a way to speed up the
frequency of TaskTracker reports, so that my scheduler has semi-accurate
slot info?
Thanks!
-Ben
Re: modify a specific TaskTracker's Map slots on the fly -
FairScheduler?
Posted by Robert Evans <ev...@yahoo-inc.com>.
Ben,
I am not completely sure how it all works, but mapred.heartbeats.in.second is the config. It controls how many heartbeats the JT will be able to process in a second from all nodes. I think the 3 sec default is actually for the NN/DN not the JT. So the number of hearbeats per second a node sends to the JT is mapred.heartbeats.in.second/# Nodes.
--Bobby Evans
On 9/29/11 11:16 AM, "Ben Clay" <rb...@ncsu.edu> wrote:
Bobby-
Is there a config option for reducing the heartbeat frequency? I think I may have a potential solution, but decreasing the heartbeat would reduce the upper bound on task launch delays.
-Ben
From: Robert Evans [mailto:evans@yahoo-inc.com]
Sent: Thursday, September 29, 2011 11:30 AM
To: mapreduce-user@hadoop.apache.org
Subject: Re: modify a specific TaskTracker's Map slots on the fly - FairScheduler?
You could increase the heartbeat frequency, although it is set to 3 sec by default, but may be more for larger clusters so your data is likely to only be about 3 sec out of date.
--Bobby Evans
On 9/29/11 7:52 AM, "Ben Clay" <rb...@ncsu.edu> wrote:
I need to modify the number of Map slots on a TaskTracker dynamically over the course of a job without restarting the TaskTracker process. If the number of allowed slots is lowered, the current Map tasks should be allowed to finish, while new tasks should be prevented until the count falls below the limit.
Conceptually, a custom scheduler should work for this, and I have a modified FairScheduler working which allows me to "turn off" TaskTrackers, disallowing new Map task assignments. To do so, the target TaskTracker's hostname is placed in a refreshable config file, and canAssignMap() always returns false for that hostname.
The problem is that when I want to raise / lower the Map slots to some value in between 0 and mapred.tasktracker.map.tasks.maximum, I need to know the currently-held number of Map tasks. However, I can't get accurate information about the current number of slots occupied. The following function calls all appear to return "stale" info:
TaskTrackerStatus.countOccupiedMapSlots()
TaskTrackerStatus.getAvailableMapSlots()
TaskTrackerStatus.countMapTasks()
I've concluded these are stale because I can see multiple quick-succession calls to canAssignMap() yield the same value for these counter functions, even though new tasks have been assigned. I thought about keeping track of the number of assigned tasks within canAssignMap() itself, but there is unfortunately no way to tell when tasks have been completed, making this moot.
Is there another approach that would work in this situation? It doesn't have to be via the FairScheduler. OR, is there a way to speed up the frequency of TaskTracker reports, so that my scheduler has semi-accurate slot info?
Thanks!
-Ben
RE: modify a specific TaskTracker's Map slots on the fly - FairScheduler?
Posted by Ben Clay <rb...@ncsu.edu>.
Bobby-
Is there a config option for reducing the heartbeat frequency? I think I
may have a potential solution, but decreasing the heartbeat would reduce the
upper bound on task launch delays.
-Ben
From: Robert Evans [mailto:evans@yahoo-inc.com]
Sent: Thursday, September 29, 2011 11:30 AM
To: mapreduce-user@hadoop.apache.org
Subject: Re: modify a specific TaskTracker's Map slots on the fly -
FairScheduler?
You could increase the heartbeat frequency, although it is set to 3 sec by
default, but may be more for larger clusters so your data is likely to only
be about 3 sec out of date.
--Bobby Evans
On 9/29/11 7:52 AM, "Ben Clay" <rb...@ncsu.edu> wrote:
I need to modify the number of Map slots on a TaskTracker dynamically over
the course of a job without restarting the TaskTracker process. If the
number of allowed slots is lowered, the current Map tasks should be allowed
to finish, while new tasks should be prevented until the count falls below
the limit.
Conceptually, a custom scheduler should work for this, and I have a modified
FairScheduler working which allows me to "turn off" TaskTrackers,
disallowing new Map task assignments. To do so, the target TaskTracker's
hostname is placed in a refreshable config file, and canAssignMap() always
returns false for that hostname.
The problem is that when I want to raise / lower the Map slots to some value
in between 0 and mapred.tasktracker.map.tasks.maximum, I need to know the
currently-held number of Map tasks. However, I can't get accurate
information about the current number of slots occupied. The following
function calls all appear to return "stale" info:
TaskTrackerStatus.countOccupiedMapSlots()
TaskTrackerStatus.getAvailableMapSlots()
TaskTrackerStatus.countMapTasks()
I've concluded these are stale because I can see multiple quick-succession
calls to canAssignMap() yield the same value for these counter functions,
even though new tasks have been assigned. I thought about keeping track of
the number of assigned tasks within canAssignMap() itself, but there is
unfortunately no way to tell when tasks have been completed, making this
moot.
Is there another approach that would work in this situation? It doesn't
have to be via the FairScheduler. OR, is there a way to speed up the
frequency of TaskTracker reports, so that my scheduler has semi-accurate
slot info?
Thanks!
-Ben
Re: modify a specific TaskTracker's Map slots on the fly -
FairScheduler?
Posted by Robert Evans <ev...@yahoo-inc.com>.
You could increase the heartbeat frequency, although it is set to 3 sec by default, but may be more for larger clusters so your data is likely to only be about 3 sec out of date.
--Bobby Evans
On 9/29/11 7:52 AM, "Ben Clay" <rb...@ncsu.edu> wrote:
I need to modify the number of Map slots on a TaskTracker dynamically over the course of a job without restarting the TaskTracker process. If the number of allowed slots is lowered, the current Map tasks should be allowed to finish, while new tasks should be prevented until the count falls below the limit.
Conceptually, a custom scheduler should work for this, and I have a modified FairScheduler working which allows me to "turn off" TaskTrackers, disallowing new Map task assignments. To do so, the target TaskTracker's hostname is placed in a refreshable config file, and canAssignMap() always returns false for that hostname.
The problem is that when I want to raise / lower the Map slots to some value in between 0 and mapred.tasktracker.map.tasks.maximum, I need to know the currently-held number of Map tasks. However, I can't get accurate information about the current number of slots occupied. The following function calls all appear to return "stale" info:
TaskTrackerStatus.countOccupiedMapSlots()
TaskTrackerStatus.getAvailableMapSlots()
TaskTrackerStatus.countMapTasks()
I've concluded these are stale because I can see multiple quick-succession calls to canAssignMap() yield the same value for these counter functions, even though new tasks have been assigned. I thought about keeping track of the number of assigned tasks within canAssignMap() itself, but there is unfortunately no way to tell when tasks have been completed, making this moot.
Is there another approach that would work in this situation? It doesn't have to be via the FairScheduler. OR, is there a way to speed up the frequency of TaskTracker reports, so that my scheduler has semi-accurate slot info?
Thanks!
-Ben