You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Adam Kramer (JIRA)" <ji...@apache.org> on 2010/02/23 07:37:28 UTC
[jira] Created: (HADOOP-6592) Scheduler: Pause button desirable
Scheduler: Pause button desirable
---------------------------------
Key: HADOOP-6592
URL: https://issues.apache.org/jira/browse/HADOOP-6592
Project: Hadoop Common
Issue Type: Wish
Reporter: Adam Kramer
It would be lovely if, from the jobtracker page, I could click a button that's not "kill" or "fail" but ..."pause."
The pause button would stop a certain task from starting any more mappers or reducers. They would all wait in the "pending" stage until the job is "un-paused." Currently-running tasks would continue to run, and then complete, thus freeing the resources for other jobs.
This would help a lot for systems (esp. Hive) in which one or two jobs are hogging a lot of mappers or reducers. The ones they have would finish, and then other jobs could "catch up," and then they could be unpaused for a while. This would also allow for user-level throttling of their jobs in instances where they need a lot of resources but have the time to spare.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6592) Scheduler: Pause button desirable
Posted by "Adam Kramer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837107#action_12837107 ]
Adam Kramer commented on HADOOP-6592:
-------------------------------------
Nearly. If there's a bunch of "very low" priority tasks, they will still vie for resources. Similarly, if I want to pause my task because I _expect_ other tasks to arrive (i.e., there are none queuing), very-low-pri doesn't quite fit the bill.
> Scheduler: Pause button desirable
> ---------------------------------
>
> Key: HADOOP-6592
> URL: https://issues.apache.org/jira/browse/HADOOP-6592
> Project: Hadoop Common
> Issue Type: Wish
> Reporter: Adam Kramer
>
> It would be lovely if, from the jobtracker page, I could click a button that's not "kill" or "fail" but ..."pause."
> The pause button would stop a certain task from starting any more mappers or reducers. They would all wait in the "pending" stage until the job is "un-paused." Currently-running tasks would continue to run, and then complete, thus freeing the resources for other jobs.
> This would help a lot for systems (esp. Hive) in which one or two jobs are hogging a lot of mappers or reducers. The ones they have would finish, and then other jobs could "catch up," and then they could be unpaused for a while. This would also allow for user-level throttling of their jobs in instances where they need a lot of resources but have the time to spare.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6592) Scheduler: Pause button desirable
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837379#action_12837379 ]
Matei Zaharia commented on HADOOP-6592:
---------------------------------------
Hi Adam,
The fair scheduler and capacity scheduler can be used to better share a cluster between large and small jobs. Instead of running jobs in FIFO order as the default scheduler does, they allow each job to have a certain share of the slots in the cluster. For example, if your cluster has 1000 slots, and you submit 1 job, it gets all the slots; but if you then submit a second job, each job's share becomes 500 slots (as tasks from job 1 finish, their slots are given to job 2). These schedulers work quite well when tasks are relatively short (10 seconds to a minute); a new job can get slots within a few seconds. The only case when you may need to do something beyond waiting for existing tasks to finish is when all your reduce slots are filled by long reduce slots, but for this case, the fair scheduler at least supports preemption in 0.21 and trunk (I believe the capacity scheduler had it in earlier versions but has now removed it; I could be wrong about that though).
> Scheduler: Pause button desirable
> ---------------------------------
>
> Key: HADOOP-6592
> URL: https://issues.apache.org/jira/browse/HADOOP-6592
> Project: Hadoop Common
> Issue Type: Wish
> Reporter: Adam Kramer
> Priority: Minor
>
> It would be lovely if, from the jobtracker page, I could click a button that's not "kill" or "fail" but ..."pause."
> The pause button would stop a certain task from starting any more mappers or reducers. They would all wait in the "pending" stage until the job is "un-paused." Currently-running tasks would continue to run, and then complete, thus freeing the resources for other jobs.
> This would help a lot for systems (esp. Hive) in which one or two jobs are hogging a lot of mappers or reducers. The ones they have would finish, and then other jobs could "catch up," and then they could be unpaused for a while. This would also allow for user-level throttling of their jobs in instances where they need a lot of resources but have the time to spare.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6592) Scheduler: Pause button desirable
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837119#action_12837119 ]
Arun C Murthy commented on HADOOP-6592:
---------------------------------------
bq. Map-Reduce assumes that all dependencies are satisfied at time of job-submission...
OTOH, a workflow manager like Oozie really sounds like what you want.
> Scheduler: Pause button desirable
> ---------------------------------
>
> Key: HADOOP-6592
> URL: https://issues.apache.org/jira/browse/HADOOP-6592
> Project: Hadoop Common
> Issue Type: Wish
> Reporter: Adam Kramer
> Priority: Minor
>
> It would be lovely if, from the jobtracker page, I could click a button that's not "kill" or "fail" but ..."pause."
> The pause button would stop a certain task from starting any more mappers or reducers. They would all wait in the "pending" stage until the job is "un-paused." Currently-running tasks would continue to run, and then complete, thus freeing the resources for other jobs.
> This would help a lot for systems (esp. Hive) in which one or two jobs are hogging a lot of mappers or reducers. The ones they have would finish, and then other jobs could "catch up," and then they could be unpaused for a while. This would also allow for user-level throttling of their jobs in instances where they need a lot of resources but have the time to spare.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6592) Scheduler: Pause button desirable
Posted by "Adam Kramer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837174#action_12837174 ]
Adam Kramer commented on HADOOP-6592:
-------------------------------------
bq. This meter would still be running even if the job is paused, so tasks or the entire job could be killed if its temp storage usage quota is reached.
In general, this seems like a good practice--when temporary storage is filled, whatever task is using said storage should be killed in FIFO order. This means that paused jobs' tasks could fail, at which point that map task would be returned to the "pending" queue.
bq. You need pre-emption, not pause.
Pre-emption doesn't address a potential desire to pause a task that's currently running when there are no other tasks waiting. Say my map task requires 3000 mappers and my cluster has 1000. Then, when 1000 start running, I may pause in case other users submit jobs, in order to have at least some mappers free when they start their job. That is what I meant by "because I expect other tasks to arrive"--I meant other JOBS, submitted by other users, to whom I would like to be polite.
Or if another user is being impolite, it would be good to "pause" that user's job until I can walk by his desk and ask him if there's a way that doesn't thrash the cluster...if it turns out there isn't, he could just keep his job paused until the workday is over (and thus the cluster is mostly free).
A good stopgap in the meantime, that gets most of the desirable qualities of pause, would be the ability to set a ceiling on the number of map tasks my job could simultaneously use.
> Scheduler: Pause button desirable
> ---------------------------------
>
> Key: HADOOP-6592
> URL: https://issues.apache.org/jira/browse/HADOOP-6592
> Project: Hadoop Common
> Issue Type: Wish
> Reporter: Adam Kramer
> Priority: Minor
>
> It would be lovely if, from the jobtracker page, I could click a button that's not "kill" or "fail" but ..."pause."
> The pause button would stop a certain task from starting any more mappers or reducers. They would all wait in the "pending" stage until the job is "un-paused." Currently-running tasks would continue to run, and then complete, thus freeing the resources for other jobs.
> This would help a lot for systems (esp. Hive) in which one or two jobs are hogging a lot of mappers or reducers. The ones they have would finish, and then other jobs could "catch up," and then they could be unpaused for a while. This would also allow for user-level throttling of their jobs in instances where they need a lot of resources but have the time to spare.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6592) Scheduler: Pause button desirable
Posted by "Adam Kramer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837109#action_12837109 ]
Adam Kramer commented on HADOOP-6592:
-------------------------------------
Similarly, if paused during the map phase, any reducers already allocated (which couldn't, in theory, begin working until all mappers complete) would be killed and freed. (so yes, the copy would need to restart, but the reducers wouldn't be held hostage by huge map tasks).
> Scheduler: Pause button desirable
> ---------------------------------
>
> Key: HADOOP-6592
> URL: https://issues.apache.org/jira/browse/HADOOP-6592
> Project: Hadoop Common
> Issue Type: Wish
> Reporter: Adam Kramer
> Priority: Minor
>
> It would be lovely if, from the jobtracker page, I could click a button that's not "kill" or "fail" but ..."pause."
> The pause button would stop a certain task from starting any more mappers or reducers. They would all wait in the "pending" stage until the job is "un-paused." Currently-running tasks would continue to run, and then complete, thus freeing the resources for other jobs.
> This would help a lot for systems (esp. Hive) in which one or two jobs are hogging a lot of mappers or reducers. The ones they have would finish, and then other jobs could "catch up," and then they could be unpaused for a while. This would also allow for user-level throttling of their jobs in instances where they need a lot of resources but have the time to spare.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6592) Scheduler: Pause button desirable
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837102#action_12837102 ]
Todd Lipcon commented on HADOOP-6592:
-------------------------------------
Can't you achieve this same thing by just changing its priority to very low?
A JT-wide pause function is useful for another reason, which is to survive a total DFS restart, but I think there's already a JIRA for that.
> Scheduler: Pause button desirable
> ---------------------------------
>
> Key: HADOOP-6592
> URL: https://issues.apache.org/jira/browse/HADOOP-6592
> Project: Hadoop Common
> Issue Type: Wish
> Reporter: Adam Kramer
>
> It would be lovely if, from the jobtracker page, I could click a button that's not "kill" or "fail" but ..."pause."
> The pause button would stop a certain task from starting any more mappers or reducers. They would all wait in the "pending" stage until the job is "un-paused." Currently-running tasks would continue to run, and then complete, thus freeing the resources for other jobs.
> This would help a lot for systems (esp. Hive) in which one or two jobs are hogging a lot of mappers or reducers. The ones they have would finish, and then other jobs could "catch up," and then they could be unpaused for a while. This would also allow for user-level throttling of their jobs in instances where they need a lot of resources but have the time to spare.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6592) Scheduler: Pause button desirable
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837135#action_12837135 ]
Amar Kamat commented on HADOOP-6592:
------------------------------------
Isnt this a duplicate of MAPREDUCE-1227?
> Scheduler: Pause button desirable
> ---------------------------------
>
> Key: HADOOP-6592
> URL: https://issues.apache.org/jira/browse/HADOOP-6592
> Project: Hadoop Common
> Issue Type: Wish
> Reporter: Adam Kramer
> Priority: Minor
>
> It would be lovely if, from the jobtracker page, I could click a button that's not "kill" or "fail" but ..."pause."
> The pause button would stop a certain task from starting any more mappers or reducers. They would all wait in the "pending" stage until the job is "un-paused." Currently-running tasks would continue to run, and then complete, thus freeing the resources for other jobs.
> This would help a lot for systems (esp. Hive) in which one or two jobs are hogging a lot of mappers or reducers. The ones they have would finish, and then other jobs could "catch up," and then they could be unpaused for a while. This would also allow for user-level throttling of their jobs in instances where they need a lot of resources but have the time to spare.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6592) Scheduler: Pause button desirable
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837117#action_12837117 ]
Arun C Murthy commented on HADOOP-6592:
---------------------------------------
This keeps coming up occasionally...
Unfortunately people do not realize that this has significant (negative) consequences to the cluster; in particular, map-outputs consume valuable temporary storage and make this feature un-viable for Map-Reduce.
bq. If there's a bunch of "very low" priority tasks, they will still vie for resources. Similarly, if I want to pause my task because I expect other tasks to arrive (i.e., there are none queuing), very-low-pri doesn't quite fit the bill.
Map-Reduce assumes that all dependencies are satisfied at time of job-submission...
bq. Similarly, if paused during the map phase, any reducers already allocated (which couldn't, in theory, begin working until all mappers complete) would be killed and freed. (so yes, the copy would need to restart, but the reducers wouldn't be held hostage by huge map tasks).
You need pre-emption, not pause.
> Scheduler: Pause button desirable
> ---------------------------------
>
> Key: HADOOP-6592
> URL: https://issues.apache.org/jira/browse/HADOOP-6592
> Project: Hadoop Common
> Issue Type: Wish
> Reporter: Adam Kramer
> Priority: Minor
>
> It would be lovely if, from the jobtracker page, I could click a button that's not "kill" or "fail" but ..."pause."
> The pause button would stop a certain task from starting any more mappers or reducers. They would all wait in the "pending" stage until the job is "un-paused." Currently-running tasks would continue to run, and then complete, thus freeing the resources for other jobs.
> This would help a lot for systems (esp. Hive) in which one or two jobs are hogging a lot of mappers or reducers. The ones they have would finish, and then other jobs could "catch up," and then they could be unpaused for a while. This would also allow for user-level throttling of their jobs in instances where they need a lot of resources but have the time to spare.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6592) Scheduler: Pause button desirable
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837163#action_12837163 ]
Hong Tang commented on HADOOP-6592:
-----------------------------------
bq. Isnt this a duplicate of MAPREDUCE-1227?
No, like Todd mentioned, MAPREDUCE-1227 is for cluster-wide pause. This one is to pause individual jobs.
There are cases where pausing individual jobs could be desirable. For instance, I am running a very large job, toward the end I found out some tasks start to fail due to disk quota limit reached. I'd like to pause the job, free up some quota space, and resume the job. Pausing a job may also be useful for debugging.
bq. Unfortunately people do not realize that this has significant (negative) consequences to the cluster; in particular, map-outputs consume valuable temporary storage and make this feature un-viable for Map-Reduce.
Agreed. However, the current implementation does not enforce an absolute time limit of tasks, so the danger of a job holding up temporary storage is still there if some of its tasks are very slow. Perhaps we should meter a job's temporary storage usage as the product of data volume and the duration (unit: "MBxMin"), and set a limit on that. This meter would still be running even if the job is paused, so tasks or the entire job could be killed if its temp storage usage quota is reached.
> Scheduler: Pause button desirable
> ---------------------------------
>
> Key: HADOOP-6592
> URL: https://issues.apache.org/jira/browse/HADOOP-6592
> Project: Hadoop Common
> Issue Type: Wish
> Reporter: Adam Kramer
> Priority: Minor
>
> It would be lovely if, from the jobtracker page, I could click a button that's not "kill" or "fail" but ..."pause."
> The pause button would stop a certain task from starting any more mappers or reducers. They would all wait in the "pending" stage until the job is "un-paused." Currently-running tasks would continue to run, and then complete, thus freeing the resources for other jobs.
> This would help a lot for systems (esp. Hive) in which one or two jobs are hogging a lot of mappers or reducers. The ones they have would finish, and then other jobs could "catch up," and then they could be unpaused for a while. This would also allow for user-level throttling of their jobs in instances where they need a lot of resources but have the time to spare.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6592) Scheduler: Pause button desirable
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837413#action_12837413 ]
Arun C Murthy commented on HADOOP-6592:
---------------------------------------
I speak from experience, at various times I baby-sit several clusters with several thousand machines and tens of thousands of jobs as my day-job... doing 'social scheduling' as you suggest very quickly breaks down - not just at the very large scale I'm used to. Once you go past tens of nodes and tens of users you cannot go find them at their desk.
As Matei suggests, please take a look at either fair-scheduler or the capacity-scheduler. They both are designed to work in a multi-tenant environment, and are reasonably adept at it... and they will need to improve.
> Scheduler: Pause button desirable
> ---------------------------------
>
> Key: HADOOP-6592
> URL: https://issues.apache.org/jira/browse/HADOOP-6592
> Project: Hadoop Common
> Issue Type: Wish
> Reporter: Adam Kramer
> Priority: Minor
>
> It would be lovely if, from the jobtracker page, I could click a button that's not "kill" or "fail" but ..."pause."
> The pause button would stop a certain task from starting any more mappers or reducers. They would all wait in the "pending" stage until the job is "un-paused." Currently-running tasks would continue to run, and then complete, thus freeing the resources for other jobs.
> This would help a lot for systems (esp. Hive) in which one or two jobs are hogging a lot of mappers or reducers. The ones they have would finish, and then other jobs could "catch up," and then they could be unpaused for a while. This would also allow for user-level throttling of their jobs in instances where they need a lot of resources but have the time to spare.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6592) Scheduler: Pause button desirable
Posted by "Adam Kramer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adam Kramer updated HADOOP-6592:
--------------------------------
Priority: Minor (was: Major)
> Scheduler: Pause button desirable
> ---------------------------------
>
> Key: HADOOP-6592
> URL: https://issues.apache.org/jira/browse/HADOOP-6592
> Project: Hadoop Common
> Issue Type: Wish
> Reporter: Adam Kramer
> Priority: Minor
>
> It would be lovely if, from the jobtracker page, I could click a button that's not "kill" or "fail" but ..."pause."
> The pause button would stop a certain task from starting any more mappers or reducers. They would all wait in the "pending" stage until the job is "un-paused." Currently-running tasks would continue to run, and then complete, thus freeing the resources for other jobs.
> This would help a lot for systems (esp. Hive) in which one or two jobs are hogging a lot of mappers or reducers. The ones they have would finish, and then other jobs could "catch up," and then they could be unpaused for a while. This would also allow for user-level throttling of their jobs in instances where they need a lot of resources but have the time to spare.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.