You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Karam Singh (JIRA)" <ji...@apache.org> on 2008/10/16 15:46:46 UTC
[jira] Created: (HADOOP-4428) Job Priorities are not handled
properly
Job Priorities are not handled properly
----------------------------------------
Key: HADOOP-4428
URL: https://issues.apache.org/jira/browse/HADOOP-4428
Project: Hadoop Core
Issue Type: Bug
Components: contrib/capacity-sched
Affects Versions: 0.19.0
Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
Single Queue=default, User Limit=25, Priorities = Yes.
Using hadoop branch 0.19 revision=705159
Reporter: Karam Singh
Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-4428:
-------------------------------------
Resolution: Duplicate
Status: Resolved (was: Patch Available)
As per discussions that have happened on this issue, there are two separate issues identified:
- Tasks get initialized without a limit on the count initialized at a given point.
- Jobs are run out of priority order.
HADOOP-4513 moves task initialization in the capacity scheduler to a separate thread. As part of that issue, we are working on limits to control the count.
HADOOP-4471 sorts jobs in the running queue as well. Hence, jobs are now sorted in priority order everywhere.
In other words, the two identified issues are resolved if both the above JIRAs are fixed. Hence, I am closing this issue as a duplicate.
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Sreekanth Ramakrishnan
> Priority: Blocker
> Fix For: 0.19.1
>
> Attachments: HADOOP-4428-20081017.1.txt, HADOOP-4428-20081020.txt, HADOOP-4428.patch
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V updated HADOOP-4428:
------------------------------
Status: Patch Available (was: Open)
Running it through Hudson.
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4428-20081017.1.txt
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640177#action_12640177 ]
Vinod K V commented on HADOOP-4428:
-----------------------------------
I think this happens because of the set-up tasks introduced in HADOOP-4261.
Earlier, jobs were initialized and moved into the running-job's queue in the order of their presence in waiting-jobs' queue which is by priority (and more).
Now, even though jobs are initialized according to their order in waiting-jobs' queue, depending on the time taken for successful return and reporting of set-up tasks. jobs may be moved out of order into the running-jobs' queue(a linked list). Also, JobTracker launches set-up tasks of initialized jobs in the order of its own jobs TreeMap data structure which stores jobs in the order of JobIDs. I think the later combined with the former explains the behavior explained by Karam.
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640547#action_12640547 ]
Vinod K V commented on HADOOP-4428:
-----------------------------------
My analysis of the cause is correct, but the proposed fix doesn't work as I've learnt that running jobs shouldn't be preempted no matter what, meaning that running jobs take higher precedence over waiting jobs irrespective of job priorities.
The correct fix, as I've discussed with Hemanth, is that when running jobs don't have anything to run, the first job in the waiting queue should be initialized, but nothing can be returned to the TT as we have to wait for job set-up. TTs that come later should also be returned no task till the job that is initialized moves to running queue after set-up is done. This way we make sure that jobs in waiting queue are served in the order of their priority.
I already have a patch for this, working on test-cases. Will put it up on the JIRA asap.
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.19.0
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala reassigned HADOOP-4428:
----------------------------------------
Assignee: Vinod K V
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-4428:
-------------------------------------
Attachment: HADOOP-4428.patch
Attaching a patch that addresses the problems raised with the previous one.
The changes are:
- We check the job state to be running before assigning a task.
- We don't look at jobs in any other queue, if the job currently initialized is not yet running.
- Modified the test case to make sure that in multiple queues scenario as well, no tasks are returned when a job in the waiting queue is initialized.
Testing in progress.
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4428-20081017.1.txt, HADOOP-4428-20081020.txt, HADOOP-4428.patch
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Vivek Ratan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641274#action_12641274 ]
Vivek Ratan commented on HADOOP-4428:
-------------------------------------
bq. We don't look at jobs in any other queue, if the job currently initialized is not yet running.
I wanted to consider this in some detail. When initTasks() returns, the job is still not in a running state, because its setup task needs to run first. For discussion sake, assume we have queues Q1, Q2 ... Qn and that we're considering queues in that order, starting with Q1. So after we call initTasks() on a job in Q1 (say, J1), we have the following options, in order to find a task to run:
1. We can look at the next job in Q1 This is not a good option since we'll face the same situation - we'll call initTasks() for the next job, and then look at the next job, and so on.
1. We can look at jobs in the next queue. This is a viable option. It does seem a bit unfair, because you're penalizing Q1 for the duration of the time it takes for J1's setup task to run, but you could equally well argue that this unfairness is temporary and is equally applicable to all queues.
1. We can return nothing to the the TT. As a result, all TTs that send heartbeats to the JT during the time that J1's setup task is running, will get nothing to run. Most setup tasks should take a couple of heartbeats to run, so this won't be a frequent problem, but if the setup task contains user code that does a bunch of stuff, the problem is exacerbated.
Upon further reflection, I'd argue for the second approach where we move on to the next queue. Returning nothing to the TTs causes unnecessary under-utilization.
The right way to do things, IMO, is get the setup/cleanup tasks out of initTasks(), which I'll argue elsewhere, but this problem (of initTasks() not necessarily changing the job's state to RUNNING) can rise up again if we decide to call initTasks() in a separate thread, the way it's done in the default scheduler.
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4428-20081017.1.txt, HADOOP-4428-20081020.txt, HADOOP-4428.patch
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640953#action_12640953 ]
Amar Kamat commented on HADOOP-4428:
------------------------------------
bq. 2) testJobRunStateChange() uses the old submitJob() api which I dont see fixed.
I somehow missed the change. Plz ignore this comment.
+1 for the patch.
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4428-20081017.1.txt, HADOOP-4428-20081020.txt
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V updated HADOOP-4428:
------------------------------
Status: Open (was: Patch Available)
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4428-20081017.1.txt, HADOOP-4428-20081020.txt
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641033#action_12641033 ]
Hemanth Yamijala commented on HADOOP-4428:
------------------------------------------
In an offline discussion with Vivek, we found that this patch is not correct, as it moves to the job in the next queue after initializing a job in the waiting queue. In fact, there are two deeper issues:
- Currently the scheduler, which calls {{initTasks}} can trigger a set up task to be run. This does not seem correct, because just like clean up tasks, setup tasks should be done outside the control of the scheduler. This should be addressed in the framework, possibly as part of HADOOP-4421 ?
- The capacity scheduler does not check for the status of a job to be running before assigning tasks from it when it initializes jobs in the waiting queue. The implicit assumption here was that after a call to {{initTasks}}, the job would be runnable. However, changes to the framework to run setup tasks broke this. Even if setup tasks are done independently of the scheduler, it is still possible that the scheduler is looking at a job whose setup task is still not complete, and hence by definition is not runnable.
The first point is more of a design issue, and hence not a blocker for 0.19, as currently nothing breaks because of the approach.
The second point must be addressed in the capacity scheduler, which should actually fix the problem raised in the bug report as well.
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4428-20081017.1.txt, HADOOP-4428-20081020.txt
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-4428:
--------------------------------
Fix Version/s: (was: 0.19.0)
0.19.1
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.19.1
>
> Attachments: HADOOP-4428-20081017.1.txt, HADOOP-4428-20081020.txt, HADOOP-4428.patch
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640751#action_12640751 ]
Hadoop QA commented on HADOOP-4428:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12392363/HADOOP-4428-20081017.1.txt
against trunk revision 705831.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3491/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3491/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3491/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3491/console
This message is automatically generated.
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4428-20081017.1.txt
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640942#action_12640942 ]
Hemanth Yamijala commented on HADOOP-4428:
------------------------------------------
+1 for the fix.
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4428-20081017.1.txt, HADOOP-4428-20081020.txt
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641276#action_12641276 ]
Owen O'Malley commented on HADOOP-4428:
---------------------------------------
I'd argue that we must look at jobs in the same queue.
1. The setup task is user code, and therefore could take an hour to run. It is unacceptable to completely block the queue while a job is being setup.
2. We would need to prevent a huge number of jobs being setup at the same time, so we should set a cap for how many jobs can be initialized at once. (3 jobs / user and 10 jobs for all users?)
Further, we must keep the running jobs sorted by priority and submit time, just as the waiting list is. The fact it is not, is also a blocker.
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4428-20081017.1.txt, HADOOP-4428-20081020.txt, HADOOP-4428.patch
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V updated HADOOP-4428:
------------------------------
Status: Patch Available (was: Open)
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4428-20081017.1.txt, HADOOP-4428-20081020.txt
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640950#action_12640950 ]
Amar Kamat commented on HADOOP-4428:
------------------------------------
Few comments :
1) Plz make sure that there are no unnecessary diffs.
For example
1)
{code}
-
- private FakeJobInProgress submitJob(int state, int maps, int reduces,
- String queue, String user) throws IOException {
+
+ private FakeJobInProgress submitJob(int state, int maps, int reduces,
+ String queue, String user, JobPriority priority)
+ throws IOException {
{code}
Here _private FakeJobInProgress submitJob(int state, int maps, int reduces,_ should not be in the diff (removed space).
2)
{code}
-
+
{code}
is useless (removed space).
3)
{code}
- FakeJobInProgress job = new FakeJobInProgress(
- new JobID("test", ++jobCounter), jobConf, taskTrackerManager, user);
+ FakeJobInProgress job =
+ new FakeJobInProgress(new JobID("test", ++jobCounter), jobConf,
+ taskTrackerManager, user);
{code}
is also unnecessarily added to the diff.
2) {{testJobRunStateChange()}} uses the old {{submitJob()}} api which I dont see fixed.
----
Just to clarify, jobs in the waiting queue can still be inited out of order when the capacity is exceeded.
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4428-20081017.1.txt, HADOOP-4428-20081020.txt
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640181#action_12640181 ]
Vinod K V commented on HADOOP-4428:
-----------------------------------
Don't know for sure if this is the right fix, but one way of solving this is by maintaining a sorted order of running-jobs queue, just like waiting job's queue. Then, irrespective of the order in which set-up tasks get launched and return, running jobs would be served in the order of priority.
One effect of the above fix is that, a higher priority job that is submitted after a lower priority job starts running, though will not preempt tasks of lower priority job, will be given more importance once there are free slots or once tasks of lower priority job started finishing.
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Karam Singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640163#action_12640163 ]
Karam Singh commented on HADOOP-4428:
-------------------------------------
TestCase-1:
Submitted 6 jobs at same time with different users as follows -:
User1 Job1 maps=reduces=212 priority=VERY_LOW
User2 Job2 maps=reduces=212 priority=LOW
User3 Job3 maps=reduces=212 priority=HIGH
User4 Job4 maps=reduces=212 priority=VERY_HIGH
User5 Job5 maps=reduces=212 priority=VERY_LOW
User6 Job6 maps=reduces=212 priority=VERY_HIGH
The JobTracker UI shows all 6 jobs as running with -:
J1, maps=reduces=53 running, Priority =VERY_LOW
J2, maps=reduces=0 running, Priority=LOW
J3, maps=, reduces=53 running, Priority=HIGH
J4, maps=reduces=0 running, Priority=VERY_HIGH
J5, maps=reduces=53, Priority=VERY_LOW
J6, maps=reduces=53, Priority=VERY_HIGH
Where expected behavior should like -:
Maps and reduces of J2, J3, J4, J6 start running while maps and reduces J1, J5 remain in queue.
TestCase-2:
First submitted J1, User7 maps=reduces=212 priority=NORMAL. (SleepJob with -rt=1)
When J1 started running, submitted 6 jobs at same time with different users -:
User1 Job2 maps=reduces=212 priority=VERY_LOW
User2 Job3 maps=reduces=212 priority=LOW
User3 Job4 maps=reduces=212 priority=HIGH
User4 Job5 maps=reduces=212 priority=VERY_HIGH
User5 Job6 maps=reduces=212 priority=VERY_LOW
User6 Job7 maps=reduces=212 priority=VERY_HIGH
"hadoop job -list" shows state of J2-J7 =4. When maps of J1 started finishing J2-J7 started running. The JobTracker UI shows status after J1 finishes as -:
J2, maps=reduces=53 running, Priority =VERY_LOW
J3, maps=reduces=53 running, Priority=LOW
J4, maps=, reduces=0 running, Priority=HIGH
J5, maps=reduces=53 running, Priority=VERY_HIGH
J6, maps=reduces=53, Priority=VERY_LOW
J7, maps=reduces=0, Priority=VERY_HIGH
Where Expected behavior should be -:
Maps and reduces of J3, J4, J5, J7 start running while maps and reduces J2, J6 remain in queue.
Note -: Preivously it was working as expected
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala reassigned HADOOP-4428:
----------------------------------------
Assignee: Sreekanth Ramakrishnan (was: Vinod K V)
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Sreekanth Ramakrishnan
> Priority: Blocker
> Fix For: 0.19.1
>
> Attachments: HADOOP-4428-20081017.1.txt, HADOOP-4428-20081020.txt, HADOOP-4428.patch
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-4428:
-------------------------------------
Priority: Blocker (was: Major)
Fix Version/s: 0.19.0
Initial diagnosis shows that this should be a blocker for Hadoop 0.19 as there is a deeper impact. The change is related to the newly introduced setup tasks. The impact seems to be that the capacity scheduler is actually initializing more jobs than it is supposed to.
The fix, at first sight, appears to be straightforward too, in the capacity scheduler.
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.19.0
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641275#action_12641275 ]
Hadoop QA commented on HADOOP-4428:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12392491/HADOOP-4428.patch
against trunk revision 706463.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3503/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3503/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3503/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3503/console
This message is automatically generated.
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4428-20081017.1.txt, HADOOP-4428-20081020.txt, HADOOP-4428.patch
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640927#action_12640927 ]
Amar Kamat commented on HADOOP-4428:
------------------------------------
Few comments :
1) The other {{submitJob}} api in {{TestCapacityScheduler}} should call the new {{submitJob}} api with {{NORMAL}} priority.
2) {{testJobPriorities()}} : What is the need to running a large job that takes up the cluster? Wouldn't a tasktracker with just one slot suffice? Something like this
{code}
1) submit 5 jobs with different priorities
2) check if they are sorted as expected
3) init the first job in the waiting queue.
4) check assignTasks() to see if no task is assigned.
5) launch setup tasks for the first job in the waiting queue
6) check assignTasks() to see if the result is as expected
7) finish the just launched task to free up the slot
8) do (5), (6) and (7) for other jobs in the waiting queue.
{code}
Rest of the code looks fine.
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4428-20081017.1.txt
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Vivek Ratan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641294#action_12641294 ]
Vivek Ratan commented on HADOOP-4428:
-------------------------------------
bq. We would need to prevent a huge number of jobs being setup at the same time, so we should set a cap for how many jobs can be initialized at once. (3 jobs / user and 10 jobs for all users?)
Agreed. I actually wanted to suggest something similar but forgot to add it to my comments. Basically, you init jobs up to a limit, then you move on to the next queue. This patch should be changed to do this.
bq. we must keep the running jobs sorted by priority and submit time, just as the waiting list is
Fair enough. I can see use cases for this. We should open another Jira for this. Implementing this functionality (sorting of run queue) will also make this Jira's solution more fair, since we won't rely on the completion times of setup tasks for jobs.
Along the same vein, there are a few more suggestions I have, for which I'll open up separate Jiras: separation of setup/clean task creation from initTasks(), and dealing with synchronous calls to initTasks() which can take a long time (thanks, Hemanth, for bringing that up).
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4428-20081017.1.txt, HADOOP-4428-20081020.txt, HADOOP-4428.patch
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640937#action_12640937 ]
Hemanth Yamijala commented on HADOOP-4428:
------------------------------------------
+1 for the code changes.
Regarding tests:
- +1 for Amar's comment on reusing submitJob API
- Regarding the testJobPriorities(), I think this is simulating the actual error condition that was reported in the bug. Hence, +1 for leaving the test as is.
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4428-20081017.1.txt
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V updated HADOOP-4428:
------------------------------
Attachment: HADOOP-4428-20081020.txt
Attaching new patch reusing the submitJob API in the test-case.
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4428-20081017.1.txt, HADOOP-4428-20081020.txt
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V updated HADOOP-4428:
------------------------------
Attachment: HADOOP-4428-20081017.1.txt
Attaching patch to fix the issue. This also has test-cases to test how waiting jobs are treated and to test that priorities are respected for jobs in the queue.
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4428-20081017.1.txt
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4428) Job Priorities are not handled
properly
Posted by "Vivek Ratan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641310#action_12641310 ]
Vivek Ratan commented on HADOOP-4428:
-------------------------------------
Summarizing the details:
We have two config values for the scheduler, regarding the number of simultaneous jobs that can be initialized: #jobs/user and #jobs/queue. These are optional settings in the scheduler's config file. Their default values are 3 and 10 respectively.
If these limits are met, or all jobs in a queue have been looked at, and no task has been found, the scheduler looks at the next queue.
HADOOP-4471 has been created to deal with the sorting of running jobs.
> Job Priorities are not handled properly
> ----------------------------------------
>
> Key: HADOOP-4428
> URL: https://issues.apache.org/jira/browse/HADOOP-4428
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Environment: Cluster: 106 TTs MapCapacity=212, ReduceCapacity=212
> Single Queue=default, User Limit=25, Priorities = Yes.
> Using hadoop branch 0.19 revision=705159
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: HADOOP-4428-20081017.1.txt, HADOOP-4428-20081020.txt, HADOOP-4428.patch
>
>
> Job Priorities are not handled properly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.