You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Karam Singh (JIRA)" <ji...@apache.org> on 2009/04/24 13:04:30 UTC
[jira] Created: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
--------------------------------------------------------------------------------------------------------
Key: HADOOP-5739
URL: https://issues.apache.org/jira/browse/HADOOP-5739
Project: Hadoop Core
Issue Type: Bug
Components: contrib/capacity-sched
Affects Versions: 0.20.0
Reporter: Karam Singh
After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5739:
----------------------------------
Attachment: hadoop-5739-5.patch
previous patch is wrong , attaching the new patch
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
> Attachments: hadoop-5739-3.patch, hadoop-5739-4.patch, hadoop-5739-5.patch, hadoop-5739-latest.patch, hadoop-5739.patch, hadoop-5739.patch
>
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712736#action_12712736 ]
Hemanth Yamijala commented on HADOOP-5739:
------------------------------------------
Some more comments:
- Change in ClusterWithCapacityScheduler seems unnecessary.
- In the test case, maybe intention is to delete testDir1, instead of testDir.
- Wrap to 80 characters.
- Why check the JobStatus only when the last event is SUCCEEDED. I think we should check for both. Also didn't understand why this is in a for loop.
- When creating the jobConf, maybe you should use the previously created job conf in MiniMRCluster.createJobConf
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
> Attachments: hadoop-5739-3.patch, hadoop-5739-latest.patch, hadoop-5739.patch, hadoop-5739.patch
>
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5739:
----------------------------------
Attachment: hadoop-5739-latest.patch
new patch with TestJobInProgressListener testcase
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
> Attachments: hadoop-5739-latest.patch, hadoop-5739.patch, hadoop-5739.patch
>
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715915#action_12715915 ]
Hadoop QA commented on HADOOP-5739:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12409551/hadoop-5739-7.patch
against trunk revision 781275.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/455/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/455/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/455/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/455/console
This message is automatically generated.
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
> Attachments: hadoop-5739-3.patch, hadoop-5739-4.patch, hadoop-5739-5.patch, hadoop-5739-6-Version20.patch, hadoop-5739-6.patch, hadoop-5739-7.patch, hadoop-5739-latest.patch, hadoop-5739.patch, hadoop-5739.patch
>
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5739:
----------------------------------
Status: Open (was: Patch Available)
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
> Attachments: hadoop-5739-3.patch, hadoop-5739-4.patch, hadoop-5739-5.patch, hadoop-5739-6-Version20.patch, hadoop-5739-6.patch, hadoop-5739-7.patch, hadoop-5739-latest.patch, hadoop-5739.patch, hadoop-5739.patch
>
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan reassigned HADOOP-5739:
----------------------------------------------
Assignee: rahul k singh
Assigning the bug to Rahul.
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5739:
----------------------------------
Attachment: hadoop-5739-4.patch
added hemanth's comment
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
> Attachments: hadoop-5739-3.patch, hadoop-5739-4.patch, hadoop-5739-latest.patch, hadoop-5739.patch, hadoop-5739.patch
>
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5739:
----------------------------------
Attachment: (was: hadoop-5419-5.patch)
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
> Attachments: hadoop-5739-3.patch, hadoop-5739-4.patch, hadoop-5739-5.patch, hadoop-5739-latest.patch, hadoop-5739.patch, hadoop-5739.patch
>
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "Karam Singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12702323#action_12702323 ]
Karam Singh commented on HADOOP-5739:
-------------------------------------
JobTracker configuration is having 5 Queues (default,Q1,Q2,Q3 and Q4), with acls enbaled. Capacity Scheduler is use as Schuder.
Submitted a job with 10000 maps and 10 reduced to Q3.
When 1000 maps get compeleted.
Restarted the JobTracker.
After successful recovery, checkec the status of job.
Q3 queue page does display then and
JobDetails page dispays following -:
Kind % Complete Num Tasks Pending Running Complete Killed Failed/Killed Task Attempts
map 10.98% 10000 8902 0 1098 0 0 / 0
reduce 4.12% 10 0 10 0 0 0 / 0
No map was schulded.
JobTracker logs says -:
[
2009-04-24 10:37:37,082 INFO org.apache.hadoop.mapred.JobInitializationPoller: Initializing job : <jobid> in Queue test_q3 For user : User1
2009-04-24 10:37:37,084 INFO org.apache.hadoop.mapred.JobInitializationPoller: Removing scheduled jobs from waiting queue<jobid>
]
It seems job is lost.
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703973#action_12703973 ]
rahul k singh commented on HADOOP-5739:
---------------------------------------
test patch results:
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 6 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
[exec]
[exec]
[exec]
[exec] ======================================================================
[exec] ======================================================================
[exec] Finished build.
[exec] ======================================================================
[exec] ======================================================================
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
> Attachments: hadoop-5739.patch, hadoop-5739.patch
>
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5739:
----------------------------------
Attachment: hadoop-5739-6.patch
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
> Attachments: hadoop-5739-3.patch, hadoop-5739-4.patch, hadoop-5739-5.patch, hadoop-5739-6-Version20.patch, hadoop-5739-6.patch, hadoop-5739-latest.patch, hadoop-5739.patch, hadoop-5739.patch
>
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5739:
----------------------------------
Status: Patch Available (was: Open)
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
> Attachments: hadoop-5739-3.patch, hadoop-5739-4.patch, hadoop-5739-5.patch, hadoop-5739-6-Version20.patch, hadoop-5739-6.patch, hadoop-5739-latest.patch, hadoop-5739.patch, hadoop-5739.patch
>
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5739:
----------------------------------
Attachment: hadoop-5739-7.patch
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
> Attachments: hadoop-5739-3.patch, hadoop-5739-4.patch, hadoop-5739-5.patch, hadoop-5739-6-Version20.patch, hadoop-5739-6.patch, hadoop-5739-7.patch, hadoop-5739-latest.patch, hadoop-5739.patch, hadoop-5739.patch
>
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-5739:
------------------------------------
Attachment: hadoop-5739-20-alt.patch
"20-alt" is an alternate version of a patch for 0.20 that is not to be committed to the branch.
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
> Attachments: hadoop-5739-20-alt.patch, hadoop-5739-3.patch, hadoop-5739-4.patch, hadoop-5739-5.patch, hadoop-5739-6-Version20.patch, hadoop-5739-6.patch, hadoop-5739-7.patch, hadoop-5739-latest.patch, hadoop-5739.patch, hadoop-5739.patch
>
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5739:
----------------------------------
Attachment: hadoop-5419-5.patch
included some more comments from hemanth
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
> Attachments: hadoop-5419-5.patch, hadoop-5739-3.patch, hadoop-5739-4.patch, hadoop-5739-latest.patch, hadoop-5739.patch, hadoop-5739.patch
>
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713854#action_12713854 ]
rahul k singh commented on HADOOP-5739:
---------------------------------------
runtest results
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
> Attachments: hadoop-5739-3.patch, hadoop-5739-4.patch, hadoop-5739-5.patch, hadoop-5739-latest.patch, hadoop-5739.patch, hadoop-5739.patch
>
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5739:
----------------------------------
Attachment: hadoop-5739-3.patch
Incorporated hemanth's comment
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
> Attachments: hadoop-5739-3.patch, hadoop-5739-latest.patch, hadoop-5739.patch, hadoop-5739.patch
>
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5739:
----------------------------------
Status: Patch Available (was: Open)
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
> Attachments: hadoop-5739-3.patch, hadoop-5739-4.patch, hadoop-5739-5.patch, hadoop-5739-6-Version20.patch, hadoop-5739-6.patch, hadoop-5739-7.patch, hadoop-5739-latest.patch, hadoop-5739.patch, hadoop-5739.patch
>
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714804#action_12714804 ]
Hadoop QA commented on HADOOP-5739:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12409376/hadoop-5739-6.patch
against trunk revision 780114.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/438/console
This message is automatically generated.
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
> Attachments: hadoop-5739-3.patch, hadoop-5739-4.patch, hadoop-5739-5.patch, hadoop-5739-6-Version20.patch, hadoop-5739-6.patch, hadoop-5739-latest.patch, hadoop-5739.patch, hadoop-5739.patch
>
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5739:
----------------------------------
Attachment: hadoop-5739-6-Version20.patch
version 20 patch
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
> Attachments: hadoop-5739-3.patch, hadoop-5739-4.patch, hadoop-5739-5.patch, hadoop-5739-6-Version20.patch, hadoop-5739-latest.patch, hadoop-5739.patch, hadoop-5739.patch
>
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5739:
----------------------------------
Attachment: hadoop-5739.patch
Adding the new patch with unit test case.
Approach: RecoveryManager in JobTracker is changed to make sure that listeners are notified in the event of any state change of Jobs.
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
> Attachments: hadoop-5739.patch, hadoop-5739.patch
>
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712636#action_12712636 ]
Hemanth Yamijala commented on HADOOP-5739:
------------------------------------------
Some comments:
- In JobTracker.processTaskAttempt, we are getting TaskAttemptID.forName(taskAttemptId) again. Can't we use the instance that's already obtained in the beginning of this method.
- MAX_COMPLETE_USER_JOBS_IN_MEMORY is being hardcoded to 0. I think this is a mistake, maybe for testing ?
- I think it is good to log that a status change event is being raised at an info level.
- shouldn't the call to updateJobInProgressListeners be synchronized on JT instance. It is everywhere else.
- Since the bug is not in Capacity scheduler, it looks like this is not required to me.
- TestJobInProgressListener: The new test will cover the code path added only if there is a taskattempt id record for the setup task in the history file. I don't see how we can guarantee that. Is there any way ?
- I think this should use the same conf parameters and same job parameters as TestJobTrackerRestart. That could be one way of ensuring that the setup task attempt is written to the history file.
- We should verify the events expected are raised. Not doing that.
- We seem to be starting datanodes twice.
- Comment "// Turn off the recovery" should be "turn on.."
- TmpJSListener -> Rename as FakeJobStatusListener.
- TmpScheduler -> FakeTaskScheduler.
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
> Attachments: hadoop-5739-latest.patch, hadoop-5739.patch, hadoop-5739.patch
>
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5739) After JobTracker restart Capacity
Schduler does not schedules pending tasks from already running tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5739:
----------------------------------
Attachment: hadoop-5739.patch
Attaching the initial solution for the problem.
Will be attaching 1 more patch with unit test
> After JobTracker restart Capacity Schduler does not schedules pending tasks from already running tasks.
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5739
> URL: https://issues.apache.org/jira/browse/HADOOP-5739
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Karam Singh
> Assignee: rahul k singh
> Attachments: hadoop-5739.patch
>
>
> After JobTracker restart Capacity Schduler does not schedules pending task from already running tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.