You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hemanth Yamijala (JIRA)" <ji...@apache.org> on 2008/11/03 06:40:44 UTC
[jira] Created: (HADOOP-4576) Modify pending tasks count in the UI
to pending jobs count in the UI
Modify pending tasks count in the UI to pending jobs count in the UI
--------------------------------------------------------------------
Key: HADOOP-4576
URL: https://issues.apache.org/jira/browse/HADOOP-4576
Project: Hadoop Core
Issue Type: Bug
Components: contrib/capacity-sched
Affects Versions: 0.19.0
Reporter: Hemanth Yamijala
The UI for capacity scheduler displays 'pending tasks' counts. However the capacity scheduler does not update these counts to be the actual values for optimization purposes, for e.g. to avoid walking all pending jobs on all heartbeats. Hence this information is not very accurate.
Also, while 'running tasks' counts are useful to compare against capacities and limits, 'pending tasks' counts do not add too much user value. A better count to display would be the number of running and pending jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4576) Modify pending tasks count in the
UI to pending jobs count in the UI
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648508#action_12648508 ]
Sreekanth Ramakrishnan commented on HADOOP-4576:
------------------------------------------------
Attaching output of ant test-patch:
{noformat}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
{noformat}
> Modify pending tasks count in the UI to pending jobs count in the UI
> --------------------------------------------------------------------
>
> Key: HADOOP-4576
> URL: https://issues.apache.org/jira/browse/HADOOP-4576
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Sreekanth Ramakrishnan
> Attachments: HADOOP-4576-1.patch
>
>
> The UI for capacity scheduler displays 'pending tasks' counts. However the capacity scheduler does not update these counts to be the actual values for optimization purposes, for e.g. to avoid walking all pending jobs on all heartbeats. Hence this information is not very accurate.
> Also, while 'running tasks' counts are useful to compare against capacities and limits, 'pending tasks' counts do not add too much user value. A better count to display would be the number of running and pending jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4576) Modify pending tasks count in the UI
to pending jobs count in the UI
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-4576:
-------------------------------------
Resolution: Fixed
Fix Version/s: 0.20.0
Hadoop Flags: [Incompatible change, Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Sreekanth !
> Modify pending tasks count in the UI to pending jobs count in the UI
> --------------------------------------------------------------------
>
> Key: HADOOP-4576
> URL: https://issues.apache.org/jira/browse/HADOOP-4576
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Sreekanth Ramakrishnan
> Fix For: 0.20.0
>
> Attachments: HADOOP-4576-1.patch, HADOOP-4576-2.patch, HADOOP-4576-3.patch, HADOOP-4576-4.patch, HADOOP-4576-5.patch, HADOOP-4576-6.patch, waitingjobs1.png
>
>
> The UI for capacity scheduler displays 'pending tasks' counts. However the capacity scheduler does not update these counts to be the actual values for optimization purposes, for e.g. to avoid walking all pending jobs on all heartbeats. Hence this information is not very accurate.
> Also, while 'running tasks' counts are useful to compare against capacities and limits, 'pending tasks' counts do not add too much user value. A better count to display would be the number of running and pending jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4576) Modify pending tasks count in the UI
to pending jobs count in the UI
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated HADOOP-4576:
-------------------------------------------
Attachment: HADOOP-4576-4.patch
Attaching latest patch incorporating Amar's comments.
With respect to the test cases:
Following is the order we are testing the scheduling information:
- Submit 5 jobs to a queue.
- Check the waiting jobs count, it should be 5.
- Then run initializationPoller(), this initializes first two jobs in the queue but does not raise the job status changed event.
- Check once again the waiting queue, it should be 5 jobs again.
- Then raise status change events.
- Assign one task to a task tracker.
- Check waiting job count, it should be 4 now.
- Then pick an initialized job but not scheduled job and fail it.
- Run the poller, since poller is responsible for removing, failed jobs and scheduled jobs from job queue maintained by job queue manager, there is no requirement for raising status changed event as the job is cleared without it by the poller once job fails before it gets scheduled. With respect to removal of the jobs which are scheduled and failing is taken care by the JobQueueManager.
- Check the waiting job count should now be 3.
- Now fail a job which has not been initialized at all.
- Run the poller, so that it can clean up the job queue.
- Check the count, the waiting job count should be 2.
I hope this answers the doubt with regarding test case, I have also mentioned the steps as inline comment in the test case.
> Modify pending tasks count in the UI to pending jobs count in the UI
> --------------------------------------------------------------------
>
> Key: HADOOP-4576
> URL: https://issues.apache.org/jira/browse/HADOOP-4576
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Sreekanth Ramakrishnan
> Attachments: HADOOP-4576-1.patch, HADOOP-4576-2.patch, HADOOP-4576-3.patch, HADOOP-4576-4.patch
>
>
> The UI for capacity scheduler displays 'pending tasks' counts. However the capacity scheduler does not update these counts to be the actual values for optimization purposes, for e.g. to avoid walking all pending jobs on all heartbeats. Hence this information is not very accurate.
> Also, while 'running tasks' counts are useful to compare against capacities and limits, 'pending tasks' counts do not add too much user value. A better count to display would be the number of running and pending jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4576) Modify pending tasks count in the UI
to pending jobs count in the UI
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated HADOOP-4576:
-------------------------------------------
Attachment: HADOOP-4576-6.patch
Changing visibility of the constructor and removing the deprecation.
ant test-patch output for the patch is :
{noformat}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 4 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
{noformat}
> Modify pending tasks count in the UI to pending jobs count in the UI
> --------------------------------------------------------------------
>
> Key: HADOOP-4576
> URL: https://issues.apache.org/jira/browse/HADOOP-4576
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Sreekanth Ramakrishnan
> Attachments: HADOOP-4576-1.patch, HADOOP-4576-2.patch, HADOOP-4576-3.patch, HADOOP-4576-4.patch, HADOOP-4576-5.patch, HADOOP-4576-6.patch
>
>
> The UI for capacity scheduler displays 'pending tasks' counts. However the capacity scheduler does not update these counts to be the actual values for optimization purposes, for e.g. to avoid walking all pending jobs on all heartbeats. Hence this information is not very accurate.
> Also, while 'running tasks' counts are useful to compare against capacities and limits, 'pending tasks' counts do not add too much user value. A better count to display would be the number of running and pending jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4576) Modify pending tasks count in the
UI to pending jobs count in the UI
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653615#action_12653615 ]
Hemanth Yamijala commented on HADOOP-4576:
------------------------------------------
I think it is not necessary to deprecate the constructor of SchedulingInfo. It is a private static inner class and hence nobody should be affected ? It would just add more code to maintain. Can you please submit a new patch removing the deprecation and just changing the api ? Please also remember to remove the checks for null in the toString method, as the object is not expected to be null after this change.
BTW, I spoke to Amar about this and we agree on this point now.
> Modify pending tasks count in the UI to pending jobs count in the UI
> --------------------------------------------------------------------
>
> Key: HADOOP-4576
> URL: https://issues.apache.org/jira/browse/HADOOP-4576
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Sreekanth Ramakrishnan
> Attachments: HADOOP-4576-1.patch, HADOOP-4576-2.patch, HADOOP-4576-3.patch, HADOOP-4576-4.patch, HADOOP-4576-5.patch
>
>
> The UI for capacity scheduler displays 'pending tasks' counts. However the capacity scheduler does not update these counts to be the actual values for optimization purposes, for e.g. to avoid walking all pending jobs on all heartbeats. Hence this information is not very accurate.
> Also, while 'running tasks' counts are useful to compare against capacities and limits, 'pending tasks' counts do not add too much user value. A better count to display would be the number of running and pending jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4576) Modify pending tasks count in the
UI to pending jobs count in the UI
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654074#action_12654074 ]
Hudson commented on HADOOP-4576:
--------------------------------
Integrated in Hadoop-trunk #680 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/680/])
> Modify pending tasks count in the UI to pending jobs count in the UI
> --------------------------------------------------------------------
>
> Key: HADOOP-4576
> URL: https://issues.apache.org/jira/browse/HADOOP-4576
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Sreekanth Ramakrishnan
> Fix For: 0.20.0
>
> Attachments: HADOOP-4576-1.patch, HADOOP-4576-2.patch, HADOOP-4576-3.patch, HADOOP-4576-4.patch, HADOOP-4576-5.patch, HADOOP-4576-6.patch, waitingjobs1.png
>
>
> The UI for capacity scheduler displays 'pending tasks' counts. However the capacity scheduler does not update these counts to be the actual values for optimization purposes, for e.g. to avoid walking all pending jobs on all heartbeats. Hence this information is not very accurate.
> Also, while 'running tasks' counts are useful to compare against capacities and limits, 'pending tasks' counts do not add too much user value. A better count to display would be the number of running and pending jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4576) Modify pending tasks count in the UI
to pending jobs count in the UI
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated HADOOP-4576:
-------------------------------------------
Attachment: waitingjobs1.png
Attaching screenshot of the scheduling information after the patch has been applied.
> Modify pending tasks count in the UI to pending jobs count in the UI
> --------------------------------------------------------------------
>
> Key: HADOOP-4576
> URL: https://issues.apache.org/jira/browse/HADOOP-4576
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Sreekanth Ramakrishnan
> Attachments: HADOOP-4576-1.patch, HADOOP-4576-2.patch, HADOOP-4576-3.patch, HADOOP-4576-4.patch, HADOOP-4576-5.patch, HADOOP-4576-6.patch, waitingjobs1.png
>
>
> The UI for capacity scheduler displays 'pending tasks' counts. However the capacity scheduler does not update these counts to be the actual values for optimization purposes, for e.g. to avoid walking all pending jobs on all heartbeats. Hence this information is not very accurate.
> Also, while 'running tasks' counts are useful to compare against capacities and limits, 'pending tasks' counts do not add too much user value. A better count to display would be the number of running and pending jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4576) Modify pending tasks count in the
UI to pending jobs count in the UI
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653209#action_12653209 ]
Amar Kamat commented on HADOOP-4576:
------------------------------------
Comments :
{{JobQueuesManager.java}} :
1) No need to import {{AtomicInteger}}. Also plz remove extra diffs.
{{CapacityTaskScheduler.java}} :
1) You cant simply change the constructor def. Overload it and deprecate the other if needed.
2) Extra diffs w.r.t.
{code}
- supportsPriority?"YES":"NO"));
+ supportsPriority?"YES":"NO"));
{code}
3) {quote}
+ sb.append(String.format("* Scheduling information can be off by " +
+ "maximum of %d seconds\n", pollingInterval/1000));
{quote}
Use {{StringUtils}} for formatting time.
{{TestCapacityScheduler.java}}
1) {quote}
+ scheduler.assignTasks(tracker("tt1")); // heartbeat
+ p.selectJobsToInitialize();
{quote}
The ordering doesnt seem right. You submit 5 jobs, try to assign tasks (which should be no-op) and then you init the jobs.
2) Shouldnt we also check/test the timing issue that after _poll-interval_ units of time the values are correct. Something like
- add jobs
- check the queue-sched-info
- allow the jobs to be inited by the poller i.e wait for _poller-interval_ time
- check again to see if the change is made.
2) Plz mark the start and end of a new sub-test using comments
> Modify pending tasks count in the UI to pending jobs count in the UI
> --------------------------------------------------------------------
>
> Key: HADOOP-4576
> URL: https://issues.apache.org/jira/browse/HADOOP-4576
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Sreekanth Ramakrishnan
> Attachments: HADOOP-4576-1.patch, HADOOP-4576-2.patch, HADOOP-4576-3.patch
>
>
> The UI for capacity scheduler displays 'pending tasks' counts. However the capacity scheduler does not update these counts to be the actual values for optimization purposes, for e.g. to avoid walking all pending jobs on all heartbeats. Hence this information is not very accurate.
> Also, while 'running tasks' counts are useful to compare against capacities and limits, 'pending tasks' counts do not add too much user value. A better count to display would be the number of running and pending jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-4576) Modify pending tasks count in the UI
to pending jobs count in the UI
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan reassigned HADOOP-4576:
----------------------------------------------
Assignee: Sreekanth Ramakrishnan
> Modify pending tasks count in the UI to pending jobs count in the UI
> --------------------------------------------------------------------
>
> Key: HADOOP-4576
> URL: https://issues.apache.org/jira/browse/HADOOP-4576
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Sreekanth Ramakrishnan
> Attachments: HADOOP-4576-1.patch
>
>
> The UI for capacity scheduler displays 'pending tasks' counts. However the capacity scheduler does not update these counts to be the actual values for optimization purposes, for e.g. to avoid walking all pending jobs on all heartbeats. Hence this information is not very accurate.
> Also, while 'running tasks' counts are useful to compare against capacities and limits, 'pending tasks' counts do not add too much user value. A better count to display would be the number of running and pending jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4576) Modify pending tasks count in the UI
to pending jobs count in the UI
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-4576:
------------------------------------
Release Note: Changed capacity scheduler UI to better present number of running and pending tasks.
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
Edit release note for publication.
> Modify pending tasks count in the UI to pending jobs count in the UI
> --------------------------------------------------------------------
>
> Key: HADOOP-4576
> URL: https://issues.apache.org/jira/browse/HADOOP-4576
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Sreekanth Ramakrishnan
> Fix For: 0.20.0
>
> Attachments: HADOOP-4576-1.patch, HADOOP-4576-2.patch, HADOOP-4576-3.patch, HADOOP-4576-4.patch, HADOOP-4576-5.patch, HADOOP-4576-6.patch, waitingjobs1.png
>
>
> The UI for capacity scheduler displays 'pending tasks' counts. However the capacity scheduler does not update these counts to be the actual values for optimization purposes, for e.g. to avoid walking all pending jobs on all heartbeats. Hence this information is not very accurate.
> Also, while 'running tasks' counts are useful to compare against capacities and limits, 'pending tasks' counts do not add too much user value. A better count to display would be the number of running and pending jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4576) Modify pending tasks count in the
UI to pending jobs count in the UI
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653282#action_12653282 ]
Amar Kamat commented on HADOOP-4576:
------------------------------------
+1. Looks good.
> Modify pending tasks count in the UI to pending jobs count in the UI
> --------------------------------------------------------------------
>
> Key: HADOOP-4576
> URL: https://issues.apache.org/jira/browse/HADOOP-4576
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Sreekanth Ramakrishnan
> Attachments: HADOOP-4576-1.patch, HADOOP-4576-2.patch, HADOOP-4576-3.patch, HADOOP-4576-4.patch
>
>
> The UI for capacity scheduler displays 'pending tasks' counts. However the capacity scheduler does not update these counts to be the actual values for optimization purposes, for e.g. to avoid walking all pending jobs on all heartbeats. Hence this information is not very accurate.
> Also, while 'running tasks' counts are useful to compare against capacities and limits, 'pending tasks' counts do not add too much user value. A better count to display would be the number of running and pending jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4576) Modify pending tasks count in the UI
to pending jobs count in the UI
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated HADOOP-4576:
-------------------------------------------
Status: Patch Available (was: Open)
> Modify pending tasks count in the UI to pending jobs count in the UI
> --------------------------------------------------------------------
>
> Key: HADOOP-4576
> URL: https://issues.apache.org/jira/browse/HADOOP-4576
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Sreekanth Ramakrishnan
> Attachments: HADOOP-4576-1.patch, HADOOP-4576-2.patch, HADOOP-4576-3.patch, HADOOP-4576-4.patch, HADOOP-4576-5.patch
>
>
> The UI for capacity scheduler displays 'pending tasks' counts. However the capacity scheduler does not update these counts to be the actual values for optimization purposes, for e.g. to avoid walking all pending jobs on all heartbeats. Hence this information is not very accurate.
> Also, while 'running tasks' counts are useful to compare against capacities and limits, 'pending tasks' counts do not add too much user value. A better count to display would be the number of running and pending jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4576) Modify pending tasks count in the
UI to pending jobs count in the UI
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652279#action_12652279 ]
Sreekanth Ramakrishnan commented on HADOOP-4576:
------------------------------------------------
Attaching output of ant test-patch :
{noformat}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
{noformat}
> Modify pending tasks count in the UI to pending jobs count in the UI
> --------------------------------------------------------------------
>
> Key: HADOOP-4576
> URL: https://issues.apache.org/jira/browse/HADOOP-4576
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Sreekanth Ramakrishnan
> Attachments: HADOOP-4576-1.patch, HADOOP-4576-2.patch
>
>
> The UI for capacity scheduler displays 'pending tasks' counts. However the capacity scheduler does not update these counts to be the actual values for optimization purposes, for e.g. to avoid walking all pending jobs on all heartbeats. Hence this information is not very accurate.
> Also, while 'running tasks' counts are useful to compare against capacities and limits, 'pending tasks' counts do not add too much user value. A better count to display would be the number of running and pending jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4576) Modify pending tasks count in the
UI to pending jobs count in the UI
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653627#action_12653627 ]
Sreekanth Ramakrishnan commented on HADOOP-4576:
------------------------------------------------
The ant test-contrib passed on the local machine.
> Modify pending tasks count in the UI to pending jobs count in the UI
> --------------------------------------------------------------------
>
> Key: HADOOP-4576
> URL: https://issues.apache.org/jira/browse/HADOOP-4576
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Sreekanth Ramakrishnan
> Attachments: HADOOP-4576-1.patch, HADOOP-4576-2.patch, HADOOP-4576-3.patch, HADOOP-4576-4.patch, HADOOP-4576-5.patch, HADOOP-4576-6.patch, waitingjobs1.png
>
>
> The UI for capacity scheduler displays 'pending tasks' counts. However the capacity scheduler does not update these counts to be the actual values for optimization purposes, for e.g. to avoid walking all pending jobs on all heartbeats. Hence this information is not very accurate.
> Also, while 'running tasks' counts are useful to compare against capacities and limits, 'pending tasks' counts do not add too much user value. A better count to display would be the number of running and pending jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4576) Modify pending tasks count in the UI
to pending jobs count in the UI
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated HADOOP-4576:
-------------------------------------------
Attachment: waitingjobs1.png
> Modify pending tasks count in the UI to pending jobs count in the UI
> --------------------------------------------------------------------
>
> Key: HADOOP-4576
> URL: https://issues.apache.org/jira/browse/HADOOP-4576
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Sreekanth Ramakrishnan
> Attachments: HADOOP-4576-1.patch, HADOOP-4576-2.patch, HADOOP-4576-3.patch, HADOOP-4576-4.patch, HADOOP-4576-5.patch, HADOOP-4576-6.patch, waitingjobs1.png
>
>
> The UI for capacity scheduler displays 'pending tasks' counts. However the capacity scheduler does not update these counts to be the actual values for optimization purposes, for e.g. to avoid walking all pending jobs on all heartbeats. Hence this information is not very accurate.
> Also, while 'running tasks' counts are useful to compare against capacities and limits, 'pending tasks' counts do not add too much user value. A better count to display would be the number of running and pending jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4576) Modify pending tasks count in the UI
to pending jobs count in the UI
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-4576:
-------------------------------------
Attachment: (was: waitingjobs1.png)
> Modify pending tasks count in the UI to pending jobs count in the UI
> --------------------------------------------------------------------
>
> Key: HADOOP-4576
> URL: https://issues.apache.org/jira/browse/HADOOP-4576
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Sreekanth Ramakrishnan
> Attachments: HADOOP-4576-1.patch, HADOOP-4576-2.patch, HADOOP-4576-3.patch, HADOOP-4576-4.patch, HADOOP-4576-5.patch, HADOOP-4576-6.patch
>
>
> The UI for capacity scheduler displays 'pending tasks' counts. However the capacity scheduler does not update these counts to be the actual values for optimization purposes, for e.g. to avoid walking all pending jobs on all heartbeats. Hence this information is not very accurate.
> Also, while 'running tasks' counts are useful to compare against capacities and limits, 'pending tasks' counts do not add too much user value. A better count to display would be the number of running and pending jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4576) Modify pending tasks count in the UI
to pending jobs count in the UI
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated HADOOP-4576:
-------------------------------------------
Attachment: HADOOP-4576-2.patch
Attaching new patch, this patch does not iterate thro' job list to find count of waiting jobs,instead introduces a counter in the QueueInfo object in JobQueueManager to maintain a count of waiting jobs which is incremented and decremented (by JobInitializationPoller). This would mean that in [HADOOP-4445|https://issues.apache.org/jira/browse/HADOOP-4445] we just need to iterate thro' running jobs queue to find number of running tasks for that queue.
> Modify pending tasks count in the UI to pending jobs count in the UI
> --------------------------------------------------------------------
>
> Key: HADOOP-4576
> URL: https://issues.apache.org/jira/browse/HADOOP-4576
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Sreekanth Ramakrishnan
> Attachments: HADOOP-4576-1.patch, HADOOP-4576-2.patch
>
>
> The UI for capacity scheduler displays 'pending tasks' counts. However the capacity scheduler does not update these counts to be the actual values for optimization purposes, for e.g. to avoid walking all pending jobs on all heartbeats. Hence this information is not very accurate.
> Also, while 'running tasks' counts are useful to compare against capacities and limits, 'pending tasks' counts do not add too much user value. A better count to display would be the number of running and pending jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4576) Modify pending tasks count in the UI
to pending jobs count in the UI
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated HADOOP-4576:
-------------------------------------------
Attachment: HADOOP-4576-5.patch
Attaching patch with merging with today's trunk.
> Modify pending tasks count in the UI to pending jobs count in the UI
> --------------------------------------------------------------------
>
> Key: HADOOP-4576
> URL: https://issues.apache.org/jira/browse/HADOOP-4576
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Sreekanth Ramakrishnan
> Attachments: HADOOP-4576-1.patch, HADOOP-4576-2.patch, HADOOP-4576-3.patch, HADOOP-4576-4.patch, HADOOP-4576-5.patch
>
>
> The UI for capacity scheduler displays 'pending tasks' counts. However the capacity scheduler does not update these counts to be the actual values for optimization purposes, for e.g. to avoid walking all pending jobs on all heartbeats. Hence this information is not very accurate.
> Also, while 'running tasks' counts are useful to compare against capacities and limits, 'pending tasks' counts do not add too much user value. A better count to display would be the number of running and pending jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4576) Modify pending tasks count in the UI
to pending jobs count in the UI
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated HADOOP-4576:
-------------------------------------------
Attachment: HADOOP-4576-1.patch
Attaching patch, which displays running jobs and waiting jobs instead of running and waiting task counts for maps and reduces.
Waiting jobs and running jobs are computed after getting all jobs which are present in the scheduler and checking their status, this is done to incorporate changes which would be made in [HADOOP-4513|https://issues.apache.org/jira/browse/HADOOP-4513]
> Modify pending tasks count in the UI to pending jobs count in the UI
> --------------------------------------------------------------------
>
> Key: HADOOP-4576
> URL: https://issues.apache.org/jira/browse/HADOOP-4576
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Attachments: HADOOP-4576-1.patch
>
>
> The UI for capacity scheduler displays 'pending tasks' counts. However the capacity scheduler does not update these counts to be the actual values for optimization purposes, for e.g. to avoid walking all pending jobs on all heartbeats. Hence this information is not very accurate.
> Also, while 'running tasks' counts are useful to compare against capacities and limits, 'pending tasks' counts do not add too much user value. A better count to display would be the number of running and pending jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4576) Modify pending tasks count in the UI
to pending jobs count in the UI
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated HADOOP-4576:
-------------------------------------------
Attachment: HADOOP-4576-3.patch
Attaching a new patch merged with current trunk.
Plus have removed the variable which I introduced in previous patch, it was not required as we can find out number of waiting jobs directly from size of the jobList instead of having a counter.
Also added a note in scheduling information stating that scheduling information can be off at the maximum by polling interval of the job initialization poller which does clean up of the waiting jobs.
> Modify pending tasks count in the UI to pending jobs count in the UI
> --------------------------------------------------------------------
>
> Key: HADOOP-4576
> URL: https://issues.apache.org/jira/browse/HADOOP-4576
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Sreekanth Ramakrishnan
> Attachments: HADOOP-4576-1.patch, HADOOP-4576-2.patch, HADOOP-4576-3.patch
>
>
> The UI for capacity scheduler displays 'pending tasks' counts. However the capacity scheduler does not update these counts to be the actual values for optimization purposes, for e.g. to avoid walking all pending jobs on all heartbeats. Hence this information is not very accurate.
> Also, while 'running tasks' counts are useful to compare against capacities and limits, 'pending tasks' counts do not add too much user value. A better count to display would be the number of running and pending jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.