You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hemanth Yamijala (JIRA)" <ji...@apache.org> on 2009/04/23 09:51:47 UTC
[jira] Created: (HADOOP-5726) Remove pre-emption from the capacity
scheduler code base
Remove pre-emption from the capacity scheduler code base
--------------------------------------------------------
Key: HADOOP-5726
URL: https://issues.apache.org/jira/browse/HADOOP-5726
Project: Hadoop Core
Issue Type: Improvement
Components: contrib/capacity-sched
Reporter: Hemanth Yamijala
Assignee: rahul k singh
In an effort to simplify the code base, we would like to remove the pre-emption related code in the capacity scheduler. We would reintroduce this (possibly with some revisions to the original design) after a while. This will be an incompatible change. Any objections ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5726) Remove pre-emption from the
capacity scheduler code base
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12704667#action_12704667 ]
rahul k singh commented on HADOOP-5726:
---------------------------------------
[exec]
[exec]
[exec]
[exec]
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 7 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
[exec]
[exec]
[exec]
[exec] ======================================================================
[exec] ======================================================================
[exec] Finished build.
[exec] ======================================================================
[exec] ======================================================================
> Remove pre-emption from the capacity scheduler code base
> --------------------------------------------------------
>
> Key: HADOOP-5726
> URL: https://issues.apache.org/jira/browse/HADOOP-5726
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/capacity-sched
> Reporter: Hemanth Yamijala
> Assignee: rahul k singh
> Attachments: cs-without-preemption-23-1-2009.patch
>
>
> In an effort to simplify the code base, we would like to remove the pre-emption related code in the capacity scheduler. We would reintroduce this (possibly with some revisions to the original design) after a while. This will be an incompatible change. Any objections ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5726) Remove pre-emption from the capacity
scheduler code base
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5726:
----------------------------------
Status: Patch Available (was: Open)
> Remove pre-emption from the capacity scheduler code base
> --------------------------------------------------------
>
> Key: HADOOP-5726
> URL: https://issues.apache.org/jira/browse/HADOOP-5726
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/capacity-sched
> Reporter: Hemanth Yamijala
> Assignee: rahul k singh
> Attachments: cs-without-preemption-23-1-2009.patch
>
>
> In an effort to simplify the code base, we would like to remove the pre-emption related code in the capacity scheduler. We would reintroduce this (possibly with some revisions to the original design) after a while. This will be an incompatible change. Any objections ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5726) Remove pre-emption from the capacity
scheduler code base
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-5726:
-------------------------------------
Resolution: Fixed
Fix Version/s: 0.20.1
Release Note: Removed pre-emption from capacity scheduler. The impact of this change is that capacities for queues can no longer be guaranteed within a given span of time. Also changed configuration variables to remove pre-emption related variables and better reflect the absence of guarantees.
Hadoop Flags: [Incompatible change, Reviewed]
Status: Resolved (was: Patch Available)
I just committed this to trunk and branch 0.20. Thanks, Rahul !
> Remove pre-emption from the capacity scheduler code base
> --------------------------------------------------------
>
> Key: HADOOP-5726
> URL: https://issues.apache.org/jira/browse/HADOOP-5726
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/capacity-sched
> Reporter: Hemanth Yamijala
> Assignee: rahul k singh
> Fix For: 0.20.1
>
> Attachments: cs-without-preemption-23-1-2009.patch
>
>
> In an effort to simplify the code base, we would like to remove the pre-emption related code in the capacity scheduler. We would reintroduce this (possibly with some revisions to the original design) after a while. This will be an incompatible change. Any objections ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5726) Remove pre-emption from the
capacity scheduler code base
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705592#action_12705592 ]
Hemanth Yamijala commented on HADOOP-5726:
------------------------------------------
At this point, it looks like there's no objection to the removal of this code. I plan to commit this soon.
On a related note, though this is a major change, it appears that committing this to the 0.20 branch will significantly simplify further development. Is there an objection to this ?
> Remove pre-emption from the capacity scheduler code base
> --------------------------------------------------------
>
> Key: HADOOP-5726
> URL: https://issues.apache.org/jira/browse/HADOOP-5726
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/capacity-sched
> Reporter: Hemanth Yamijala
> Assignee: rahul k singh
> Attachments: cs-without-preemption-23-1-2009.patch
>
>
> In an effort to simplify the code base, we would like to remove the pre-emption related code in the capacity scheduler. We would reintroduce this (possibly with some revisions to the original design) after a while. This will be an incompatible change. Any objections ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5726) Remove pre-emption from the
capacity scheduler code base
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12706152#action_12706152 ]
Hudson commented on HADOOP-5726:
--------------------------------
Integrated in Hadoop-trunk #827 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/827/])
. Remove pre-emption from capacity scheduler code base. Contributed by Rahul Kumar Singh.
> Remove pre-emption from the capacity scheduler code base
> --------------------------------------------------------
>
> Key: HADOOP-5726
> URL: https://issues.apache.org/jira/browse/HADOOP-5726
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/capacity-sched
> Reporter: Hemanth Yamijala
> Assignee: rahul k singh
> Fix For: 0.20.1
>
> Attachments: cs-without-preemption-23-1-2009.patch
>
>
> In an effort to simplify the code base, we would like to remove the pre-emption related code in the capacity scheduler. We would reintroduce this (possibly with some revisions to the original design) after a while. This will be an incompatible change. Any objections ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5726) Remove pre-emption from the
capacity scheduler code base
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705591#action_12705591 ]
Hemanth Yamijala commented on HADOOP-5726:
------------------------------------------
bq. I'd propose we take this opportunity to further improve the capacity-scheduler by simplifying it's design, refactoring inner-classes into standalone classes, using the JobChangeEvents more extensively rather than re-sort queues on every heartbeat etc. Maybe a new jira?
Arun, I agree. Particularly with the refactoring of inner classes into standalone classes. Also, I think we should look at combining all queue state into a single class and promote it to a first class object. Today, this state is split between JobQueueManager and QueueSchedulingInfo.
Can you please open a new jira with your ideas and we can discuss there ?
> Remove pre-emption from the capacity scheduler code base
> --------------------------------------------------------
>
> Key: HADOOP-5726
> URL: https://issues.apache.org/jira/browse/HADOOP-5726
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/capacity-sched
> Reporter: Hemanth Yamijala
> Assignee: rahul k singh
> Attachments: cs-without-preemption-23-1-2009.patch
>
>
> In an effort to simplify the code base, we would like to remove the pre-emption related code in the capacity scheduler. We would reintroduce this (possibly with some revisions to the original design) after a while. This will be an incompatible change. Any objections ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5726) Remove pre-emption from the capacity
scheduler code base
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5726:
----------------------------------
Attachment: cs-without-preemption-23-1-2009.patch
Attaching the patch .
> Remove pre-emption from the capacity scheduler code base
> --------------------------------------------------------
>
> Key: HADOOP-5726
> URL: https://issues.apache.org/jira/browse/HADOOP-5726
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/capacity-sched
> Reporter: Hemanth Yamijala
> Assignee: rahul k singh
> Attachments: cs-without-preemption-23-1-2009.patch
>
>
> In an effort to simplify the code base, we would like to remove the pre-emption related code in the capacity scheduler. We would reintroduce this (possibly with some revisions to the original design) after a while. This will be an incompatible change. Any objections ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5726) Remove pre-emption from the
capacity scheduler code base
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12702304#action_12702304 ]
Arun C Murthy commented on HADOOP-5726:
---------------------------------------
+1 for the direction.
I'd propose we take this opportunity to further improve the capacity-scheduler by simplifying it's design, refactoring inner-classes into standalone classes, using the JobChangeEvents more extensively rather than re-sort queues on every heartbeat etc. Maybe a new jira?
Thoughts?
> Remove pre-emption from the capacity scheduler code base
> --------------------------------------------------------
>
> Key: HADOOP-5726
> URL: https://issues.apache.org/jira/browse/HADOOP-5726
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/capacity-sched
> Reporter: Hemanth Yamijala
> Assignee: rahul k singh
> Attachments: cs-without-preemption-23-1-2009.patch
>
>
> In an effort to simplify the code base, we would like to remove the pre-emption related code in the capacity scheduler. We would reintroduce this (possibly with some revisions to the original design) after a while. This will be an incompatible change. Any objections ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.