You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "rahul k singh (JIRA)" <ji...@apache.org> on 2009/05/26 13:03:46 UTC
[jira] Created: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Allow administrators to be able to start and stop queues
---------------------------------------------------------
Key: HADOOP-5913
URL: https://issues.apache.org/jira/browse/HADOOP-5913
Project: Hadoop Core
Issue Type: New Feature
Components: mapred
Reporter: rahul k singh
This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718799#action_12718799 ]
Hemanth Yamijala commented on HADOOP-5913:
------------------------------------------
I tried to see if this is ready for a commit. But I had a few comments.
- In the commands manual, the documentation of the refreshQueues command seems a little too terse. We should expand on it a bit more.
- In cluster_setup documentation, it appears that mapred.queue.names property is described *after* the ACLs. This order should be reversed.
- QueueManager.isRunning() should be synchronized.
- In checkDeprecation, the checks for mapred.queue.names is using a getStrings() with a default value, which means it will never be null. Hence, it doesn't serve the purpose.
- Methods in Queue don't seem to need to be public.
- Should they also be synchronized ?
- The QueueState enum has a method called 'equalTo'. Why doesn't the idiomatic 'equals' apply here ?
- QueueManager.getQueueAcls() should be similarly delegated like hasAccess(), no ?
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-10.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch, hadoop-5913-9.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5913:
----------------------------------
Attachment: hadoop-5913-7.patch
reformated Queue.java and QueueManager.java
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan reassigned HADOOP-5913:
----------------------------------------------
Assignee: rahul k singh
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717233#action_12717233 ]
rahul k singh commented on HADOOP-5913:
---------------------------------------
run test patch
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 11 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] -1 release audit. The applied patch generated 494 release audit warnings (more than the trunk's current 493 warnings).
[exec]
[exec]
[exec]
-1 in release audit is due to addition of new xml file , and with out Apache license text at the top
[exec]
[exec] ======================================================================
[exec] ======================================================================
[exec] Finished build.
[exec] ======================================================================
[exec] ======================================================================
[exec]
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-10.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch, hadoop-5913-9.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5913:
----------------------------------
Attachment: hadoop-5913-8.patch
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5913:
----------------------------------
Attachment: hadoop-5913-9.patch
runtest
[exec]
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 11 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] -1 release audit. The applied patch generated 494 release audit warnings (more than the trunk's current 493 warnings).
-1 in release audit is due to addition of new xml file , and with out Apache license text at the top
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch, hadoop-5913-9.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5913:
----------------------------------
Attachment: hadoop-5913-1.patch
Submitting the first cut.
Would be submitting the patch again with state related testcases.
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Attachments: hadoop-5913-1.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5913:
----------------------------------
Attachment: hadoop-5913-2.patch
added testcase for state.
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-2.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719260#action_12719260 ]
Hemanth Yamijala commented on HADOOP-5913:
------------------------------------------
test-patch results:
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 11 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-10.patch, HADOOP-5913-11.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch, hadoop-5913-9.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712955#action_12712955 ]
rahul k singh commented on HADOOP-5913:
---------------------------------------
All the running jobs in the queue will be completed , and any further new jobs will not be accepted.
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714404#action_12714404 ]
Sreekanth Ramakrishnan commented on HADOOP-5913:
------------------------------------------------
* Please recheck mapred-queues.xml.template file. The file seems to have been duplicated from the last patch to this.
* The file Queue.java which is a new file seems to be missing in the diff.
Can you please correct this and upload a new patch?
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719923#action_12719923 ]
rahul k singh commented on HADOOP-5913:
---------------------------------------
a small comment
-In Queue.java schedulingInfo need to be set to "N/A" as done in JobQueueInfo
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-10.patch, HADOOP-5913-11.patch, HADOOP-5913-12.patch, HADOOP-5913-13.patch, HADOOP-5913-14.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch, hadoop-5913-9.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713413#action_12713413 ]
rahul k singh commented on HADOOP-5913:
---------------------------------------
Summary:
-This feature would provide functionality to stop and start queues in Hadoop at runtime. All the running jobs in the queue will be
completed , and any further new jobs will not be accepted. This state would be persisted in the configuration file.
Requirements:
- Administrators should be able to stop(stop accepting jobs) and start(start accepting jobs)
queues at runtime.
- if a queue is stopped at runtime , it should complete all the existing running job and stop
accepting any new jobs.
- Administators should be able to change the queue configuration to start and stop.
- Once configuration is updated , administrators run " hadoop refreshQueues" command to refresh the existing
queues.This configuration would be persisted across the Jobtracker restart.
- Moving all the queue info(ACLS and state(stop or running)) in a common xml file , this would reduce the no of
commands required to change any queue settings , we can simply change this xml file and execute "hadoop refreshQueues"
to update the queue information.
- In order to do the above , we would be renaming mapred-queue-acls.xml file to mapred-queues.xml.
Design
-Assumptions:
- Administrators would have permission to execute the "hadoop refreshQueues" command.
- Only settings which are mentioned in mapred-queue.xml would be changed , any queue settings which are not part of
mapred-queue.xml(For ex. settings in scheduler's configuration) will not be effected.
-Summary
- Rename the"mapred-queue-acls.xml" to "mapred-queues.xml"
- Move all the queue related data from mapred-site.xml to mapred-queues.xml
Following tags would be moved.
- mapred.queue.names
- mapred.acls.enabled
- mapred.queue.<queueName>.acl-submit-job //This setting is for each queue
- mapred.queue.<queueName>.acl-administer-job //This setting is for each queue
- Refactor the existing code to encapsulate this new mapred-queues.xml file
- Introduce the new command "hadoop refreshQueues" which reads from the existing mapred-queues.xml
- Introduce new api in QueuesManager.java to check for state of the Queue.
- Introduction of new property for Queue in mapred-queues.xml.
"mapred.queue.<queueName>.state" values being "stopped" | "running"
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5913:
----------------------------------
Attachment: hadoop-5913-3.patch
Attaching new patch with doc changes
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-2.patch, hadoop-5913-3.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-5913:
-------------------------------------
Attachment: HADOOP-5913-13.patch
Added queue state information to the UI. Rahul, can you please take a look at the changes ?
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-10.patch, HADOOP-5913-11.patch, HADOOP-5913-12.patch, HADOOP-5913-13.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch, hadoop-5913-9.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5913:
----------------------------------
Attachment: hadoop-5913-4.patch
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-5913:
-------------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Release Note: Added an ability to stop/start queues. After a queue is stopped, no jobs can be submitted to the queue, though running jobs will continue to completion. Added the ability to refresh the state of the queue at runtime using the 'mapred mradmin -refreshQueues' command. In order to support the refresh, all queue configuration (including ACLs) is moved to a new configuration file called mapred-queues.xml. Modified 'mapred queue' commands and the Web UI to display the queue's state information.
Hadoop Flags: [Incompatible change, Reviewed]
Status: Resolved (was: Patch Available)
I just committed this to trunk. Thanks, Rahul !
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Fix For: 0.21.0
>
> Attachments: hadoop-5913-1.patch, hadoop-5913-10.patch, HADOOP-5913-11.patch, HADOOP-5913-12.patch, HADOOP-5913-13.patch, HADOOP-5913-14.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch, hadoop-5913-9.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721307#action_12721307 ]
Hudson commented on HADOOP-5913:
--------------------------------
Integrated in Hadoop-trunk #870 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/870/])
. Provide ability to an administrator to stop and start job queues. Contributed by Rahul Kumar Singh and Hemanth Yamijala.
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Fix For: 0.21.0
>
> Attachments: hadoop-5913-1.patch, hadoop-5913-10.patch, HADOOP-5913-11.patch, HADOOP-5913-12.patch, HADOOP-5913-13.patch, HADOOP-5913-14.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch, hadoop-5913-9.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-5913:
-------------------------------------
Status: Patch Available (was: Open)
Running tests manually, but just in case hudson can pick it up...
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-10.patch, HADOOP-5913-11.patch, HADOOP-5913-12.patch, HADOOP-5913-13.patch, HADOOP-5913-14.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch, hadoop-5913-9.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5913:
----------------------------------
Attachment: hadoop-5913-10.patch
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-10.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch, hadoop-5913-9.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-5913:
-------------------------------------
Attachment: HADOOP-5913-12.patch
A new patch that cleans up the code a bit, also fixed some test failures in the HADOOP-5913-11.patch I uploaded. Improves javadocs. ant test-patch continues passing with this patch. I also ran all tests and they pass locally.
I will run a few manual tests for sanity, and if things work fine, I will commit this patch.
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-10.patch, HADOOP-5913-11.patch, HADOOP-5913-12.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch, hadoop-5913-9.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-5913:
-------------------------------------
Attachment: HADOOP-5913-11.patch
Modified patch that incorporates the comments I raised. I also changed the refactoring slightly. Now, the Queue class is a simple data class with accessors. All the logic still rests with the QueueManager. I moved it to this model because it seemed like the logic was split between Queue and QueueManager and was making it more confusing that way.
Queue related tests pass. Running test-patch and tests.
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-10.patch, HADOOP-5913-11.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch, hadoop-5913-9.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5913:
----------------------------------
Attachment: hadoop-5913-5.patch
implemented sreekanth's offline comments
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715374#action_12715374 ]
Sreekanth Ramakrishnan commented on HADOOP-5913:
------------------------------------------------
+1 to the latest patch.
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719994#action_12719994 ]
rahul k singh commented on HADOOP-5913:
---------------------------------------
+1
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-10.patch, HADOOP-5913-11.patch, HADOOP-5913-12.patch, HADOOP-5913-13.patch, HADOOP-5913-14.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch, hadoop-5913-9.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719995#action_12719995 ]
rahul k singh commented on HADOOP-5913:
---------------------------------------
+1
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-10.patch, HADOOP-5913-11.patch, HADOOP-5913-12.patch, HADOOP-5913-13.patch, HADOOP-5913-14.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch, hadoop-5913-9.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720539#action_12720539 ]
Hemanth Yamijala commented on HADOOP-5913:
------------------------------------------
test-patch output:
{noformat}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 18 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{noformat}
I ran all tests. Some tests timed out on my local machine. But they timed out on trunk as well, or were clearly not related to the patch. These are HADOOP-6061, HADOOP-6062 and HADOOP-6064. Based on this, I am planning to commit this patch soon.
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-10.patch, HADOOP-5913-11.patch, HADOOP-5913-12.patch, HADOOP-5913-13.patch, HADOOP-5913-14.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch, hadoop-5913-9.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-5913:
-------------------------------------
Attachment: HADOOP-5913-14.patch
I missed incrementing version number for JobSubmissionProtocol. This patch takes care of that.
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-10.patch, HADOOP-5913-11.patch, HADOOP-5913-12.patch, HADOOP-5913-13.patch, HADOOP-5913-14.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch, hadoop-5913-9.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5913:
----------------------------------
Attachment: hadoop-5913-6.patch
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719497#action_12719497 ]
Hemanth Yamijala commented on HADOOP-5913:
------------------------------------------
Sigh ! The manual tests worked, but unfortunately, I realized the UI does not reflect the state of the queues, which I think is necessary for completeness of this patch.
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-10.patch, HADOOP-5913-11.patch, HADOOP-5913-12.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch, hadoop-5913-9.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719960#action_12719960 ]
Hemanth Yamijala commented on HADOOP-5913:
------------------------------------------
Rahul, the schedulingInfo member in Queue is an Object and not a string. Therefore, it cannot be set to a default value like in JobQueueInfo. I verified that if the object is null, it will not cause any damage elsewhere in code. This seems to be a valid assumption. For instance, all calls to Queue.getSchedulingInfo (except in test cases) check for null before using the object. Does this make sense for you ?
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Assignee: rahul k singh
> Attachments: hadoop-5913-1.patch, hadoop-5913-10.patch, HADOOP-5913-11.patch, HADOOP-5913-12.patch, HADOOP-5913-13.patch, HADOOP-5913-14.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch, hadoop-5913-9.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713577#action_12713577 ]
rahul k singh commented on HADOOP-5913:
---------------------------------------
Administrator can change the property values at runtime by
1.changing the property values in mapred-queues.xml
2. running "hadoop mradmin -refreshQueues"
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
> Attachments: hadoop-5913-1.patch
>
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5913) Allow administrators to be able to
start and stop queues
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713574#action_12713574 ]
rahul k singh commented on HADOOP-5913:
---------------------------------------
the state would be part of queue configuration. Earlier statement of state being persisted in configuration is not correct.
The queue "state" would be a configuration entry , administrator can change this entry and execute "hadoop mradmin -refreshQueues" command to update this value
> Allow administrators to be able to start and stop queues
> ---------------------------------------------------------
>
> Key: HADOOP-5913
> URL: https://issues.apache.org/jira/browse/HADOOP-5913
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: rahul k singh
>
> This feature would provide functionality to stop and start queues in Hadoop at runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.