You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hemanth Yamijala (JIRA)" <ji...@apache.org> on 2008/09/09 09:05:44 UTC
[jira] Created: (HADOOP-4124) Changing priority of a job should be
available in CLI and available on the web UI only along with the Kill Job
actions
Changing priority of a job should be available in CLI and available on the web UI only along with the Kill Job actions
----------------------------------------------------------------------------------------------------------------------
Key: HADOOP-4124
URL: https://issues.apache.org/jira/browse/HADOOP-4124
Project: Hadoop Core
Issue Type: Improvement
Components: mapred
Reporter: Hemanth Yamijala
Assignee: Hemanth Yamijala
Fix For: 0.19.0
Currently, a job's priority can only be changed from the job tracker web UI. However, it is really similar (if not as destructive) as killing a job, in the sense that is should not be exposed publicly. The use case for this kind of restriction is where Hadoop is shared in production environments by different users, and they should not be allowed to change another user's priority. At the same time, it should be possible to still change a job's priority in some manner, if a user is authorized to do so, and the CLI can provide a way to do this, along with mechanisms introduced for ACLs in HADOOP-3698.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4124) Changing priority of a job should be
available in CLI and available on the web UI only along with the Kill Job
actions
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-4124:
----------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Hemanth!
> Changing priority of a job should be available in CLI and available on the web UI only along with the Kill Job actions
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4124
> URL: https://issues.apache.org/jira/browse/HADOOP-4124
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.19.0
>
> Attachments: HADOOP-4124.patch, HADOOP-4124.patch, HADOOP-4124.patch, HADOOP-4124.patch
>
>
> Currently, a job's priority can only be changed from the job tracker web UI. However, it is really similar (if not as destructive) as killing a job, in the sense that is should not be exposed publicly. The use case for this kind of restriction is where Hadoop is shared in production environments by different users, and they should not be allowed to change another user's priority. At the same time, it should be possible to still change a job's priority in some manner, if a user is authorized to do so, and the CLI can provide a way to do this, along with mechanisms introduced for ACLs in HADOOP-3698.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4124) Changing priority of a job should be
available in CLI and available on the web UI only along with the Kill Job
actions
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-4124:
-------------------------------------
Status: Patch Available (was: Open)
Running tests on my machine. Submitting to Hudson in the meantime.
> Changing priority of a job should be available in CLI and available on the web UI only along with the Kill Job actions
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4124
> URL: https://issues.apache.org/jira/browse/HADOOP-4124
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.19.0
>
> Attachments: HADOOP-4124.patch, HADOOP-4124.patch, HADOOP-4124.patch, HADOOP-4124.patch
>
>
> Currently, a job's priority can only be changed from the job tracker web UI. However, it is really similar (if not as destructive) as killing a job, in the sense that is should not be exposed publicly. The use case for this kind of restriction is where Hadoop is shared in production environments by different users, and they should not be allowed to change another user's priority. At the same time, it should be possible to still change a job's priority in some manner, if a user is authorized to do so, and the CLI can provide a way to do this, along with mechanisms introduced for ACLs in HADOOP-3698.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4124) Changing priority of a job should be
available in CLI and available on the web UI only along with the Kill Job
actions
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-4124:
-------------------------------------
Attachment: HADOOP-4124.patch
Modified patch to incorporate Arun's comments.
> Changing priority of a job should be available in CLI and available on the web UI only along with the Kill Job actions
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4124
> URL: https://issues.apache.org/jira/browse/HADOOP-4124
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.19.0
>
> Attachments: HADOOP-4124.patch, HADOOP-4124.patch, HADOOP-4124.patch
>
>
> Currently, a job's priority can only be changed from the job tracker web UI. However, it is really similar (if not as destructive) as killing a job, in the sense that is should not be exposed publicly. The use case for this kind of restriction is where Hadoop is shared in production environments by different users, and they should not be allowed to change another user's priority. At the same time, it should be possible to still change a job's priority in some manner, if a user is authorized to do so, and the CLI can provide a way to do this, along with mechanisms introduced for ACLs in HADOOP-3698.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4124) Changing priority of a job should be
available in CLI and available on the web UI only along with the Kill Job
actions
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-4124:
-------------------------------------
Attachment: HADOOP-4124.patch
> Changing priority of a job should be available in CLI and available on the web UI only along with the Kill Job actions
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4124
> URL: https://issues.apache.org/jira/browse/HADOOP-4124
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.19.0
>
> Attachments: HADOOP-4124.patch
>
>
> Currently, a job's priority can only be changed from the job tracker web UI. However, it is really similar (if not as destructive) as killing a job, in the sense that is should not be exposed publicly. The use case for this kind of restriction is where Hadoop is shared in production environments by different users, and they should not be allowed to change another user's priority. At the same time, it should be possible to still change a job's priority in some manner, if a user is authorized to do so, and the CLI can provide a way to do this, along with mechanisms introduced for ACLs in HADOOP-3698.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4124) Changing priority of a job should
be available in CLI and available on the web UI only along with the Kill
Job actions
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631265#action_12631265 ]
Hemanth Yamijala commented on HADOOP-4124:
------------------------------------------
The attached patch implements the proposal above.
Specific changes:
- Introduced a new command line option in {{JobClient}} to change the priority
- Modified the Job list operation to display the Job Priority as a column so that the user can see the current priority. In order to do so, needed to introduce a field in the {{JobStatus}} class to hold the job's priority.
- Added a new API in {{JobSubmissionProtocol}} to change the job priority.
{code}
public void changeJobPriority(JobID jobid, String newPriority) throws IOException;
{code}
The priority is passed as a string matching the priority names. The help message of job client lists these options for ease of use.
- Implemented same in JobTracker to call the existing {{setJobPriority}}. This includes ACL checks
- Modified the web UI to show priority links only if {{webinterface.private.actions}} is set to true.
- Added new tests in {{TestJobClient}} to verify that job priority is being displayed, and in {{TestQueueManager}} to ensure job priority change is controlled by ACLs.
Can someone please review ?
> Changing priority of a job should be available in CLI and available on the web UI only along with the Kill Job actions
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4124
> URL: https://issues.apache.org/jira/browse/HADOOP-4124
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.19.0
>
> Attachments: HADOOP-4124.patch
>
>
> Currently, a job's priority can only be changed from the job tracker web UI. However, it is really similar (if not as destructive) as killing a job, in the sense that is should not be exposed publicly. The use case for this kind of restriction is where Hadoop is shared in production environments by different users, and they should not be allowed to change another user's priority. At the same time, it should be possible to still change a job's priority in some manner, if a user is authorized to do so, and the CLI can provide a way to do this, along with mechanisms introduced for ACLs in HADOOP-3698.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4124) Changing priority of a job should be
available in CLI and available on the web UI only along with the Kill Job
actions
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-4124:
-------------------------------------
Attachment: HADOOP-4124.patch
Fixes a findbugs warning in the previous patch due to an incorrect test case.
test-patch output:
[exec] +1 overall.
[exec] +1 @author. The patch does not contain any @author tags.
[exec] +1 tests included. The patch appears to include 6 new or modified tests.
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
> Changing priority of a job should be available in CLI and available on the web UI only along with the Kill Job actions
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4124
> URL: https://issues.apache.org/jira/browse/HADOOP-4124
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.19.0
>
> Attachments: HADOOP-4124.patch, HADOOP-4124.patch, HADOOP-4124.patch, HADOOP-4124.patch
>
>
> Currently, a job's priority can only be changed from the job tracker web UI. However, it is really similar (if not as destructive) as killing a job, in the sense that is should not be exposed publicly. The use case for this kind of restriction is where Hadoop is shared in production environments by different users, and they should not be allowed to change another user's priority. At the same time, it should be possible to still change a job's priority in some manner, if a user is authorized to do so, and the CLI can provide a way to do this, along with mechanisms introduced for ACLs in HADOOP-3698.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4124) Changing priority of a job should
be available in CLI and available on the web UI only along with the Kill
Job actions
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629404#action_12629404 ]
Hemanth Yamijala commented on HADOOP-4124:
------------------------------------------
Initial proposal:
- Introduce the following CLI:
{code}
hadoop job -change-priority <job-id> <new-priority>
{code}
This can be controlled via the ADMINISTER_JOBS ACL if they are turned on.
- Make the links for changing priority appear only if {{webinterface.private.actions}} is set to true.
Comments ?
> Changing priority of a job should be available in CLI and available on the web UI only along with the Kill Job actions
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4124
> URL: https://issues.apache.org/jira/browse/HADOOP-4124
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.19.0
>
>
> Currently, a job's priority can only be changed from the job tracker web UI. However, it is really similar (if not as destructive) as killing a job, in the sense that is should not be exposed publicly. The use case for this kind of restriction is where Hadoop is shared in production environments by different users, and they should not be allowed to change another user's priority. At the same time, it should be possible to still change a job's priority in some manner, if a user is authorized to do so, and the CLI can provide a way to do this, along with mechanisms introduced for ACLs in HADOOP-3698.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4124) Changing priority of a job should be
available in CLI and available on the web UI only along with the Kill Job
actions
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-4124:
-------------------------------------
Attachment: HADOOP-4124.patch
New patch merged with trunk. Adds a test case to test changing priority via job client. Fixes find bugs warnings. Rest of the implementation remains.
Here's the output of test-patch:
[exec] +1 overall.
[exec] +1 @author. The patch does not contain any @author tags.
[exec] +1 tests included. The patch appears to include 6 new or modified tests.
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
I am going to run tests to make sure nothing's broken.
Would really appreciate if this makes 0.19.
> Changing priority of a job should be available in CLI and available on the web UI only along with the Kill Job actions
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4124
> URL: https://issues.apache.org/jira/browse/HADOOP-4124
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.19.0
>
> Attachments: HADOOP-4124.patch, HADOOP-4124.patch
>
>
> Currently, a job's priority can only be changed from the job tracker web UI. However, it is really similar (if not as destructive) as killing a job, in the sense that is should not be exposed publicly. The use case for this kind of restriction is where Hadoop is shared in production environments by different users, and they should not be allowed to change another user's priority. At the same time, it should be possible to still change a job's priority in some manner, if a user is authorized to do so, and the CLI can provide a way to do this, along with mechanisms introduced for ACLs in HADOOP-3698.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4124) Changing priority of a job should
be available in CLI and available on the web UI only along with the Kill
Job actions
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12632538#action_12632538 ]
Arun C Murthy commented on HADOOP-4124:
---------------------------------------
Minor nits:
1. -change-priority should be '-set-priority' or '-priority'
2. 'changeJobPriority' should be 'setJobPriority'
3. JobStatus.setJobPriority should throw an IllegalArgumentException if passed-in 'priority' parameter is null.
> Changing priority of a job should be available in CLI and available on the web UI only along with the Kill Job actions
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4124
> URL: https://issues.apache.org/jira/browse/HADOOP-4124
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.19.0
>
> Attachments: HADOOP-4124.patch, HADOOP-4124.patch
>
>
> Currently, a job's priority can only be changed from the job tracker web UI. However, it is really similar (if not as destructive) as killing a job, in the sense that is should not be exposed publicly. The use case for this kind of restriction is where Hadoop is shared in production environments by different users, and they should not be allowed to change another user's priority. At the same time, it should be possible to still change a job's priority in some manner, if a user is authorized to do so, and the CLI can provide a way to do this, along with mechanisms introduced for ACLs in HADOOP-3698.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4124) Changing priority of a job should
be available in CLI and available on the web UI only along with the Kill
Job actions
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12633343#action_12633343 ]
Hudson commented on HADOOP-4124:
--------------------------------
Integrated in Hadoop-trunk #611 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/611/])
> Changing priority of a job should be available in CLI and available on the web UI only along with the Kill Job actions
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4124
> URL: https://issues.apache.org/jira/browse/HADOOP-4124
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.19.0
>
> Attachments: HADOOP-4124.patch, HADOOP-4124.patch, HADOOP-4124.patch, HADOOP-4124.patch
>
>
> Currently, a job's priority can only be changed from the job tracker web UI. However, it is really similar (if not as destructive) as killing a job, in the sense that is should not be exposed publicly. The use case for this kind of restriction is where Hadoop is shared in production environments by different users, and they should not be allowed to change another user's priority. At the same time, it should be possible to still change a job's priority in some manner, if a user is authorized to do so, and the CLI can provide a way to do this, along with mechanisms introduced for ACLs in HADOOP-3698.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4124) Changing priority of a job should
be available in CLI and available on the web UI only along with the Kill
Job actions
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12632634#action_12632634 ]
Hemanth Yamijala commented on HADOOP-4124:
------------------------------------------
Ran core tests and contrib tests on my machine.
Contrib tests all passed.
Following core tests failed:
org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement: This test timed out. There is no relation to this test in the patch.
Running org.apache.hadoop.mapred.TestLimitTasksPerJobTaskScheduler: Same as HADOOP-4213.
> Changing priority of a job should be available in CLI and available on the web UI only along with the Kill Job actions
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4124
> URL: https://issues.apache.org/jira/browse/HADOOP-4124
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.19.0
>
> Attachments: HADOOP-4124.patch, HADOOP-4124.patch, HADOOP-4124.patch, HADOOP-4124.patch
>
>
> Currently, a job's priority can only be changed from the job tracker web UI. However, it is really similar (if not as destructive) as killing a job, in the sense that is should not be exposed publicly. The use case for this kind of restriction is where Hadoop is shared in production environments by different users, and they should not be allowed to change another user's priority. At the same time, it should be possible to still change a job's priority in some manner, if a user is authorized to do so, and the CLI can provide a way to do this, along with mechanisms introduced for ACLs in HADOOP-3698.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.