You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "rahul k singh (JIRA)" <ji...@apache.org> on 2009/09/04 11:47:57 UTC
[jira] Created: (MAPREDUCE-953) Generate configuration dump for
hierarchial queue configuration
Generate configuration dump for hierarchial queue configuration
---------------------------------------------------------------
Key: MAPREDUCE-953
URL: https://issues.apache.org/jira/browse/MAPREDUCE-953
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: rahul k singh
Generate configuration dump for hierarchial queue configuration
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-953) Generate configuration dump for
hierarchial queue configuration
Posted by "V.V.Chaitanya Krishna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
V.V.Chaitanya Krishna updated MAPREDUCE-953:
--------------------------------------------
Attachment: MAPREDUCE-953-2.patch
Uploading the patch with the above suggested modifications done.
In order to dump acls_enabled, the json format is changed to the following:
{"acls_enabled":true, "queues":[
{"name":<queue_name>,"state":<queue_state>,"submit_job":<submit-job acls>,
"administer_jobs":<administer-jobs-acls>,"properties"<properties>,"children":[ {"name":<child queue name>,"state":<child_queue_state>, "submit_job":<child_submit-job acls>, "administer_jobs":<child_administer-jobs-acls>,"properties":<child_properties>,"children":[<child queues>] } ]
}
] }
> Generate configuration dump for hierarchial queue configuration
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-953
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: rahul k singh
> Assignee: V.V.Chaitanya Krishna
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-953-1.patch, MAPREDUCE-953-2.patch
>
>
> Generate configuration dump for hierarchial queue configuration
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-953) Generate configuration dump for
hierarchial queue configuration
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12757062#action_12757062 ]
Hudson commented on MAPREDUCE-953:
----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #49 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/49/])
. Fix QueueManager to dump queue configuration in JSON format. Contributed by V.V. Chaitanya Krishna.
> Generate configuration dump for hierarchial queue configuration
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-953
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: rahul k singh
> Assignee: V.V.Chaitanya Krishna
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-953-1.patch, MAPREDUCE-953-2.patch
>
>
> Generate configuration dump for hierarchial queue configuration
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-953) Generate configuration dump for
hierarchial queue configuration
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756976#action_12756976 ]
Hadoop QA commented on MAPREDUCE-953:
-------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12419909/MAPREDUCE-953-2.patch
against trunk revision 816439.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/47/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/47/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/47/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/47/console
This message is automatically generated.
> Generate configuration dump for hierarchial queue configuration
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-953
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: rahul k singh
> Assignee: V.V.Chaitanya Krishna
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-953-1.patch, MAPREDUCE-953-2.patch
>
>
> Generate configuration dump for hierarchial queue configuration
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-953) Generate configuration dump for
hierarchial queue configuration
Posted by "V.V.Chaitanya Krishna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
V.V.Chaitanya Krishna reassigned MAPREDUCE-953:
-----------------------------------------------
Assignee: V.V.Chaitanya Krishna
> Generate configuration dump for hierarchial queue configuration
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-953
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: rahul k singh
> Assignee: V.V.Chaitanya Krishna
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-953-1.patch
>
>
> Generate configuration dump for hierarchial queue configuration
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-953) Generate configuration dump for
hierarchial queue configuration
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated MAPREDUCE-953:
---------------------------------------
Component/s: jobtracker
Affects Version/s: 0.21.0
> Generate configuration dump for hierarchial queue configuration
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-953
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-953-1.patch
>
>
> Generate configuration dump for hierarchial queue configuration
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-953) Generate configuration dump for
hierarchial queue configuration
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated MAPREDUCE-953:
---------------------------------------
Fix Version/s: 0.21.0
> Generate configuration dump for hierarchial queue configuration
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-953
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-953-1.patch
>
>
> Generate configuration dump for hierarchial queue configuration
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-953) Generate configuration dump for
hierarchial queue configuration
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated MAPREDUCE-953:
---------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Chaitanya !
> Generate configuration dump for hierarchial queue configuration
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-953
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: rahul k singh
> Assignee: V.V.Chaitanya Krishna
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-953-1.patch, MAPREDUCE-953-2.patch
>
>
> Generate configuration dump for hierarchial queue configuration
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-953) Generate configuration dump for
hierarchial queue configuration
Posted by "V.V.Chaitanya Krishna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
V.V.Chaitanya Krishna updated MAPREDUCE-953:
--------------------------------------------
Attachment: MAPREDUCE-953-1.patch
Uploading patch taking the above mentioned format.
Proper test cases are also included.
> Generate configuration dump for hierarchial queue configuration
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-953
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPREDUCE-953-1.patch
>
>
> Generate configuration dump for hierarchial queue configuration
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-953) Generate configuration dump for
hierarchial queue configuration
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-953:
------------------------------------
Priority: Blocker (was: Major)
> Generate configuration dump for hierarchial queue configuration
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-953
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: rahul k singh
> Priority: Blocker
>
> Generate configuration dump for hierarchial queue configuration
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-953) Generate configuration dump for
hierarchial queue configuration
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated MAPREDUCE-953:
---------------------------------------
Status: Patch Available (was: Open)
This looks good to me. Passing through Hudson.
> Generate configuration dump for hierarchial queue configuration
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-953
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: rahul k singh
> Assignee: V.V.Chaitanya Krishna
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-953-1.patch, MAPREDUCE-953-2.patch
>
>
> Generate configuration dump for hierarchial queue configuration
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-953) Generate configuration dump for
hierarchial queue configuration
Posted by "V.V.Chaitanya Krishna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756426#action_12756426 ]
V.V.Chaitanya Krishna commented on MAPREDUCE-953:
-------------------------------------------------
The proposal of the JSONN string format is as follows:
{"queues":[
{"name":<queue_name>,"state":<queue_state>,"submit_job":<submit-job acls>,
"administer_jobs":<administer-jobs-acls>,"properties"<properties>,"children":[ {"name":<child queue name>,"state":<child_queue_state>,
"submit_job":<child_submit-job acls>, "administer_jobs":<child_administer-jobs-acls>,"properties":<child_properties>,"children":[<child queues>]
} ]
}
] }
Note: The <children queues> will be again in the same format as its parent is. So, it is a tree-like structure.
> Generate configuration dump for hierarchial queue configuration
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-953
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: rahul k singh
> Priority: Blocker
>
> Generate configuration dump for hierarchial queue configuration
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-953) Generate configuration dump for
hierarchial queue configuration
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756569#action_12756569 ]
Hemanth Yamijala commented on MAPREDUCE-953:
--------------------------------------------
Looked at the test case also. One comment I have is that we should verify the ACLs, properties and children in a different manner I think. Because right now, we are constructing the ACLs in the test case using similar code as in the actual code. This leads to a chance that we could make the same mistake at both places. I think it may be OK to hardcode what we are verifying, rather than writing generically. For e.g. we know what the queue configuration is. We know which queues have what ACLs and what properties, we can hardcode that verification. That way, the test will be simpler and it will be correct.
For e.g., now because both test and code make the same mistake about the ACL key, the test doesn't fail, but it doesn't catch the bug as well.
> Generate configuration dump for hierarchial queue configuration
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-953
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: rahul k singh
> Assignee: V.V.Chaitanya Krishna
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-953-1.patch
>
>
> Generate configuration dump for hierarchial queue configuration
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-953) Generate configuration dump for
hierarchial queue configuration
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756544#action_12756544 ]
Hemanth Yamijala commented on MAPREDUCE-953:
--------------------------------------------
- QueueManager.dumpConfiguration currently assumes that the new configuration is being used because it will call getQueueConfigurationParser with a null configuration, that loads the new QueueConfigurationParser. It should take the Configuration instance from JobTracker, and check if the queues are configured there. If yes, it should return silently as the config would have been dumped in the JT itself. Otherwise, it can use a QueueConfigurationParser
- We are not dumping whether acls are enabled or not.
- To get the ACLs from the ACLs map, you should use the API toFullPropertyName.
- We have no way to separate ACL user names and groups - it will be difficult for admins to separate them, Why not use the existing format itself: u1,u2 g1,g2 - will
having a comma be a problem for json ?
- At any rate the code to build the list of users and groups should be pulled into a separate utility API and called for both the ACLs.
- Use acl_submit_job and acl_administer_jobs as keys.
> Generate configuration dump for hierarchial queue configuration
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-953
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: rahul k singh
> Assignee: V.V.Chaitanya Krishna
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-953-1.patch
>
>
> Generate configuration dump for hierarchial queue configuration
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.