You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Ivan A. Veselovsky (JIRA)" <ji...@apache.org> on 2012/11/01 17:55:16 UTC
[jira] [Created] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Ivan A. Veselovsky created MAPREDUCE-4764:
---------------------------------------------
Summary: repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Key: MAPREDUCE-4764
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Ivan A. Veselovsky
the test is @Ignore-ed, and fails being enabled.
Suggested to repair it to fill the coverage gap.
Problems fixed in the test:
(1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
(2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
(3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
(4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Ivan A. Veselovsky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan A. Veselovsky updated MAPREDUCE-4764:
------------------------------------------
Attachment: MAPREDUCE-4764-trunk.patch
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492715#comment-13492715 ]
Daryn Sharp commented on MAPREDUCE-4764:
----------------------------------------
I was mistaken. I thought the job token for the umbilical was not provided to the task's context, but apparently it is.
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Ivan A. Veselovsky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan A. Veselovsky updated MAPREDUCE-4764:
------------------------------------------
Status: Patch Available (was: Open)
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Ivan A. Veselovsky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan A. Veselovsky updated MAPREDUCE-4764:
------------------------------------------
Status: Open (was: Patch Available)
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Ivan A. Veselovsky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488903#comment-13488903 ]
Ivan A. Veselovsky commented on MAPREDUCE-4764:
-----------------------------------------------
The failed test org.apache.hadoop.mapred.TestClusterMRNotification.testMR appears to be flaky: it fails in many previous pre-commit builds.
Re-submitting the patch to try again.
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492436#comment-13492436 ]
Daryn Sharp commented on MAPREDUCE-4764:
----------------------------------------
bq. are we revealing an implementation bug, or our expectation is wrong?
I'm a bit confused about what you think might be a bug? Ie. What did you expect as output?
BTW, I think Bobby's right about the task not running in the same process, you'll have to pass the filename.
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Ivan A. Veselovsky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan A. Veselovsky updated MAPREDUCE-4764:
------------------------------------------
Attachment: MAPREDUCE-4764.patch
the patch is applicable to trunk, branch-2, branch-0.23.
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764.patch, MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496479#comment-13496479 ]
Hadoop QA commented on MAPREDUCE-4764:
--------------------------------------
{color:green}+1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12553341/MAPREDUCE-4764.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3024//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3024//console
This message is automatically generated.
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764.patch, MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Ivan A. Veselovsky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan A. Veselovsky updated MAPREDUCE-4764:
------------------------------------------
Attachment: (was: MAPREDUCE-4764-trunk.patch)
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Ivan A. Veselovsky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan A. Veselovsky updated MAPREDUCE-4764:
------------------------------------------
Status: Patch Available (was: Open)
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Ivan A. Veselovsky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan A. Veselovsky updated MAPREDUCE-4764:
------------------------------------------
Status: Open (was: Patch Available)
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504589#comment-13504589 ]
Hudson commented on MAPREDUCE-4764:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #1239 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1239/])
MAPREDUCE-4764. repair TestBinaryTokenFile (Ivan A. Veselovsky via bobby) (Revision 1413739)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1413739
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestBinaryTokenFile.java
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Fix For: 3.0.0, 2.0.3-alpha, 0.23.6
>
> Attachments: MAPREDUCE-4764.patch, MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489579#comment-13489579 ]
Robert Joseph Evans commented on MAPREDUCE-4764:
------------------------------------------------
Daryn,
For the hdfs mini cluster it is all running in the same process, but for MR it is not. They actually do launch separate processes, even though the NM and RM are running the same process the AM and tasks are not.
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504606#comment-13504606 ]
Hudson commented on MAPREDUCE-4764:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #1270 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1270/])
MAPREDUCE-4764. repair TestBinaryTokenFile (Ivan A. Veselovsky via bobby) (Revision 1413739)
Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1413739
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestBinaryTokenFile.java
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Fix For: 3.0.0, 2.0.3-alpha, 0.23.6
>
> Attachments: MAPREDUCE-4764.patch, MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489282#comment-13489282 ]
Hadoop QA commented on MAPREDUCE-4764:
--------------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12551827/MAPREDUCE-4764-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapred.TestClusterMRNotification
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2982//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2982//console
This message is automatically generated.
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504581#comment-13504581 ]
Hudson commented on MAPREDUCE-4764:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #448 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/448/])
svn merge -c 1413739 FIXES: MAPREDUCE-4764. repair TestBinaryTokenFile (Ivan A. Veselovsky via bobby) (Revision 1413742)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1413742
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestBinaryTokenFile.java
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Fix For: 3.0.0, 2.0.3-alpha, 0.23.6
>
> Attachments: MAPREDUCE-4764.patch, MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504521#comment-13504521 ]
Hudson commented on MAPREDUCE-4764:
-----------------------------------
Integrated in Hadoop-Yarn-trunk #49 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/49/])
MAPREDUCE-4764. repair TestBinaryTokenFile (Ivan A. Veselovsky via bobby) (Revision 1413739)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1413739
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestBinaryTokenFile.java
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Fix For: 3.0.0, 2.0.3-alpha, 0.23.6
>
> Attachments: MAPREDUCE-4764.patch, MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Ivan A. Veselovsky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491690#comment-13491690 ]
Ivan A. Veselovsky commented on MAPREDUCE-4764:
-----------------------------------------------
Hi, Daryn,
regarding your suggestion about UGI.getCurrebntUser():
if inside the job I execute the following code:
{noformat}
final UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
final Collection<Token<? extends TokenIdentifier>> ugiTokenCollection = ugi.getTokens();
for (Token t: ugiTokenCollection) {
System.out.println("UGI token: [" + t + "]");
}
{noformat}
, i receive the following output:
{noformat}
UGI token: [Kind: HDFS_DELEGATION_TOKEN, Service: 127.0.0.1:52538, Ident: (HDFS_DELEGATION_TOKEN token 1 for ivan)]
UGI token: [Kind: mapreduce.job, Service: 172.18.128.36:59358, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@3f56e5ed)]
{noformat}
So, are we revealing an implementation bug, or our expectation is wrong?
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Ivan A. Veselovsky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan A. Veselovsky updated MAPREDUCE-4764:
------------------------------------------
Status: Patch Available (was: Open)
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Ivan A. Veselovsky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan A. Veselovsky updated MAPREDUCE-4764:
------------------------------------------
Attachment: MAPREDUCE-4764-trunk.patch
The patch is applicable to all the main branches: trunk, branch-2, branch-0.23.
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489507#comment-13489507 ]
Daryn Sharp commented on MAPREDUCE-4764:
----------------------------------------
This test is a bit misleadingly named. It should have been called {{TestTokenPropagation}}. It overlaps with other tests, so I think it can be simplified:
# create credentials, fill with a few tokens
# create job with those creds and submit
# in the task, check that the tokens in the context AND the UGI.getCurrentUser() contain the tokens supplied at job submission. Notably the user's UGI should not have the job token, only what was submitted.
# since it's a minicluster, no need to write out the tokens and set a conf key for the task to read back in the tokens. it's all in the same process so just use a static class field to hold the creds used to submit the job.
Then do the same thing, but this time:
# write the creds out and set the binary token file key
# create a job with empty creds and submit
# do the same context and ugi checks as before
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495376#comment-13495376 ]
Daryn Sharp commented on MAPREDUCE-4764:
----------------------------------------
Sure. Just be sure to also check that the tokens are in the current user's UGI.
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503927#comment-13503927 ]
Hudson commented on MAPREDUCE-4764:
-----------------------------------
Integrated in Hadoop-trunk-Commit #3056 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3056/])
MAPREDUCE-4764. repair TestBinaryTokenFile (Ivan A. Veselovsky via bobby) (Revision 1413739)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1413739
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestBinaryTokenFile.java
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Fix For: 3.0.0, 2.0.3-alpha, 0.23.6
>
> Attachments: MAPREDUCE-4764.patch, MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488900#comment-13488900 ]
Hadoop QA commented on MAPREDUCE-4764:
--------------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12551730/MAPREDUCE-4764-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapred.TestClusterMRNotification
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2978//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2978//console
This message is automatically generated.
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Ivan A. Veselovsky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493959#comment-13493959 ]
Ivan A. Veselovsky commented on MAPREDUCE-4764:
-----------------------------------------------
Hi, Daryn,
I'd like to clarify our plan of improvements in this test.
Currently the test writes the token into a file, then sets the file name as MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY value in the config, and also passes the same file name as a value of a dedicated config property (KEY_SECURITY_TOKEN).
In the job: it gets the tokens from the job context (context.getCredentials().getAllTokens()), and gets the delegation token from there by the known key: let it be token X.
After that it gets the binary file name from the job config (key KEY_SECURITY_TOKEN), reads the file, de-serializing the token: let it be token Y.
Then the job asserts X.equals(Y).
This way the binary token propagation and serialization/de-serialization is checked, and this pretty much corresponds to the test name.
As I understand, you suggested to check also that the same delegation token is present in UserGroupInformation.getCurrentUser().getTokens(), right?
So, If I add this check, will you be okay with that test? Or, do you have other suggestions on how to improve it?
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Ivan A. Veselovsky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496391#comment-13496391 ]
Ivan A. Veselovsky commented on MAPREDUCE-4764:
-----------------------------------------------
the patch MAPREDUCE-4764 addresses the last comments: adds also checking of the UGI tokens.
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764.patch, MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503917#comment-13503917 ]
Robert Joseph Evans commented on MAPREDUCE-4764:
------------------------------------------------
The changes look good to me +1.
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764.patch, MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4764:
-------------------------------------------
Resolution: Fixed
Fix Version/s: 0.23.6
2.0.3-alpha
3.0.0
Status: Resolved (was: Patch Available)
Thanks Ivan,
I put this in trunk, branch-2, and branch-0.23
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Fix For: 3.0.0, 2.0.3-alpha, 0.23.6
>
> Attachments: MAPREDUCE-4764.patch, MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4764) repair test
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Posted by "Ivan A. Veselovsky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492534#comment-13492534 ]
Ivan A. Veselovsky commented on MAPREDUCE-4764:
-----------------------------------------------
Daryn, I meant the following your comment:
bq. "Notably the user's UGI should not have the job token, only what was submitted."
Please correct me if I misinterpreted something. "User's UGI" is not what we get with UserGroupInformation.getCurrentUser()?
> repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-4764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ivan A. Veselovsky
> Attachments: MAPREDUCE-4764-trunk.patch
>
>
> the test is @Ignore-ed, and fails being enabled.
> Suggested to repair it to fill the coverage gap.
> Problems fixed in the test:
> (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties must be correctly set in the configuration to correctly enable the security in the way this test implies.
> (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not passed into the Job configuration -- it is intentionally deleted from there. So, we pass the binary file name in another dedicated property.
> (3) The test was using deprecated cluster classes. All them are updated to the modern analogs.
> (4) The delegation token found in the job context is now correctly compared to the one deserialized from the binary file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira