You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Alejandro Abdelnur (Created) (JIRA)" <ji...@apache.org> on 2011/12/02 01:04:40 UTC
[jira] [Created] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
----------------------------------------------------------------------------------------------
Key: MAPREDUCE-3499
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 0.23.1, 0.24.0
Reporter: Alejandro Abdelnur
Priority: Critical
Fix For: 0.23.1, 0.24.0
The new MiniMR implementation is not taking proxyuser settings.
Because of this, testcases using/testing doAs functionality fail.
This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162669#comment-13162669 ]
Mahadev konar commented on MAPREDUCE-3499:
------------------------------------------
Sid,
I am not sure I understand 3. Why does the proxy user need to read the AppJar? When referring to the Proxy user, it is the user who is doing the doAs right? I like to use effective user and real user (real user is the user who does the doAs imitating other users and effective user is the one being imitated).
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Priority: Critical
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194918#comment-13194918 ]
Alejandro Abdelnur commented on MAPREDUCE-3499:
-----------------------------------------------
Hi Mahadev, any luck with this?
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Critical
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-3499:
-------------------------------------
Attachment: MR-3499.patch
I had forgotten to attach the test file in the previous patch. This patch has the test case.
I havent updated the patch that Alejandro suggested. John, you want to take it from here?
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Critical
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "John George (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195029#comment-13195029 ]
John George commented on MAPREDUCE-3499:
----------------------------------------
@Mahadev - I will incorporate your patch and submit a new one with the changes alejandro proposed
@Sid - How does this work in a real cluster? I mean when oozie submits a job as a proxy user, I dint think the user had to be the same group or user? Or am i wrong?
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Critical
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198003#comment-13198003 ]
Hudson commented on MAPREDUCE-3499:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #474 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/474/])
Merge -r 1239206:1239207 from trunk to branch. FIXES: MAPREDUCE-3499
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239210
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-3499:
-------------------------------------
Component/s: mrv2
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Priority: Critical
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197150#comment-13197150 ]
Hadoop QA commented on MAPREDUCE-3499:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12512598/MR-3499.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 3721 javac compiler warnings (more than the trunk's current 3706 warnings).
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1725//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1725//console
This message is automatically generated.
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198001#comment-13198001 ]
Hudson commented on MAPREDUCE-3499:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1648 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1648/])
MAPREDUCE-3499. New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work. (johnvijoe via tucu)
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239207
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "John George (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John George updated MAPREDUCE-3499:
-----------------------------------
Attachment: MR-3499.patch
Uploading a patch that passes the attached test by Alejandro. Not sure if this is the best way, hence comments or suggestions welcome
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Critical
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196726#comment-13196726 ]
Alejandro Abdelnur commented on MAPREDUCE-3499:
-----------------------------------------------
I'm getting a bit further, now I'm getting the following messages
{code}
22:04:34,129 WARN Configuration:639 - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
22:04:34,133 WARN Configuration:639 - fs.default.name is deprecated. Instead, use fs.defaultFS
22:04:34,135 WARN Configuration:639 - user.name is deprecated. Instead, use mapreduce.job.user.name
22:04:34,139 ERROR UserGroupInformation:1180 - PriviledgedActionException as:test (auth:PROXY) via tucu (auth:SIMPLE) cause:org.apache.hadoop.fs.UnsupportedFileSystemException: No AbstractFileSystem for scheme: hdfs
22:04:34,141 INFO Cluster:116 - Failed to use org.apache.hadoop.mapred.YarnClientProtocolProvider due to error: Error in instantiating YarnClient
22:04:34,141 ERROR UserGroupInformation:1180 - PriviledgedActionException as:test (auth:PROXY) via tucu (auth:SIMPLE) cause:java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
{code}
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Alejandro Abdelnur (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated MAPREDUCE-3499:
------------------------------------------
Attachment: TestForMAPREDUCE-3499.patch
The attached testcase should pass if proxyuser settings work.
Using the MR1 bridge API as I've written/verify the testcase with 0.20.20#.
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.23.1, 0.24.0
>
> Attachments: TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197967#comment-13197967 ]
Hudson commented on MAPREDUCE-3499:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #460 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/460/])
Merge -r 1239206:1239207 from trunk to branch. FIXES: MAPREDUCE-3499
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239210
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Siddharth Seth (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194996#comment-13194996 ]
Siddharth Seth commented on MAPREDUCE-3499:
-------------------------------------------
John, I took a very brief look at the patch. Changing the default history dir permissions is avoidable - they were specifically setup to prevent users having access to others history files. For MiniCluster, maybe these directories can pre-created with the required permissions?
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Critical
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196731#comment-13196731 ]
Mahadev konar commented on MAPREDUCE-3499:
------------------------------------------
@Alejandro,
Weird. This is coming from an unsupported file system. Is there someway I can try it out and debug on my own?
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "John George (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197045#comment-13197045 ]
John George commented on MAPREDUCE-3499:
----------------------------------------
Thanks for the update Alejandro. I will upload a patch with changes suggested by you, Sid and include Mahadev's patch in the next few minutes.
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197021#comment-13197021 ]
Alejandro Abdelnur commented on MAPREDUCE-3499:
-----------------------------------------------
Mahadev,
I was missing something, now I got things working. I think we are good with this patch for now.
I'm able to successfully submit job to MiniCluster from Oozie, I'm now hitting the Counter's API change, MAPREDUCE-3697
{code}
java.lang.NoSuchMethodError: org.apache.hadoop.mapred.Counters.getGroup(Ljava/lang/String;)Lorg/apache/hadoop/mapred/Counters$Group;
at org.apache.oozie.action.hadoop.LauncherMapper.hasIdSwap(LauncherMapper.java:273)
at org.apache.oozie.action.hadoop.TestMapReduceActionExecutor.testSetExecutionStats_when_user_has_specified_stats_write_FALSE(TestMapReduceActionExecutor.java:562)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at com.intellij.junit3.JUnit3IdeaTestRunner.doRun(JUnit3IdeaTestRunner.java:139)
at com.intellij.junit3.JUnit3IdeaTestRunner.startRunnerWithArgs(JUnit3IdeaTestRunner.java:52)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:199)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
{code}
Thanks.
Alejandro
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194326#comment-13194326 ]
Alejandro Abdelnur commented on MAPREDUCE-3499:
-----------------------------------------------
I've applied this patch locally and tested Oozie with it. The patch seems to do the trick to pass the proxyuser issue.
However I'm not running into the following error when trying run an Oozie testcase that submits a MR job to MiniMR:
{code}
testRecovery(org.apache.oozie.action.hadoop.TestJavaActionExecutor) Time elapsed: 0.994 sec <<< ERROR!
org.apache.oozie.action.ActionExecutorException: NullPointerException: null
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:386)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:670)
at org.apache.oozie.action.hadoop.TestJavaActionExecutor.submitAction(TestJavaActionExecutor.java:326)
at org.apache.oozie.action.hadoop.TestJavaActionExecutor.testRecovery(TestJavaActionExecutor.java:578)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.mapred.YARNRunner.getDelegationTokenFromHS(YARNRunner.java:193)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:270)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:379)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1212)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1212)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:611)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:606)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:606)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:655)
... 30 more
{code}
Is this still related or ...?
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Critical
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "John George (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193954#comment-13193954 ]
John George commented on MAPREDUCE-3499:
----------------------------------------
@Mona - I have a patch that is almost ready, I will post it sometime today after I run somoe more tests and verify the fix.
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Critical
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197960#comment-13197960 ]
Hudson commented on MAPREDUCE-3499:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1703 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1703/])
MAPREDUCE-3499. New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work. (johnvijoe via tucu)
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239207
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197358#comment-13197358 ]
Mahadev konar commented on MAPREDUCE-3499:
------------------------------------------
@Alejandro,
Any pointers to re create the counters issue? Trying to fix it but would help if I can test the fix easily via oozie.
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "John George (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John George updated MAPREDUCE-3499:
-----------------------------------
Status: Patch Available (was: Open)
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Critical
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194345#comment-13194345 ]
Mahadev konar commented on MAPREDUCE-3499:
------------------------------------------
I think I know what the issue above is. Ill upload an updated patch with the fix by tonight.
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Critical
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "John George (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John George updated MAPREDUCE-3499:
-----------------------------------
Attachment: MR-3499.patch
Sorry for that. Attaching a patch with all files included.
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197060#comment-13197060 ]
Mahadev konar commented on MAPREDUCE-3499:
------------------------------------------
Thanks Alejandro. I am working on a patch for MAPREDUCE-3697. Will update soon. Can you please verify John's latest patch?
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197572#comment-13197572 ]
Hadoop QA commented on MAPREDUCE-3499:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12512712/MR-3499.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1731//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1731//console
This message is automatically generated.
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197513#comment-13197513 ]
Alejandro Abdelnur commented on MAPREDUCE-3499:
-----------------------------------------------
Mahadev, John,
Apologies for the delay. I could not reproduce the problem (the Counters$Group related exception) until Tom gave me a hint and thus the solution. I'll update MAPREDUCE-3697 with my the details of the Counters$Group exception.
The problem I'm seeing now is that I'm getting different Hadoop built in counters, before I used to get a "Task$Counter" counter, now I'm getting "org.apache.hadoop.mapreduce.TaskCounter"
But other than that, things seem to work for what this patch is fixing.
Thanks.
Patch looks good. A couple of minor code-style things, YARNRunner changes have wrong indentation and the TestMiniMRProxyUser has some trailing spaces.
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198743#comment-13198743 ]
Hudson commented on MAPREDUCE-3499:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #944 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/944/])
MAPREDUCE-3499. New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work. (johnvijoe via tucu)
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239207
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Roman Shaposhnik (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roman Shaposhnik updated MAPREDUCE-3499:
----------------------------------------
Labels: bigtop (was: )
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Priority: Critical
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198781#comment-13198781 ]
Hudson commented on MAPREDUCE-3499:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #179 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/179/])
Merge -r 1239206:1239207 from trunk to branch. FIXES: MAPREDUCE-3499
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239210
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Siddharth Seth (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161340#comment-13161340 ]
Siddharth Seth commented on MAPREDUCE-3499:
-------------------------------------------
Tucu, after MR-3443, proxy users should work - at least the doAs part.
There's several issues after that
1. MiniMRYarnCluster - set the default app staging dir to the test Working dir. Ran into
testValidProxyUser(org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser): Permission denied: user=u1, access=EXECUTE, inode="/home":<REPLACED_USER>:supergroup:drwxr-x---
getJobConf needs -> conf.set(MRJobConfig.MR_AM_STAGING_DIR, "/tmp");
2. The host list for the proxy user - needs to include the host IP, instead of just "localhost"
3. After this, the proxy user cannot read the AppJar - since it was written out by the original user when creating the cluster (MiniMRClientClusterFactory).
4. Another possible issue after this, would be creation of additional dirs for job history on hdfs.
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Priority: Critical
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "John George (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John George updated MAPREDUCE-3499:
-----------------------------------
Attachment: MR-3499.patch
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "John George (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John George reassigned MAPREDUCE-3499:
--------------------------------------
Assignee: John George
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Critical
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "John George (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John George updated MAPREDUCE-3499:
-----------------------------------
Status: Open (was: Patch Available)
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Critical
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198815#comment-13198815 ]
Hudson commented on MAPREDUCE-3499:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #977 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/977/])
MAPREDUCE-3499. New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work. (johnvijoe via tucu)
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239207
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Mona Chitnis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191827#comment-13191827 ]
Mona Chitnis commented on MAPREDUCE-3499:
-----------------------------------------
I'm hitting the problem 1
testValidProxyUser(org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser): org.apache.hadoop.security.AccessControlException: Permission denied: user=u1, access=EXECUTE, inode="/home":<REPLACED_USER>:supergroup:drwxr-x---
Is the patch for this JIRA underway?
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Critical
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194138#comment-13194138 ]
Alejandro Abdelnur commented on MAPREDUCE-3499:
-----------------------------------------------
John,
A couple of things:
#1
You changed the setting for the staging dir of *MiniMRYarnCluster* to '/tmp'. Why not the original resolution to the getTestWorkingDir() ? I know it resolves to 'target/...' and in projects like Pig/Hive which use ant and 'build/' dir this would break; but should you create a fresh dir under /tmp then? (using the File.createTempFile() method -you have to create the file, then delete it and then create a dir with that name).
#2
I had to tweak the test (how to create the list of valid hosts) to get it to work:
Instead doing *conf.set("hadoop.proxyuser." + proxyUser + ".hosts", "10.74.90.219,127.0.0.1");* I've done:
{code}
StringBuilder sb = new StringBuilder();
sb.append("127.0.0.1,localhost");
for (InetAddress i : InetAddress.getAllByName(InetAddress.getLocalHost().getHostName())) {
sb.append(",").append(i.getCanonicalHostName());
}
conf.set("hadoop.proxyuser." + proxyUser + ".hosts", sb.toString());
{code}
And then the test works.
Thanks.
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Critical
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197916#comment-13197916 ]
Alejandro Abdelnur commented on MAPREDUCE-3499:
-----------------------------------------------
+1
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "John George (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John George updated MAPREDUCE-3499:
-----------------------------------
Status: Patch Available (was: Open)
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197598#comment-13197598 ]
Alejandro Abdelnur commented on MAPREDUCE-3499:
-----------------------------------------------
John, the patch is missing the testcase.
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "John George (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John George updated MAPREDUCE-3499:
-----------------------------------
Attachment: MR-3499.patch
Thanks Alejandro. Removed trailing spaces from Test program. I could not see any indentation issues in YARNRunner.
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197971#comment-13197971 ]
Hudson commented on MAPREDUCE-3499:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #450 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/450/])
Merge -r 1239206:1239207 from trunk to branch. FIXES: MAPREDUCE-3499
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239210
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194113#comment-13194113 ]
Hadoop QA commented on MAPREDUCE-3499:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12512020/MR-3499.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 3732 javac compiler warnings (more than the trunk's current 3717 warnings).
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1687//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1687//console
This message is automatically generated.
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Critical
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198760#comment-13198760 ]
Hudson commented on MAPREDUCE-3499:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #157 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/157/])
Merge -r 1239206:1239207 from trunk to branch. FIXES: MAPREDUCE-3499
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239210
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195032#comment-13195032 ]
Alejandro Abdelnur commented on MAPREDUCE-3499:
-----------------------------------------------
Mahadev's tweaks get me a bit futher, but I'm still hitting some other issues. still digging.
Thanks.
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Critical
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197973#comment-13197973 ]
Hudson commented on MAPREDUCE-3499:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1632 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1632/])
MAPREDUCE-3499. New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work. (johnvijoe via tucu)
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239207
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-3499:
-------------------------------------
Attachment: MR-3499.patch
Updated the patch with my changes on top of John's changes.
@Alejandro, want to give this a try?
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Critical
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Alejandro Abdelnur (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated MAPREDUCE-3499:
------------------------------------------
Resolution: Fixed
Fix Version/s: (was: 0.24.0)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks John & Mahadev for the work on this one. Committed to trunk and branch-0.23
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196613#comment-13196613 ]
Mahadev konar commented on MAPREDUCE-3499:
------------------------------------------
@Alejandro,
Any more details on the issues?
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "John George (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John George updated MAPREDUCE-3499:
-----------------------------------
Priority: Blocker (was: Critical)
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: John George
> Priority: Blocker
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup
proxyuser configuration correctly, thus tests using doAs do not work
Posted by "Siddharth Seth (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163063#comment-13163063 ]
Siddharth Seth commented on MAPREDUCE-3499:
-------------------------------------------
The appjar would need to be read by the proxy (effective) user - so that it can be localized on the NM.
4. Based on the johistory dir configuration - a single effective user may run, additional effective users per cluster could run into jobhistory dir permission issues.
> New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Priority: Critical
> Labels: bigtop
> Fix For: 0.23.1, 0.24.0
>
> Attachments: TestForMAPREDUCE-3499.patch
>
>
> The new MiniMR implementation is not taking proxyuser settings.
> Because of this, testcases using/testing doAs functionality fail.
> This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira