You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Harsh J (JIRA)" <ji...@apache.org> on 2012/06/07 18:27:22 UTC
[jira] [Created] (MAPREDUCE-4324) JobClient can perhaps set
mapreduce.job.credentials.binary rather than expect its presence?
Harsh J created MAPREDUCE-4324:
----------------------------------
Summary: JobClient can perhaps set mapreduce.job.credentials.binary rather than expect its presence?
Key: MAPREDUCE-4324
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4324
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1, mrv2
Affects Versions: 2.0.0-alpha, 0.22.0
Reporter: Harsh J
Assignee: Harsh J
HDFS-1007 added in this requirement property "mapreduce.job.credentials.binary", that has lead Oozie to add the following duplicate snippet to all its Job-launching main classes such as the Pig, Hive, MR and Sqoop actions:
{code}
if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
jobConf.set("mapreduce.job.credentials.binary", System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
}
{code}
Same is required for any client program that launches a job from within a task.
Why can't this simply be set by the JobClient initialization bits itself? If no one imagines it causing issues, I'd like to add this snippet somewhere in JobSubmitter before it requests NN/JT, as otherwise we'd get…
{code}
org.apache.hadoop.ipc.RemoteException: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:5509)
at org.apache.hadoop.hdfs.server.namenode.NameNode.getDelegationToken(NameNode.java:536)
at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
at org.apache.hadoop.ipc.Client.call(Client.java:1107)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
at $Proxy6.getDelegationToken(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy6.getDelegationToken(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:331)
at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:605)
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:115)
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:79)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:851)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1242)
{code}
… or similar errors when a user submits a job from a task running in a secured cluster.
Let me know your thoughts on this!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4324) JobClient can perhaps set
mapreduce.job.credentials.binary rather than expect its presence?
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404589#comment-13404589 ]
Hadoop QA commented on MAPREDUCE-4324:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12534118/MAPREDUCE-4324.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2537//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2537//console
This message is automatically generated.
> JobClient can perhaps set mapreduce.job.credentials.binary rather than expect its presence?
> -------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4324
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4324
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2, security
> Affects Versions: 0.22.0, 2.0.0-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: MAPREDUCE-4324.patch
>
>
> HDFS-1007 added in this requirement property "mapreduce.job.credentials.binary", that has lead Oozie to add the following duplicate snippet to all its Job-launching main classes such as the Pig, Hive, MR and Sqoop actions:
> {code}
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary", System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> {code}
> Same is required for any client program that launches a job from within a task.
> Why can't this simply be set by the JobClient initialization bits itself? If no one imagines it causing issues, I'd like to add this snippet somewhere in JobSubmitter before it requests NN/JT, as otherwise we'd get…
> {code}
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:5509)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.getDelegationToken(NameNode.java:536)
> at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
> at $Proxy6.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy6.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:331)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:605)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:115)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:79)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:851)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1242)
> {code}
> … or similar errors when a user submits a job from a task running in a secured cluster.
> Let me know your thoughts on this!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4324) JobClient can perhaps set
mapreduce.job.credentials.binary rather than expect its presence?
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-4324:
-------------------------------
Component/s: security
> JobClient can perhaps set mapreduce.job.credentials.binary rather than expect its presence?
> -------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4324
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4324
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2, security
> Affects Versions: 0.22.0, 2.0.0-alpha
> Reporter: Harsh J
> Assignee: Harsh J
>
> HDFS-1007 added in this requirement property "mapreduce.job.credentials.binary", that has lead Oozie to add the following duplicate snippet to all its Job-launching main classes such as the Pig, Hive, MR and Sqoop actions:
> {code}
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary", System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> {code}
> Same is required for any client program that launches a job from within a task.
> Why can't this simply be set by the JobClient initialization bits itself? If no one imagines it causing issues, I'd like to add this snippet somewhere in JobSubmitter before it requests NN/JT, as otherwise we'd get…
> {code}
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:5509)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.getDelegationToken(NameNode.java:536)
> at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
> at $Proxy6.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy6.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:331)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:605)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:115)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:79)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:851)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1242)
> {code}
> … or similar errors when a user submits a job from a task running in a secured cluster.
> Let me know your thoughts on this!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4324) JobClient can perhaps set
mapreduce.job.credentials.binary rather than expect its presence?
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291137#comment-13291137 ]
Ashutosh Chauhan commented on MAPREDUCE-4324:
---------------------------------------------
As someone working on higher up the stack, I have seen this {{if}} code block in all the clients. Ideally, {{jobclient}} should do it, freeing apps from this unnecessary requirement. Thanks, Harsh for picking this up!
> JobClient can perhaps set mapreduce.job.credentials.binary rather than expect its presence?
> -------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4324
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4324
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2, security
> Affects Versions: 0.22.0, 2.0.0-alpha
> Reporter: Harsh J
> Assignee: Harsh J
>
> HDFS-1007 added in this requirement property "mapreduce.job.credentials.binary", that has lead Oozie to add the following duplicate snippet to all its Job-launching main classes such as the Pig, Hive, MR and Sqoop actions:
> {code}
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary", System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> {code}
> Same is required for any client program that launches a job from within a task.
> Why can't this simply be set by the JobClient initialization bits itself? If no one imagines it causing issues, I'd like to add this snippet somewhere in JobSubmitter before it requests NN/JT, as otherwise we'd get…
> {code}
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:5509)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.getDelegationToken(NameNode.java:536)
> at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
> at $Proxy6.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy6.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:331)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:605)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:115)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:79)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:851)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1242)
> {code}
> … or similar errors when a user submits a job from a task running in a secured cluster.
> Let me know your thoughts on this!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4324) JobClient can perhaps set
mapreduce.job.credentials.binary rather than expect its presence?
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-4324:
-------------------------------
Attachment: MAPREDUCE-4324.patch
Here's a simple attempt, which modifies JobClient's initializer method alone.
With YARNRunner involved (via JobSubmitter, which ignores conf when sought), I am unsure where to place it. I _think_ its sufficient for JobClient alone to have it, but if the code has diverged much beyond JobClient paths when using Job (new API), then this patch may not suffice.
> JobClient can perhaps set mapreduce.job.credentials.binary rather than expect its presence?
> -------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4324
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4324
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2, security
> Affects Versions: 0.22.0, 2.0.0-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: MAPREDUCE-4324.patch
>
>
> HDFS-1007 added in this requirement property "mapreduce.job.credentials.binary", that has lead Oozie to add the following duplicate snippet to all its Job-launching main classes such as the Pig, Hive, MR and Sqoop actions:
> {code}
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary", System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> {code}
> Same is required for any client program that launches a job from within a task.
> Why can't this simply be set by the JobClient initialization bits itself? If no one imagines it causing issues, I'd like to add this snippet somewhere in JobSubmitter before it requests NN/JT, as otherwise we'd get…
> {code}
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:5509)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.getDelegationToken(NameNode.java:536)
> at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
> at $Proxy6.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy6.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:331)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:605)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:115)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:79)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:851)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1242)
> {code}
> … or similar errors when a user submits a job from a task running in a secured cluster.
> Let me know your thoughts on this!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4324) JobClient can perhaps set
mapreduce.job.credentials.binary rather than expect its presence?
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-4324:
-------------------------------
Resolution: Won't Fix
Status: Resolved (was: Patch Available)
Doing this would break (or be defunct due to) MAPREDUCE-3727 I think. Resolving as Won't Fix at this point.
Users should continue manually configuring this.
> JobClient can perhaps set mapreduce.job.credentials.binary rather than expect its presence?
> -------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4324
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4324
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2, security
> Affects Versions: 0.22.0, 2.0.0-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: MAPREDUCE-4324.patch
>
>
> HDFS-1007 added in this requirement property "mapreduce.job.credentials.binary", that has lead Oozie to add the following duplicate snippet to all its Job-launching main classes such as the Pig, Hive, MR and Sqoop actions:
> {code}
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary", System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> {code}
> Same is required for any client program that launches a job from within a task.
> Why can't this simply be set by the JobClient initialization bits itself? If no one imagines it causing issues, I'd like to add this snippet somewhere in JobSubmitter before it requests NN/JT, as otherwise we'd get…
> {code}
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:5509)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.getDelegationToken(NameNode.java:536)
> at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
> at $Proxy6.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy6.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:331)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:605)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:115)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:79)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:851)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1242)
> {code}
> … or similar errors when a user submits a job from a task running in a secured cluster.
> Let me know your thoughts on this!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4324) JobClient can perhaps set
mapreduce.job.credentials.binary rather than expect its presence?
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-4324:
-------------------------------
Status: Patch Available (was: Open)
> JobClient can perhaps set mapreduce.job.credentials.binary rather than expect its presence?
> -------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4324
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4324
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2, security
> Affects Versions: 2.0.0-alpha, 0.22.0
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: MAPREDUCE-4324.patch
>
>
> HDFS-1007 added in this requirement property "mapreduce.job.credentials.binary", that has lead Oozie to add the following duplicate snippet to all its Job-launching main classes such as the Pig, Hive, MR and Sqoop actions:
> {code}
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary", System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> {code}
> Same is required for any client program that launches a job from within a task.
> Why can't this simply be set by the JobClient initialization bits itself? If no one imagines it causing issues, I'd like to add this snippet somewhere in JobSubmitter before it requests NN/JT, as otherwise we'd get…
> {code}
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:5509)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.getDelegationToken(NameNode.java:536)
> at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
> at $Proxy6.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy6.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:331)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:605)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:115)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:79)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:851)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1242)
> {code}
> … or similar errors when a user submits a job from a task running in a secured cluster.
> Let me know your thoughts on this!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira