You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org> on 2011/12/01 18:54:40 UTC
[jira] [Commented] (HADOOP-7733) Mapreduce jobs are failing when JT
has hadoop.security.token.service.use_ip=false and client has
hadoop.security.token.service.use_ip=true
[ https://issues.apache.org/jira/browse/HADOOP-7733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161004#comment-13161004 ]
Alejandro Abdelnur commented on HADOOP-7733:
--------------------------------------------
In some Oozie setups, I'm seeing this exception at tgt renewal time. This means, the original login is successful and access to HDFS is OK, but when renewal time comes this exception pops in the logs and HDFS access is no more. In other words, same same but different.
Wondering if this could be related to this misconfiguration.
> Mapreduce jobs are failing when JT has hadoop.security.token.service.use_ip=false and client has hadoop.security.token.service.use_ip=true
> ------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7733
> URL: https://issues.apache.org/jira/browse/HADOOP-7733
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
> Affects Versions: 0.20.205.0
> Reporter: Rajit Saha
> Assignee: Daryn Sharp
>
> I have added following property in core-site.xml of all the nodes in cluster and restarted
> <property>
> <name>hadoop.security.token.service.use_ip</name>
> <value>false</value>
> <description>desc</description>
> <final></final>
> </property>
> Then ran a randomwriter, distcp jobs, they are all failing
> $HADOOP_HOME/bin/hadoop --config $HADOOP_CONFIG_DIR jar $HADOOP_HOME/hadoop-examples.jar randomwriter -Dtest.randomwrite.bytes_per_map=256000 input_1318325953
> Running 140 maps.
> Job started: Tue Oct 11 09:48:09 UTC 2011
> 11/10/11 09:48:09 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 14 for <USERNAME> on <Namenode IP>:8020
> 11/10/11 09:48:09 INFO security.TokenCache: Got dt for
> hdfs://<Namenode Hostname>/user/<USERNAME>/.staging/job_201110110946_0001;uri=<Namenode IP>:8020;t.service=<Namenode IP>:8020
> 11/10/11 09:48:09 INFO mapred.JobClient: Cleaning up the staging area
> hdfs://<Namenode Hostname>/user/<USERNAME>/.staging/job_201110110946_0001
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.io.IOException: Call to
> <Namenode Hostname>/<Namenode IP>:8020 failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)]
> at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3943)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
> Caused by: java.io.IOException: Call to <Namenode Hostname>/<Namenode IP>:8020 failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid
> credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103)
> at org.apache.hadoop.ipc.Client.call(Client.java:1071)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> at $Proxy7.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
> at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:222)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:187)
> at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
> at org.apache.hadoop.mapred.JobInProgress$2.run(JobInProgress.java:401)
> at org.apache.hadoop.mapred.JobInProgress$2.run(JobInProgress.java:399)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:399)
> at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3941)
> ... 11 more
> Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No
> valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:539)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:484)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:586)
> at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202)
> at org.apache.hadoop.ipc.Client.call(Client.java:1046)
> ... 31 more
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials
> provided (Mechanism level: Failed to find any Kerberos tgt)]
> at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
> at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:134)
> at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:381)
> at org.apache.hadoop.ipc.Client$Connection.access$1100(Client.java:184)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:579)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:576)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:575)
> ... 34 more
> Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
> at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130)
> at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
> at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
> at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
> at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
> ... 43 more
> at org.apache.hadoop.ipc.Client.call(Client.java:1066)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> at org.apache.hadoop.mapred.$Proxy7.submitJob(Unknown Source)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:913)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:842)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:842)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:816)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1253)
> at org.apache.hadoop.examples.RandomWriter.run(RandomWriter.java:272)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.examples.RandomWriter.main(RandomWriter.java:283)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> At the sametime Jobtracker log says
> ===================================
> 2011-10-11 09:48:09,486 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)]
> 2011-10-11 09:48:09,487 INFO org.apache.hadoop.ipc.Server: IPC Server handler 26 on 50300, call
> submitJob(job_201110110946_0001, hdfs://<Namenode hostname> /user/<USER>/.staging/job_201110110946_0001,
> org.apache.hadoop.security.Credentials@16381a53) from <client IP> :46859: error: java.io.IOException:
> java.io.IOException: Call to <Namenode hostname> /<Namenode IP> :8020 failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid
> credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
> java.io.IOException: java.io.IOException: Call to <Namenode hostname> /<Namenode IP> :8020 failed on local
> exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No
> valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
> at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3943)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
> Caused by: java.io.IOException: Call to <Namenode hostname> /<Namenode IP> :8020 failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid
> credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103)
> at org.apache.hadoop.ipc.Client.call(Client.java:1071)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> at $Proxy7.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
> at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:222)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:187)
> at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
> at org.apache.hadoop.mapred.JobInProgress$2.run(JobInProgress.java:401)
> at org.apache.hadoop.mapred.JobInProgress$2.run(JobInProgress.java:399)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:399)
> at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3941)
> ... 11 more
> Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No
> valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:539)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:484)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:586)
> at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202)
> at org.apache.hadoop.ipc.Client.call(Client.java:1046)
> ... 31 more
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials
> provided (Mechanism level: Failed to find any Kerberos tgt)]
> at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
> at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:134)
> at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:381)
> at org.apache.hadoop.ipc.Client$Connection.access$1100(Client.java:184)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:579)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:576)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:575)
> ... 34 more
> Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
> at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130)
> at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
> at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
> at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
> at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
> ... 43 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira