You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Yuren Wu (JIRA)" <ji...@apache.org> on 2016/05/18 14:16:12 UTC
[jira] [Updated] (HADOOP-13173) SecurityUtil.buildtokenService
broke token authentication in multi-home network env
[ https://issues.apache.org/jira/browse/HADOOP-13173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuren Wu updated HADOOP-13173:
------------------------------
Description:
Hadoop cluster: multihomed network with KMS and Kerberos.
Sympton: mapreduce mapper will fail with kerberos error. (stack trace later)
Let's assume hadoop cluster has two networks, PRIVATE_NET and PUBLIC_NET.
When mapreduce job submitted to the cluster from an outside ndoe (node only has visibility to PUBLIC_NET, it will acquire KMS tokens based on PUBLIC_NET ip address. Such token in Credential's token list using IP will not be correct resolved in the cluster. Therefore, we are using hadoop.security.token.service.use_ip=false in the client side core-site.xml file.
However, once job is accepted by yarn, the mapper container still throws kerberos error. A detailed trace shows that when DFSClient is calling decryptEncryptedDataencryptionKey function, it will reach to org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection method. This method will call SecurityUtil.buildtokenService to obtain the service name and use it to lookup the token in Credentials.getToken(service) call. This call returned the service in the IP:PORT format. Therefore, no token can be retrieved from credential's token hashmap.
After read SecurityUtil class, I think this class does not load configuration from configuration file at all. It seems to me that this class create a default configuration object and the useIpForTokenService variable is always set false.
I do not have enough experience to provide a fix. In my mind, somewhere we need to provide the correct value of useIpForTokenService based on job setup.
Stack trace provided here. this is from wordcount job from example.
2016-05-17 22:11:49,428 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:503)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:766)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:185)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:181)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:181)
at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1419)
at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1489)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:310)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:304)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:775)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:318)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:203)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:373)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:497)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:493)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:493)
... 22 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:297)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:275)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:275)
... 32 more
was:
Hadoop cluster: multihomed network with KMS and Kerberos.
Sympton: mapreduce mapper will fail with kerberos error. (stack trace later)
Let's assume hadoop cluster has two networks, PRIVATE_NET and PUBLIC_NET.
When mapreduce job submitted to the cluster from an outside ndoe (node only has visibility to PUBLIC_NET, it will acquire KMS tokens based on PUBLIC_NET ip address. Such token in Credential's token list using IP will not be correct resolved in the cluster. Therefore, we are using hadoop.security.token.service.use_ip=false in the client side core-site.xml file.
However, once job is accepted by yarn, the mapper container still throws kerberos error. A detailed trace shows that when DFSClient is calling decryptEncryptedDataencryptionKey function, it will reach to org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection method. This method will call SecurityUtil.buildtokenService to obtain the service name and use it to lookup the token in Credentials.getToken(service) call. This call returned the service in the IP:PORT format. Therefore, no token can be retrieved from credential's token hashmap.
After read SecurityUtil class, I think this class does not load configuration from configuration file at all. It seems to me that this class create a default configuration object and the useIpForTokenService variable is always set false.
I do not have enough experience to provide a fix. In my mind, somewhere we need to provide the correct value of useIpForTokenService based on job setup.
> SecurityUtil.buildtokenService broke token authentication in multi-home network env
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-13173
> URL: https://issues.apache.org/jira/browse/HADOOP-13173
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
> Affects Versions: 2.6.0
> Environment: multi-homed network cluster on SLES 11
> Reporter: Yuren Wu
>
> Hadoop cluster: multihomed network with KMS and Kerberos.
> Sympton: mapreduce mapper will fail with kerberos error. (stack trace later)
> Let's assume hadoop cluster has two networks, PRIVATE_NET and PUBLIC_NET.
> When mapreduce job submitted to the cluster from an outside ndoe (node only has visibility to PUBLIC_NET, it will acquire KMS tokens based on PUBLIC_NET ip address. Such token in Credential's token list using IP will not be correct resolved in the cluster. Therefore, we are using hadoop.security.token.service.use_ip=false in the client side core-site.xml file.
> However, once job is accepted by yarn, the mapper container still throws kerberos error. A detailed trace shows that when DFSClient is calling decryptEncryptedDataencryptionKey function, it will reach to org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection method. This method will call SecurityUtil.buildtokenService to obtain the service name and use it to lookup the token in Credentials.getToken(service) call. This call returned the service in the IP:PORT format. Therefore, no token can be retrieved from credential's token hashmap.
> After read SecurityUtil class, I think this class does not load configuration from configuration file at all. It seems to me that this class create a default configuration object and the useIpForTokenService variable is always set false.
> I do not have enough experience to provide a fix. In my mind, somewhere we need to provide the correct value of useIpForTokenService based on job setup.
> Stack trace provided here. this is from wordcount job from example.
> 2016-05-17 22:11:49,428 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
> at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:503)
> at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:766)
> at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:185)
> at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:181)
> at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94)
> at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:181)
> at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
> at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1419)
> at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1489)
> at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:310)
> at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:304)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:775)
> at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85)
> at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
> at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:318)
> at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:203)
> at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
> at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
> at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:373)
> at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:497)
> at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:493)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:493)
> ... 22 more
> Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
> at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
> at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
> at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:297)
> at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:275)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:275)
> ... 32 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org