You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Xiaoyu Yao (JIRA)" <ji...@apache.org> on 2019/03/18 19:06:00 UTC
[jira] [Comment Edited] (HADOOP-16199) KMSLoadBlanceClientProvider
does not select token correctly
[ https://issues.apache.org/jira/browse/HADOOP-16199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795315#comment-16795315 ]
Xiaoyu Yao edited comment on HADOOP-16199 at 3/18/19 7:05 PM:
--------------------------------------------------------------
As can be seen below, a kms-dt with service field *Service: kms://[http@c316-node3.raghav.com|mailto:http@c316-node3.raghav.com];[c316-node4.raghav.com:9292/kms|http://c316-node4.raghav.com:9292/kms]* can't be selected for LoadBalancingKMSClientProvider because it does not match its *canonical service: 172.25.36.130:9292*. Subsequent matching with individual KMSClientProvider also failed in this case. The proposed fix it to allow LoadBalancingKMSClientProvider#selectDelegationToken to match not only the canonical service but also the delegation token service.
Also, the comments on reason of hard code canonical service of LoadBalancingKMSClientProvider to the ip:port of the first KMSClientProvider instance can be improved.
Below is detailed failure log for reference:
{code:java}
2019-03-13 18:51:33,056 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.ipc.ProtobufRpcEngine: Call: getServerDefaults took 5ms
2019-03-13 18:51:33,086 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: KMSClientProvider created for KMS url: [http://c316-node3.raghav.com:9292/kms/v1/] delegation token service: [kms://http@c316-node3].[raghav.com:9292/kms|http://raghav.com:9292/kms]canonical service: 172.25.36.130:9292.
2019-03-13 18:51:33,087 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: KMSClientProvider created for KMS url: [http://c316-node4.raghav.com:9292/kms/v1/] delegation token service: [kms://http@c316-node4].[raghav.com:9292/kms|http://raghav.com:9292/kms]canonical service: 172.25.38.4:9292.
2019-03-13 18:51:33,089 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider: Created LoadBalancingKMSClientProvider for KMS url: kms://[http@c316-node3.raghav.com|mailto:http@c316-node3.raghav.com];[c316-node4.raghav.com:9292/kms|http://c316-node4.raghav.com:9292/kms] with 2 providers. delegation token service: kms://[http@c316-node3.raghav.com|mailto:http@c316-node3.raghav.com];[c316-node4.raghav.com:9292/kms|http://c316-node4.raghav.com:9292/kms], canonical service: 172.25.36.130:9292
...
2019-03-13 18:51:33,112 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: Current UGI: hr1 (auth:SIMPLE)
2019-03-13 18:51:33,141 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: Localizer, Service: , Ident: (org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.security.LocalizerTokenIdentifier@54604a95)
*2019-03-13 18:51:33,141 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: kms-dt, Service: kms://[http@c316-node3.raghav.com|mailto:http@c316-node3.raghav.com];[c316-node4.raghav.com:9292/kms|http://c316-node4.raghav.com:9292/kms], Ident: (kms-dt owner=hr1, renewer=yarn, realUser=oozie, issueDate=1552503090542, maxDate=1553107890542, sequenceNumber=27, masterKeyId=30)*
2019-03-13 18:51:33,142 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: HDFS_DELEGATION_TOKEN, Service: 172.25.35.133:8020, Ident: (token for hr1: HDFS_DELEGATION_TOKEN owner=hr1, renewer=yarn, realUser=oozie/c316-node1.[raghav.com@RAGHAV.COM|mailto:raghav.com@RAGHAV.COM], issueDate=1552503090263, maxDate=1553107890263, sequenceNumber=443, masterKeyId=93)
2019-03-13 18:51:33,142 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.token.Token: Cannot find class for token kind HIVE_DELEGATION_TOKEN
2019-03-13 18:51:33,142 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: HIVE_DELEGATION_TOKEN, Service: hiveserver2ClientToken, Ident: 00 03 68 72 31 04 68 69 76 65 05 6f 6f 7a 69 65 8a 01 69 78 65 2b a8 8a 01 69 9c 71 af a8 03 8f 84
2019-03-13 18:51:33,143 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: RM_DELEGATION_TOKEN, Service: 172.25.35.133:8050, Ident: (RM_DELEGATION_TOKEN owner=hr1, renewer=yarn, realUser=oozie/c316-node1.[raghav.com@RAGHAV.COM|mailto:raghav.com@RAGHAV.COM], issueDate=1552503090238, maxDate=1553107890238, sequenceNumber=21, masterKeyId=139)
2019-03-13 18:51:33,143 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: MR_DELEGATION_TOKEN, Service: 172.25.35.133:10020, Ident: (MR_DELEGATION_TOKEN owner=hr1, renewer=yarn, realUser=oozie/c316-node1.[raghav.com@RAGHAV.COM|mailto:raghav.com@RAGHAV.COM], issueDate=1552503090488, maxDate=1553107890488, sequenceNumber=5, masterKeyId=107)
2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: Login UGI: hr1 (auth:SIMPLE)
2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: Searching for KMS delegation token in user hr1 (auth:SIMPLE)'s credentials
2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by alias=172.25.36.130:9292 token=null
2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by service=172.25.36.130:9292 token=null
2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by alias=[kms://http@c316-node4].[raghav.com:9292/kms|http://raghav.com:9292/kms]token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by service=[kms://http@c316-node4].[raghav.com:9292/kms|http://raghav.com:9292/kms]token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by alias=172.25.38.4:9292 token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by service=172.25.38.4:9292 token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by alias=[kms://http@c316-node3].[raghav.com:9292/kms|http://raghav.com:9292/kms]token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by service=[kms://http@c316-node3].[raghav.com:9292/kms|http://raghav.com:9292/kms]token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by alias=172.25.36.130:9292 token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by service=172.25.36.130:9292 token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: Using loginUser when Kerberos is enabled but the actual user does not have either KMS Delegation Token or Kerberos Credentials
2019-03-13 18:51:33,146 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.UserGroupInformation: PrivilegedAction as:hr1 (auth:SIMPLE) from:org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:506)
2019-03-13 18:51:33,150 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL: Connecting to url [http://c316-node4.raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt]with token as null
2019-03-13 18:51:33,150 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL: Token not set, looking for delegation token. Creds:[], size:0
2019-03-13 18:51:33,150 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator: No delegation token found for url=[http://c316-node4|http://c316-node4/].[raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt|http://raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt], token=, authenticating with class org.apache.hadoop.security.token.delegation.web.KerberosDelegationTokenAuthenticator$1
2019-03-13 18:51:33,178 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.authentication.client.KerberosAuthenticator: Performing our own SPNEGO sequence.
2019-03-13 18:51:33,179 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.authentication.client.KerberosAuthenticator: No subject in context, logging in
2019-03-13 18:51:33,179 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.authentication.client.KerberosAuthenticator: Using subject: Subject:
Principal: UnixPrincipal: hr1
Principal: UnixNumericUserPrincipal: 1016
Principal: UnixNumericGroupPrincipal [Primary Group]: 1016
2019-03-13 18:51:33,182 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.UserGroupInformation: PrivilegedActionException as:hr1 (auth:SIMPLE) cause:org.apache.hadoop.security.authentication.client.AuthenticationException: Error while authenticating with endpoint: [http://c316-node4.raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt]
2019-03-13 18:51:33,182 WARN [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider: KMS provider at [[http://c316-node4.raghav.com:9292/kms/v1/]] threw an IOException:
java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Error while authenticating with endpoint: [http://c316-node4.raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt]
{code}
was (Author: xyao):
As can be seen below, a kms-dt with service field *Service: kms://[http@c316-node3.raghav.com|mailto:http@c316-node3.raghav.com];[c316-node4.raghav.com:9292/kms|http://c316-node4.raghav.com:9292/kms]* can't be selected for LoadBalancingKMSClientProvider because it does not match its *canonical service: 172.25.36.130:9292*. Subsequent matching with individual KMSClientProvider also failed in this case. The proposed fix it to allow LoadBalancingKMSClientProvider#selectDelegationToken to match not only the canonical service but also the delegation token service.
Below is detailed failure log for reference:
{code}
2019-03-13 18:51:33,056 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.ipc.ProtobufRpcEngine: Call: getServerDefaults took 5ms
2019-03-13 18:51:33,086 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: KMSClientProvider created for KMS url: [http://c316-node3.raghav.com:9292/kms/v1/] delegation token service: [kms://http@c316-node3].[raghav.com:9292/kms|http://raghav.com:9292/kms]canonical service: 172.25.36.130:9292.
2019-03-13 18:51:33,087 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: KMSClientProvider created for KMS url: [http://c316-node4.raghav.com:9292/kms/v1/] delegation token service: [kms://http@c316-node4].[raghav.com:9292/kms|http://raghav.com:9292/kms]canonical service: 172.25.38.4:9292.
2019-03-13 18:51:33,089 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider: Created LoadBalancingKMSClientProvider for KMS url: kms://[http@c316-node3.raghav.com|mailto:http@c316-node3.raghav.com];[c316-node4.raghav.com:9292/kms|http://c316-node4.raghav.com:9292/kms] with 2 providers. delegation token service: kms://[http@c316-node3.raghav.com|mailto:http@c316-node3.raghav.com];[c316-node4.raghav.com:9292/kms|http://c316-node4.raghav.com:9292/kms], canonical service: 172.25.36.130:9292
...
2019-03-13 18:51:33,112 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: Current UGI: hr1 (auth:SIMPLE)
2019-03-13 18:51:33,141 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: Localizer, Service: , Ident: (org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.security.LocalizerTokenIdentifier@54604a95)
*2019-03-13 18:51:33,141 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: kms-dt, Service: kms://[http@c316-node3.raghav.com|mailto:http@c316-node3.raghav.com];[c316-node4.raghav.com:9292/kms|http://c316-node4.raghav.com:9292/kms], Ident: (kms-dt owner=hr1, renewer=yarn, realUser=oozie, issueDate=1552503090542, maxDate=1553107890542, sequenceNumber=27, masterKeyId=30)*
2019-03-13 18:51:33,142 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: HDFS_DELEGATION_TOKEN, Service: 172.25.35.133:8020, Ident: (token for hr1: HDFS_DELEGATION_TOKEN owner=hr1, renewer=yarn, realUser=oozie/c316-node1.[raghav.com@RAGHAV.COM|mailto:raghav.com@RAGHAV.COM], issueDate=1552503090263, maxDate=1553107890263, sequenceNumber=443, masterKeyId=93)
2019-03-13 18:51:33,142 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.token.Token: Cannot find class for token kind HIVE_DELEGATION_TOKEN
2019-03-13 18:51:33,142 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: HIVE_DELEGATION_TOKEN, Service: hiveserver2ClientToken, Ident: 00 03 68 72 31 04 68 69 76 65 05 6f 6f 7a 69 65 8a 01 69 78 65 2b a8 8a 01 69 9c 71 af a8 03 8f 84
2019-03-13 18:51:33,143 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: RM_DELEGATION_TOKEN, Service: 172.25.35.133:8050, Ident: (RM_DELEGATION_TOKEN owner=hr1, renewer=yarn, realUser=oozie/c316-node1.[raghav.com@RAGHAV.COM|mailto:raghav.com@RAGHAV.COM], issueDate=1552503090238, maxDate=1553107890238, sequenceNumber=21, masterKeyId=139)
2019-03-13 18:51:33,143 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: MR_DELEGATION_TOKEN, Service: 172.25.35.133:10020, Ident: (MR_DELEGATION_TOKEN owner=hr1, renewer=yarn, realUser=oozie/c316-node1.[raghav.com@RAGHAV.COM|mailto:raghav.com@RAGHAV.COM], issueDate=1552503090488, maxDate=1553107890488, sequenceNumber=5, masterKeyId=107)
2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: Login UGI: hr1 (auth:SIMPLE)
2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: Searching for KMS delegation token in user hr1 (auth:SIMPLE)'s credentials
2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by alias=172.25.36.130:9292 token=null
2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by service=172.25.36.130:9292 token=null
2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by alias=[kms://http@c316-node4].[raghav.com:9292/kms|http://raghav.com:9292/kms]token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by service=[kms://http@c316-node4].[raghav.com:9292/kms|http://raghav.com:9292/kms]token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by alias=172.25.38.4:9292 token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by service=172.25.38.4:9292 token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by alias=[kms://http@c316-node3].[raghav.com:9292/kms|http://raghav.com:9292/kms]token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by service=[kms://http@c316-node3].[raghav.com:9292/kms|http://raghav.com:9292/kms]token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by alias=172.25.36.130:9292 token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by service=172.25.36.130:9292 token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: Using loginUser when Kerberos is enabled but the actual user does not have either KMS Delegation Token or Kerberos Credentials
2019-03-13 18:51:33,146 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.UserGroupInformation: PrivilegedAction as:hr1 (auth:SIMPLE) from:org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:506)
2019-03-13 18:51:33,150 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL: Connecting to url [http://c316-node4.raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt]with token as null
2019-03-13 18:51:33,150 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL: Token not set, looking for delegation token. Creds:[], size:0
2019-03-13 18:51:33,150 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator: No delegation token found for url=[http://c316-node4|http://c316-node4/].[raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt|http://raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt], token=, authenticating with class org.apache.hadoop.security.token.delegation.web.KerberosDelegationTokenAuthenticator$1
2019-03-13 18:51:33,178 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.authentication.client.KerberosAuthenticator: Performing our own SPNEGO sequence.
2019-03-13 18:51:33,179 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.authentication.client.KerberosAuthenticator: No subject in context, logging in
2019-03-13 18:51:33,179 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.authentication.client.KerberosAuthenticator: Using subject: Subject:
Principal: UnixPrincipal: hr1
Principal: UnixNumericUserPrincipal: 1016
Principal: UnixNumericGroupPrincipal [Primary Group]: 1016
2019-03-13 18:51:33,182 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.UserGroupInformation: PrivilegedActionException as:hr1 (auth:SIMPLE) cause:org.apache.hadoop.security.authentication.client.AuthenticationException: Error while authenticating with endpoint: [http://c316-node4.raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt]
2019-03-13 18:51:33,182 WARN [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider: KMS provider at [[http://c316-node4.raghav.com:9292/kms/v1/]] threw an IOException:
java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Error while authenticating with endpoint: [http://c316-node4.raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt]
{code}
> KMSLoadBlanceClientProvider does not select token correctly
> -----------------------------------------------------------
>
> Key: HADOOP-16199
> URL: https://issues.apache.org/jira/browse/HADOOP-16199
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Xiaoyu Yao
> Assignee: Xiaoyu Yao
> Priority: Major
>
> After HADOOP-14445 and HADOOP-15997, there are still cases where KMSLoadBlanceClientProvider does not select token correctly.
> Here is the use case:
> The new configuration key hadoop.security.kms.client.token.use.uri.format=true is set cross all the cluster, including both Submitter and Yarn RM(renewer), which is not covered in the test matrix in this [HADOOP-14445 comment|https://issues.apache.org/jira/browse/HADOOP-14445?focusedCommentId=16505761&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16505761].
> I will post the debug log and the proposed fix shortly, cc: [~xiaochen] and [~jojochuang].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org