You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Xiao Chen (JIRA)" <ji...@apache.org> on 2018/10/15 03:46:00 UTC
[jira] [Comment Edited] (HADOOP-14445) Use DelegationTokenIssuer to
create KMS delegation tokens that can authenticate to all KMS instances
[ https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649691#comment-16649691 ]
Xiao Chen edited comment on HADOOP-14445 at 10/15/18 3:45 AM:
--------------------------------------------------------------
[~daryn] Do you mind another review? Sadly this needs an addendum for 2 things:
* {{DelegationTokenIssuer}} class was recursively 'org/apache/hadoop/security/token' package twice... sorry didn't catch this during review
* It caused 2 test failures in TestEncryptionZones. Pre-commit smartly skipped hadoop-hdfs (only ran hadoop-hdfs-client and hadoop-common), and it's caught when I try to backport to CDH where a full unit test was carried out. Out of the 2 failures, {{testDelegationToken}} needs to update the way it's mocked, and {{addMockKmsToken}} (another test method) caused mockito to give up, refusing to call the method on interface...
(For thoroughness, internal pre-commit also complained about API compat, saying {{addDelegationTokens}} is removed from FileSystem and DistributedFileSystem; it also noted the same method is added to DelegationTokenIssuer, but not able to use the latter as a clue to cross off the former. So this part is clearly to be overruled)
was (Author: xiaochen):
[~daryn] sadly this needs an addendum for 2 things:
* {{DelegationTokenIssuer}} class was recursively 'org/apache/hadoop/security/token' package twice... sorry didn't catch this during review
* It caused 2 test failures in TestEncryptionZones. Pre-commit smartly skipped hadoop-hdfs (only ran hadoop-hdfs-client and hadoop-common), and it's caught when I try to backport to CDH where a full unit test was carried out. Out of the 2 failures, {{testDelegationToken}} needs to update the way it's mocked, and {{addMockKmsToken}} (another test method) caused mockito to give up, refusing to call the method on interface...
(For thoroughness, internal pre-commit also complained about API compat, saying {{addDelegationTokens}} is removed from FileSystem and DistributedFileSystem; it also noted the same method is added to DelegationTokenIssuer, but not able to use the latter as a clue to cross off the former. So this part is clearly to be overruled)
> Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances
> ----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
> Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
> Reporter: Wei-Chiu Chuang
> Assignee: Xiao Chen
> Priority: Major
> Fix For: 3.2.0, 3.0.4, 3.1.2
>
> Attachments: HADOOP-14445-branch-2.8.002.patch, HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch, HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch, HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch, HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch, HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch, HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch, HADOOP-14445.18.patch, HADOOP-14445.19.patch, HADOOP-14445.20.patch, HADOOP-14445.addemdum.patch, HADOOP-14445.branch-2.000.precommit.patch, HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch, HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch, HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch, HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch, HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch, HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch, HADOOP-14445.branch-3.0.001.patch, HADOOP-14445.compat.patch, HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do not share delegation tokens. (a client uses KMS address/port as the key for delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation tokens too.
> Under HA, A KMS instance must verify the delegation token given by another KMS instance, by checking the shared secret used to sign the delegation token. To do this, all KMS instances must be able to retrieve the shared secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org