You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "yuyanlei (Jira)" <ji...@apache.org> on 2022/09/29 02:17:00 UTC
[jira] [Commented] (HDFS-14509) DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x
[ https://issues.apache.org/jira/browse/HDFS-14509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17610768#comment-17610768 ]
yuyanlei commented on HDFS-14509:
---------------------------------
When Kerberos is enabled and Hadoop is upgraded from 2.7.2 to 3.3.4, when Acitve Namenode version is 3.3.4 and Datanode version is 2.7.2, The BlockToken authentication between Namenode and Datanode fails. As a result, the client cannot read and write.
The datanode error:
org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with block_token_identifier (expiryDate=1664452892587, keyId=2032735264, userId=work, blockPoolId=BP-874546658-10.48.20.234-1660635316009, blockId=1152681184, access modes=[READ]) doesn't have the correct token password
at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:303)
at org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:97)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.checkAccess(DataXceiver.java:1296)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:521)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:116)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:253)
at java.lang.Thread.run(Thread.java:745)
This phenomenon like https://issues.apache.org/jira/browse/HDFS-14509, but can't merge the issues on the version 2.7.2 patch, so now can't in the case of open kerberos, Perform a rolling upgrade of Hadoop (2.7.2 upgrading 3.3.4)
I think it's a problem. What do you think
> DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x
> -------------------------------------------------------------------------------
>
> Key: HDFS-14509
> URL: https://issues.apache.org/jira/browse/HDFS-14509
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Yuxuan Wang
> Assignee: Yuxuan Wang
> Priority: Blocker
> Labels: release-blocker
> Fix For: 2.10.0, 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-14509-001.patch, HDFS-14509-002.patch, HDFS-14509-003.patch, HDFS-14509-branch-2.001.patch
>
>
> According to the doc, if we want to upgrade cluster from 2.x to 3.x, we need upgrade NN first. And there will be a intermediate state that NN is 3.x and DN is 2.x. At that moment, if a client reads (or writes) a block, it will get a block token from NN and then deliver the token to DN who can verify the token. But the verification in the code now is :
> {code:title=BlockTokenSecretManager.java|borderStyle=solid}
> public void checkAccess(...)
> {
> ...
> id.readFields(new DataInputStream(new ByteArrayInputStream(token.getIdentifier())));
> ...
> if (!Arrays.equals(retrievePassword(id), token.getPassword())) {
> throw new InvalidToken("Block token with " + id.toString()
> + " doesn't have the correct token password");
> }
> }
> {code}
> And {{retrievePassword(id)}} is:
> {code}
> public byte[] retrievePassword(BlockTokenIdentifier identifier)
> {
> ...
> return createPassword(identifier.getBytes(), key.getKey());
> }
> {code}
> So, if NN's identifier add new fields, DN will lose the fields and compute wrong password.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org