[jira] [Comment Edited] (HDFS-15191) EOF when reading legacy buffer in BlockTokenIdentifier

["Steven Rand (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-15191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17063999#comment-17063999 ] 

Steven Rand edited comment on HDFS-15191 at 3/21/20, 5:57 PM:
--------------------------------------------------------------

Also, I noticed that the original implementation of {{BlockTokenIdentifier#readFieldsLegacy}} from HDFS-11026 stops at exactly the right place, and it was only after HDFS-6708 that we can get this EOF, because it added more to that method: [https://github.com/apache/hadoop/commit/2f73396b5901fd5fe29f6cd76fc1b3134b854b37#diff-4bd9e663e048a075423d94ba53b8d315R217].
 
It seems like we should remove the additions to {{BlockTokenIdentifier#readFieldsLegacy}} from HDFS-6708, since if we're reading a legacy block token, then we know that the DataNode was on too old of a version to add the information about storages to the block token. Does that sound right?


was (Author: steven rand):
Also, I noticed that the original implementation of {{BlockTokenIdentifier#readFieldsLegacy}} from HDFS-11026 stops at exactly the right place, and it was only after HDFS-6708 that we can get this EOF, because it added more to that method: https://github.com/apache/hadoop/commit/2f73396b5901fd5fe29f6cd76fc1b3134b854b37#diff-4bd9e663e048a075423d94ba53b8d315R217.

> EOF when reading legacy buffer in BlockTokenIdentifier
> ------------------------------------------------------
>
>                 Key: HDFS-15191
>                 URL: https://issues.apache.org/jira/browse/HDFS-15191
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs
>    Affects Versions: 3.2.1
>            Reporter: Steven Rand
>            Priority: Major
>
> We have an HDFS client application which recently upgraded from 3.2.0 to 3.2.1. After this upgrade (but not before), we sometimes see these errors when this application is used with clusters still running Hadoop 2.x (more specifically CDH 5.12.1):
> {code}
> WARN  [2020-02-24T00:54:32.856Z] org.apache.hadoop.hdfs.client.impl.BlockReaderFactory: I/O error constructing remote block reader. (_sampled: true)
> java.io.EOFException:
>         at java.io.DataInputStream.readByte(DataInputStream.java:272)
>         at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308)
>         at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329)
>         at org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.readFieldsLegacy(BlockTokenIdentifier.java:240)
>         at org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.readFields(BlockTokenIdentifier.java:221)
>         at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:200)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:530)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:342)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:276)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:245)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:227)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.peerSend(SaslDataTransferClient.java:170)
>         at org.apache.hadoop.hdfs.DFSUtilClient.peerFromSocketAndKey(DFSUtilClient.java:730)
>         at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2942)
>         at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:822)
>         at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:747)
>         at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:380)
>         at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:644)
>         at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:575)
>         at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:757)
>         at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:829)
>         at java.io.DataInputStream.read(DataInputStream.java:100)
>         at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2314)
>         at org.apache.commons.io.IOUtils.copy(IOUtils.java:2270)
>         at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2291)
>         at org.apache.commons.io.IOUtils.copy(IOUtils.java:2246)
>         at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:765)
> {code}
> We get this warning for all DataNodes with a copy of the block, so the read fails.
> I haven't been able to figure out what changed between 3.2.0 and 3.2.1 to cause this, but HDFS-13617 and HDFS-14611 seem related, so tagging [~vagarychen] in case you have any ideas.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org