[jira] [Created] (HDFS-15191) EOF when reading legacy buffer in BlockTokenIdentifier

["Steven Rand (Jira)" <ji...@apache.org>]

Steven Rand created HDFS-15191:
----------------------------------

             Summary: EOF when reading legacy buffer in BlockTokenIdentifier
                 Key: HDFS-15191
                 URL: https://issues.apache.org/jira/browse/HDFS-15191
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: hdfs
    Affects Versions: 3.2.1
            Reporter: Steven Rand


We have an HDFS client application which recently upgraded from 3.2.0 to 3.2.1. After this upgrade (but not before), we sometimes see these errors when this application is used with clusters still running Hadoop 2.x (more specifically CDH 5.12.1):

{code}
WARN  [2020-02-24T00:54:32.856Z] org.apache.hadoop.hdfs.client.impl.BlockReaderFactory: I/O error constructing remote block reader. (_sampled: true)
java.io.EOFException:
        at java.io.DataInputStream.readByte(DataInputStream.java:272)
        at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308)
        at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329)
        at org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.readFieldsLegacy(BlockTokenIdentifier.java:240)
        at org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.readFields(BlockTokenIdentifier.java:221)
        at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:200)
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:530)
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:342)
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:276)
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:245)
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:227)
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.peerSend(SaslDataTransferClient.java:170)
        at org.apache.hadoop.hdfs.DFSUtilClient.peerFromSocketAndKey(DFSUtilClient.java:730)
        at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2942)
        at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:822)
        at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:747)
        at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:380)
        at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:644)
        at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:575)
        at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:757)
        at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:829)
        at java.io.DataInputStream.read(DataInputStream.java:100)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2314)
        at org.apache.commons.io.IOUtils.copy(IOUtils.java:2270)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2291)
        at org.apache.commons.io.IOUtils.copy(IOUtils.java:2246)
        at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:765)
{code}

We get this warning for all DataNodes with a copy of the block, so the read fails.

I haven't been able to figure out what changed between 3.2.0 and 3.2.1 to cause this, but HDFS-13617 and HDFS-14611 seem related, so tagging [~vagarychen] in case you have any ideas.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org