You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "Xiaoqiao He (Jira)" <ji...@apache.org> on 2024/04/11 10:07:00 UTC
[jira] [Resolved] (HDFS-17455) Fix Client throw IndexOutOfBoundsException in DFSInputStream#fetchBlockAt
[ https://issues.apache.org/jira/browse/HDFS-17455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiaoqiao He resolved HDFS-17455.
--------------------------------
Fix Version/s: 3.5.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Fix Client throw IndexOutOfBoundsException in DFSInputStream#fetchBlockAt
> -------------------------------------------------------------------------
>
> Key: HDFS-17455
> URL: https://issues.apache.org/jira/browse/HDFS-17455
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Haiyang Hu
> Assignee: Haiyang Hu
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> When the client read data, connect to the datanode, because at this time the datanode access token is invalid will throw InvalidBlockTokenException. At this time, when call fetchBlockAt method will throw java.lang.IndexOutOfBoundsException causing read data failed.
> *Root case:*
> * The HDFS file contains only one RBW block, with a block data size of 2048KB.
> * The client open this file and seeks to the offset of 1024KB to read data.
> * Call DFSInputStream#getBlockReader method connect to the datanode, because at this time the datanode access token is invalid will throw InvalidBlockTokenException., and call DFSInputStream#fetchBlockAt will throw java.lang.IndexOutOfBoundsException.
> {code:java}
> private synchronized DatanodeInfo blockSeekTo(long target)
> throws IOException {
> if (target >= getFileLength()) {
> // the target size is smaller than fileLength (completeBlockSize + lastBlockBeingWrittenLength),
> // here at this time target is 1024 and getFileLength is 2048
> throw new IOException("Attempted to read past end of file");
> }
> ...
> while (true) {
> ...
> try {
> blockReader = getBlockReader(targetBlock, offsetIntoBlock,
> targetBlock.getBlockSize() - offsetIntoBlock, targetAddr,
> storageType, chosenNode);
> if(connectFailedOnce) {
> DFSClient.LOG.info("Successfully connected to " + targetAddr +
> " for " + targetBlock.getBlock());
> }
> return chosenNode;
> } catch (IOException ex) {
> ...
> } else if (refetchToken > 0 && tokenRefetchNeeded(ex, targetAddr)) {
> refetchToken--;
> // Here will catch InvalidBlockTokenException.
> fetchBlockAt(target);
> } else {
> ...
> }
> }
> }
> }
> private LocatedBlock fetchBlockAt(long offset, long length, boolean useCache)
> throws IOException {
> maybeRegisterBlockRefresh();
> synchronized(infoLock) {
> // Here the locatedBlocks only contains one locatedBlock, at this time the offset is 1024 and fileLength is 0,
> // so the targetBlockIdx is -2
> int targetBlockIdx = locatedBlocks.findBlock(offset);
> if (targetBlockIdx < 0) { // block is not cached
> targetBlockIdx = LocatedBlocks.getInsertIndex(targetBlockIdx);
> // Here the targetBlockIdx is 1;
> useCache = false;
> }
> if (!useCache) { // fetch blocks
> final LocatedBlocks newBlocks = (length == 0)
> ? dfsClient.getLocatedBlocks(src, offset)
> : dfsClient.getLocatedBlocks(src, offset, length);
> if (newBlocks == null || newBlocks.locatedBlockCount() == 0) {
> throw new EOFException("Could not find target position " + offset);
> }
> // Update the LastLocatedBlock, if offset is for last block.
> if (offset >= locatedBlocks.getFileLength()) {
> setLocatedBlocksFields(newBlocks, getLastBlockLength(newBlocks));
> } else {
> locatedBlocks.insertRange(targetBlockIdx,
> newBlocks.getLocatedBlocks());
> }
> }
> // Here the locatedBlocks only contains one locatedBlock, so will throw java.lang.IndexOutOfBoundsException: Index 1 out of bounds for length 1
> return locatedBlocks.get(targetBlockIdx);
> }
> }
> {code}
> The client exception:
> {code:java}
> java.lang.IndexOutOfBoundsException: Index 1 out of bounds for length 1
> at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
> at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
> at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:266)
> at java.base/java.util.Objects.checkIndex(Objects.java:359)
> at java.base/java.util.ArrayList.get(ArrayList.java:427)
> at org.apache.hadoop.hdfs.protocol.LocatedBlocks.get(LocatedBlocks.java:87)
> at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockAt(DFSInputStream.java:569)
> at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockAt(DFSInputStream.java:540)
> at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:704)
> at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:884)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:957)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:804)
> {code}
> The datanode exception:
> {code:java}
> 2024-03-27 15:56:35,477 WARN datanode.DataNode (DataXceiver.java:checkAccess(1487)) [DataXceiver for client DFSClient_NONMAPREDUCE_475786505_1 at /xxx [Sending block BP-xxx:blk_1138933918_65194340]] - Block token verification failed: op=READ_BLOCK, remoteAddress=/XXX, message=Can't re-compute password for block_token_identifier (expiryDate=1711562193469, keyId=1775816931, userId=test, blockPoolId=BP-xxx-xxx-xxx, blockId=1138933918, access modes=[READ], storageTypes= [SSD, SSD, SSD], storageIds= [DS-xxx1, DS-xxx2,DS-xxx3]), since the required block key (keyID=1775816931) doesn't exist
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org