[jira] [Commented] (HDFS-11797) BlockManager#createLocatedBlocks() can throw ArrayIndexOutofBoundsException when corrupt replicas are inconsistent

["Wei-Chiu Chuang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-11797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203894#comment-16203894 ] 

Wei-Chiu Chuang commented on HDFS-11797:
----------------------------------------

For future reference, I can confirm an occurrence of this bug happened to a customer of us, and I was able to find the sequence of incidences leading to this bug, which is exactly what HDFS-11445 fixes.

# A datanode was shutdown, making the replica stale.
# NameNode detected the staleness, adding it to corruptReplicaMap. Because the replica was on a DataNode that was out of date, the replica was not invalidated. So the corruptReplicaMap had the replica, and blockMap had the replica as well.
# The block was updated, causing the stale replica removed from blockMap. *but it was not removed from corruptReplicaMap*
# a client calling getBlockLocations caused AIOOE because of the mismatch.
{noformat}
2017-10-10 14:48:10,664 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Inconsistent number of corrupt replicas for blk_1041920008_1133174794 blockMap has 0 but corrupt replicas map has 1
2017-10-10 14:48:10,665 WARN org.apache.hadoop.ipc.Server: IPC Server handler 5 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 10.103.4.11:56487 Call#5239908 Retry#0
java.lang.ArrayIndexOutOfBoundsException: 2
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:982)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:929)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlocks(BlockManager.java:1031)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2059)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2008)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1921)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
        at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)
{noformat}

> BlockManager#createLocatedBlocks() can throw ArrayIndexOutofBoundsException when corrupt replicas are inconsistent
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-11797
>                 URL: https://issues.apache.org/jira/browse/HDFS-11797
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Kuhu Shukla
>            Assignee: Kuhu Shukla
>            Priority: Critical
>         Attachments: HDFS-11797.001.patch
>
>
> The calculation for {{numMachines}} can be too less (causing ArrayIndexOutOfBoundsException) or too many (causing NPE (HDFS-9958)) if data structures find inconsistent number of corrupt replicas. This was earlier found related to failed storages. This JIRA tracks a change that works for all possible cases of inconsistencies.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org