You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Henrique Barros (JIRA)" <ji...@apache.org> on 2018/08/17 17:27:00 UTC
[jira] [Comment Edited] (HDFS-10453) ReplicationMonitor thread
could stuck for long time due to the race between replication and delete of
same file in a large cluster.
[ https://issues.apache.org/jira/browse/HDFS-10453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16584189#comment-16584189 ]
Henrique Barros edited comment on HDFS-10453 at 8/17/18 5:26 PM:
-----------------------------------------------------------------
I have the same problem with Hadoop 2.6.0-cdh5.15.0
With Cloudera CDH-5.15.0-1.cdh5.15.0.p0.21
In my case we are getting this error very randomly and with only one Datanode (for now).
Here is the Log.
{code:java}
Choosing random from 1 available nodes on node /default, scope=/default, excludedScope=null, excludeNodes=[]
2:38:20.527 PM DEBUG NetworkTopology
Choosing random from 0 available nodes on node /default, scope=/default, excludedScope=null, excludeNodes=[192.168.220.53:50010]
2:38:20.527 PM DEBUG NetworkTopology
chooseRandom returning null
2:38:20.527 PM DEBUG BlockPlacementPolicy
[
Node /default/192.168.220.53:50010 [
Datanode 192.168.220.53:50010 is not chosen since the node is too busy (load: 8 > 0.0).
2:38:20.527 PM DEBUG NetworkTopology
chooseRandom returning 192.168.220.53:50010
2:38:20.527 PM INFO BlockPlacementPolicy
Not enough replicas was chosen. Reason:{NODE_TOO_BUSY=1}
2:38:20.527 PM DEBUG StateChange
closeFile: /mobi.me/development/apps/flink/checkpoints/a5a6806866c1640660924ea1453cbe34/chk-2118/eef8bff6-75a9-43c1-ae93-4b1a9ca31ad9 with 1 blocks is persisted to the file system
2:38:20.527 PM DEBUG StateChange
*BLOCK* NameNode.addBlock: file /mobi.me/development/apps/flink/checkpoints/a5a6806866c1640660924ea1453cbe34/chk-2118/1cfe900d-6f45-4b55-baaa-73c02ace2660 fileId=129628869 for DFSClient_NONMAPREDUCE_467616914_65
2:38:20.527 PM DEBUG BlockPlacementPolicy
Failed to choose from local rack (location = /default); the second replica is not found, retry choosing ramdomly
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException:
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:784)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:694)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalRack(BlockPlacementPolicyDefault.java:601)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalStorage(BlockPlacementPolicyDefault.java:561)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder(BlockPlacementPolicyDefault.java:464)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:395)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:270)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:142)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:158)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1715)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3505)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:694)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:219)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:507)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275)
{code}
This part makes no sense at all:
{code:java}
load: 8 > 0.0{code}
I created a dedicated Bug for this case since it could not have anything to do with this one:
https://issues.apache.org/jira/browse/HDFS-13833
was (Author: rikeppb100):
I have the same problem with Hadoop 2.6.0-cdh5.15.0
With Cloudera CDH-5.15.0-1.cdh5.15.0.p0.21
In my case we are getting this error very randomly and with only one Datanode (for now).
Here is the Log.
{code:java}
Choosing random from 1 available nodes on node /default, scope=/default, excludedScope=null, excludeNodes=[]
2:38:20.527 PM DEBUG NetworkTopology
Choosing random from 0 available nodes on node /default, scope=/default, excludedScope=null, excludeNodes=[192.168.220.53:50010]
2:38:20.527 PM DEBUG NetworkTopology
chooseRandom returning null
2:38:20.527 PM DEBUG BlockPlacementPolicy
[
Node /default/192.168.220.53:50010 [
Datanode 192.168.220.53:50010 is not chosen since the node is too busy (load: 8 > 0.0).
2:38:20.527 PM DEBUG NetworkTopology
chooseRandom returning 192.168.220.53:50010
2:38:20.527 PM INFO BlockPlacementPolicy
Not enough replicas was chosen. Reason:{NODE_TOO_BUSY=1}
2:38:20.527 PM DEBUG StateChange
closeFile: /mobi.me/development/apps/flink/checkpoints/a5a6806866c1640660924ea1453cbe34/chk-2118/eef8bff6-75a9-43c1-ae93-4b1a9ca31ad9 with 1 blocks is persisted to the file system
2:38:20.527 PM DEBUG StateChange
*BLOCK* NameNode.addBlock: file /mobi.me/development/apps/flink/checkpoints/a5a6806866c1640660924ea1453cbe34/chk-2118/1cfe900d-6f45-4b55-baaa-73c02ace2660 fileId=129628869 for DFSClient_NONMAPREDUCE_467616914_65
2:38:20.527 PM DEBUG BlockPlacementPolicy
Failed to choose from local rack (location = /default); the second replica is not found, retry choosing ramdomly
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException:
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:784)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:694)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalRack(BlockPlacementPolicyDefault.java:601)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalStorage(BlockPlacementPolicyDefault.java:561)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder(BlockPlacementPolicyDefault.java:464)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:395)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:270)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:142)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:158)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1715)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3505)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:694)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:219)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:507)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275)
{code}
This part makes no sense at all:
{code:java}
load: 8 > 0.0{code}
> ReplicationMonitor thread could stuck for long time due to the race between replication and delete of same file in a large cluster.
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-10453
> URL: https://issues.apache.org/jira/browse/HDFS-10453
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 2.4.1, 2.5.2, 2.7.1, 2.6.4
> Reporter: He Xiaoqiao
> Assignee: He Xiaoqiao
> Priority: Major
> Fix For: 3.1.0, 2.10.0, 2.9.1, 2.8.4, 2.7.6, 3.0.3
>
> Attachments: HDFS-10453-branch-2.001.patch, HDFS-10453-branch-2.003.patch, HDFS-10453-branch-2.7.004.patch, HDFS-10453-branch-2.7.005.patch, HDFS-10453-branch-2.7.006.patch, HDFS-10453-branch-2.7.007.patch, HDFS-10453-branch-2.7.008.patch, HDFS-10453-branch-2.7.009.patch, HDFS-10453-branch-2.8.001.patch, HDFS-10453-branch-2.8.002.patch, HDFS-10453-branch-2.9.001.patch, HDFS-10453-branch-2.9.002.patch, HDFS-10453-branch-3.0.001.patch, HDFS-10453-branch-3.0.002.patch, HDFS-10453-trunk.001.patch, HDFS-10453-trunk.002.patch, HDFS-10453.001.patch
>
>
> ReplicationMonitor thread could stuck for long time and loss data with little probability. Consider the typical scenario：
> (1) create and close a file with the default replicas(3);
> (2) increase replication (to 10) of the file.
> (3) delete the file while ReplicationMonitor is scheduling blocks belong to that file for replications.
> if ReplicationMonitor stuck reappeared, NameNode will print log as:
> {code:xml}
> 2016-04-19 10:20:48,083 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 7 to reach 10 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=false) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy
> ......
> 2016-04-19 10:21:17,184 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 7 to reach 10 (unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=false) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy
> 2016-04-19 10:21:17,184 WARN org.apache.hadoop.hdfs.protocol.BlockStoragePolicy: Failed to place enough replicas: expected size is 7 but only 0 storage types can be selected (replication=10, selected=[], unavailable=[DISK, ARCHIVE], removed=[DISK, DISK, DISK, DISK, DISK, DISK, DISK], policy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]})
> 2016-04-19 10:21:17,184 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 7 to reach 10 (unavailableStorages=[DISK, ARCHIVE], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=false) All required storage types are unavailable: unavailableStorages=[DISK, ARCHIVE], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}
> {code}
> This is because 2 threads (#NameNodeRpcServer and #ReplicationMonitor) process same block at the same moment.
> (1) ReplicationMonitor#computeReplicationWorkForBlocks get blocks to replicate and leave the global lock.
> (2) FSNamesystem#delete invoked to delete blocks then clear the reference in blocksmap, needReplications, etc. the block's NumBytes will set NO_ACK(Long.MAX_VALUE) which is used to indicate that the block deletion does not need explicit ACK from the node.
> (3) ReplicationMonitor#computeReplicationWorkForBlocks continue to chooseTargets for the same blocks and no node will be selected after traverse whole cluster because no node choice satisfy the goodness criteria (remaining spaces achieve required size Long.MAX_VALUE).
> During of stage#3 ReplicationMonitor stuck for long time, especial in a large cluster. invalidateBlocks & neededReplications continues to grow and no consumes. it will loss data at the worst.
> This can mostly be avoided by skip chooseTarget for BlockCommand.NO_ACK block and remove it from neededReplications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org