You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Wei-Chiu Chuang (Jira)" <ji...@apache.org> on 2019/10/02 21:06:00 UTC

[jira] [Commented] (HDFS-14527) Stop all DataNodes may result in NN terminate

    [ https://issues.apache.org/jira/browse/HDFS-14527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16943165#comment-16943165 ] 

Wei-Chiu Chuang commented on HDFS-14527:
----------------------------------------

Patch applies cleanly in branch-3.2 also.
But it doesn't compile in branch-3.1. I'll provide a patch shortly.

> Stop all DataNodes may result in NN terminate
> ---------------------------------------------
>
>                 Key: HDFS-14527
>                 URL: https://issues.apache.org/jira/browse/HDFS-14527
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>            Reporter: Xiaoqiao He
>            Assignee: Xiaoqiao He
>            Priority: Major
>             Fix For: 3.3.0, 3.2.2
>
>         Attachments: HDFS-14527.001.patch, HDFS-14527.002.patch, HDFS-14527.003.patch, HDFS-14527.004.patch, HDFS-14527.005.patch
>
>
> If we stop all datanodes of cluster, BlockPlacementPolicyDefault#chooseTarget may get ArithmeticException when calling #getMaxNodesPerRack, which throws the runtime exception out to BlockManager's ReplicationMonitor thread and then terminate the NN.
> The root cause is that BlockPlacementPolicyDefault#chooseTarget not hold the global lock, and if all DataNodes are dead between {{clusterMap.getNumberOfLeaves()}} and {{getMaxNodesPerRack}} then it meet {{ArithmeticException}} while invoke {{getMaxNodesPerRack}}.
> {code:java}
>   private DatanodeStorageInfo[] chooseTarget(int numOfReplicas,
>                                     Node writer,
>                                     List<DatanodeStorageInfo> chosenStorage,
>                                     boolean returnChosenNodes,
>                                     Set<Node> excludedNodes,
>                                     long blocksize,
>                                     final BlockStoragePolicy storagePolicy,
>                                     EnumSet<AddBlockFlag> addBlockFlags,
>                                     EnumMap<StorageType, Integer> sTypes) {
>     if (numOfReplicas == 0 || clusterMap.getNumOfLeaves()==0) {
>       return DatanodeStorageInfo.EMPTY_ARRAY;
>     }
>     ......
>     int[] result = getMaxNodesPerRack(chosenStorage.size(), numOfReplicas);
>     ......
> }
> {code}
> Some detailed log show as following.
> {code:java}
> 2019-05-31 12:29:21,803 ERROR org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: ReplicationMonitor thread received Runtime exception. 
> java.lang.ArithmeticException: / by zero
>         at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.getMaxNodesPerRack(BlockPlacementPolicyDefault.java:282)
>         at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:228)
>         at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:132)
>         at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationWork.chooseTargets(BlockManager.java:4533)
>         at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationWork.access$1800(BlockManager.java:4493)
>         at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWorkForBlocks(BlockManager.java:1954)
>         at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWork(BlockManager.java:1830)
>         at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:4453)
>         at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:4388)
>         at java.lang.Thread.run(Thread.java:745)
> 2019-05-31 12:29:21,805 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
> {code}
> To be honest, this is not serious bug and not reprod easily, since if we stop all Datanodes and only keep NameNode lives, HDFS could be not offer service normally and we could only retrieve directory. It may be one corner case.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org