[jira] [Created] (HDFS-16058) If the balancer cannot update the datanode's storage type in time, the balancer will throw NullPointerException

["lei w (Jira)" <ji...@apache.org>]

lei w created HDFS-16058:
----------------------------

             Summary: If the balancer cannot update the datanode's storage type in time, the balancer will throw NullPointerException
                 Key: HDFS-16058
                 URL: https://issues.apache.org/jira/browse/HDFS-16058
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: balancer & mover
            Reporter: lei w


The main logic of the balancer is to initialize the cluster information (number of datanodes, datanode's storage type and network topology etc)、calculate the source DataNode and target DataNode that need to move the block and then obtain the movable block through the getblocks(). Finally , move blocks. If a DataNode adds another type of storage and writes an EC block after the cluster information is initialized, and then returns the newly written EC block to balancer by getblocks(), it will cause balancer to throw NullPointerException .The main reason is that the balancer does not record the newly added storage type of the DataNode, which causes the current DataNode information not to be recorded when generate an object of type DBlockStriped . Finally, null is returned when the block of the current DataNode is obtained through the object of type DBlockStriped.
{code:java}
2021-04-23 19:38:21,233 WARN org.apache.hadoop.hdfs.server.balancer.Dispatcher: Dispatcher thread failed
java.lang.NullPointerException
        at org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.chooseProxySource(Dispatcher.java:325)
        at org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.markMovedIfGoodBlock(Dispatcher.java:291)
        at org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.chooseBlockAndProxy(Dispatcher.java:271)
        at org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.access$2500(Dispatcher.java:235)
        at org.apache.hadoop.hdfs.server.balancer.Dispatcher$Source.chooseNextMove(Dispatcher.java:886)
        at org.apache.hadoop.hdfs.server.balancer.Dispatcher$Source.dispatchBlocks(Dispatcher.java:943)
        at org.apache.hadoop.hdfs.server.balancer.Dispatcher$Source.access$3200(Dispatcher.java:751)
        at org.apache.hadoop.hdfs.server.balancer.Dispatcher$2.run(Dispatcher.java:1221)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org