[jira] [Created] (HDFS-14503) ThrottledAsyncChecker throws NPE during block pool initialization

["Yiqun Lin (JIRA)" <ji...@apache.org>]

Yiqun Lin created HDFS-14503:
--------------------------------

             Summary: ThrottledAsyncChecker throws NPE during block pool initialization 
                 Key: HDFS-14503
                 URL: https://issues.apache.org/jira/browse/HDFS-14503
             Project: Hadoop HDFS
          Issue Type: Improvement
    Affects Versions: 3.3.0
            Reporter: Yiqun Lin


ThrottledAsyncChecker throws NPE during block pool initialization. The error leads the block pool registration failure.

The exception
{noformat}
2019-05-20 01:02:36,003 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected exception in block pool Block pool <registering> (Datanode Uuid xxxxx) service to xx.xx.xx.xx/xx.xx.xx.xx
java.lang.NullPointerException
        at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker$LastCheckResult.access$000(ThrottledAsyncChecker.java:211)
        at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker.schedule(ThrottledAsyncChecker.java:129)
        at org.apache.hadoop.hdfs.server.datanode.checker.DatasetVolumeChecker.checkAllVolumes(DatasetVolumeChecker.java:209)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.checkDiskError(DataNode.java:3387)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1508)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:319)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:272)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:768)
        at java.lang.Thread.run(Thread.java:745)
{noformat}

Looks like this error due to {{WeakHashMap}} type map {{completedChecks}} has removed the target entry while we still get that entry. Although we have done a check before we get it, there is still a chance the entry is got as null. 

We met a corner case for this: A federation mode, two block pools in DN, {{ThrottledAsyncChecker}} schedules two same health checks for same volume.
{noformat}
2019-05-20 01:02:36,000 INFO org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker: Scheduling a check for /hadoop/2/hdfs/data/current
2019-05-20 01:02:36,000 INFO org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker: Scheduling a check for /hadoop/2/hdfs/data/current
{noformat}





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org