[jira] [Created] (HDFS-15018) DataNode doesn't shutdown although the number of failed disks reaches dfs.datanode.failed.volumes.tolerated

["Toshihiro Suzuki (Jira)" <ji...@apache.org>]

Toshihiro Suzuki created HDFS-15018:
---------------------------------------

             Summary: DataNode doesn't shutdown although the number of failed disks reaches dfs.datanode.failed.volumes.tolerated
                 Key: HDFS-15018
                 URL: https://issues.apache.org/jira/browse/HDFS-15018
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: datanode
    Affects Versions: 2.7.3
         Environment: HDP-2.6.5
            Reporter: Toshihiro Suzuki
         Attachments: thread_dumps.txt

In our case, we set dfs.datanode.failed.volumes.tolerated=0 but a DataNode didn't shutdown when a disk in the DataNode host got failed for some reason.

The the following log messages were shown in the DataNode log which indicates the DataNode detected the disk failure, but the DataNode didn't shutdown:
{code}
2019-09-17T13:15:43.262-0400 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskErrorAsync callback got 1 failed volumes: [/data2/hdfs/current]
2019-09-17T13:15:43.262-0400 INFO org.apache.hadoop.hdfs.server.datanode.BlockScanner: Removing scanner for volume /data2/hdfs (StorageID DS-329dec9d-a476-4334-9570-651a7e4d1f44)
2019-09-17T13:15:43.263-0400 INFO org.apache.hadoop.hdfs.server.datanode.VolumeScanner: VolumeScanner(/data2/hdfs, DS-329dec9d-a476-4334-9570-651a7e4d1f44) exiting.
{code}

Looking at the HDFS code, it looks like when the DataNode detects a disk failure, DataNode waits until the volume reference of the disk is released.
https://github.com/hortonworks/hadoop/blob/HDP-2.6.5.0-292-tag/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java#L246

I'm suspecting that the volume reference is not released after the failure detection, but not sure the reason.

And we took when the issue was happening. Attaching it to this Jira.

It looks like the following thread is waiting for the volume reference of the disk to be released:
{code}
"pool-4-thread-1" #174 daemon prio=5 os_prio=0 tid=0x00007f9e7c7bf800 nid=0x8325 in Object.wait() [0x00007f9e629cb000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.waitVolumeRemoved(FsVolumeList.java:262)
        at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.handleVolumeFailures(FsVolumeList.java:246)
        - locked <0x0000000670559278> (a java.lang.Object)
        at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.handleVolumeFailures(FsDatasetImpl.java:2178)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.handleVolumeFailures(DataNode.java:3410)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.access$100(DataNode.java:248)
        at org.apache.hadoop.hdfs.server.datanode.DataNode$4.call(DataNode.java:2013)
        at org.apache.hadoop.hdfs.server.datanode.checker.DatasetVolumeChecker$ResultHandler.invokeCallback(DatasetVolumeChecker.java:394)
        at org.apache.hadoop.hdfs.server.datanode.checker.DatasetVolumeChecker$ResultHandler.cleanup(DatasetVolumeChecker.java:387)
        at org.apache.hadoop.hdfs.server.datanode.checker.DatasetVolumeChecker$ResultHandler.onFailure(DatasetVolumeChecker.java:370)
        at com.google.common.util.concurrent.Futures$6.run(Futures.java:977)
        at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
        at org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.executeListener(AbstractFuture.java:991)
        at org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.complete(AbstractFuture.java:885)
        at org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.setException(AbstractFuture.java:739)
        at org.apache.hadoop.hdfs.server.datanode.checker.TimeoutFuture$Fire.run(TimeoutFuture.java:137)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
{code}

We found a similar issue HDFS-13339, but we didn't see any dead lock from the thread dump.

Attaching the full thread dumps of the problematic DataNode.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org