[jira] [Created] (HDFS-15069) DecommissionMonitor thread will block forever while it encountered an unchecked exception.

["Xudong Cao (Jira)" <ji...@apache.org>]

Xudong Cao created HDFS-15069:
---------------------------------

             Summary: DecommissionMonitor thread will block forever while it encountered an unchecked exception.
                 Key: HDFS-15069
                 URL: https://issues.apache.org/jira/browse/HDFS-15069
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: namenode
    Affects Versions: 3.1.3
            Reporter: Xudong Cao
            Assignee: Xudong Cao
         Attachments: stack_on_16_12.png, stack_on_16_42.png

More than once, we have observed that during decommissioning of a large number of dns, the thread DecommissionMonitor-0 will stop scheduling, blocking for a long time, and there will be no exception logs or notifications at all.

e.g. Recently, we are decommissioning 65 DNs at the same time, each DN about 10TB, and the DecommissionMonitor-0 thread blocked for about 15 days.

The stack of DecommissionMonitor-0 looks like this:
1. stack on 2019.12.17 16:12 !stack_on_16_12.png!

2. stack on 2019.12.17 16:42 !stack_on_16_42.png!

It can be seen that during half an hour, this thread has not been scheduled at all, its Waited count has not changed.

We think the cause of the problem is:
1. The DecommissionMonitor task submitted by NameNode encounters an unchecked exception during its running , and then this task will be never executed again.
2. But NameNode does not care about the ScheduledFuture of this task, and never calls ScheduledFuture.get(), so the unchecked exception thrown by the task above will always be placed there, no one knows.

After that, the subsequent phenomenon is:
1.The ScheduledExecutorService thread DecommissionMonitor-0 will block forever in ThreadPoolExecutor.getTask ().
2. The previously submitted task DecommissionMonitor will be never executed again.
3. No logs or notifications let us know exactly what had happened.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org