You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "He Xiaoqiao (JIRA)" <ji...@apache.org> on 2019/05/16 13:05:00 UTC
[jira] [Commented] (HDFS-14186) blockreport storm slow down
namenode restart seriously in large cluster
[ https://issues.apache.org/jira/browse/HDFS-14186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16841310#comment-16841310 ]
He Xiaoqiao commented on HDFS-14186:
------------------------------------
Hi wathers, Sorry for reopen this issue again.
After apply HADOOP-12173 and HDFS-9198, this issue has resolved temporary. when cluster scale continued, it expose again.
[~elgoiri] offer one solution about lifeline, I believe it could solve DataNode set DEAD and avoid block report storm after exit safemode while namenode startup (sorry for no verify), However it could not solve this issue completely, NameNode could not service normally due to lock hold by block report for long times.
In one word, I think may be we should upgrade the current exit mechanism of safe mode as mentioned earlier. welcome to some suggestions.
> blockreport storm slow down namenode restart seriously in large cluster
> -----------------------------------------------------------------------
>
> Key: HDFS-14186
> URL: https://issues.apache.org/jira/browse/HDFS-14186
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 2.7.1
> Reporter: He Xiaoqiao
> Assignee: He Xiaoqiao
> Priority: Major
> Attachments: HDFS-14186.001.patch
>
>
> In the current implementation, the datanode sends blockreport immediately after register to namenode successfully when restart, and the blockreport storm will make namenode high load to process them. One result is some received RPC have to skip because queue time is timeout. If some datanodes' heartbeat RPC are continually skipped for long times (default is heartbeatExpireInterval=630s) it will be set DEAD, then datanode has to re-register and send blockreport again, aggravate blockreport storm and trap in a vicious circle, and slow down (more than one hour and even more) namenode startup seriously in a large (several thousands of datanodes) and busy cluster especially. Although there are many work to optimize namenode startup, the issue still exists.
> I propose to postpone dead datanode check when namenode have finished startup.
> Any comments and suggestions are welcome.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org