You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Jinglun (Jira)" <ji...@apache.org> on 2021/03/06 10:52:00 UTC
[jira] [Comment Edited] (HDFS-15809) DeadNodeDetector doesn't
remove live nodes from dead node set.
[ https://issues.apache.org/jira/browse/HDFS-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17296505#comment-17296505 ]
Jinglun edited comment on HDFS-15809 at 3/6/21, 10:51 AM:
----------------------------------------------------------
After a discussion with [~leosun08], we decide to abandon the Collections.synchronizedSet(new LinkedHashSet<>()) plan because the interface is not friendly and makes the patch more complicated. Also it makes the probe queue hard to spy(set wrapped by synchronized would be final).
Thanks [~leosun08]'s suggestions for design and unit tests. Submit v05 fixing unit test.
was (Author: lijinglun):
After a offline discussion with [~leosun08], we decide to abandon the Collections.synchronizedSet(new LinkedHashSet<>()) plan because the interface is not friendly and makes the patch more complicated. Also it makes the probe queue hard to spy(set wrapped by synchronized would be final).
Thanks [~leosun08]'s suggestions for design and unit tests. Submit v05 fixing unit test.
> DeadNodeDetector doesn't remove live nodes from dead node set.
> --------------------------------------------------------------
>
> Key: HDFS-15809
> URL: https://issues.apache.org/jira/browse/HDFS-15809
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Jinglun
> Assignee: Jinglun
> Priority: Major
> Attachments: HDFS-15809.001.patch, HDFS-15809.002.patch, HDFS-15809.003.patch, HDFS-15809.004.patch, HDFS-15809.005.patch
>
>
> We found the dead node detector might never remove the alive nodes from the dead node set in a big cluster. For example:
> # 200 nodes are added to the dead node set by DeadNodeDetector.
> # DeadNodeDetector#checkDeadNodes() adds 100 nodes to the deadNodesProbeQueue because the queue limited length is 100.
> # The probe threads start working and probe 30 nodes.
> # DeadNodeDetector#checkDeadNodes() is scheduled again. It iterates the dead node set and adds 30 nodes to the deadNodesProbeQueue. But the order is the same as the last time. So the 30 nodes that has already been probed are added to the queue again.
> # Repeat 3 and 4. But we always add the first 30 nodes from the dead set. If they are all dead then the live nodes behind them could never be recovered.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org