You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Hudson (Jira)" <ji...@apache.org> on 2019/11/16 03:49:00 UTC
[jira] [Commented] (HDFS-14648) Implement DeadNodeDetector basic
model
[ https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16975566#comment-16975566 ]
Hudson commented on HDFS-14648:
-------------------------------
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17649 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17649/])
HDFS-14648. Implement DeadNodeDetector basic model. Contributed by (yqlin: rev b3119b9ab60a19d624db476c4e1c53410870c7a6)
* (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
* (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientContext.java
* (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
* (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DeadNodeDetector.java
* (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDeadNodeDetection.java
* (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
* (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
> Implement DeadNodeDetector basic model
> --------------------------------------
>
> Key: HDFS-14648
> URL: https://issues.apache.org/jira/browse/HDFS-14648
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Lisheng Sun
> Assignee: Lisheng Sun
> Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14648.001.patch, HDFS-14648.002.patch, HDFS-14648.003.patch, HDFS-14648.004.patch, HDFS-14648.005.patch, HDFS-14648.006.patch, HDFS-14648.007.patch, HDFS-14648.008.patch, HDFS-14648.009.patch, HDFS-14648.010.patch, HDFS-14648.011.patch
>
>
> This Jira constructs DeadNodeDetector state machine model. The function it implements as follow:
> # When a DFSInputstream is opened, a BlockReader is opened. If some DataNode of the block is found to inaccessible, put the DataNode into DeadNodeDetector#deadnode.(HDFS-14649) will optimize this part. Because when DataNode is not accessible, it is likely that the replica has been removed from the DataNode.Therefore, it needs to be confirmed by re-probing and requires a higher priority processing.
> # DeadNodeDetector will periodically detect the Node in DeadNodeDetector#deadnode, If the access is successful, the Node will be moved from DeadNodeDetector#deadnode. Continuous detection of the dead node is necessary. The DataNode need rejoin the cluster due to a service restart/machine repair. The DataNode may be permanently excluded if there is no added probe mechanism.
> # DeadNodeDetector#dfsInputStreamNodes Record the DFSInputstream using DataNode. When the DFSInputstream is closed, it will be moved from DeadNodeDetector#dfsInputStreamNodes.
> # Every time get the global deanode, update the DeadNodeDetector#deadnode. The new DeadNodeDetector#deadnode Equals to the intersection of the old DeadNodeDetector#deadnode and the Datanodes are by DeadNodeDetector#dfsInputStreamNodes.
> # DeadNodeDetector has a switch that is turned off by default. When it is closed, each DFSInputstream still uses its own local deadnode.
> # This feature has been used in the XIAOMI production environment for a long time. Reduced hbase read stuck, due to node hangs.
> # Just open the DeadNodeDetector switch and you can use it directly. No other restrictions. Don't want to use DeadNodeDetector, just close it.
> {code:java}
> if (sharedDeadNodesEnabled && deadNodeDetector == null) {
> deadNodeDetector = new DeadNodeDetector(name);
> deadNodeDetectorThr = new Daemon(deadNodeDetector);
> deadNodeDetectorThr.start();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org