[jira] [Comment Edited] (HDFS-14563) Enhance interface about recommissioning/decommissioning

["He Xiaoqiao (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16862322#comment-16862322 ] 

He Xiaoqiao edited comment on HDFS-14563 at 6/12/19 5:58 PM:
-------------------------------------------------------------

Thanks [~elgoiri],[~kihwal],[~jojochuang] for your valuable comments. I am sure that we have to resolve consistency problems between new interface and existing approach if we provide lightweight interface.
{quote}In our experience, decommissioning and adding new nodes are manageable and the latency is okay even for 5,000+ node clusters. We avoid recommissioning as it is expensive. It is much better to wipe out and add as a new node.{quote}
correct, the latency is related with number nodes in include/exclude files, however we do not limit the size currently, so it is possible that size could be expanded without wipe out for long time.
[~elgoiri] offer a good idea that maybe we could make this state store pluggable just like Router does. Another way, we could maintain this state just in memory and sync to fsimage/editlog, Of course, this approach is heavyweight. FYI. Thanks again for your suggestions.


was (Author: hexiaoqiao):
Thanks [~elgoiri],[~kihwal] for your valuable comments. I am sure that we have to resolve consistency problems between new interface and existing approach if we provide lightweight interface.
{quote}In our experience, decommissioning and adding new nodes are manageable and the latency is okay even for 5,000+ node clusters. We avoid recommissioning as it is expensive. It is much better to wipe out and add as a new node.{quote}
correct, the latency is related with number nodes in include/exclude files, however we do not limit the size currently, so it is possible that size could be expanded without wipe out for long time.
[~elgoiri] offer a good idea that maybe we could make this state store pluggable just like Router does. Another way, we could maintain this state just in memory and sync to fsimage/editlog, Of course, this approach is heavyweight. FYI. Thanks again for your suggestions.

> Enhance interface about recommissioning/decommissioning
> -------------------------------------------------------
>
>                 Key: HDFS-14563
>                 URL: https://issues.apache.org/jira/browse/HDFS-14563
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client, namenode
>            Reporter: He Xiaoqiao
>            Assignee: He Xiaoqiao
>            Priority: Major
>
> In current implementation, if we need to decommissioning or recommissioning one datanode, the only way is add the datanode to include or exclude file under namenode configuration path then execute command `bin/hadoop dfsadmin -refreshNodes` and trigger namenode to reload include/exclude and start to recommissioning or decommissioning datanode.
> The shortcomings of this approach is that:
> a. namenode reload include/exclude configuration file from devices, if I/O load is high, handler may be blocked.
> b. namenode has to process every datnodes in include and exclude configurations, if there are many datanodes (very common for large cluster) pending to process, namenode will be hung for hundred seconds to wait recommision/decommision finish at the worst since holding write lock.
> I think we should expose one lightweight interface to support recommissioning or decommissioning single datanode, thus we can operate datanode using dfsadmin more smooth.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org