[jira] [Commented] (HDFS-7396) Revisit synchronization in Namenode

["Ming Ma (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-7396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14236420#comment-14236420 ] 

Ming Ma commented on HDFS-7396:
-------------------------------

It might be nice if we can enforce the correctness via unit tests. For example, https://issues.apache.org/jira/browse/HDFS-7182 tries to assert any JMX metrics shouldn't require FSNamesystem read lock with the modified unit test.

Federated NN is sort of orthogonal to this discussion given it is still useful to improve the overall synchronization in NN even for federated NN. However it does make synchronization improvement less critical; e.g., you can get the NN JVM heap size under 32G and use compressed pointer, etc. There is still improvement we need to make to federated NN, such as dynamic reallocation amount namespaces, have each NN host several volumes for efficient move, etc. It will be interesting to understand if there is any plan to scale HDFS in that direction.

> Revisit synchronization in Namenode
> -----------------------------------
>
>                 Key: HDFS-7396
>                 URL: https://issues.apache.org/jira/browse/HDFS-7396
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>
> HDFS-2106 separated block management to a new package from namenode.  As part of it, some code was refactored to new classes such as DatanodeManager, HeartbeatManager, etc.  There are opportunities for improve locking in namenode while currently the synchronization in namenode is mainly done by a single global FSNamesystem lock. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)