Does NameNode directory need fine-grained path lock?

[Denny Ye <de...@gmail.com>]

hi guys,
    Currently, NameNode uses read-write lock at top root folder. In my
opinion, it's too huge for whole namespace with million of file/folder.
Meanwhile, a few level folder on top of the namespace directory has been
set with different application. Does we need lesser lock level for such
application sub-directory to reduce impact each other? It can be used with
five(or less) top level to HDFS path with individual lock tree-like
structure.
     Can anybody provide your feedback or advice? Thanks

-Regards
Denny Ye

Re: Does NameNode directory need fine-grained path lock?

[Todd Lipcon <to...@cloudera.com>]

Hi Denny,

Do you see an issue in practice?

Fine grained locking makes sense if you hold the lock while doing disk
IO. But since the structure is all in-memory, it's far less important,
and in fact could reduce performance in some cases.

Until you hit a few thousand nodes in your cluster, lock contention in
the NN is not a bottleneck. And the read-write lock introduced in
later versions gets past this issue even for ~4k node clusters for the
most part.

Not to say it won't ever make sense, but seems like a low priority change.

-Todd

On Tue, May 8, 2012 at 1:15 AM, Denny Ye <de...@gmail.com> wrote:
> hi guys,
>     Currently, NameNode uses read-write lock at top root folder. In my
> opinion, it's too huge for whole namespace with million of file/folder.
> Meanwhile, a few level folder on top of the namespace directory has been set
> with different application. Does we need lesser lock level for such
> application sub-directory to reduce impact each other? It can be used with
> five(or less) top level to HDFS path with individual lock tree-like
> structure.
>      Can anybody provide your feedback or advice? Thanks
>
> -Regards
> Denny Ye



-- 
Todd Lipcon
Software Engineer, Cloudera