You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ZanderXu (Jira)" <ji...@apache.org> on 2024/02/02 03:03:00 UTC
[jira] [Updated] (HDFS-17366) NameNode Fine-Grained Locking via Namespace Tree
[ https://issues.apache.org/jira/browse/HDFS-17366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ZanderXu updated HDFS-17366:
----------------------------
Description:
As we all known, the write performance of NameNode is limited by the global lock. We target to enable fine-grained locking based on the Namespace tree to improve the performance of NameNode write operations.
There are multiple motivations for creating this ticket:
* We have implemented this fine-grained locking and gained nearly 7x performance improvements in our prod environment
* Other companies made similar improvements based on their internal branch. Internal branches are quite different from the community, so few feedback and discussions in the community.
* The topic of fine-grained locking has been discussed for a very long time, but still without any results.
We implemented this fine-gained locking based on the namespace tree to maximize the number of concurrency for disjoint or independent operations.
was:
As we all known, the write performance of NameNode is limited by the global lock. We target to enable fine-grained locking based on the Namespace tree to improve the performance of NameNode write operations.
There are multiple motivations for creating this ticket:
* We have implemented this fine-grained locking and gained nearly 7x performance improvements in our prod environment
* Other companies made similar improvements based on their internal branch. Internal branches are quite different from the community, so few feedback and discussions in the community.
* The topic off fine-grained locking has been discussed for a very long time without progress.
We implemented this fine-gained locking based on the namespace tree to maximize the number of concurrency for disjoint or independent operations.
> NameNode Fine-Grained Locking via Namespace Tree
> ------------------------------------------------
>
> Key: HDFS-17366
> URL: https://issues.apache.org/jira/browse/HDFS-17366
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, namenode
> Reporter: ZanderXu
> Priority: Major
>
> As we all known, the write performance of NameNode is limited by the global lock. We target to enable fine-grained locking based on the Namespace tree to improve the performance of NameNode write operations.
> There are multiple motivations for creating this ticket:
> * We have implemented this fine-grained locking and gained nearly 7x performance improvements in our prod environment
> * Other companies made similar improvements based on their internal branch. Internal branches are quite different from the community, so few feedback and discussions in the community.
> * The topic of fine-grained locking has been discussed for a very long time, but still without any results.
>
> We implemented this fine-gained locking based on the namespace tree to maximize the number of concurrency for disjoint or independent operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org