You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Renukaprasad C (Jira)" <ji...@apache.org> on 2021/06/08 12:18:00 UTC
[jira] [Comment Edited] (HDFS-14703) NameNode Fine-Grained Locking
via Metadata Partitioning
[ https://issues.apache.org/jira/browse/HDFS-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17359306#comment-17359306 ]
Renukaprasad C edited comment on HDFS-14703 at 6/8/21, 12:17 PM:
-----------------------------------------------------------------
[~shv]/ [~xinglin]
We have implemented FGL for Create API and done basic testing & captured the performance reading. With the create API we could see around 25% of improvement.
I have created PR - [https://github.com/apache/hadoop/pull/3013] for the same. Can you please review & feedback when you get time?
Command:
/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs [file:///] -op create -threads 200 -files 1000000 -filesPerDir 40
Result:
||Iteration||Base||Patch||
|Itr-1|27124|32712|
|Itr-2|26460|31312|
|Itr-3|24166|32276|
|Avg|25916.66|32100|
|Improvement| |23.86|
was (Author: prasad-acit):
[~shv]/ [~xinglin]
We have implemented FGL for Create API and done basic testing & captured the performance reading. With the create API we could see around 25% of improvement.
I have created PR - [https://github.com/apache/hadoop/pull/3013] for the same. Can you please review & feedback when you get time?
Command:
/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs file:/// -op create -threads 200 -files 1000000 -filesPerDir 40
Result:
||Iteration||Heading 1||Heading 2||
|Itr-1|27124|32712|
|Itr-2|26460|31312|
|Itr-3|24166|32276|
|Avg|25916.66|32100|
|Improvement| |23.86|
> NameNode Fine-Grained Locking via Metadata Partitioning
> -------------------------------------------------------
>
> Key: HDFS-14703
> URL: https://issues.apache.org/jira/browse/HDFS-14703
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, namenode
> Reporter: Konstantin Shvachko
> Priority: Major
> Attachments: 001-partitioned-inodeMap-POC.tar.gz, 002-partitioned-inodeMap-POC.tar.gz, 003-partitioned-inodeMap-POC.tar.gz, NameNode Fine-Grained Locking.pdf, NameNode Fine-Grained Locking.pdf
>
>
> We target to enable fine-grained locking by splitting the in-memory namespace into multiple partitions each having a separate lock. Intended to improve performance of NameNode write operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org