You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2025/12/12 10:14:00 UTC
[jira] [Commented] (HDFS-17864) Improve fsimage load time by making LightWeightGSet and NameCache thread-safe
[ https://issues.apache.org/jira/browse/HDFS-17864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18044637#comment-18044637 ]
ASF GitHub Bot commented on HDFS-17864:
---------------------------------------
khazhen opened a new pull request, #8130:
URL: https://github.com/apache/hadoop/pull/8130
Refer to [HDFS-17864](https://issues.apache.org/jira/browse/HDFS-17864).
HDFS-14617 allows the inode and inode directory sections of the fsimage to be loaded in parallel.
However, increasing the configured number of sections and threads has diminishing returns as there are some synchronized points in the loading code to protect some in memory structures.
Currently, there are mainly 3 data structures that need to be protected by synchronized blocks:
# INodeMap (internally based on LightWeightGSet, but it is not thread-safe)
# BlocksMap (internally based on LightWeightGSet, but it is not thread-safe)
# NameCache (it is not thread-safe by itself)
To further improve FSImage loading speed, this PR attempts to make the above 3 data structures thread-safe, and then use multiple threads to initialize them when NameNode starts.
Additionally, some optimizations have been made to reduce GC overheads during FSImage parsing.
In our tests, the FSImage loading time (165M inodes & 258M blocks) was reduced from 180s to 73s.
*1. Making LightWeightGSet thread-safe*
LightWeightGSet is a HashMap-like data structure that uses a fixed-length array as hash buckets, with each array element storing the head node of an independent linked list.
Since each linked list is independent, we can allocate a lock for each bucket to protect the corresponding linked list.
To trade off between memory consumption and concurrency, we can let multiple buckets share a lock and use a hash-based mapping.
To minimize changes, we don't plan to implement a completely thread-safe GSet to replace LightWeightGSet, as this would require significant changes and is unnecessary since all operations on LightWeightGSet are synchronized once NameNode finishes starting up.
We introduced an external synchronization tool GSetConcurrencyController to ensure the thread safety of LightWeightGSet during NameNode startup.
Another issue that needs to be addressed is the GSet's size. Currently, the size in LightWeightGSet is not an atomic variable, and even if we use segmented locks to protect hash buckets, the size is still inaccurate.
Fortunately, in the FSImage loading scenario, we can clearly know the expected size of INodeMap and BlocksMap after loading, so we can correct its size after loading is complete.
*2. Making NameCache thread-safe*
This is simpler compared to LightWeightGSet. We only need to combine ConcurrentHashMap and AtomicInteger to implement a thread-safe version of NameCache.
*3. Reducing GC pressure during FSImage loading*
After completing steps 1 and 2, we found that GC gradually became a new bottleneck. After analysis, we discovered that the parseDelimitedFrom method in ProtoBuffer creates a 4096-byte array as cache when parsing each INode object.
To optimize this issue, we introduced the DelimitedProtoBufParseHelper utility class to reuse the cache array.
Appendix: Test environment and configuration information
*Hadoop version*: current master, including previous fsimage loading optimizations: HDFS-13694, HDFS-14617, HDFS-15493
*FSImage information*:
Size: 20G (165M inodes & 258M blocks)
*Config:*
dfs.image.parallel.threads=16
dfs.image.parallel.target.sections=128
dfs.image.parallel.load=true
* new config in this patch:*
dfs.image.concurrent.init.inode.map.enable=true
dfs.image.name.cache.init.thread.num=16
dfs.image.block.map.init.thread.num=16
> Improve fsimage load time by making LightWeightGSet and NameCache thread-safe
> -----------------------------------------------------------------------------
>
> Key: HDFS-17864
> URL: https://issues.apache.org/jira/browse/HDFS-17864
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: khazhen
> Priority: Major
>
> HDFS-14617 allows the inode and inode directory sections of the fsimage to be loaded in parallel.
> However, increasing the configured number of sections and threads has diminishing returns as there are some synchronized points in the loading code to protect some in memory structures.
> Currently, there are mainly 3 data structures that need to be protected by synchronized blocks:
> # INodeMap (internally based on LightWeightGSet, but it is not thread-safe)
> # BlocksMap (internally based on LightWeightGSet, but it is not thread-safe)
> # NameCache (it is not thread-safe by itself)
> To further improve FSImage loading speed, this PR attempts to make the above 3 data structures thread-safe, and then use multiple threads to initialize them when NameNode starts.
> Additionally, some optimizations have been made to reduce GC overheads during FSImage parsing.
> In our tests, the FSImage loading time (165M inodes & 258M blocks) was reduced from 180s to 73s.
> *1. Making LightWeightGSet thread-safe*
> LightWeightGSet is a HashMap-like data structure that uses a fixed-length array as hash buckets, with each array element storing the head node of an independent linked list.
> Since each linked list is independent, we can allocate a lock for each bucket to protect the corresponding linked list.
> To trade off between memory consumption and concurrency, we can let multiple buckets share a lock and use a hash-based mapping.
> To minimize changes, we don't plan to implement a completely thread-safe GSet to replace LightWeightGSet, as this would require significant changes and is unnecessary since all operations on LightWeightGSet are synchronized once NameNode finishes starting up.
> We introduced an external synchronization tool GSetConcurrencyController to ensure the thread safety of LightWeightGSet during NameNode startup.
> Another issue that needs to be addressed is the GSet's size. Currently, the size in LightWeightGSet is not an atomic variable, and even if we use segmented locks to protect hash buckets, the size is still inaccurate.
> Fortunately, in the FSImage loading scenario, we can clearly know the expected size of INodeMap and BlocksMap after loading, so we can correct its size after loading is complete.
> *2. Making NameCache thread-safe*
> This is simpler compared to LightWeightGSet. We only need to combine ConcurrentHashMap and AtomicInteger to implement a thread-safe version of NameCache.
> *3. Reducing GC pressure during FSImage loading*
> After completing steps 1 and 2, we found that GC gradually became a new bottleneck. After analysis, we discovered that the parseDelimitedFrom method in ProtoBuffer creates a 4096-byte array as cache when parsing each INode object.
> To optimize this issue, we introduced the DelimitedProtoBufParseHelper utility class to reuse the cache array.
> Appendix: Test environment and configuration information
> *Hadoop version*: current master, including previous fsimage loading optimizations: HDFS-13694, HDFS-14617, HDFS-15493
> *FSImage information*:
> Size: 20G (165M inodes & 258M blocks)
> *Config:*
> dfs.image.parallel.threads=16
> dfs.image.parallel.target.sections=128
> dfs.image.parallel.load=true
> * new config in this patch:*
> dfs.image.concurrent.init.inode.map.enable=true
> dfs.image.name.cache.init.thread.num=16
> dfs.image.block.map.init.thread.num=16
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org