You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Yanlei Yu (Jira)" <ji...@apache.org> on 2022/12/02 02:26:00 UTC
[jira] [Commented] (HDFS-14657) Refine NameSystem lock usage during processing FBR
[ https://issues.apache.org/jira/browse/HDFS-14657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17642233#comment-17642233 ]
Yanlei Yu commented on HDFS-14657:
----------------------------------
hi [~zhangchen] ,Whether the new patch can be uploaded?
> Refine NameSystem lock usage during processing FBR
> --------------------------------------------------
>
> Key: HDFS-14657
> URL: https://issues.apache.org/jira/browse/HDFS-14657
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Chen Zhang
> Assignee: Chen Zhang
> Priority: Major
> Attachments: HDFS-14657-001.patch, HDFS-14657.002.patch
>
>
> The disk with 12TB capacity is very normal today, which means the FBR size is much larger than before, Namenode holds the NameSystemLock during processing block report for each storage, which might take quite a long time.
> On our production environment, processing large FBR usually cause a longer RPC queue time, which impacts client latency, so we did some simple work on refining the lock usage, which improved the p99 latency significantly.
> In our solution, BlockManager release the NameSystem write lock and request it again for every 5000 blocks(by default) during processing FBR, with the fair lock, all the RPC request can be processed before BlockManager re-acquire the write lock.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org