You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Longping Jie (Jira)" <ji...@apache.org> on 2024/11/29 02:14:00 UTC
[jira] [Commented] (HDFS-17676) The frequency of submitting DFSInputStream refresh block locations in the LocatedBlocksRefresher thread is too fast.
[ https://issues.apache.org/jira/browse/HDFS-17676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17901827#comment-17901827 ]
Longping Jie commented on HDFS-17676:
-------------------------------------
I'd be happy to submit this code if necessary.
> The frequency of submitting DFSInputStream refresh block locations in the LocatedBlocksRefresher thread is too fast.
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-17676
> URL: https://issues.apache.org/jira/browse/HDFS-17676
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: dfsclient
> Affects Versions: 3.3.6
> Reporter: Longping Jie
> Priority: Major
> Attachments: image-2024-11-29-10-12-45-483.png
>
>
> LocatedBlocksRefresher thread is responsible for submitting the task of DFSInputStream refresh block locations.
> When a region server opens tens of thousands of DFSInputstreams, these DFSInputstream refresh block locations will be executed in a short period of time, which will put great pressure on the datanode, create a large number of DataXceiver threads in a short period of time, and cause GC fluctuations, resulting in time-consuming hbase queries. Burr appears
> !image-2024-11-29-10-12-45-483.png!
> Even if you set the size of refreshThreadPool to 1, you can't limit the rate well, consider adding a rate limiting mechanism, which can be enabled by parameter configuration to limit the refresh frequency.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org