[jira] [Created] (HDFS-17676) The frequency of submitting DFSInputStream refresh block locations in the LocatedBlocksRefresher thread is too fast.

["Longping Jie (Jira)" <ji...@apache.org>]

Longping Jie created HDFS-17676:
-----------------------------------

             Summary: The frequency of submitting DFSInputStream refresh block locations in the LocatedBlocksRefresher thread is too fast. 
                 Key: HDFS-17676
                 URL: https://issues.apache.org/jira/browse/HDFS-17676
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: dfsclient
    Affects Versions: 3.3.6
            Reporter: Longping Jie
         Attachments: image-2024-11-29-10-12-45-483.png

LocatedBlocksRefresher thread is responsible for submitting the task of DFSInputStream refresh block locations.

When a region server opens tens of thousands of DFSInputstreams, these DFSInputstream refresh block locations will be executed in a short period of time, which will put great pressure on the datanode, create a large number of DataXceiver threads in a short period of time, and cause GC fluctuations, resulting in time-consuming hbase queries. Burr appears

!image-2024-11-29-10-12-45-483.png!

Even if you set the size of refreshThreadPool to 1, you can't limit the rate well, consider adding a rate limiting mechanism, which can be enabled by parameter configuration to limit the refresh frequency.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org