You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-dev@hadoop.apache.org by "HuangTao (JIRA)" <ji...@apache.org> on 2019/04/09 10:19:00 UTC

[jira] [Created] (HDFS-14419) Avoid repeated calls to the listOpenFiles function

HuangTao created HDFS-14419:
-------------------------------

             Summary: Avoid repeated calls to the listOpenFiles function
                 Key: HDFS-14419
                 URL: https://issues.apache.org/jira/browse/HDFS-14419
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: namenode, performance
    Affects Versions: 3.3.0
            Reporter: HuangTao
            Assignee: HuangTao
             Fix For: 3.3.0


`hdfs dfsadmin -listOpenFiles -path /any/path` will request all opened files. In the NameNode side, the function LeaseManager.java#getUnderConstructionFiles will be called.

When there are only N(<maxListOpenFilesResponses, default is 100) files meet the conditions, but the leaseManager contains M(>maxListOpenFilesResponses) files, we will scan all leases. Finally, the hasMore will be set true and the openFileEntries contains N files, the scenario will cause listOpenFiles() will be called again.

If M is greater more than N, the two calls to getUnderConstructionFiles will impact the NameNode performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org