[jira] [Updated] (HDFS-4995) Make getContentSummary() less expensive

["Kihwal Lee (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDFS-4995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kihwal Lee updated HDFS-4995:
-----------------------------

    Attachment: HDFS-4995.trunk1.patch

Here is a candidate patch, which adds a runtime context object to keep track of progress at various inodes. When it goes over the configured limit, locks are relinquished and reacquired after a delay.  If this happens, all affected inodes should check for safety.  

With this change, the consistency is not guaranteed. But it is no different from what is expected from "du" command against normal file systems. If a tree is actively being modified, the result may not be consistent, but still useful. If the tree is not changing, it will give a consistent result.  Even without this patch, the result of getContentSummary() will get stale if the tree is constantly being modified. So I think the strict consistency is not required.

By setting the limit to something like 750,000, the max blocking time becomes about 15-25ms on todays typical Intel servers.  This is set to 0 by default, which makes the getContentSummary() calls blocking, which is the behavior before this patch.

> Make getContentSummary() less expensive
> ---------------------------------------
>
>                 Key: HDFS-4995
>                 URL: https://issues.apache.org/jira/browse/HDFS-4995
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 0.23.9, 2.3.0
>            Reporter: Kihwal Lee
>         Attachments: HDFS-4995.trunk1.patch
>
>
> When users call du or count DFS command, getContentSummary() method is called against namenode. If the directory has many directories and files, it could hold the namesystem lock for a long time. We've seen it taking over 20 seconds. Namenode should not allow regular users to cause extended locking.



--
This message was sent by Atlassian JIRA
(v6.1#6144)