You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Kihwal Lee (JIRA)" <ji...@apache.org> on 2013/07/15 23:30:49 UTC
[jira] [Commented] (HDFS-4995) Make getContentSummary() less
expensive
[ https://issues.apache.org/jira/browse/HDFS-4995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13709012#comment-13709012 ]
Kihwal Lee commented on HDFS-4995:
----------------------------------
Here is my first cut of idea.
If the result of these commands don't need to be strictly consistent, we can probably avoid potentially long lock region.
- Lock directory when obtaining a copy of children list (ReadOnlyList).
- Lock namesystem when computing in INodeFile.
- Handle stale children in the depth first traversing.
Recursion may need to be done at FSDirectory-level. This also makes it easy to ensure the namespace lock is not acquired while the directory lock is held. If we do this at the finest granularity, lock contention will increase and at minimum getContentSummary() will take considerably longer. We may internally batch (increased lock granularity) to reduce the impact. In this case, processing of a batch needs to be in the same namespace lock. Locking may look less tangled if the recursion is in FSNamesystem.
> Make getContentSummary() less expensive
> ---------------------------------------
>
> Key: HDFS-4995
> URL: https://issues.apache.org/jira/browse/HDFS-4995
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 0.23.9, 2.3.0
> Reporter: Kihwal Lee
>
> When users call du or count DFS command, getContentSummary() method is called against namenode. If the directory has many directories and files, it could hold the namesystem lock for a long time. We've seen it taking over 20 seconds. Namenode should not allow regular users to cause extended locking.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira