You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Colin Patrick McCabe (JIRA)" <ji...@apache.org> on 2015/09/26 13:21:04 UTC

[jira] [Commented] (HDFS-8873) Allow the directoryScanner to be rate-limited

    [ https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14909206#comment-14909206 ] 

Colin Patrick McCabe commented on HDFS-8873:
--------------------------------------------

+1.  Thanks, [~templedf].  The test failures are unrelated (just more noclassfound jenkins environment issues), and all pass for me locally.  Committed to 2.8 and trunk

> Allow the directoryScanner to be rate-limited
> ---------------------------------------------
>
>                 Key: HDFS-8873
>                 URL: https://issues.apache.org/jira/browse/HDFS-8873
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>    Affects Versions: 2.7.1
>            Reporter: Nathan Roberts
>            Assignee: Daniel Templeton
>         Attachments: HDFS-8873.001.patch, HDFS-8873.002.patch, HDFS-8873.003.patch, HDFS-8873.004.patch, HDFS-8873.005.patch, HDFS-8873.006.patch, HDFS-8873.007.patch, HDFS-8873.008.patch, HDFS-8873.009.patch
>
>
> The new 2-level directory layout can make directory scans expensive in terms of disk seeks (see HDFS-8791) for details. 
> It would be good if the directoryScanner() had a configurable duty cycle that would reduce its impact on disk performance (much like the approach in HDFS-8617). 
> Without such a throttle, disks can go 100% busy for many minutes at a time (assuming the common case of all inodes in cache but no directory blocks cached, 64K seeks are required for full directory listing which translates to 655 seconds) 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)