You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Gang Xie (JIRA)" <ji...@apache.org> on 2018/06/01 10:19:00 UTC

[jira] [Commented] (HDFS-8865) Improve quota initialization performance

    [ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497816#comment-16497816 ] 

Gang Xie commented on HDFS-8865:
--------------------------------

We want to port this path to our 2.4 release. But we have a little concern about the mem pressure after multi-threaded, especially when it starts and transition to active state immediately, I.E. after the full block report. Do we have any mem pressure test result about this?

One idea to optimize this is not to held the children list since the namespace is not changed during this update.

> Improve quota initialization performance
> ----------------------------------------
>
>                 Key: HDFS-8865
>                 URL: https://issues.apache.org/jira/browse/HDFS-8865
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Major
>             Fix For: 2.8.0, 3.0.0-alpha1, 2.6.6, 2.7.5
>
>         Attachments: HDFS-8865.branch-2.6.01.patch, HDFS-8865.branch-2.6.patch, HDFS-8865.branch-2.7.patch, HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch, HDFS-8865_branch-2.6.patch, HDFS-8865_branch-2.7.patch
>
>
> After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time.  Since this is done during namenode failover, it also affects failover latency.
> By using the Fork-Join framework, I was able to greatly reduce the initialization time.  The following is the test result using the fsimage from one of the big name nodes we have.
> || threads || seconds||
> | 1 (existing) | 55|
> | 1 (fork-join) | 68 |
> | 4 | 16 |
> | 8 | 8 |
> | 12 | 6 |
> | 16 | 5 |
> | 20 | 4 |



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org