You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-issues@hadoop.apache.org by "Haohui Mai (JIRA)" <ji...@apache.org> on 2015/02/12 23:33:12 UTC

[jira] [Comment Edited] (HDFS-7784) load fsimage in parallel

    [ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319115#comment-14319115 ] 

Haohui Mai edited comment on HDFS-7784 at 2/12/15 10:32 PM:
------------------------------------------------------------

I've done some experiments in HDFS-5698. Parallelism does improve the performance, however, my feeling is that the improvement is *not* significant enough to justify the the complexity, especially having one race / bug here could easily lead to data loss.


was (Author: wheat9):
I've done some experiments in HDFS-5698. Parallelism does improve the performance, however, my feeling is that the improvement is  significant enough to justify the the complexity, especially having one race / bug here could easily lead to data loss.

> load fsimage in parallel
> ------------------------
>
>                 Key: HDFS-7784
>                 URL: https://issues.apache.org/jira/browse/HDFS-7784
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: Walter Su
>            Assignee: Walter Su
>
> When single Namenode has huge amount of files, without using federation, the startup/restart speed is slow. The fsimage loading step takes the most of the time. fsimage loading can seperate to two parts, deserialization and object construction(mostly map insertion). Deserialization takes the most of CPU time. So we can do deserialization in parallel, and add to hashmap in serial.  It will significantly reduce the NN start time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)