You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Lisheng Sun (JIRA)" <ji...@apache.org> on 2019/08/04 10:24:00 UTC
[jira] [Updated] (HDFS-13700) The process of loading image can be
done in a pipeline model
[ https://issues.apache.org/jira/browse/HDFS-13700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lisheng Sun updated HDFS-13700:
-------------------------------
Description:
The process of loading a file system image involves reading inodes section, deserializing inodes, initializing inodes, adding inodes to the global map, reading directories section, adding inodes to their parents' map, cache name etc. These steps can be done in a pipeline model to reduce the total duration.
Test the patch against a fsimage of a 70PB 2.4 cluster (200million files and 300million blocks, the fsimage is around 22GB), the image loading time be reduced from 1210 seconds to 739 seconds.
was:The process of loading a file system image involves reading inodes section, deserializing inodes, initializing inodes, adding inodes to the global map, reading directories section, adding inodes to their parents' map, cache name etc. These steps can be done in a pipeline model to reduce the total duration.
> The process of loading image can be done in a pipeline model
> ------------------------------------------------------------
>
> Key: HDFS-13700
> URL: https://issues.apache.org/jira/browse/HDFS-13700
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: zhouyingchao
> Assignee: Lisheng Sun
> Priority: Major
> Attachments: HDFS-13700-001.patch
>
>
> The process of loading a file system image involves reading inodes section, deserializing inodes, initializing inodes, adding inodes to the global map, reading directories section, adding inodes to their parents' map, cache name etc. These steps can be done in a pipeline model to reduce the total duration.
> Test the patch against a fsimage of a 70PB 2.4 cluster (200million files and 300million blocks, the fsimage is around 22GB), the image loading time be reduced from 1210 seconds to 739 seconds.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org