You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "JiangHua Zhu (Jira)" <ji...@apache.org> on 2021/10/02 11:56:00 UTC

[jira] [Comment Edited] (HDFS-16245) Record the number of INodeDirectory when loading the FSImage file

    [ https://issues.apache.org/jira/browse/HDFS-16245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17423507#comment-17423507 ] 

JiangHua Zhu edited comment on HDFS-16245 at 10/2/21, 11:55 AM:
----------------------------------------------------------------

Here are some examples when loading the FSImage file is complete.
E.g:
2021-10-02 18:30:29,380 [Listener at localhost/55058] INFO namenode.FSImageFormatPBINode (FSImageFormatPBINode.java:loadINodeSectionInParallel(432))-Loading the INode section in parallel with 4 sub-sections
2021-10-02 18:30:29,381 [Listener at localhost/55058] INFO namenode.FSImageFormatPBINode (FSImageFormatPBINode.java:loadINodeSectionHeader(423))-Loading xxxx INodes.
2021-10-02 18:30:29,388 [Listener at localhost/55058] INFO namenode.FSImageFormatPBINode (FSImageFormatPBINode.java:loadINodeSectionInParallel(479))-Completed loading all INode sections. Loaded xxxx inodes.
2021-10-02 18:30:29,388 [Listener at localhost/55058] INFO namenode.FSImageFormatPBINode (FSImageFormatPBINode.java:loadINodeDirectorySectionInParallel(227))-Loading the INodeDirectory section in parallel with 4 sub-sections
2021-10-02 18:30:29,389 [Listener at localhost/55058] INFO namenode.FSImageFormatPBINode (FSImageFormatPBINode.java:loadINodeDirectorySectionInParallel(268))-*Completed loading all INodeDirectory sub-sections. Loaded xxxx inodes.*


was (Author: jianghuazhu):
Here are some examples when loading the FSImage file is complete.
E.g:
2021-10-02 18:30:29,380 [Listener at localhost/55058] INFO namenode.FSImageFormatPBINode (FSImageFormatPBINode.java:loadINodeSectionInParallel(432))-Loading the INode section in parallel with 4 sub-sections
2021-10-02 18:30:29,381 [Listener at localhost/55058] INFO namenode.FSImageFormatPBINode (FSImageFormatPBINode.java:loadINodeSectionHeader(423))-Loading xxxx INodes.
2021-10-02 18:30:29,388 [Listener at localhost/55058] INFO namenode.FSImageFormatPBINode (FSImageFormatPBINode.java:loadINodeSectionInParallel(479))-Completed loading all INode sections. Loaded xxxx inodes.
2021-10-02 18:30:29,388 [Listener at localhost/55058] INFO namenode.FSImageFormatPBINode (FSImageFormatPBINode.java:loadINodeDirectorySectionInParallel(227))-Loading the INodeDirectory section in parallel with 4 sub-sections
2021-10-02 18:30:29,389 [Listener at localhost/55058] INFO namenode.FSImageFormatPBINode (FSImageFormatPBINode.java:loadINodeDirectorySectionInParallel(268))-Completed loading all INodeDirectory sub-sections. Loaded xxxx inodes.

> Record the number of INodeDirectory when loading the FSImage file
> -----------------------------------------------------------------
>
>                 Key: HDFS-16245
>                 URL: https://issues.apache.org/jira/browse/HDFS-16245
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: JiangHua Zhu
>            Assignee: JiangHua Zhu
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> When starting parallel loading of FsImage (dfs.image.parallel.load=true), you can see in the startup log that the number of loaded INodes has been printed, but it is impossible to know the specific number of loaded INodeDirectory, because it is not printed now.
> For example, here are some startup information:
> 2021-09-30 19:12:55,012 [15587]-INFO [main:FSImageFormatProtobuf$Loader@340]-The fsimage will be loaded in parallel using 4 threads
> 2021-09-30 19:12:55,031 [15606]-INFO [main:FSImageFormatPBINode$Loader@418]-Loading the INode section in parallel with 12 sub-sections
> 2021-09-30 19:12:55,034 [15609]-INFO [main:FSImageFormatPBINode$Loader@409]-Loading xxxx INodes.
> ......
> 2021-09-30 19:30:37,080 [1077655]-INFO [main:FSImageFormatPBINode$Loader@465]-Completed loading all INode sections. Loaded xxxx inodes.
> 2021-09-30 19:30:37,086 [1077661]-INFO [main:FSImageFormatPBINode$Loader@222]-Loading the INodeDirectory section in parallel with 12 sub-sections
> ......
> 2021-09-30 19:36:58,074 [1458649]-INFO [main:FSImageFormatPBINode$Loader@261]-Completed loading all INodeDirectory sub-sections
> 2021-09-30 19:36:58,076 [1458651]-INFO [main:FSImageFormatPBINode$Loader@339]-Completed update blocks map and name cache, total waiting duration 1 ms.
> 2021-09-30 19:36:58,111 [1458686]-INFO [main:FSImageFormatProtobuf$Loader@248]-Loaded FSImage in xxxx seconds.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org