[jira] [Created] (HDFS-17894) A new tool has been added to synchronize the latest FsImage

["JiangHua Zhu (Jira)" <ji...@apache.org>]

JiangHua Zhu created HDFS-17894:
-----------------------------------

             Summary: A new tool has been added to synchronize the latest FsImage
                 Key: HDFS-17894
                 URL: https://issues.apache.org/jira/browse/HDFS-17894
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: namenode
    Affects Versions: 3.4.1
            Reporter: JiangHua Zhu


The observer namenode stores the following fsimage files:
{code:java}
-rw-rw-r-- 1 hadoop hadoop 12075926817 3月  11 18:20 fsimage_0000000000810972885
-rw-rw-r-- 1 hadoop hadoop          62 3月  11 18:20 fsimage_0000000000810972885.md5
{code}

The standby namenode stores the latest fsimage file:
{code:java}
-rw-rw-r-- 1 hadoop hadoop 12142544499 Mar 20 04:42 fsimage_0000000000816122358
-rw-rw-r-- 1 hadoop hadoop          62 Mar 20 04:42 fsimage_0000000000816122358.md5
{code}

When the observer namenode has been running for a long time and needs to be restarted, it will prompt that the edit log is missing. The log is as follows:
{code:java}
2026-03-20 09:56:44,259 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception loading fsimage

java.io.IOException: There appears to be a gap in the edit log.  We expected txid 810972886, but got txid 814100454.

at org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:95)

at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:269)

at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:183)

at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:917)

at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:762)

at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:339)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1275)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:834)

at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:817)

at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:904)

at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1158)

at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1133)

at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1906)

at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1971)
{code}

We can add a tool to synchronize the latest fsimage from the standby namenode. This tool is mainly used by the administrator.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org