You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "liuguanghua (Jira)" <ji...@apache.org> on 2026/02/25 02:30:00 UTC
[jira] [Created] (HDFS-17886) Fix namenode storageDirectory errors when doCheckpoint updateStorageVersion failed because of doCheckpoint thread interrupted when standby namenode ha failover to active
liuguanghua created HDFS-17886:
----------------------------------
Summary: Fix namenode storageDirectory errors when doCheckpoint updateStorageVersion failed because of doCheckpoint thread interrupted when standby namenode ha failover to active
Key: HDFS-17886
URL: https://issues.apache.org/jira/browse/HDFS-17886
Project: Hadoop HDFS
Issue Type: Bug
Reporter: liuguanghua
Assignee: liuguanghua
When namenode ha failover occurs, the standby namenode convert to active namenode，it will interrupt doCheckpoint thread. There is an extremely small probability that doCheckpoint updateStorageVersion() will throw java.nio.channels.ClosedByInterruptException. It will lead to the storage directory errors and remove from available list.
The relevant error log is as follows：
2026-01-29 20:13:38,234 WARN org.apache.hadoop.hdfs.server.common.Storage: Error during write properties to the VERSION file to Storage Directory root= /data/hadoop/hdfs/namenode; location= null
java.nio.channels.ClosedByInterruptException
at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:199)
at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:162)
at java.base/sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:342)
at org.apache.hadoop.hdfs.server.common.Storage.writeProperties(Storage.java:1284)
at org.apache.hadoop.hdfs.server.common.Storage.writeProperties(Storage.java:1263)
at org.apache.hadoop.hdfs.server.common.Storage.writeProperties(Storage.java:1254)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.writeAll(NNStorage.java:1169)
at org.apache.hadoop.hdfs.server.namenode.FSImage.updateStorageVersion(FSImage.java:1106)
at org.apache.hadoop.hdfs.server.namenode.FSImage.saveNamespace(FSImage.java:1165)
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer.doCheckpoint(StandbyCheckpointer.java:227)
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer.access$1300(StandbyCheckpointer.java:64)
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread.doWork(StandbyCheckpointer.java:480)
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread.access$600(StandbyCheckpointer.java:383)
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread$1.run(StandbyCheckpointer.java:403)
at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:503)
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread.run(StandbyCheckpointer.java:399)
2026-01-29 20:13:38,238 ERROR org.apache.hadoop.hdfs.server.common.Storage: Error reported on storage directory Storage Directory root= /data/hadoop/hdfs/namenode; location= null
2026-01-29 20:13:38,238 WARN org.apache.hadoop.hdfs.server.common.Storage: About to remove corresponding storage: /data/hadoop/hdfs/namenode
2026-01-29 20:13:38,245 ERROR org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer: Exception in doCheckpoint
java.io.IOException: All the storage failed while writing properties to VERSION file
at org.apache.hadoop.hdfs.server.namenode.NNStorage.writeAll(NNStorage.java:1175)
at org.apache.hadoop.hdfs.server.namenode.FSImage.updateStorageVersion(FSImage.java:1106)
at org.apache.hadoop.hdfs.server.namenode.FSImage.saveNamespace(FSImage.java:1165)
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer.doCheckpoint(StandbyCheckpointer.java:227)
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer.access$1300(StandbyCheckpointer.java:64)
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread.doWork(StandbyCheckpointer.java:480)
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread.access$600(StandbyCheckpointer.java:383)
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread$1.run(StandbyCheckpointer.java:403)
at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:503)
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread.run(StandbyCheckpointer.java:399)
And java.nio.channels.ClosedByInterruptException is not a disk errors , so it should not remove from available storage list.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org