You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "baijixue (Jira)" <ji...@apache.org> on 2021/07/09 18:46:00 UTC
[jira] [Updated] (HDFS-16123) NameNode standby checkpoint is
abnormal, because the data disk is full, even if the disk space is restored
[ https://issues.apache.org/jira/browse/HDFS-16123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
baijixue updated HDFS-16123:
----------------------------
Affects Version/s: (was: 3.1.1)
3.1.3
Environment:
Hadoop 3.1.1
When dfs.namenode.name.dir disk is full，standby nn ERROR:
2021-05-28 10:15:20,206 | WARN | IPC Server handler 48 on 25000 | Space available on volume '/dev/vdb' is 0, which is below the configured reserved amount 104857600 | NameNodeResourceChecker.java:91
2021-05-28 10:15:20,206 | INFO | IPC Server handler 48 on 25000 | IPC Server handler 48 on 25000, call org.apache.hadoop.ha.HAServiceProtocol.monitorHealth from 192.168.0.142:35128 Call#193407129 Retry#0 | Server.java
org.apache.hadoop.ha.HealthCheckFailedException: The NameNode has no resources available
at org.apache.hadoop.hdfs.server.namenode.NameNode.monitorHealth(NameNode.java:1781)
After hard disk space recovery，but checkpoint cannot be restored; the ERROR is：
2021-05-28 15:19:34,494 | ERROR | Standby State Checkpointer | Exception in doCheckpoint | StandbyCheckpointer.java:452
java.io.IOException: No image directories available!
at org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1219)
> NameNode standby checkpoint is abnormal, because the data disk is full, even if the disk space is restored
> -----------------------------------------------------------------------------------------------------------
>
> Key: HDFS-16123
> URL: https://issues.apache.org/jira/browse/HDFS-16123
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
> Affects Versions: 3.1.3
> Environment: Hadoop 3.1.1
> When dfs.namenode.name.dir disk is full，standby nn ERROR:
> 2021-05-28 10:15:20,206 | WARN | IPC Server handler 48 on 25000 | Space available on volume '/dev/vdb' is 0, which is below the configured reserved amount 104857600 | NameNodeResourceChecker.java:91
> 2021-05-28 10:15:20,206 | INFO | IPC Server handler 48 on 25000 | IPC Server handler 48 on 25000, call org.apache.hadoop.ha.HAServiceProtocol.monitorHealth from 192.168.0.142:35128 Call#193407129 Retry#0 | Server.java
> org.apache.hadoop.ha.HealthCheckFailedException: The NameNode has no resources available
> at org.apache.hadoop.hdfs.server.namenode.NameNode.monitorHealth(NameNode.java:1781)
>
> After hard disk space recovery，but checkpoint cannot be restored; the ERROR is：
> 2021-05-28 15:19:34,494 | ERROR | Standby State Checkpointer | Exception in doCheckpoint | StandbyCheckpointer.java:452
> java.io.IOException: No image directories available!
> at org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1219)
> Reporter: baijixue
> Priority: Major
>
> When the disk space of fsimages dir is located is full, standby NameNode reports the error： "Space available on volume'/dev/vdb' is 0"; after the space is restored, the standby nn's checkpoint cannot be restored, and the error is "java.io.IOException: No image directories available!"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org