You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Yiqun Lin (Jira)" <ji...@apache.org> on 2020/03/14 12:07:00 UTC
[jira] [Created] (HDDS-3180) Datanode shutdown due to inconsistent
volume state without helpful error message
Yiqun Lin created HDDS-3180:
-------------------------------
Summary: Datanode shutdown due to inconsistent volume state without helpful error message
Key: HDDS-3180
URL: https://issues.apache.org/jira/browse/HDDS-3180
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Affects Versions: 0.4.1
Reporter: Yiqun Lin
Assignee: Yiqun Lin
I meet an error in my testing ozone cluster when I restart datanode. From the log, it throws inconsistent volume state but without other detailed helpful info:
{noformat}
2020-03-14 02:31:46,204 [main] INFO (LogAdapter.java:51) - registered UNIX signal handlers for [TERM, HUP, INT]
2020-03-14 02:31:46,736 [main] INFO (HddsDatanodeService.java:204) - HddsDatanodeService host:lyq-xx.xx.xx.xx ip:xx.xx.xx.xx
2020-03-14 02:31:46,784 [main] INFO (HddsVolume.java:177) - Creating Volume: /tmp/hadoop-hdfs/dfs/data/hdds of storage type : DISK and capacity : 20063645696
2020-03-14 02:31:46,786 [main] ERROR (MutableVolumeSet.java:202) - Failed to parse the storage location: file:///tmp/hadoop-hdfs/dfs/data
java.io.IOException: Volume is in an INCONSISTENT state. Skipped loading volume: /tmp/hadoop-hdfs/dfs/data/hdds
at org.apache.hadoop.ozone.container.common.volume.HddsVolume.initialize(HddsVolume.java:226)
at org.apache.hadoop.ozone.container.common.volume.HddsVolume.<init>(HddsVolume.java:180)
at org.apache.hadoop.ozone.container.common.volume.HddsVolume.<init>(HddsVolume.java:71)
at org.apache.hadoop.ozone.container.common.volume.HddsVolume$Builder.build(HddsVolume.java:158)
at org.apache.hadoop.ozone.container.common.volume.MutableVolumeSet.createVolume(MutableVolumeSet.java:336)
at org.apache.hadoop.ozone.container.common.volume.MutableVolumeSet.initializeVolumeSet(MutableVolumeSet.java:183)
at org.apache.hadoop.ozone.container.common.volume.MutableVolumeSet.<init>(MutableVolumeSet.java:139)
at org.apache.hadoop.ozone.container.common.volume.MutableVolumeSet.<init>(MutableVolumeSet.java:111)
at org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.<init>(OzoneContainer.java:97)
at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.<init>(DatanodeStateMachine.java:128)
at org.apache.hadoop.ozone.HddsDatanodeService.start(HddsDatanodeService.java:235)
at org.apache.hadoop.ozone.HddsDatanodeService.start(HddsDatanodeService.java:179)
at org.apache.hadoop.ozone.HddsDatanodeService.call(HddsDatanodeService.java:154)
at org.apache.hadoop.ozone.HddsDatanodeService.call(HddsDatanodeService.java:78)
at picocli.CommandLine.execute(CommandLine.java:1173)
at picocli.CommandLine.access$800(CommandLine.java:141)
at picocli.CommandLine$RunLast.handle(CommandLine.java:1367)
at picocli.CommandLine$RunLast.handle(CommandLine.java:1335)
at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243)
at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526)
at picocli.CommandLine.parseWithHandler(CommandLine.java:1465)
at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:65)
at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:56)
at org.apache.hadoop.ozone.HddsDatanodeService.main(HddsDatanodeService.java:137)
2020-03-14 02:31:46,795 [shutdown-hook-0] INFO (LogAdapter.java:51) - SHUTDOWN_MSG:
{noformat}
Then I look into the code and the root cause is that the version file was lost in that node.
We need to log key message as well to help user quickly know the root cause of this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org