You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Sammi Chen (Jira)" <ji...@apache.org> on 2021/08/19 07:54:00 UTC
[jira] [Created] (HDDS-5641) Empty CURRENT file of rocksDB after DN
restart
Sammi Chen created HDDS-5641:
--------------------------------
Summary: Empty CURRENT file of rocksDB after DN restart
Key: HDDS-5641
URL: https://issues.apache.org/jira/browse/HDDS-5641
Project: Apache Ozone
Issue Type: Bug
Reporter: Sammi Chen
When restart a DN, found following LOGs,
2021-08-19 15:41:56,893 [Thread-6] ERROR org.apache.hadoop.ozone.container.common.utils.ContainerCache: Error opening DB. Container:54469 ContainerPath:/data11/ozone/hdds/hdds/CID-97a03be3-7554-46dc-ae39-58d600bfb107/current/containerDir106/54469/metadata/54469-dn-container.db
java.io.IOException: Failed init RocksDB, db path : /data11/ozone/hdds/hdds/CID-97a03be3-7554-46dc-ae39-58d600bfb107/current/containerDir106/54469/metadata/54469-dn-container.db, exception :org.rocksdb.RocksDBException CURRENT file does not end with newline; status : Corruption; message : CURRENT file does not end with newline
at org.apache.hadoop.hdds.utils.HddsServerUtil.toIOException(HddsServerUtil.java:545)
at org.apache.hadoop.hdds.utils.db.RDBStore.<init>(RDBStore.java:163)
at org.apache.hadoop.hdds.utils.db.DBStoreBuilder.build(DBStoreBuilder.java:191)
at org.apache.hadoop.ozone.container.metadata.AbstractDatanodeStore.start(AbstractDatanodeStore.java:131)
at org.apache.hadoop.ozone.container.metadata.AbstractDatanodeStore.<init>(AbstractDatanodeStore.java:106)
at org.apache.hadoop.ozone.container.metadata.DatanodeStoreSchemaTwoImpl.<init>(DatanodeStoreSchemaTwoImpl.java:48)
at org.apache.hadoop.ozone.container.keyvalue.helpers.BlockUtils.getUncachedDatanodeStore(BlockUtils.java:75)
at org.apache.hadoop.ozone.container.common.utils.ContainerCache.getDB(ContainerCache.java:159)
at org.apache.hadoop.ozone.container.keyvalue.helpers.BlockUtils.getDB(BlockUtils.java:121)
at org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil.parseKVContainerData(KeyValueContainerUtil.java:201)
at org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.upgradeContainerLayout(ContainerReader.java:246)
at org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.verifyAndFixupContainerData(ContainerReader.java:271)
at org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.verifyContainerFile(ContainerReader.java:215)
at org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.readVolume(ContainerReader.java:159)
at org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.run(ContainerReader.java:109)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.rocksdb.RocksDBException: CURRENT file does not end with newline
at org.rocksdb.RocksDB.open(Native Method)
at org.rocksdb.RocksDB.open(RocksDB.java:306)
at org.apache.hadoop.hdds.utils.db.RDBStore.<init>(RDBStore.java:118)
... 14 more
2021-08-19 15:41:56,895 [Thread-6] ERROR org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader: Failed to parse ContainerFile for ContainerID: 54469
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Error opening DB. Container:54469 ContainerPath:/data11/ozone/hdds/hdds/CID-97a03be3-7554-46dc-ae39-58d600bfb107/current/containerDir106/54469/metadata/54469-dn-container.db
at org.apache.hadoop.ozone.container.keyvalue.helpers.BlockUtils.getDB(BlockUtils.java:129)
at org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil.parseKVContainerData(KeyValueContainerUtil.java:201)
at org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.upgradeContainerLayout(ContainerReader.java:246)
at org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.verifyAndFixupContainerData(ContainerReader.java:271)
at org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.verifyContainerFile(ContainerReader.java:215)
at org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.readVolume(ContainerReader.java:159)
at org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.run(ContainerReader.java:109)
at java.lang.Thread.run(Thread.java:748)
A futher investigation shows that the CURRENT file is empty.
Here is the content of 54469.container,
!<KeyValueContainerData>
checksum: 4abf4aeeb494c96d47b8c5b0da3a7e8de8c18fe418efd150bc88531cb813fe5a
chunksPath: /data11/ozone/hdds/hdds/CID-97a03be3-7554-46dc-ae39-58d600bfb107/current/containerDir106/54469/chunks
containerDBType: RocksDB
containerID: 54469
containerType: KeyValueContainer
layOutVersion: 2
maxSize: 5368709120
metadata: {}
metadataPath: /data11/ozone/hdds/hdds/CID-97a03be3-7554-46dc-ae39-58d600bfb107/current/containerDir106/54469/metadata
originNodeId: 634a1474-5580-48ac-b652-618ba2a7a27e
originPipelineId: e397a2d3-0ee3-4d57-bad1-5914fb0fa3fa
schemaVersion: '2'
state: CLOSED
And here is the output of container info CLI
{
"state" : "CLOSED",
"replicationFactor" : "THREE",
"replicationType" : "RATIS",
"usedBytes" : 4026743214,
"numberOfKeys" : 31,
"lastUsed" : "2021-08-19T07:52:59.758Z",
"stateEnterTime" : "2021-05-14T06:39:18.675Z",
"owner" : "BP-374564002-11.26.14.197-1614743795885",
"containerID" : 54469,
"deleteTransactionId" : 0,
"sequenceId" : 3595566,
"open" : false
}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org