You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Dave Teng (Jira)" <ji...@apache.org> on 2023/06/08 03:30:00 UTC
[jira] [Commented] (HDDS-8786) Encounter error of OzoneManagerDoubleBuffer Terminating during snapshot creation
[ https://issues.apache.org/jira/browse/HDDS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17730366#comment-17730366 ]
Dave Teng commented on HDDS-8786:
---------------------------------
Two other exceptions during snapshot creation:
{code:java}
2023-06-08 02:01:14,769 [OMDoubleBufferFlushThread] ERROR org.apache.hadoop.hdds.utils.db.RDBCheckpointManager: Unable to create RocksDB Snapsho
t.
java.io.IOException: RocksDatabase[/var/lib/hadoop-ozone/om/data/om.db]: Failed to flush; status : IOError; message : While open a file for appe
nding: /var/lib/hadoop-ozone/om/data/om.db/050410.sst: Too many open files
at org.apache.hadoop.hdds.utils.HddsServerUtil.toIOException(HddsServerUtil.java:588)
at org.apache.hadoop.hdds.utils.db.RocksDatabase.toIOException(RocksDatabase.java:88)
at org.apache.hadoop.hdds.utils.db.RocksDatabase.flush(RocksDatabase.java:502)
at org.apache.hadoop.hdds.utils.db.RDBCheckpointManager.createCheckpoint(RDBCheckpointManager.java:81)
at org.apache.hadoop.hdds.utils.db.RDBStore.getSnapshot(RDBStore.java:325)
at org.apache.hadoop.ozone.om.OmSnapshotManager.createOmSnapshotCheckpoint(OmSnapshotManager.java:391)
at org.apache.hadoop.ozone.om.response.snapshot.OMSnapshotCreateResponse.addToDBBatch(OMSnapshotCreateResponse.java:81)
at org.apache.hadoop.ozone.om.response.OMClientResponse.checkAndUpdateDB(OMClientResponse.java:73)
at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$5(OzoneManagerDoubleBuffer.java:414)
at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatchWithTrace(OzoneManagerDoubleBuffer.java:237)
at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatch(OzoneManagerDoubleBuffer.java:412)
at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushBatch(OzoneManagerDoubleBuffer.java:333)
at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushCurrentBuffer(OzoneManagerDoubleBuffer.java:312)
at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:277)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.rocksdb.RocksDBException: While open a file for appending: /var/lib/hadoop-ozone/om/data/om.db/050410.sst: Too many open files
at org.rocksdb.RocksDB.flush(Native Method)
at org.rocksdb.RocksDB.flush(RocksDB.java:3785)
at org.rocksdb.RocksDB.flush(RocksDB.java:3763)
at org.apache.hadoop.hdds.utils.db.RocksDatabase.flush(RocksDatabase.java:498)
... 12 more {code}
> Encounter error of OzoneManagerDoubleBuffer Terminating during snapshot creation
> --------------------------------------------------------------------------------
>
> Key: HDDS-8786
> URL: https://issues.apache.org/jira/browse/HDDS-8786
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Manager
> Reporter: Dave Teng
> Priority: Major
> Labels: ozone-snapshot
>
> When creating snapshot, encountered following error in OM:
> {code:java}
> 2023-06-08 02:01:14,771 [OMDoubleBufferFlushThread] ERROR org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer: Terminating with exit statu
> s 1: During flush to DB encountered error in OMDoubleBuffer flush thread OMDoubleBufferFlushThread when handling OMRequest: cmdType: CreateSnaps
> hot
> traceID: ""
> success: true
> status: OK
> CreateSnapshotResponse {
> snapshotInfo {
> snapshotID: "3ce30204-da71-41ad-9e78-607c688e8c5c"
> name: "snap5670"
> volumeName: "vol2"
> bucketName: "bucket2"
> snapshotStatus: SNAPSHOT_ACTIVE
> creationTime: 1686189673557
> deletionTime: 18446744073709551615
> pathPreviousSnapshotID: "180bd9f4-a83e-4bfb-97b3-a19d0eef0450"
> globalPreviousSnapshotID: "180bd9f4-a83e-4bfb-97b3-a19d0eef0450"
> snapshotPath: "vol2/bucket2"
> checkpointDir: "-3ce30204-da71-41ad-9e78-607c688e8c5c"
> dbTxSequenceNumber: 3269138
> }
> }
> java.io.IOException: Rocks Database is closed
> at org.apache.hadoop.hdds.utils.db.RocksDatabase.assertClose(RocksDatabase.java:442)
> at org.apache.hadoop.hdds.utils.db.RocksDatabase.newIterator(RocksDatabase.java:827)
> at org.apache.hadoop.hdds.utils.db.RDBTable.iterator(RDBTable.java:194)
> at org.apache.hadoop.hdds.utils.db.TypedTable.iterator(TypedTable.java:361)
> at org.apache.hadoop.ozone.om.OmSnapshotManager.deleteKeysFromDelKeyTableInSnapshotScope(OmSnapshotManager.java:573)
> at org.apache.hadoop.ozone.om.OmSnapshotManager.createOmSnapshotCheckpoint(OmSnapshotManager.java:394)
> at org.apache.hadoop.ozone.om.response.snapshot.OMSnapshotCreateResponse.addToDBBatch(OMSnapshotCreateResponse.java:81)
> at org.apache.hadoop.ozone.om.response.OMClientResponse.checkAndUpdateDB(OMClientResponse.java:73)
> at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$5(OzoneManagerDoubleBuffer.java:414)
> at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatchWithTrace(OzoneManagerDoubleBuffer.java:237)
> at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatch(OzoneManagerDoubleBuffer.java:412)
> at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushBatch(OzoneManagerDoubleBuffer.java:333)
> at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushCurrentBuffer(OzoneManagerDoubleBuffer.java:312)
> at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:277)
> at java.lang.Thread.run(Thread.java:748) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org