You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ethan Rose (Jira)" <ji...@apache.org> on 2022/11/22 21:32:00 UTC
[jira] [Commented] (HDDS-4226) Cleanup OM snapshots left after a failed installSnapshot
[ https://issues.apache.org/jira/browse/HDDS-4226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17637467#comment-17637467 ]
Ethan Rose commented on HDDS-4226:
----------------------------------
We have seen a similar issue in a cluster recently where the snapshot directory filled up the disk. Unfortunately the cluster was wiped before any evidence could be retrieved. The snapshots the follower downloads from the leader are never deleted if the move fails for any reason.
> Cleanup OM snapshots left after a failed installSnapshot
> --------------------------------------------------------
>
> Key: HDDS-4226
> URL: https://issues.apache.org/jira/browse/HDDS-4226
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Manager
> Reporter: Mukul Kumar Singh
> Priority: Major
> Labels: MiniOzoneChaosCluster
>
> Ozonemanager tries to install the snapshot
> {code:java}
> 2020-09-09 22:07:14,830 [pool-144-thread-1] INFO om.OzoneManager (OzoneManager.java:installCheckpoint(3159)) - Installing checkpoint with OMTransactionInfo 2#68754
> 2020-09-09 22:07:14,831 [grpc-default-executor-50] INFO impl.RaftServerImpl (RaftServerImpl.java:installSnapshot(1127)) - omNode-2@group-D62218D261DE: reply installSnapshot: omNode-1<-omNode-2#0:FAIL-t2,IN
> _PROGRESS
> {code}
> It failed because of the issues from HDDS-4224.
> {code:java}
> 2020-09-09 22:07:14,831 [pool-144-thread-1] ERROR om.OzoneManager (OzoneManager.java:installSnapshotFromLeader(3141)) - Failed to install snapshot from Leader OM: {}
> java.lang.NullPointerException
> at org.apache.hadoop.ozone.om.OzoneManager.installCheckpoint(OzoneManager.java:3168)
> at org.apache.hadoop.ozone.om.OzoneManager.installCheckpoint(OzoneManager.java:3162)
> at org.apache.hadoop.ozone.om.OzoneManager.installSnapshotFromLeader(OzoneManager.java:3139)
> at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$notifyInstallSnapshotFromLeader$4(OzoneManagerStateMachine.java:372)
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
> The checkpoint is left in the snapshot directory.
> {code:java}
> ➜ chaos-2020-09-09-22-05-33-IST ls MiniOzoneClusterImpl-71baac34-2321-4756-ba1e-5834c5628047/omNode-2/ratis/snapshot/om.db-omNode-1-1599669
> om.db-omNode-1-1599669432684/ om.db-omNode-1-1599669451421/ om.db-omNode-1-1599669478149/ om.db-omNode-1-1599669504818/ om.db-omNode-1-1599669533577/ om.db-omNode-1-1599669566509/
> om.db-omNode-1-1599669433775/ om.db-omNode-1-1599669453030/ om.db-omNode-1-1599669480273/ om.db-omNode-1-1599669507385/ om.db-omNode-1-1599669535603/ om.db-omNode-1-1599669568325/
> om.db-omNode-1-1599669434867/ om.db-omNode-1-1599669454688/ om.db-omNode-1-1599669482206/ om.db-omNode-1-1599669509373/ om.db-omNode-1-1599669537716/ om.db-omNode-1-1599669570186/
> om.db-omNode-1-1599669435886/ om.db-omNode-1-1599669456346/ om.db-omNode-1-1599669484256/ om.db-omNode-1-1599669511241/ om.db-omNode-1-1599669540574/ om.db-omNode-1-1599669572150/
> om.db-omNode-1-1599669437199/ om.db-omNode-1-1599669458194/ om.db-omNode-1-1599669486200/ om.db-omNode-1-1599669513051/ om.db-omNode-1-1599669543136/ om.db-omNode-1-1599669574811/
> om.db-omNode-1-1599669438519/ om.db-omNode-1-1599669459992/ om.db-omNode-1-1599669487968/ om.db-omNode-1-1599669515343/ om.db-omNode-1-1599669546272/ om.db-omNode-1-1599669576833/
> om.db-omNode-1-1599669439819/ om.db-omNode-1-1599669461897/ om.db-omNode-1-1599669490218/ om.db-omNode-1-1599669517332/ om.db-omNode-1-1599669548363/ om.db-omNode-1-1599669578680/
> om.db-omNode-1-1599669441209/ om.db-omNode-1-1599669463871/ om.db-omNode-1-1599669492005/ om.db-omNode-1-1599669519320/ om.db-omNode-1-1599669551596/ om.db-omNode-1-1599669580427/
> om.db-omNode-1-1599669442606/ om.db-omNode-1-1599669465810/ om.db-omNode-1-1599669493727/ om.db-omNode-1-1599669521491/ om.db-omNode-1-1599669554153/ om.db-omNode-1-1599669582124/
> om.db-omNode-1-1599669443967/ om.db-omNode-1-1599669467909/ om.db-omNode-1-1599669495587/ om.db-omNode-1-1599669523436/ om.db-omNode-1-1599669556370/ om.db-omNode-1-1599669583768/
> om.db-omNode-1-1599669445468/ om.db-omNode-1-1599669470054/ om.db-omNode-1-1599669497445/ om.db-omNode-1-1599669525567/ om.db-omNode-1-1599669558461/ om.db-omNode-1-1599669585501/
> om.db-omNode-1-1599669446937/ om.db-omNode-1-1599669472125/ om.db-omNode-1-1599669499362/ om.db-omNode-1-1599669527648/ om.db-omNode-1-1599669560578/
> om.db-omNode-1-1599669448360/ om.db-omNode-1-1599669474051/ om.db-omNode-1-1599669501269/ om.db-omNode-1-1599669529648/ om.db-omNode-1-1599669562666/
> om.db-omNode-1-1599669449867/ om.db-omNode-1-1599669476078/ om.db-omNode-1-1599669503036/ om.db-omNode-1-1599669531573/ om.db-omNode-1-1599669564620/ {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org