You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Wei-Chiu Chuang (Jira)" <ji...@apache.org> on 2026/09/21 22:19:00 UTC
[jira] [Updated] (HDDS-16534) [Docs] Bootstrap a faulty OM
[ https://issues.apache.org/jira/browse/HDDS-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wei-Chiu Chuang updated HDDS-16534:
-----------------------------------
Description:
If an OM fails to start due to corrupt OM db, or corrupt Ratis logs, it is possible to bootstrap its metadata from scratch.
Let's say OM fails with this error message (HDDS-15068):
{noformat}
2026-09-21 21:49:56,340 ERROR [main]-org.apache.hadoop.ozone.om.OzoneManagerStarter: OM start failed with exception
java.util.concurrent.CompletionException: java.lang.IllegalStateException: gap between start index 92074675 and first entry to append 131115267
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
at java.base/java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1498)
at java.base/java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1219)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:191)
at org.apache.ratis.util.ConcurrentUtils.lambda$null$4(ConcurrentUtils.java:180)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840){noformat}
Steps:
1. Stop the faulty OM if it's not stopped yet.
2. Take a back up of Ozone Manager Metadata Directory (ozone.om.db.dirs
) e.g. /var/lib/hadoop-ozone/om/data and delete it.
3. If the OM db is very large, apply the tunings ozone.om.ratis.log.purge.preservation.log.num to a high value (e.g. 1000000) and increase ozone.om.snapshot.checkpoint.dir.creation.poll.timeout to 1 minute. (Described in details in Troubleshooting OM HA snapshot installation issues)
4. Restart the other 2 healthy OM as well to apply the change.
5. Once the other 2 OMs are up and healthy, on the faulty OM, do
{code:java}
ozone om --init
{code} or copy the VERSION file from the backup directory to ${ozone.om.db.dirs}/om/current/VERSION
.
The faulty OM should download the OM db checkpoint, load it up and start from there.
was:
If an OM fails to start due to corrupt OM db, or corrupt Ratis logs, it is possible to bootstrap its metadata from scratch.
Let's say OM fails with this error message (HDDS-15068):
{noformat}
2026-09-21 21:49:56,340 ERROR [main]-org.apache.hadoop.ozone.om.OzoneManagerStarter: OM start failed with exception
java.util.concurrent.CompletionException: java.lang.IllegalStateException: gap between start index 92074675 and first entry to append 131115267
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
at java.base/java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1498)
at java.base/java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1219)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:191)
at org.apache.ratis.util.ConcurrentUtils.lambda$null$4(ConcurrentUtils.java:180)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840){noformat}
Steps:
1. Stop the faulty OM if it's not stopped yet.
2. Take a back up of Ozone Manager Metadata Directory (ozone.om.db.dirs
) e.g. /var/lib/hadoop-ozone/om/data and delete it.
3. If the OM db is very large, apply the tunings ozone.om.ratis.log.purge.preservation.log.num to a high value (e.g. 1000000) and increase ozone.om.snapshot.checkpoint.dir.creation.poll.timeout to 1 minute. (Described in details in Troubleshooting OM HA snapshot installation issues)
4. Restart the other 2 healthy OM as well to apply the change.
5. Once the other 2 OMs are up and healthy, restart the faulty OM.
The faulty OM should download the OM db checkpoint, load it up and start from there.
> [Docs] Bootstrap a faulty OM
> ----------------------------
>
> Key: HDDS-16534
> URL: https://issues.apache.org/jira/browse/HDDS-16534
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: documentation
> Reporter: Wei-Chiu Chuang
> Priority: Major
>
> If an OM fails to start due to corrupt OM db, or corrupt Ratis logs, it is possible to bootstrap its metadata from scratch.
>
> Let's say OM fails with this error message (HDDS-15068):
>
> {noformat}
> 2026-09-21 21:49:56,340 ERROR [main]-org.apache.hadoop.ozone.om.OzoneManagerStarter: OM start failed with exception
> java.util.concurrent.CompletionException: java.lang.IllegalStateException: gap between start index 92074675 and first entry to append 131115267
> at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> at java.base/java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1498)
> at java.base/java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1219)
> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> at org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:191)
> at org.apache.ratis.util.ConcurrentUtils.lambda$null$4(ConcurrentUtils.java:180)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:840){noformat}
>
> Steps:
> 1. Stop the faulty OM if it's not stopped yet.
> 2. Take a back up of Ozone Manager Metadata Directory (ozone.om.db.dirs
> ) e.g. /var/lib/hadoop-ozone/om/data and delete it.
> 3. If the OM db is very large, apply the tunings ozone.om.ratis.log.purge.preservation.log.num to a high value (e.g. 1000000) and increase ozone.om.snapshot.checkpoint.dir.creation.poll.timeout to 1 minute. (Described in details in Troubleshooting OM HA snapshot installation issues)
> 4. Restart the other 2 healthy OM as well to apply the change.
> 5. Once the other 2 OMs are up and healthy, on the faulty OM, do
> {code:java}
> ozone om --init
> {code} or copy the VERSION file from the backup directory to ${ozone.om.db.dirs}/om/current/VERSION
> .
> The faulty OM should download the OM db checkpoint, load it up and start from there.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org