You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Wei-Chiu Chuang (Jira)" <ji...@apache.org> on 2026/09/21 22:19:00 UTC

[jira] [Updated] (HDDS-16534) [Docs] Bootstrap a faulty OM

     [ https://issues.apache.org/jira/browse/HDDS-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wei-Chiu Chuang updated HDDS-16534:
-----------------------------------
    Description: 
If an OM fails to start due to corrupt OM db, or corrupt Ratis logs, it is possible to bootstrap its metadata from scratch.

 

Let's say OM fails with this error message (HDDS-15068):

 
{noformat}
2026-09-21 21:49:56,340 ERROR [main]-org.apache.hadoop.ozone.om.OzoneManagerStarter: OM start failed with exception
java.util.concurrent.CompletionException: java.lang.IllegalStateException: gap between start index 92074675 and first entry to append 131115267
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
        at java.base/java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1498)
        at java.base/java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1219)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
        at org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:191)
        at org.apache.ratis.util.ConcurrentUtils.lambda$null$4(ConcurrentUtils.java:180)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840){noformat}
 

Steps:

1. Stop the faulty OM if it's not stopped yet.
2. Take a back up of Ozone Manager Metadata Directory (ozone.om.db.dirs
) e.g. /var/lib/hadoop-ozone/om/data and delete it.
3. If the OM db is very large, apply the tunings ozone.om.ratis.log.purge.preservation.log.num to a high value (e.g. 1000000) and increase ozone.om.snapshot.checkpoint.dir.creation.poll.timeout to 1 minute. (Described in details in Troubleshooting OM HA snapshot installation issues)
4. Restart the other 2 healthy OM as well to apply the change.
5. Once the other 2 OMs are up and healthy, on the faulty OM, do 
{code:java}
ozone om --init
{code} or copy the VERSION file from the backup directory to ${ozone.om.db.dirs}/om/current/VERSION
.

The faulty OM should download the OM db checkpoint, load it up and start from there.

  was:
If an OM fails to start due to corrupt OM db, or corrupt Ratis logs, it is possible to bootstrap its metadata from scratch.

 

Let's say OM fails with this error message (HDDS-15068):

 
{noformat}
2026-09-21 21:49:56,340 ERROR [main]-org.apache.hadoop.ozone.om.OzoneManagerStarter: OM start failed with exception
java.util.concurrent.CompletionException: java.lang.IllegalStateException: gap between start index 92074675 and first entry to append 131115267
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
        at java.base/java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1498)
        at java.base/java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1219)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
        at org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:191)
        at org.apache.ratis.util.ConcurrentUtils.lambda$null$4(ConcurrentUtils.java:180)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840){noformat}
 

Steps:

1. Stop the faulty OM if it's not stopped yet.
2. Take a back up of Ozone Manager Metadata Directory (ozone.om.db.dirs
) e.g. /var/lib/hadoop-ozone/om/data and delete it.
3. If the OM db is very large, apply the tunings ozone.om.ratis.log.purge.preservation.log.num to a high value (e.g. 1000000) and increase ozone.om.snapshot.checkpoint.dir.creation.poll.timeout to 1 minute. (Described in details in Troubleshooting OM HA snapshot installation issues)
4. Restart the other 2 healthy OM as well to apply the change.
5. Once the other 2 OMs are up and healthy, restart the faulty OM.

The faulty OM should download the OM db checkpoint, load it up and start from there.


> [Docs] Bootstrap a faulty OM
> ----------------------------
>
>                 Key: HDDS-16534
>                 URL: https://issues.apache.org/jira/browse/HDDS-16534
>             Project: Apache Ozone
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Wei-Chiu Chuang
>            Priority: Major
>
> If an OM fails to start due to corrupt OM db, or corrupt Ratis logs, it is possible to bootstrap its metadata from scratch.
>  
> Let's say OM fails with this error message (HDDS-15068):
>  
> {noformat}
> 2026-09-21 21:49:56,340 ERROR [main]-org.apache.hadoop.ozone.om.OzoneManagerStarter: OM start failed with exception
> java.util.concurrent.CompletionException: java.lang.IllegalStateException: gap between start index 92074675 and first entry to append 131115267
>         at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
>         at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
>         at java.base/java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1498)
>         at java.base/java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1219)
>         at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
>         at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
>         at org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:191)
>         at org.apache.ratis.util.ConcurrentUtils.lambda$null$4(ConcurrentUtils.java:180)
>         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>         at java.base/java.lang.Thread.run(Thread.java:840){noformat}
>  
> Steps:
> 1. Stop the faulty OM if it's not stopped yet.
> 2. Take a back up of Ozone Manager Metadata Directory (ozone.om.db.dirs
> ) e.g. /var/lib/hadoop-ozone/om/data and delete it.
> 3. If the OM db is very large, apply the tunings ozone.om.ratis.log.purge.preservation.log.num to a high value (e.g. 1000000) and increase ozone.om.snapshot.checkpoint.dir.creation.poll.timeout to 1 minute. (Described in details in Troubleshooting OM HA snapshot installation issues)
> 4. Restart the other 2 healthy OM as well to apply the change.
> 5. Once the other 2 OMs are up and healthy, on the faulty OM, do 
> {code:java}
> ozone om --init
> {code} or copy the VERSION file from the backup directory to ${ozone.om.db.dirs}/om/current/VERSION
> .
> The faulty OM should download the OM db checkpoint, load it up and start from there.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org