You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Jyotirmoy Sinha (Jira)" <ji...@apache.org> on 2025/06/04 05:43:00 UTC
[jira] [Updated] (HDDS-13172) OM down due to CompletionException
[ https://issues.apache.org/jira/browse/HDDS-13172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jyotirmoy Sinha updated HDDS-13172:
-----------------------------------
Issue Type: Bug (was: Improvement)
> OM down due to CompletionException
> ----------------------------------
>
> Key: HDDS-13172
> URL: https://issues.apache.org/jira/browse/HDDS-13172
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Manager
> Reporter: Jyotirmoy Sinha
> Priority: Major
>
> Steps :
> * Execute snapshot load workloads
> * Workload executed for 3+ days
> * Workload details -
> ** Volume - 100
> ** Buckets - 400 (4 per volume)
> *** Ratis-fso
> *** Ratis-obs
> *** Ec-fso
> *** Ec-obs
> ** Keys -
> *** 500-1000 per snapshot
> *** Size = 10kb
> * Configs -
> ** ozone.compaction.service.enabled = True
> ** ozone.om.compaction.service.run.interval = 1m
> ** ozone.om.snapshot.compaction.dag.prune.daemon.run.interval = 5m
> ** ozone.scm.block.size = 1KB
>
> OM Error logs -
> {{}}
> {code:java}
> 2025-06-03 05:22:20,972 INFO [om1546335944-impl-thread1]-org.apache.ratis.server.storage.RaftStorageDirectory: Lock on /var/lib/hadoop-ozone/om/ratis/2b34c18d-ed7a-3124-99bd-ca13a35e4b88/in_use.lock acquired by nodename 1680214@ccycloud-5.quasar-vckttz.root.comops.site
> 2025-06-03 05:22:20,978 ERROR [main]-org.apache.hadoop.ozone.om.OzoneManagerStarter: OM start failed with exception
> java.util.concurrent.CompletionException: java.io.IOException: Failed to load /var/lib/hadoop-ozone/om/ratis/2b34c18d-ed7a-3124-99bd-ca13a35e4b88/current/raft-meta
> at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> at java.base/java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1498)
> at java.base/java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1219)
> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> at org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:191)
> at org.apache.ratis.util.ConcurrentUtils.lambda$null$4(ConcurrentUtils.java:180)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:833)
> Caused by: java.io.IOException: Failed to load /var/lib/hadoop-ozone/om/ratis/2b34c18d-ed7a-3124-99bd-ca13a35e4b88/current/raft-meta
> at org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.load(RaftStorageMetadataFileImpl.java:118)
> at org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.lambda$getMetadata$0(RaftStorageMetadataFileImpl.java:55)
> at org.apache.ratis.util.ConcurrentUtils.lambda$updateAndGet$0(ConcurrentUtils.java:51)
> at java.base/java.util.concurrent.atomic.AtomicReference.updateAndGet(AtomicReference.java:210)
> at org.apache.ratis.util.ConcurrentUtils.updateAndGet(ConcurrentUtils.java:49)
> at org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.getMetadata(RaftStorageMetadataFileImpl.java:55)
> at org.apache.ratis.server.storage.RaftStorageImpl.analyzeAndRecoverStorage(RaftStorageImpl.java:116)
> at org.apache.ratis.server.storage.RaftStorageImpl.initialize(RaftStorageImpl.java:63)
> at org.apache.ratis.server.impl.ServerState.initialize(ServerState.java:169)
> at org.apache.ratis.server.impl.RaftServerImpl.start(RaftServerImpl.java:338)
> at org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:188)
> ... 4 more
> Caused by: java.io.IOException: Failed to parse 'term' from properties: {}
> at org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.getTerm(RaftStorageMetadataFileImpl.java:96)
> at org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.load(RaftStorageMetadataFileImpl.java:116)
> ... 14 more
> Caused by: java.io.IOException: 'term' not found in properties: {}
> at org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.lambda$getValue$2(RaftStorageMetadataFileImpl.java:89)
> at java.base/java.util.Optional.orElseThrow(Optional.java:403)
> at org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.getValue(RaftStorageMetadataFileImpl.java:88)
> at org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.getTerm(RaftStorageMetadataFileImpl.java:94)
> ... 15 more
> 2025-06-03 05:22:20,982 INFO [shutdown-hook-0]-org.apache.ranger.audit.provider.AuditProviderFactory: ==> JVMShutdownHook.run() {code}
> {{}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org