You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2019/11/15 05:41:04 UTC
[GitHub] [pulsar] kaykillyou opened a new issue #5668: bookie server's log
error: Failed to restore rocksdb
000000000000000000/000000000000000001/000000000000000000
kaykillyou opened a new issue #5668: bookie server's log error: Failed to restore rocksdb 000000000000000000/000000000000000001/000000000000000000
URL: https://github.com/apache/pulsar/issues/5668
**Describe the bug**
i have a pulsar instance consisting of two clusters, there is a bookie server background log in each cluster that continuously outputs the following error:
Failed to restore rocksdb 000000000000000008/000000000000000001/000000000000000000
Now producer and consumer are stopped, and bookie's heap memory usage reaches 90%.
**To Reproduce**
[root@iZbp1h590z6jc4f0ib1fjyZ logs]# pwd
/opt/bookieCluster/logs
[root@iZbp1h590z6jc4f0ib1fjyZ logs]# tail -f pulsar-bookie-iZbp1h590z6jc4f0ib1fjyZ.log
get error:
11:55:49.583 [io-write-scheduler-OrderedScheduler-1-0] WARN org.apache.bookkeeper.stream.storage.impl.sc.ZkStorageContainerManager - Failed to start storage container (8)
java.util.concurrent.CompletionException: org.apache.bookkeeper.statelib.api.exceptions.StateStoreException: Failed to restore rocksdb 000000000000000008/000000000000000001/000000000000000000
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_221]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_221]
at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:943) ~[?:1.8.0_221]
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926) ~[?:1.8.0_221]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_221]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_221]
at org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal.lambda$executeIO$16(AbstractStateStoreWithJournal.java:474) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_221]
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111) [com.google.guava-guava-21.0.jar:?]
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58) [com.google.guava-guava-21.0.jar:?]
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75) [com.google.guava-guava-21.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_221]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_221]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_221]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_221]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_221]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_221]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_221]
Caused by: org.apache.bookkeeper.statelib.api.exceptions.StateStoreException: Failed to restore rocksdb 000000000000000008/000000000000000001/000000000000000000
at org.apache.bookkeeper.statelib.impl.rocksdb.checkpoint.RocksCheckpointer.restore(RocksCheckpointer.java:84) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.kv.RocksdbKVStore.loadRocksdbFromCheckpointStore(RocksdbKVStore.java:161) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.kv.RocksdbKVStore.init(RocksdbKVStore.java:223) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal.lambda$initializeLocalStore$5(AbstractStateStoreWithJournal.java:202) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal.lambda$executeIO$16(AbstractStateStoreWithJournal.java:471) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
... 12 more
Caused by: java.io.FileNotFoundException: 000000000000000008/000000000000000001/000000000000000000/checkpoints/7723596f-fd11-4342-93a1-69cd30afa9dc/metadata
at org.apache.bookkeeper.statelib.impl.rocksdb.checkpoint.dlog.DLCheckpointStore.openInputStream(DLCheckpointStore.java:92) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.rocksdb.checkpoint.RocksCheckpointer.getLatestCheckpoint(RocksCheckpointer.java:117) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.rocksdb.checkpoint.RocksCheckpointer.restore(RocksCheckpointer.java:52) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.kv.RocksdbKVStore.loadRocksdbFromCheckpointStore(RocksdbKVStore.java:161) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.kv.RocksdbKVStore.init(RocksdbKVStore.java:223) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal.lambda$initializeLocalStore$5(AbstractStateStoreWithJournal.java:202) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal.lambda$executeIO$16(AbstractStateStoreWithJournal.java:471) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
... 12 more
I want to know what caused the issue.How should I solve it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services