You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2019/10/12 02:21:52 UTC

[GitHub] [pulsar] ronaldofreitas opened a new issue #5373: Pulsar Standalone is shutdown before complete starting

ronaldofreitas opened a new issue #5373: Pulsar Standalone is shutdown before complete starting
URL: https://github.com/apache/pulsar/issues/5373
 
 
   **Describe the bug**
   Before this problem happened, I was using it normally. I had to restart the machine at brute force and after I restarted my computer the pulsar started to catch this problem. I removed and installed the pulsar from scratch and the problem continues.
   
   **To Reproduce**
   ./pulsar standalone
   
   
   23:14:51.138 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.distributedlog.BKLogSegmentWriter - Stream streams_000000000000000001_000000000000000001_000000000000000000:<default>:inprogress_000000000000000002 aborted 0 writes
   23:14:51.156 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1000044fedc0007
   23:14:51.156 [Thread-1] INFO  org.apache.zookeeper.ZooKeeper - Session: 0x1000044fedc0007 closed
   23:14:51.158 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO  org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:46064 which had sessionid 0x1000044fedc0007
   23:14:51.160 [Thread-1] INFO  org.apache.distributedlog.BookKeeperClient - BookKeeper Client closed bk:distributedlog://127.0.0.1:2181/stream/storage:factory_writer_shared
   23:14:51.160 [Thread-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 127.0.0.1:3181
   23:14:51.164 [DL-io-0] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0x9988bc98, L:/127.0.0.1:37070 ! R:localhost/127.0.0.1:3181]
   23:14:51.165 [Thread-1] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=8,loopId=3d11891f) Error writing metadata to store
   org.apache.bookkeeper.client.BKException$BKClientClosedException: BookKeeper client is closed
   	at org.apache.bookkeeper.meta.CleanupLedgerManager.close(CleanupLedgerManager.java:245) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.client.BookKeeper.close(BookKeeper.java:1410) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.BookKeeperClient.close(BookKeeperClient.java:270) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.impl.BKNamespaceDriver.doClose(BKNamespaceDriver.java:404) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.impl.BKNamespaceDriver.close(BKNamespaceDriver.java:385) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at com.google.common.io.Closeables.close(Closeables.java:76) ~[com.google.guava-guava-21.0.jar:?]
   	at org.apache.distributedlog.util.Utils.close(Utils.java:544) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.BKDistributedLogNamespace.close(BKDistributedLogNamespace.java:341) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.stream.server.service.DLNamespaceProviderService.doClose(DLNamespaceProviderService.java:135) ~[org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$close$4(LifecycleComponentStack.java:123) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:408) [com.google.guava-guava-21.0.jar:?]
   	at org.apache.bookkeeper.common.component.LifecycleComponentStack.close(LifecycleComponentStack.java:123) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent.doClose(StreamStorageLifecycleComponent.java:61) [org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.stop(LocalBookkeeperEnsemble.java:405) [org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:2.4.1]
   	at org.apache.pulsar.PulsarStandaloneStarter$1.run(PulsarStandaloneStarter.java:104) [org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
   23:14:51.165 [Thread-1] ERROR org.apache.distributedlog.BKAbstractLogWriter - Completing Log segments encountered exception
   java.io.IOException: Failed to close ledger for streams_000000000000000000_000000000000000001_000000000000000000:<default>:inprogress_000000000000000002 : BookKeeper client is closed
   	at org.apache.distributedlog.BKLogSegmentWriter$6.closeComplete(BKLogSegmentWriter.java:660) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.client.LedgerHandle$5.lambda$safeRun$0(LedgerHandle.java:550) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_202]
   	at org.apache.bookkeeper.client.LedgerHandle$5.lambda$safeRun$3(LedgerHandle.java:612) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_202]
   	at org.apache.bookkeeper.client.MetadataUpdateLoop.lambda$writeLoop$1(MetadataUpdateLoop.java:146) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_202]
   	at org.apache.bookkeeper.meta.CleanupLedgerManager.lambda$close$1(CleanupLedgerManager.java:246) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4649) ~[?:1.8.0_202]
   	at org.apache.bookkeeper.meta.CleanupLedgerManager.close(CleanupLedgerManager.java:246) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.client.BookKeeper.close(BookKeeper.java:1410) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.BookKeeperClient.close(BookKeeperClient.java:270) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.impl.BKNamespaceDriver.doClose(BKNamespaceDriver.java:404) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.impl.BKNamespaceDriver.close(BKNamespaceDriver.java:385) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at com.google.common.io.Closeables.close(Closeables.java:76) ~[com.google.guava-guava-21.0.jar:?]
   	at org.apache.distributedlog.util.Utils.close(Utils.java:544) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.BKDistributedLogNamespace.close(BKDistributedLogNamespace.java:341) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.stream.server.service.DLNamespaceProviderService.doClose(DLNamespaceProviderService.java:135) ~[org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$close$4(LifecycleComponentStack.java:123) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:408) [com.google.guava-guava-21.0.jar:?]
   	at org.apache.bookkeeper.common.component.LifecycleComponentStack.close(LifecycleComponentStack.java:123) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent.doClose(StreamStorageLifecycleComponent.java:61) [org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.stop(LocalBookkeeperEnsemble.java:405) [org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:2.4.1]
   	at org.apache.pulsar.PulsarStandaloneStarter$1.run(PulsarStandaloneStarter.java:104) [org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
   23:14:51.167 [bookie-io-1-2] INFO  org.apache.bookkeeper.proto.BookieRequestHandler - Channels disconnected: [id: 0x93825262, L:/127.0.0.1:3181 ! R:/127.0.0.1:37070]
   23:14:51.167 [DLM-/stream/storage-OrderedScheduler-1-0] INFO  org.apache.distributedlog.bk.SimpleLedgerAllocator - Abort ledger allocator without cleaning up on /stream/storage/streams_000000000000000000_000000000000000001_000000000000000000/<default>/allocation.
   23:14:51.169 [io-read-scheduler-OrderedScheduler-1-0] INFO  org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal - Successfully close the log stream of state store 000000000000000000/000000000000000001/000000000000000000
   23:14:51.170 [Thread-1] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=9,loopId=69cdb660) Error writing metadata to store
   org.apache.bookkeeper.client.BKException$BKClientClosedException: BookKeeper client is closed
   	at org.apache.bookkeeper.meta.CleanupLedgerManager.close(CleanupLedgerManager.java:245) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.client.BookKeeper.close(BookKeeper.java:1410) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.BookKeeperClient.close(BookKeeperClient.java:270) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.impl.BKNamespaceDriver.doClose(BKNamespaceDriver.java:404) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.impl.BKNamespaceDriver.close(BKNamespaceDriver.java:385) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at com.google.common.io.Closeables.close(Closeables.java:76) ~[com.google.guava-guava-21.0.jar:?]
   	at org.apache.distributedlog.util.Utils.close(Utils.java:544) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.BKDistributedLogNamespace.close(BKDistributedLogNamespace.java:341) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.stream.server.service.DLNamespaceProviderService.doClose(DLNamespaceProviderService.java:135) ~[org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$close$4(LifecycleComponentStack.java:123) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:408) [com.google.guava-guava-21.0.jar:?]
   	at org.apache.bookkeeper.common.component.LifecycleComponentStack.close(LifecycleComponentStack.java:123) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent.doClose(StreamStorageLifecycleComponent.java:61) [org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.stop(LocalBookkeeperEnsemble.java:405) [org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:2.4.1]
   	at org.apache.pulsar.PulsarStandaloneStarter$1.run(PulsarStandaloneStarter.java:104) [org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
   23:14:51.171 [Thread-1] ERROR org.apache.distributedlog.BKAbstractLogWriter - Completing Log segments encountered exception
   java.io.IOException: Failed to close ledger for streams_000000000000000001_000000000000000001_000000000000000000:<default>:inprogress_000000000000000002 : BookKeeper client is closed
   	at org.apache.distributedlog.BKLogSegmentWriter$6.closeComplete(BKLogSegmentWriter.java:660) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.client.LedgerHandle$5.lambda$safeRun$0(LedgerHandle.java:550) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_202]
   	at org.apache.bookkeeper.client.LedgerHandle$5.lambda$safeRun$3(LedgerHandle.java:612) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_202]
   	at org.apache.bookkeeper.client.MetadataUpdateLoop.lambda$writeLoop$1(MetadataUpdateLoop.java:146) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_202]
   	at org.apache.bookkeeper.meta.CleanupLedgerManager.lambda$close$1(CleanupLedgerManager.java:246) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4649) ~[?:1.8.0_202]
   	at org.apache.bookkeeper.meta.CleanupLedgerManager.close(CleanupLedgerManager.java:246) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.client.BookKeeper.close(BookKeeper.java:1410) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.BookKeeperClient.close(BookKeeperClient.java:270) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.impl.BKNamespaceDriver.doClose(BKNamespaceDriver.java:404) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.impl.BKNamespaceDriver.close(BKNamespaceDriver.java:385) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at com.google.common.io.Closeables.close(Closeables.java:76) ~[com.google.guava-guava-21.0.jar:?]
   	at org.apache.distributedlog.util.Utils.close(Utils.java:544) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.BKDistributedLogNamespace.close(BKDistributedLogNamespace.java:341) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.stream.server.service.DLNamespaceProviderService.doClose(DLNamespaceProviderService.java:135) ~[org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$close$4(LifecycleComponentStack.java:123) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:408) [com.google.guava-guava-21.0.jar:?]
   	at org.apache.bookkeeper.common.component.LifecycleComponentStack.close(LifecycleComponentStack.java:123) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent.doClose(StreamStorageLifecycleComponent.java:61) [org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.stop(LocalBookkeeperEnsemble.java:405) [org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:2.4.1]
   	at org.apache.pulsar.PulsarStandaloneStarter$1.run(PulsarStandaloneStarter.java:104) [org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
   23:14:51.173 [DLM-/stream/storage-OrderedScheduler-0-0] INFO  org.apache.distributedlog.bk.SimpleLedgerAllocator - Abort ledger allocator without cleaning up on /stream/storage/streams_000000000000000001_000000000000000001_000000000000000000/<default>/allocation.
   23:14:51.173 [Thread-1] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=6,loopId=6610c33f) Error writing metadata to store
   org.apache.bookkeeper.client.BKException$BKClientClosedException: BookKeeper client is closed
   	at org.apache.bookkeeper.meta.CleanupLedgerManager.close(CleanupLedgerManager.java:245) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.client.BookKeeper.close(BookKeeper.java:1410) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.BookKeeperClient.close(BookKeeperClient.java:270) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.impl.BKNamespaceDriver.doClose(BKNamespaceDriver.java:404) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.impl.BKNamespaceDriver.close(BKNamespaceDriver.java:385) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at com.google.common.io.Closeables.close(Closeables.java:76) ~[com.google.guava-guava-21.0.jar:?]
   	at org.apache.distributedlog.util.Utils.close(Utils.java:544) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.BKDistributedLogNamespace.close(BKDistributedLogNamespace.java:341) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.stream.server.service.DLNamespaceProviderService.doClose(DLNamespaceProviderService.java:135) ~[org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$close$4(LifecycleComponentStack.java:123) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:408) [com.google.guava-guava-21.0.jar:?]
   	at org.apache.bookkeeper.common.component.LifecycleComponentStack.close(LifecycleComponentStack.java:123) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent.doClose(StreamStorageLifecycleComponent.java:61) [org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.stop(LocalBookkeeperEnsemble.java:405) [org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:2.4.1]
   	at org.apache.pulsar.PulsarStandaloneStarter$1.run(PulsarStandaloneStarter.java:104) [org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
   23:14:51.174 [io-read-scheduler-OrderedScheduler-1-0] INFO  org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal - Successfully close the log stream of state store 000000000000000001/000000000000000001/000000000000000000
   23:14:51.178 [io-read-scheduler-OrderedScheduler-1-0] INFO  org.apache.bookkeeper.stream.storage.impl.sc.StorageContainerImpl - Successfully stopped storage container (1).
   23:14:51.174 [Thread-1] ERROR org.apache.distributedlog.BKAbstractLogWriter - Completing Log segments encountered exception
   java.io.IOException: Failed to close ledger for streams_000000000000000000_000000000000000000_000000000000000000:<default>:inprogress_000000000000000002 : BookKeeper client is closed
   	at org.apache.distributedlog.BKLogSegmentWriter$6.closeComplete(BKLogSegmentWriter.java:660) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.client.LedgerHandle$5.lambda$safeRun$0(LedgerHandle.java:550) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_202]
   	at org.apache.bookkeeper.client.LedgerHandle$5.lambda$safeRun$3(LedgerHandle.java:612) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_202]
   	at org.apache.bookkeeper.client.MetadataUpdateLoop.lambda$writeLoop$1(MetadataUpdateLoop.java:146) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_202]
   	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_202]
   	at org.apache.bookkeeper.meta.CleanupLedgerManager.lambda$close$1(CleanupLedgerManager.java:246) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4649) ~[?:1.8.0_202]
   	at org.apache.bookkeeper.meta.CleanupLedgerManager.close(CleanupLedgerManager.java:246) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.client.BookKeeper.close(BookKeeper.java:1410) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.BookKeeperClient.close(BookKeeperClient.java:270) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.impl.BKNamespaceDriver.doClose(BKNamespaceDriver.java:404) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.impl.BKNamespaceDriver.close(BKNamespaceDriver.java:385) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at com.google.common.io.Closeables.close(Closeables.java:76) ~[com.google.guava-guava-21.0.jar:?]
   	at org.apache.distributedlog.util.Utils.close(Utils.java:544) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.distributedlog.BKDistributedLogNamespace.close(BKDistributedLogNamespace.java:341) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.stream.server.service.DLNamespaceProviderService.doClose(DLNamespaceProviderService.java:135) ~[org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$close$4(LifecycleComponentStack.java:123) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:408) [com.google.guava-guava-21.0.jar:?]
   	at org.apache.bookkeeper.common.component.LifecycleComponentStack.close(LifecycleComponentStack.java:123) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent.doClose(StreamStorageLifecycleComponent.java:61) [org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2]
   	at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
   	at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.stop(LocalBookkeeperEnsemble.java:405) [org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:2.4.1]
   	at org.apache.pulsar.PulsarStandaloneStarter$1.run(PulsarStandaloneStarter.java:104) [org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
   23:14:51.179 [io-read-scheduler-OrderedScheduler-1-0] INFO  org.apache.bookkeeper.stream.storage.impl.sc.ZkStorageContainerManager - Successfully stopped storage container (1)
   23:14:51.179 [io-read-scheduler-OrderedScheduler-1-0] INFO  org.apache.bookkeeper.stream.storage.impl.sc.ZkStorageContainerManager - Storage container (1) is removed from live set.
   23:14:51.179 [DLM-/stream/storage-OrderedScheduler-3-0] INFO  org.apache.distributedlog.bk.SimpleLedgerAllocator - Abort ledger allocator without cleaning up on /stream/storage/streams_000000000000000000_000000000000000000_000000000000000000/<default>/allocation.
   23:14:51.180 [io-read-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal - Successfully close the log stream of state store 000000000000000000/000000000000000000/000000000000000000
   23:14:51.183 [Thread-1] INFO  org.apache.distributedlog.ZooKeeperClient - Close zookeeper client bk:distributedlog://127.0.0.1:2181/stream/storage:factory_writer_shared:zk.
   23:14:51.183 [Thread-1] INFO  org.apache.distributedlog.ZooKeeperClient - Closing zookeeper client bk:distributedlog://127.0.0.1:2181/stream/storage:factory_writer_shared:zk.
   23:14:51.184 [ProcessThread(sid:0 cport:2181):] INFO  org.apache.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x1000044fedc0008
   23:14:51.352 [Thread-1] INFO  org.apache.zookeeper.ZooKeeper - Session: 0x1000044fedc0008 closed
   23:14:51.352 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO  org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:46070 which had sessionid 0x1000044fedc0008
   23:14:51.352 [Thread-1] INFO  org.apache.distributedlog.ZooKeeperClient - Closed zookeeper client bk:distributedlog://127.0.0.1:2181/stream/storage:factory_writer_shared:zk.
   23:14:51.352 [io-write-scheduler-OrderedScheduler-0-0-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1000044fedc0008
   23:14:51.352 [Thread-1] INFO  org.apache.distributedlog.ZooKeeperClient - Close zookeeper client dlzk:distributedlog://127.0.0.1:2181/stream/storage:factory_writer_shared.
   23:14:51.352 [Thread-1] INFO  org.apache.distributedlog.ZooKeeperClient - Closing zookeeper client dlzk:distributedlog://127.0.0.1:2181/stream/storage:factory_writer_shared.
   23:14:51.353 [ProcessThread(sid:0 cport:2181):] INFO  org.apache.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x1000044fedc0006
   23:14:51.465 [Thread-1] INFO  org.apache.zookeeper.ZooKeeper - Session: 0x1000044fedc0006 closed
   23:14:51.466 [Thread-1] INFO  org.apache.distributedlog.ZooKeeperClient - Closed zookeeper client dlzk:distributedlog://127.0.0.1:2181/stream/storage:factory_writer_shared.
   23:14:51.466 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1000044fedc0006
   23:14:51.466 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO  org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:46062 which had sessionid 0x1000044fedc0006
   23:14:51.467 [io-read-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.stream.storage.impl.sc.StorageContainerImpl - Successfully stopped storage container (0).
   23:14:51.467 [io-read-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.stream.storage.impl.sc.ZkStorageContainerManager - Successfully stopped storage container (0)
   23:14:51.467 [io-read-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.stream.storage.impl.sc.ZkStorageContainerManager - Storage container (0) is removed from live set.
   23:14:51.468 [Thread-1] INFO  org.apache.distributedlog.impl.BKNamespaceDriver - Release external resources used by channel factory.
   23:14:51.469 [Thread-1] INFO  org.apache.distributedlog.impl.BKNamespaceDriver - Stopped request timer
   23:14:51.470 [Thread-1] INFO  org.apache.distributedlog.BKDistributedLogNamespace - Executor Service Stopped.
   23:14:51.471 [Curator-Framework-0] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
   23:14:51.474 [ProcessThread(sid:0 cport:2181):] INFO  org.apache.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x1000044fedc0004
   23:14:51.794 [Thread-1] INFO  org.apache.zookeeper.ZooKeeper - Session: 0x1000044fedc0004 closed
   23:14:51.794 [Thread-1] INFO  org.apache.bookkeeper.proto.BookieServer - Shutting down BookieServer
   23:14:51.794 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1000044fedc0004
   23:14:51.794 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO  org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:46058 which had sessionid 0x1000044fedc0004
   23:14:51.795 [Thread-1] INFO  org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
   23:14:51.823 [Thread-1] INFO  org.apache.bookkeeper.bookie.Bookie - Shutting down Bookie-3181 with exitCode 0
   23:14:51.823 [Thread-1] INFO  org.apache.bookkeeper.bookie.Bookie - Turning bookie to read only during shut down
   23:14:51.823 [Thread-1] INFO  org.apache.bookkeeper.bookie.SyncThread - Shutting down SyncThread
   23:14:51.827 [SyncThread-7-1] INFO  org.apache.bookkeeper.bookie.EntryLogManagerBase - Creating a new entry log file because current active log channel has not initialized yet
   23:14:51.836 [SyncThread-7-1] INFO  org.apache.bookkeeper.bookie.EntryLoggerAllocator - Created new entry log file data/standalone/bookkeeper0/current/2.log for logId 2.
   23:14:51.842 [pool-5-thread-1] INFO  org.apache.bookkeeper.bookie.EntryLoggerAllocator - Created new entry log file data/standalone/bookkeeper0/current/3.log for logId 3.
   23:14:52.351 [SyncThread-7-1] INFO  org.apache.bookkeeper.bookie.SyncThread - Flush ledger storage at checkpoint CheckpointList{checkpoints=[LogMark: logFileId - 1570846318841 , logFileOffset - 6656]}.
   23:14:52.454 [Thread-1] INFO  org.apache.bookkeeper.bookie.Journal - Shutting down Journal
   23:14:52.455 [ForceWriteThread] INFO  org.apache.bookkeeper.bookie.Journal - ForceWrite thread interrupted
   23:14:52.456 [BookieJournal-3181] INFO  org.apache.bookkeeper.bookie.Journal - Journal exits when shutting down
   23:14:52.456 [BookieJournal-3181] INFO  org.apache.bookkeeper.bookie.Journal - Journal exited loop!
   23:14:52.457 [Thread-1] INFO  org.apache.bookkeeper.bookie.Journal - Finished Shutting down Journal thread
   23:14:52.458 [Bookie-3181] INFO  org.apache.bookkeeper.bookie.Bookie - Journal thread(s) quit.
   23:14:52.843 [Thread-1] INFO  org.apache.bookkeeper.bookie.GarbageCollectorThread - Shutting down GarbageCollectorThread
   23:14:52.844 [Thread-1] INFO  org.apache.bookkeeper.bookie.EntryLogger - Stopping EntryLogger
   23:14:52.846 [Thread-1] INFO  org.apache.bookkeeper.bookie.EntryLoggerAllocator - Stopped entry logger preallocator.
   23:14:52.850 [Thread-1] INFO  org.apache.bookkeeper.bookie.LedgerDirsMonitor - Shutting down LedgerDirsMonitor
   23:14:52.853 [ProcessThread(sid:0 cport:2181):] INFO  org.apache.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x1000044fedc0001
   23:14:53.026 [Thread-1] INFO  org.apache.zookeeper.ZooKeeper - Session: 0x1000044fedc0001 closed
   23:14:53.026 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1000044fedc0001
   23:14:53.026 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO  org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:46052 which had sessionid 0x1000044fedc0001
   23:14:53.032 [ProcessThread(sid:0 cport:2181):] INFO  org.apache.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x1000044fedc0000
   23:14:53.059 [Thread-1] INFO  org.apache.zookeeper.ZooKeeper - Session: 0x1000044fedc0000 closed
   23:14:53.059 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO  org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:46050 which had sessionid 0x1000044fedc0000
   23:14:53.059 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1000044fedc0000
   23:14:53.059 [Thread-1] INFO  org.apache.zookeeper.server.ZooKeeperServer - shutting down
   23:14:53.059 [Thread-1] INFO  org.apache.zookeeper.server.SessionTrackerImpl - Shutting down
   23:14:53.060 [Thread-1] INFO  org.apache.zookeeper.server.PrepRequestProcessor - Shutting down
   23:14:53.060 [Thread-1] INFO  org.apache.zookeeper.server.SyncRequestProcessor - Shutting down
   23:14:53.060 [ProcessThread(sid:0 cport:2181):] INFO  org.apache.zookeeper.server.PrepRequestProcessor - PrepRequestProcessor exited loop!
   23:14:53.060 [SyncThread:0] INFO  org.apache.zookeeper.server.SyncRequestProcessor - SyncRequestProcessor exited!
   23:14:53.060 [Thread-1] INFO  org.apache.zookeeper.server.FinalRequestProcessor - shutdown of request processor complete
   23:14:53.063 [Thread-1] INFO  org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:46078 which had sessionid 0x1000044fedc000b
   23:14:53.063 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO  org.apache.zookeeper.server.NIOServerCnxnFactory - NIOServerCnxn factory exited run method
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services