You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/05/24 13:30:48 UTC
[GitHub] [pulsar] kirill-kosenko opened a new issue #10688: [Transactions] Errors during shutting down the broker
kirill-kosenko opened a new issue #10688:
URL: https://github.com/apache/pulsar/issues/10688
Hello
I noticed errors in log during shutting down the broker
It's reproduced on master if **standalone.conf** has `transactionCoordinatorEnabled=true` (seems to be all alright if set to `false`)
```
16:02:26.207 [Thread-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 127.0.0.1:3181
16:02:26.207 [Thread-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 127.0.0.1:3181
16:02:26.207 [Thread-1] ERROR io.netty.util.concurrent.DefaultPromise.rejectedExecution - Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) ~[netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) ~[netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346) ~[netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828) ~[netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818) ~[netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:842) [netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:499) [netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184) [netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95) [netty-transport-4.1.63.Final.jar:4.1.63.Final]
at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30) [netty-transport-4.1.63.Final.jar:4.1.63.Final]
at org.apache.bookkeeper.proto.PerChannelBookieClient.closeChannel(PerChannelBookieClient.java:1102) [bookkeeper-server-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.proto.PerChannelBookieClient.closeInternal(PerChannelBookieClient.java:1091) [bookkeeper-server-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.proto.PerChannelBookieClient.close(PerChannelBookieClient.java:1075) [bookkeeper-server-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.close(DefaultPerChannelBookieClientPool.java:157) [bookkeeper-server-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.proto.BookieClientImpl.close(BookieClientImpl.java:587) [bookkeeper-server-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.client.BookKeeper.close(BookKeeper.java:1435) [bookkeeper-server-4.13.0.jar:4.13.0]
at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.close(BookkeeperSchemaStorage.java:241) [pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.broker.service.schema.SchemaRegistryServiceImpl.close(SchemaRegistryServiceImpl.java:226) [pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.broker.service.schema.validator.SchemaRegistryServiceWithSchemaDataValidator.close(SchemaRegistryServiceWithSchemaDataValidator.java:47) [pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.broker.PulsarService.closeAsync(PulsarService.java:453) [pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.broker.PulsarService.close(PulsarService.java:322) [pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.PulsarStandaloneStarter$1.run(PulsarStandaloneStarter.java:98) [pulsar-broker.jar:2.8.0-SNAPSHOT]
16:02:26.207 [Thread-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 127.0.0.1:3181
16:02:26.207 [Thread-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 127.0.0.1:3181
16:02:26.208 [Thread-1] ERROR io.netty.util.concurrent.DefaultPromise.rejectedExecution - Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) ~[netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) ~[netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346) ~[netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828) ~[netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818) ~[netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:842) [netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:499) [netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184) [netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95) [netty-transport-4.1.63.Final.jar:4.1.63.Final]
at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30) [netty-transport-4.1.63.Final.jar:4.1.63.Final]
at org.apache.bookkeeper.proto.PerChannelBookieClient.closeChannel(PerChannelBookieClient.java:1102) [bookkeeper-server-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.proto.PerChannelBookieClient.closeInternal(PerChannelBookieClient.java:1091) [bookkeeper-server-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.proto.PerChannelBookieClient.close(PerChannelBookieClient.java:1075) [bookkeeper-server-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.close(DefaultPerChannelBookieClientPool.java:157) [bookkeeper-server-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.proto.BookieClientImpl.close(BookieClientImpl.java:587) [bookkeeper-server-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.client.BookKeeper.close(BookKeeper.java:1435) [bookkeeper-server-4.13.0.jar:4.13.0]
at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.close(BookkeeperSchemaStorage.java:241) [pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.broker.service.schema.SchemaRegistryServiceImpl.close(SchemaRegistryServiceImpl.java:226) [pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.broker.service.schema.validator.SchemaRegistryServiceWithSchemaDataValidator.close(SchemaRegistryServiceWithSchemaDataValidator.java:47) [pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.broker.PulsarService.closeAsync(PulsarService.java:453) [pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.broker.PulsarService.close(PulsarService.java:322) [pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.PulsarStandaloneStarter$1.run(PulsarStandaloneStarter.java:98) [pulsar-broker.jar:2.8.0-SNAPSHOT]
16:02:26.210 [Thread-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 127.0.0.1:3181
16:02:26.212 [main-SendThread(127.0.0.1:2181)] WARN org.apache.zookeeper.ClientCnxn - An exception was thrown while closing send thread for session 0x10000c1b984000e.
org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional data from server sessionid 0x10000c1b984000e, likely server has closed socket
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) ~[zookeeper-3.6.2.jar:3.6.2]
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[zookeeper-3.6.2.jar:3.6.2]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1275) [zookeeper-3.6.2.jar:3.6.2]
16:02:26.313 [Thread-1] INFO org.apache.zookeeper.ZooKeeper - Session: 0x10000c1b984000e closed
16:02:26.313 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x10000c1b984000e
16:02:26.324 [main-SendThread(127.0.0.1:2181)] WARN org.apache.zookeeper.ClientCnxn - An exception was thrown while closing send thread for session 0x10000c1b9840009.
org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional data from server sessionid 0x10000c1b9840009, likely server has closed socket
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) ~[zookeeper-3.6.2.jar:3.6.2]
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[zookeeper-3.6.2.jar:3.6.2]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1275) [zookeeper-3.6.2.jar:3.6.2]
16:02:26.424 [Thread-1] INFO org.apache.zookeeper.ZooKeeper - Session: 0x10000c1b9840009 closed
...
16:02:26.671 [Thread-1] INFO org.apache.zookeeper.ZooKeeper - Session: 0x10000c1b9840007 closed
16:02:26.671 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x10000c1b9840007
16:02:26.672 [Thread-1] INFO org.apache.distributedlog.BookKeeperClient - BookKeeper Client closed bk:distributedlog://127.0.0.1:2181/stream/storage:factory_writer_shared
16:02:26.672 [Thread-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 127.0.0.1:3181
16:02:26.672 [DL-io-0] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0x0c5cd1ee, L:/127.0.0.1:50182 ! R:127.0.0.1/127.0.0.1:3181]
16:02:26.673 [bookie-io-1-2] INFO org.apache.bookkeeper.proto.BookieRequestHandler - Channels disconnected: [id: 0xb906cf79, L:/127.0.0.1:3181 ! R:/127.0.0.1:50182]
16:02:26.673 [Thread-1] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=2,loopId=41952edd) Error writing metadata to store
org.apache.bookkeeper.client.BKException$BKClientClosedException: BookKeeper client is closed
at org.apache.bookkeeper.meta.CleanupLedgerManager.close(CleanupLedgerManager.java:245) ~[bookkeeper-server-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.client.BookKeeper.close(BookKeeper.java:1439) ~[bookkeeper-server-4.13.0.jar:4.13.0]
at org.apache.distributedlog.BookKeeperClient.close(BookKeeperClient.java:273) ~[distributedlog-core-4.13.0.jar:4.13.0]
at org.apache.distributedlog.impl.BKNamespaceDriver.doClose(BKNamespaceDriver.java:404) ~[distributedlog-core-4.13.0.jar:4.13.0]
at org.apache.distributedlog.impl.BKNamespaceDriver.close(BKNamespaceDriver.java:385) ~[distributedlog-core-4.13.0.jar:4.13.0]
at com.google.common.io.Closeables.close(Closeables.java:78) ~[guava-30.1-jre.jar:?]
at org.apache.distributedlog.util.Utils.close(Utils.java:544) ~[distributedlog-core-4.13.0.jar:4.13.0]
at org.apache.distributedlog.BKDistributedLogNamespace.close(BKDistributedLogNamespace.java:340) ~[distributedlog-core-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.stream.server.service.DLNamespaceProviderService.doClose(DLNamespaceProviderService.java:135) ~[stream-storage-server-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:123) ~[bookkeeper-common-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$close$6(LifecycleComponentStack.java:154) ~[bookkeeper-common-4.13.0.jar:4.13.0]
at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:406) [guava-30.1-jre.jar:?]
at org.apache.bookkeeper.common.component.LifecycleComponentStack.close(LifecycleComponentStack.java:154) [bookkeeper-common-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent.doClose(StreamStorageLifecycleComponent.java:60) [stream-storage-server-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:123) [bookkeeper-common-4.13.0.jar:4.13.0]
at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.stop(LocalBookkeeperEnsemble.java:492) [pulsar-zookeeper-utils.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.PulsarStandaloneStarter$1.run(PulsarStandaloneStarter.java:102) [pulsar-broker.jar:2.8.0-SNAPSHOT]
16:02:26.674 [Thread-1] ERROR org.apache.distributedlog.BKAbstractLogWriter - Completing Log segments encountered exception
java.io.IOException: Failed to close ledger for streams_000000000000000000_000000000000000001_000000000000000000:<default>:inprogress_000000000000000001 : BookKeeper client is closed
at org.apache.distributedlog.BKLogSegmentWriter$6.closeComplete(BKLogSegmentWriter.java:661) ~[distributedlog-core-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.client.LedgerHandle$5.lambda$safeRun$0(LedgerHandle.java:552) ~[bookkeeper-server-4.13.0.jar:4.13.0]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_201]
at org.apache.bookkeeper.client.LedgerHandle$5.lambda$safeRun$3(LedgerHandle.java:614) ~[bookkeeper-server-4.13.0.jar:4.13.0]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_201]
at org.apache.bookkeeper.client.MetadataUpdateLoop.lambda$writeLoop$1(MetadataUpdateLoop.java:161) ~[bookkeeper-server-4.13.0.jar:4.13.0]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_201]
at org.apache.bookkeeper.meta.CleanupLedgerManager.lambda$close$1(CleanupLedgerManager.java:246) ~[bookkeeper-server-4.13.0.jar:4.13.0]
at java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4649) ~[?:1.8.0_201]
at org.apache.bookkeeper.meta.CleanupLedgerManager.close(CleanupLedgerManager.java:246) ~[bookkeeper-server-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.client.BookKeeper.close(BookKeeper.java:1439) ~[bookkeeper-server-4.13.0.jar:4.13.0]
at org.apache.distributedlog.BookKeeperClient.close(BookKeeperClient.java:273) ~[distributedlog-core-4.13.0.jar:4.13.0]
at org.apache.distributedlog.impl.BKNamespaceDriver.doClose(BKNamespaceDriver.java:404) ~[distributedlog-core-4.13.0.jar:4.13.0]
at org.apache.distributedlog.impl.BKNamespaceDriver.close(BKNamespaceDriver.java:385) ~[distributedlog-core-4.13.0.jar:4.13.0]
at com.google.common.io.Closeables.close(Closeables.java:78) ~[guava-30.1-jre.jar:?]
at org.apache.distributedlog.util.Utils.close(Utils.java:544) ~[distributedlog-core-4.13.0.jar:4.13.0]
at org.apache.distributedlog.BKDistributedLogNamespace.close(BKDistributedLogNamespace.java:340) ~[distributedlog-core-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.stream.server.service.DLNamespaceProviderService.doClose(DLNamespaceProviderService.java:135) ~[stream-storage-server-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:123) ~[bookkeeper-common-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$close$6(LifecycleComponentStack.java:154) ~[bookkeeper-common-4.13.0.jar:4.13.0]
at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:406) [guava-30.1-jre.jar:?]
at org.apache.bookkeeper.common.component.LifecycleComponentStack.close(LifecycleComponentStack.java:154) [bookkeeper-common-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent.doClose(StreamStorageLifecycleComponent.java:60) [stream-storage-server-4.13.0.jar:4.13.0]
at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:123) [bookkeeper-common-4.13.0.jar:4.13.0]
at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.stop(LocalBookkeeperEnsemble.java:492) [pulsar-zookeeper-utils.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.PulsarStandaloneStarter$1.run(PulsarStandaloneStarter.java:102) [pulsar-broker.jar:2.8.0-SNAPSHOT]
16:02:26.676 [DLM-/stream/storage-OrderedScheduler-5-0] INFO org.apache.distributedlog.bk.SimpleLedgerAllocator - Abort ledger allocator without cleaning up on /stream/storage/streams_000000000000000000_000000000000000001_000000000000000000/<default>/allocation.
16:02:26.677 [main-EventThread] INFO org.apache.distributedlog.logsegment.PerStreamLogSegmentCache - streams_000000000000000000_000000000000000000_000000000000000000 added log segment (logrecs_000000000000000001 : [LogSegmentId:1, firstTxId:1, lastTxId:2, version:VERSION_V5_SEQUENCE_ID, completionTime:1621861346571, recordCount:2, regionId:0, status:0, logSegmentSequenceNumber:1, lastEntryId:2, lastSlotId:0, inprogress:false, minActiveDLSN:DLSN{logSegmentSequenceNo=1, entryId=0, slotId=0}, startSequenceId:0]) to cache.
16:02:26.677 [Thread-1] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=0,loopId=12241a52) Error writing metadata to store
org.apache.bookkeeper.client.BKException$BKClientClosedException: BookKeeper client is closed
```
Once broker is restarted:
```
16:16:26.085 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/functions/persistent/__transaction_buffer_snapshot-partition-0-multiTopicsReader-3051ff2ca4__transaction_pending_ack][__pending_ack_state] Successfully closed & deleted ledger 55 in cursor
16:16:26.086 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/functions/persistent/__transaction_buffer_snapshot-partition-0-multiTopicsReader-0bfc9ccc21__transaction_pending_ack][__pending_ack_state] Successfully closed & deleted ledger 57 in cursor
16:16:26.089 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/functions/persistent/__transaction_buffer_snapshot-partition-0-multiTopicsReader-6139cc7303__transaction_pending_ack][__pending_ack_state] Successfully closed & deleted ledger 56 in cursor
16:16:26.089 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/functions/persistent/__transaction_buffer_snapshot-partition-0-multiTopicsReader-4a02dc376c__transaction_pending_ack][__pending_ack_state] Successfully closed & deleted ledger 58 in cursor
16:16:26.701 [pulsar-io-30-12] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/functions/persistent/coordinate-participants] Rewind from 46:0 to 46:0
16:16:26.703 [pulsar-external-listener-85-1] INFO org.apache.pulsar.functions.worker.LeaderService - Worker c-standalone-fw-localhost-8080:localhost:8080 became the leader.
16:17:25.635 [pulsar-client-io-84-1] WARN org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/functions/metadata][c-standalone-fw-localhost-8080-function-metadata-tailer-reader-868cf7ae66] Failed to subscribe to topic on localhost/127.0.0.1:6650
16:17:25.636 [pulsar-client-io-84-1] WARN org.apache.pulsar.client.impl.PulsarClientImpl - [persistent://public/functions/metadata] Failed to get create topic reader
java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException$TimeoutException: 5 request timedout after ms 30000
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.uniRun(CompletableFuture.java:700) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:687) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_201]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_201]
at org.apache.pulsar.client.impl.ClientCnx.checkRequestTimeout(ClientCnx.java:1145) ~[pulsar-client-original.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.client.impl.ClientCnx.lambda$channelActive$0(ClientCnx.java:211) ~[pulsar-client-original.jar:2.8.0-SNAPSHOT]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) [netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:176) [netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) [netty-transport-native-epoll-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.63.Final.jar:4.1.63.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: org.apache.pulsar.client.api.PulsarClientException$TimeoutException: 5 request timedout after ms 30000
... 11 more
16:17:25.637 [pulsar-client-io-84-1] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0xa7109229, L:/127.0.0.1:44998 - R:localhost/127.0.0.1:6650] 5 request timedout after ms 30000
16:17:25.639 [main] ERROR org.apache.pulsar.functions.worker.FunctionMetaDataManager - Failed to initialize meta data store
org.apache.pulsar.client.api.PulsarClientException$TimeoutException: 5 request timedout after ms 30000
at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:961) ~[pulsar-client-api.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.client.impl.ReaderBuilderImpl.create(ReaderBuilderImpl.java:78) ~[pulsar-client-original.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.functions.worker.FunctionMetaDataTopicTailer.createReader(FunctionMetaDataTopicTailer.java:135) ~[pulsar-functions-worker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:107) [pulsar-functions-worker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:496) [pulsar-functions-worker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1487) [pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:790) [pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:295) [pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:121) [pulsar-broker.jar:2.8.0-SNAPSHOT]
16:17:25.640 [main] ERROR org.apache.pulsar.functions.worker.PulsarWorkerService - Error Starting up in worker
java.lang.RuntimeException: Failed to initialize Metadata Manager
at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:116) ~[pulsar-functions-worker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:496) [pulsar-functions-worker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1487) [pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:790) [pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:295) [pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:121) [pulsar-broker.jar:2.8.0-SNAPSHOT]
Caused by: org.apache.pulsar.client.api.PulsarClientException$TimeoutException: 5 request timedout after ms 30000
at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:961) ~[pulsar-client-api.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.client.impl.ReaderBuilderImpl.create(ReaderBuilderImpl.java:78) ~[pulsar-client-original.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.functions.worker.FunctionMetaDataTopicTailer.createReader(FunctionMetaDataTopicTailer.java:135) ~[pulsar-functions-worker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:107) ~[pulsar-functions-worker.jar:2.8.0-SNAPSHOT]
... 5 more
16:17:25.640 [main] ERROR org.apache.pulsar.broker.PulsarService - java.lang.RuntimeException: Failed to initialize Metadata Manager
java.lang.RuntimeException: java.lang.RuntimeException: Failed to initialize Metadata Manager
at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:571) ~[pulsar-functions-worker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1487) ~[pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:790) [pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:295) [pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:121) [pulsar-broker.jar:2.8.0-SNAPSHOT]
Caused by: java.lang.RuntimeException: Failed to initialize Metadata Manager
at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:116) ~[pulsar-functions-worker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:496) ~[pulsar-functions-worker.jar:2.8.0-SNAPSHOT]
... 4 more
Caused by: org.apache.pulsar.client.api.PulsarClientException$TimeoutException: 5 request timedout after ms 30000
at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:961) ~[pulsar-client-api.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.client.impl.ReaderBuilderImpl.create(ReaderBuilderImpl.java:78) ~[pulsar-client-original.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.functions.worker.FunctionMetaDataTopicTailer.createReader(FunctionMetaDataTopicTailer.java:135) ~[pulsar-functions-worker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:107) ~[pulsar-functions-worker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:496) ~[pulsar-functions-worker.jar:2.8.0-SNAPSHOT]
... 4 more
16:17:25.641 [main] ERROR org.apache.pulsar.PulsarStandaloneStarter - Failed to start pulsar service.
org.apache.pulsar.broker.PulsarServerException: java.lang.RuntimeException: java.lang.RuntimeException: Failed to initialize Metadata Manager
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:818) ~[pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:295) ~[pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:121) [pulsar-broker.jar:2.8.0-SNAPSHOT]
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Failed to initialize Metadata Manager
at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:571) ~[pulsar-functions-worker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1487) ~[pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:790) ~[pulsar-broker.jar:2.8.0-SNAPSHOT]
... 2 more
Caused by: java.lang.RuntimeException: Failed to initialize Metadata Manager
at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:116) ~[pulsar-functions-worker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:496) ~[pulsar-functions-worker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1487) ~[pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:790) ~[pulsar-broker.jar:2.8.0-SNAPSHOT]
... 2 more
Caused by: org.apache.pulsar.client.api.PulsarClientException$TimeoutException: 5 request timedout after ms 30000
at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:961) ~[pulsar-client-api.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.client.impl.ReaderBuilderImpl.create(ReaderBuilderImpl.java:78) ~[pulsar-client-original.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.functions.worker.FunctionMetaDataTopicTailer.createReader(FunctionMetaDataTopicTailer.java:135) ~[pulsar-functions-worker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:107) ~[pulsar-functions-worker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:496) ~[pulsar-functions-worker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1487) ~[pulsar-broker.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:790) ~[pulsar-broker.jar:2.8.0-SNAPSHOT]
... 2 more
```
After this exception falls - broker is not available anymore.
Kindly take a look once you find a spare minute
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #10688: [Transactions] Errors during shutting down the pulsar standalone
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #10688:
URL: https://github.com/apache/pulsar/issues/10688#issuecomment-1058890072
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org