You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/12/24 08:10:08 UTC

[GitHub] [pulsar] ericsyh edited a comment on issue #13479: [bug](broker): brokerDeleteInactiveTopic should exclude pulsar internal topics

ericsyh edited a comment on issue #13479:
URL: https://github.com/apache/pulsar/issues/13479#issuecomment-1000709708


   > Should we check why this topic become inactive, I think this topic should not become inactive.
   
   In below broker log, you can get the life cycle of healthcheck from creating to deleting by inactivity.
   
   <details>
   <summary>broker.log</summary>
   
   ```
   00:00:43.618 [pulsar-ordered-OrderedExecutor-3-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger pulsar/test-cluster/192.168.5.165:8080/persistent/healthcheck
   00:00:43.618 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.mledger.impl.MetaStoreImpl - Creating '/managed-ledgers/pulsar/test-cluster/192.168.5.165:8080/persistent/healthcheck'
   00:00:43.621 [pulsar-ordered-OrderedExecutor-3-0-EventThread] INFO  org.apache.pulsar.zookeeper.ZooKeeperManagedLedgerCache - [State:CONNECTED Timeout:30000 sessionid:0x17d9494218224e1 local:/192.168.5.165:39556 remoteserver:168.63.65.196/168.63.65.196:2182 lastZxid:228663556426 xid:517168 sent:517168 recv:532669 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/managed-ledgers/pulsar/test-cluster/192.168.5.165:8080/persistent
   00:00:43.621 [pulsar-ordered-OrderedExecutor-3-0-EventThread] INFO  org.apache.pulsar.zookeeper.ZooKeeperManagedLedgerCache - invalidate called in zookeeperChildrenCache for path /managed-ledgers/pulsar/test-cluster/192.168.5.165:8080/persistent
   00:00:43.621 [metadata-store-6-1] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [pulsar/test-cluster/192.168.5.165:8080/persistent/healthcheck] Creating ledger, metadata: {component=[109, 97, 110, 97, 103, 101, 100, 45, 108, 101, 100, 103, 101, 114], pulsar/managed-ledger=[112, 117, 108, 115, 97, 114, 47, 112, 117, 108, 115, 97, 114, 45, 99, 108, 117, 115, 116, 101, 114, 45, 104, 116, 115, 99, 45, 50, 47, 49, 54, 56, 46, 54, 52, 46, 53, 46, 49, 54, 53, 58, 56, 48, 56, 48, 47, 112, 101, 114, 115, 105, 115, 116, 101, 110, 116, 47, 104, 101, 97, 108, 116, 104, 99, 104, 101, 99, 107], application=[112, 117, 108, 115, 97, 114]} - metadata ops timeout : 60 seconds
   00:00:43.621 [metadata-store-6-1] WARN  org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [168.64.5.167:3181, 168.64.5.168:3181] is not adhering to Placement Policy. quarantinedBookies: []
   00:00:43.625 [main-EventThread] INFO  org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [168.64.5.167:3181, 168.64.5.168:3181] for ledger: 275969
   00:00:43.625 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [pulsar/test-cluster/192.168.5.165:8080/persistent/healthcheck] Created ledger 275969
   00:00:43.627 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl - [pulsar/test-cluster/192.168.5.165:8080/persistent/healthcheck] Successfully initialize managed ledger
   00:00:43.628 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.pulsar.broker.service.AbstractTopic - Disabling publish throttling for persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck
   00:00:43.629 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck] Error getting policies KeeperErrorCode = NoNode and isEncryptionRequired will be set to false
   00:00:43.629 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck] There are no replicated subscriptions on the topic
   00:00:43.629 [ForkJoinPool.commonPool-worker-11] INFO  org.apache.pulsar.broker.service.BrokerService - Created topic persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck - dedup is disabled
   00:00:43.631 [pulsar-io-4-7] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer status recorder with config: {
     "topicNames" : [ "persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck" ],
     "topicsPattern" : null,
     "subscriptionName" : "reader-5f29fb5984",
     "subscriptionType" : "Exclusive",
     "subscriptionMode" : "NonDurable",
     "receiverQueueSize" : 1000,
     "acknowledgementsGroupTimeMicros" : 100000,
     "negativeAckRedeliveryDelayMicros" : 60000000,
     "maxTotalReceiverQueueSizeAcrossPartitions" : 50000,
     "consumerName" : null,
     "ackTimeoutMillis" : 0,
     "tickDurationMillis" : 1000,
     "priorityLevel" : 0,
     "maxPendingChunkedMessage" : 10,
     "autoAckOldestChunkedMessageOnQueueFull" : false,
     "expireTimeOfIncompleteChunkedMessageMillis" : 60000,
     "cryptoFailureAction" : "FAIL",
     "properties" : { },
     "readCompacted" : false,
     "subscriptionInitialPosition" : "Latest",
     "patternAutoDiscoveryPeriod" : 60,
     "regexSubscriptionMode" : "PersistentOnly",
     "deadLetterPolicy" : null,
     "retryEnable" : false,
     "autoUpdatePartitions" : true,
     "autoUpdatePartitionsIntervalSeconds" : 60,
     "replicateSubscriptionState" : false,
     "resetIncludeHead" : false,
     "keySharedPolicy" : null,
     "batchIndexAckEnabled" : false,
     "ackReceiptEnabled" : false,
     "poolMessages" : false,
     "maxPendingChuckedMessage" : 10
   }
   00:00:43.631 [pulsar-io-4-7] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: {
     "serviceUrl" : "pulsar://192.168.5.165:6650",
     "authPluginClassName" : "org.apache.pulsar.client.impl.auth.AuthenticationDisabled",
     "authParams" : "*****",
     "authParamMap" : null,
     "operationTimeoutMs" : 30000,
     "statsIntervalSeconds" : 60,
     "numIoThreads" : 1,
     "numListenerThreads" : 1,
     "connectionsPerBroker" : 1,
     "useTcpNoDelay" : true,
     "useTls" : false,
     "tlsTrustCertsFilePath" : null,
     "tlsAllowInsecureConnection" : false,
     "tlsHostnameVerificationEnable" : false,
     "concurrentLookupRequest" : 5000,
     "maxLookupRequest" : 50000,
     "maxLookupRedirects" : 20,
     "maxNumberOfRejectedRequestPerConnection" : 50,
     "keepAliveIntervalSeconds" : 30,
     "connectionTimeoutMs" : 10000,
     "requestTimeoutMs" : 60000,
     "initialBackoffIntervalNanos" : 100000000,
     "maxBackoffIntervalNanos" : 60000000000,
     "enableBusyWait" : false,
     "listenerName" : null,
     "useKeyStoreTls" : false,
     "sslProvider" : null,
     "tlsTrustStoreType" : "JKS",
     "tlsTrustStorePath" : null,
     "tlsTrustStorePassword" : null,
     "tlsCiphers" : [ ],
     "tlsProtocols" : [ ],
     "memoryLimitBytes" : 0,
     "proxyServiceUrl" : null,
     "proxyProtocol" : null,
     "enableTransaction" : false
   }
   00:00:43.632 [pulsar-io-4-7] INFO  org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Starting Pulsar producer perf with config: {
     "topicName" : "persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck",
     "producerName" : null,
     "sendTimeoutMs" : 30000,
     "blockIfQueueFull" : false,
     "maxPendingMessages" : 1000,
     "maxPendingMessagesAcrossPartitions" : 50000,
     "messageRoutingMode" : "RoundRobinPartition",
     "hashingScheme" : "JavaStringHash",
     "cryptoFailureAction" : "FAIL",
     "batchingMaxPublishDelayMicros" : 1000,
     "batchingPartitionSwitchFrequencyByPublishDelay" : 10,
     "batchingMaxMessages" : 1000,
     "batchingMaxBytes" : 131072,
     "batchingEnabled" : true,
     "chunkingEnabled" : false,
     "compressionType" : "NONE",
     "initialSequenceId" : null,
     "autoUpdatePartitions" : true,
     "autoUpdatePartitionsIntervalSeconds" : 60,
     "multiSchema" : true,
     "accessMode" : "Shared",
     "properties" : { }
   }
   00:00:43.632 [pulsar-io-4-7] INFO  org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Pulsar client config: {
     "serviceUrl" : "pulsar://192.168.5.165:6650",
     "authPluginClassName" : "org.apache.pulsar.client.impl.auth.AuthenticationDisabled",
     "authParams" : "*****",
     "authParamMap" : null,
     "operationTimeoutMs" : 30000,
     "statsIntervalSeconds" : 60,
     "numIoThreads" : 1,
     "numListenerThreads" : 1,
     "connectionsPerBroker" : 1,
     "useTcpNoDelay" : true,
     "useTls" : false,
     "tlsTrustCertsFilePath" : null,
     "tlsAllowInsecureConnection" : false,
     "tlsHostnameVerificationEnable" : false,
     "concurrentLookupRequest" : 5000,
     "maxLookupRequest" : 50000,
     "maxLookupRedirects" : 20,
     "maxNumberOfRejectedRequestPerConnection" : 50,
     "keepAliveIntervalSeconds" : 30,
     "connectionTimeoutMs" : 10000,
     "requestTimeoutMs" : 60000,
     "initialBackoffIntervalNanos" : 100000000,
     "maxBackoffIntervalNanos" : 60000000000,
     "enableBusyWait" : false,
     "listenerName" : null,
     "useKeyStoreTls" : false,
     "sslProvider" : null,
     "tlsTrustStoreType" : "JKS",
     "tlsTrustStorePath" : null,
     "tlsTrustStorePassword" : null,
     "tlsCiphers" : [ ],
     "tlsProtocols" : [ ],
     "memoryLimitBytes" : 0,
     "proxyServiceUrl" : null,
     "proxyProtocol" : null,
     "enableTransaction" : false
   }
   00:00:43.632 [pulsar-io-4-7] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck][reader-5f29fb5984] Subscribing to topic on cnx [id: 0x2686dc4b, L:/192.168.5.165:60218 - R:192.168.5.165/192.168.5.165:6650], consumerId 10627
   00:00:43.632 [pulsar-io-4-7] INFO  org.apache.pulsar.client.impl.ProducerImpl - [persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck] [null] Creating producer on cnx [id: 0x2686dc4b, L:/192.168.5.165:60218 - R:192.168.5.165/192.168.5.165:6650]
   00:00:43.632 [pulsar-io-4-8] INFO  org.apache.pulsar.broker.service.ServerCnx - [/192.168.5.165:60218] Subscribing on topic persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck / reader-5f29fb5984
   00:00:43.633 [pulsar-io-4-8] INFO  org.apache.pulsar.broker.service.ServerCnx - [/192.168.5.165:60218][persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck] Creating producer. producerId=10627
   00:00:43.634 [ForkJoinPool.commonPool-worker-11] WARN  org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [168.64.5.167:3181, 168.64.5.169:3181] is not adhering to Placement Policy. quarantinedBookies: []
   00:00:43.637 [ZKC-connect-executor-0-EventThread] INFO  org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [168.64.5.167:3181, 168.64.5.169:3181] for ledger: 275970
   00:00:43.637 [pulsar-io-4-8] WARN  org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [168.64.5.168:3181, 168.64.5.169:3181] is not adhering to Placement Policy. quarantinedBookies: []
   00:00:43.640 [ZKC-connect-executor-0-EventThread] INFO  org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [168.64.5.168:3181, 168.64.5.169:3181] for ledger: 275971
   00:00:43.641 [metadata-store-6-1] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck][reader-5f29fb5984] Creating non-durable subscription at msg id 9223372036854775807:9223372036854775807:-1:-1
   00:00:43.641 [metadata-store-6-1] INFO  org.apache.bookkeeper.mledger.impl.NonDurableCursorImpl - [pulsar/test-cluster/192.168.5.165:8080/persistent/healthcheck] Created non-durable cursor read-position=275969:0 mark-delete-position=275969:-1
   00:00:43.641 [metadata-store-6-1] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [pulsar/test-cluster/192.168.5.165:8080/persistent/healthcheck] Opened new cursor: NonDurableCursorImpl{ledger=pulsar/test-cluster/192.168.5.165:8080/persistent/healthcheck, ackPos=275969:-1, readPos=275969:0}
   00:00:43.642 [metadata-store-6-1] INFO  org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [pulsar/test-cluster/192.168.5.165:8080/persistent/healthcheck-reader-5f29fb5984] Rewind from 275969:0 to 275969:0
   00:00:43.642 [metadata-store-6-1] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck] There are no replicated subscriptions on the topic
   00:00:43.642 [metadata-store-6-1] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck][reader-5f29fb5984] Created new subscription for 10627
   00:00:43.642 [metadata-store-6-1] INFO  org.apache.pulsar.broker.service.ServerCnx - [/192.168.5.165:60218] Created subscription on topic persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck / reader-5f29fb5984
   00:00:43.642 [pulsar-io-4-7] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck][reader-5f29fb5984] Subscribed to topic on 192.168.5.165/192.168.5.165:6650 -- consumer: 10627
   00:00:43.660 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO  org.apache.bookkeeper.client.ReadOnlyLedgerHandle - Closing recovered ledger 275970 at entry 0
   00:00:43.663 [BookKeeperClientWorker-OrderedExecutor-2-0] WARN  org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [168.64.5.167:3181, 168.64.5.169:3181] is not adhering to Placement Policy. quarantinedBookies: []
   00:00:43.665 [ZKC-connect-executor-0-EventThread] INFO  org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [168.64.5.167:3181, 168.64.5.169:3181] for ledger: 275972
   00:00:43.670 [metadata-store-6-1] INFO  org.apache.pulsar.broker.service.ServerCnx - [/192.168.5.165:60218] Created new producer: Producer{topic=PersistentTopic{topic=persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck}, client=/192.168.5.165:60218, producerName=test-cluster-4-10639, producerId=10627}
   00:00:43.670 [pulsar-io-4-7] INFO  org.apache.pulsar.client.impl.ProducerImpl - [persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck] [test-cluster-4-10639] Created producer on cnx [id: 0x2686dc4b, L:/192.168.5.165:60218 - R:192.168.5.165/192.168.5.165:6650]
   00:00:43.673 [pulsar-io-4-8] INFO  org.apache.pulsar.broker.service.ServerCnx - [PersistentTopic{topic=persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck}][test-cluster-4-10639] Closing producer on cnx /192.168.5.165:60218. producerId=10627
   00:00:43.673 [pulsar-io-4-8] INFO  org.apache.pulsar.broker.service.ServerCnx - [PersistentTopic{topic=persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck}][test-cluster-4-10639] Closed producer on cnx /192.168.5.165:60218. producerId=10627
   00:00:43.673 [pulsar-io-4-8] INFO  org.apache.pulsar.broker.service.ServerCnx - [/192.168.5.165:60218] Closing consumer: consumerId=10627
   00:00:43.673 [pulsar-io-4-7] INFO  org.apache.pulsar.client.impl.ProducerImpl - [persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck] [test-cluster-4-10639] Closed Producer
   00:00:43.673 [pulsar-io-4-8] INFO  org.apache.pulsar.broker.service.AbstractDispatcherSingleActiveConsumer - Removing consumer Consumer{subscription=PersistentSubscription{topic=persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck, name=reader-5f29fb5984}, consumerId=10627, consumerName=6ad6f, address=/192.168.5.165:60218}
   00:00:43.674 [pulsar-io-4-8] INFO  org.apache.pulsar.broker.service.persistent.PersistentSubscription - [persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck][reader-5f29fb5984] Successfully closed subscription [NonDurableCursorImpl{ledger=pulsar/test-cluster/192.168.5.165:8080/persistent/healthcheck, ackPos=275969:-1, readPos=275969:1}]
   00:00:43.674 [pulsar-io-4-8] INFO  org.apache.pulsar.broker.service.persistent.PersistentSubscription - [persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck][reader-5f29fb5984] Successfully closed dispatcher for reader
   00:00:43.673 [pulsar-client-internal-44-1] INFO  org.eclipse.jetty.server.RequestLog - 168.63.193.104 - - [20/Dec/2021:00:00:43 +0800] "GET /admin/v2/brokers/health HTTP/1.1" 200 2 "-" "Pulsar-Java-v2.5.2" 58
   00:00:43.674 [pulsar-io-4-8] INFO  org.apache.pulsar.broker.service.ServerCnx - [/192.168.5.165:60218] Closed consumer, consumerId=10627
   00:00:43.674 [pulsar-io-4-7] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck] [reader-5f29fb5984] Closed consumer
   00:00:55.186 [pulsar-web-40-15] INFO  org.eclipse.jetty.server.RequestLog - 168.64.5.169 - - [20/Dec/2021:00:00:55 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.17.2" 1
   00:00:55.189 [prometheus-stats-41-1] INFO  org.eclipse.jetty.server.RequestLog - 168.64.5.169 - - [20/Dec/2021:00:00:55 +0800] "GET /metrics/ HTTP/1.1" 200 78985 "http://192.168.5.165:8080/metrics" "Prometheus/2.17.2" 3
   00:01:10.186 [pulsar-web-40-3] INFO  org.eclipse.jetty.server.RequestLog - 168.64.5.169 - - [20/Dec/2021:00:01:10 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.17.2" 1
   00:01:10.189 [prometheus-stats-41-1] INFO  org.eclipse.jetty.server.RequestLog - 168.64.5.169 - - [20/Dec/2021:00:01:10 +0800] "GET /metrics/ HTTP/1.1" 200 78986 "http://192.168.5.165:8080/metrics" "Prometheus/2.17.2" 3
   00:01:25.186 [pulsar-web-40-4] INFO  org.eclipse.jetty.server.RequestLog - 168.64.5.169 - - [20/Dec/2021:00:01:25 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.17.2" 1
   00:01:25.190 [prometheus-stats-41-1] INFO  org.eclipse.jetty.server.RequestLog - 168.64.5.169 - - [20/Dec/2021:00:01:25 +0800] "GET /metrics/ HTTP/1.1" 200 79126 "http://192.168.5.165:8080/metrics" "Prometheus/2.17.2" 4
   00:01:40.186 [pulsar-web-40-5] INFO  org.eclipse.jetty.server.RequestLog - 168.64.5.169 - - [20/Dec/2021:00:01:40 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.17.2" 1
   00:01:40.189 [prometheus-stats-41-1] INFO  org.eclipse.jetty.server.RequestLog - 168.64.5.169 - - [20/Dec/2021:00:01:40 +0800] "GET /metrics/ HTTP/1.1" 200 79126 "http://192.168.5.165:8080/metrics" "Prometheus/2.17.2" 3
   00:01:47.846 [bookkeeper-ml-scheduler-OrderedScheduler-3-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [pulsar/test-cluster/192.168.5.165:8080/persistent/healthcheck] Start checking if current ledger is full
   00:01:55.185 [pulsar-web-40-14] INFO  org.eclipse.jetty.server.RequestLog - 168.64.5.169 - - [20/Dec/2021:00:01:55 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.17.2" 0
   00:01:55.189 [prometheus-stats-41-1] INFO  org.eclipse.jetty.server.RequestLog - 168.64.5.169 - - [20/Dec/2021:00:01:55 +0800] "GET /metrics/ HTTP/1.1" 200 79126 "http://192.168.5.165:8080/metrics" "Prometheus/2.17.2" 3
   00:02:08.400 [bookkeeper-ml-scheduler-OrderedScheduler-5-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [pulsar/test-cluster/192.168.5.165:8080/persistent/healthcheck] Start checking if current ledger is full
   00:02:08.403 [main-EventThread] WARN  org.apache.bookkeeper.meta.AbstractZkLedgerManager - Ledger node does not exist in ZooKeeper: ledgerId=275436
   00:02:08.403 [main-EventThread] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=275436,loopId=2f4893cb) Error writing metadata to store
   org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server
   	at org.apache.bookkeeper.meta.AbstractZkLedgerManager$4.processResult(AbstractZkLedgerManager.java:505) [org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
   	at org.apache.bookkeeper.zookeeper.ZooKeeperClient$22$1.processResult(ZooKeeperClient.java:1094) [org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
   	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:638) [org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
   	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:563) [org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
   00:02:08.403 [main-EventThread] WARN  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Error when closing ledger 275436. Status=No such ledger exists on Metadata Server
   00:02:08.403 [main-EventThread] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [pulsar/test-cluster/192.168.5.165:8080/persistent/healthcheck] Creating ledger, metadata: {component=[109, 97, 110, 97, 103, 101, 100, 45, 108, 101, 100, 103, 101, 114], pulsar/managed-ledger=[112, 117, 108, 115, 97, 114, 47, 112, 117, 108, 115, 97, 114, 45, 99, 108, 117, 115, 116, 101, 114, 45, 104, 116, 115, 99, 45, 50, 47, 49, 54, 56, 46, 54, 52, 46, 53, 46, 49, 54, 53, 58, 56, 48, 56, 48, 47, 112, 101, 114, 115, 105, 115, 116, 101, 110, 116, 47, 104, 101, 97, 108, 116, 104, 99, 104, 101, 99, 107], application=[112, 117, 108, 115, 97, 114]} - metadata ops timeout : 60 seconds
   00:02:08.403 [main-EventThread] WARN  org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [168.64.5.168:3181, 168.64.5.167:3181] is not adhering to Placement Policy. quarantinedBookies: []
   00:02:08.405 [main-EventThread] INFO  org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [168.64.5.168:3181, 168.64.5.167:3181] for ledger: 275973
   00:02:08.405 [main-EventThread] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [pulsar/test-cluster/192.168.5.165:8080/persistent/healthcheck] Created new ledger 275973
   00:02:08.406 [pulsar-ordered-OrderedExecutor-3-0-EventThread] INFO  org.apache.pulsar.zookeeper.ZooKeeperCache - [State:CONNECTED Timeout:30000 sessionid:0x17d9494218224e1 local:/192.168.5.165:39556 remoteserver:168.63.65.196/168.63.65.196:2182 lastZxid:228663567828 xid:517211 sent:517211 recv:532713 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: WatchedEvent state:SyncConnected type:NodeDataChanged path:/managed-ledgers/pulsar/test-cluster/192.168.5.165:8080/persistent/healthcheck
   00:02:10.186 [pulsar-web-40-1] INFO  org.eclipse.jetty.server.RequestLog - 168.64.5.169 - - [20/Dec/2021:00:02:10 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.17.2" 0
   00:02:10.189 [prometheus-stats-41-1] INFO  org.eclipse.jetty.server.RequestLog - 168.64.5.169 - - [20/Dec/2021:00:02:10 +0800] "GET /metrics/ HTTP/1.1" 200 79123 "http://192.168.5.165:8080/metrics" "Prometheus/2.17.2" 3
   00:02:21.952 [BookKeeperClientWorker-OrderedExecutor-4-0] INFO  org.apache.bookkeeper.client.ReadOnlyLedgerHandle - Closing recovered ledger 275972 at entry 0
   00:02:21.954 [ForkJoinPool.commonPool-worker-11] INFO  org.apache.pulsar.broker.service.AbstractTopic - Delete schema storage of id: pulsar/192.168.5.165:8080/healthcheck
   00:02:22.018 [BookKeeperClientWorker-OrderedExecutor-1-0] INFO  org.apache.bookkeeper.mledger.impl.MetaStoreImpl - [pulsar/test-cluster/192.168.5.165:8080/persistent/healthcheck] Remove ManagedLedger
   00:02:22.028 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [pulsar/test-cluster/192.168.5.165:8080/persistent/healthcheck] Successfully deleted managed ledger
   00:02:22.028 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck] Topic deleted
   00:02:22.029 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://pulsar/test-cluster/192.168.5.165:8080/healthcheck] Topic deleted successfully due to inactivity
   00:02:25.186 [pulsar-web-40-1] INFO  org.eclipse.jetty.server.RequestLog - 168.64.5.169 - - [20/Dec/2021:00:02:25 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.17.2" 1
   00:02:25.190 [prometheus-stats-41-1] INFO  org.eclipse.jetty.server.RequestLog - 168.64.5.169 - - [20/Dec/2021:00:02:25 +0800] "GET /metrics/ HTTP/1.1" 200 59128 "http://192.168.5.165:8080/metrics" "Prometheus/2.17.2" 4
   00:02:40.185 [pulsar-web-40-7] INFO  org.eclipse.jetty.server.RequestLog - 168.64.5.169 - - [20/Dec/2021:00:02:40 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.17.2" 0
   00:02:40.188 [prometheus-stats-41-1] INFO  org.eclipse.jetty.server.RequestLog - 168.64.5.169 - - [20/Dec/2021:00:02:40 +0800] "GET /metrics/ HTTP/1.1" 200 59127 "http://192.168.5.165:8080/metrics" "Prometheus/2.17.2" 2
   00:02:43.811 [pulsar-web-40-16] INFO  org.eclipse.jetty.server.RequestLog - 168.63.193.104 - - [20/Dec/2021:00:02:43 +0800] "GET /admin/v2/clusters/test-cluster/failureDomains HTTP/1.1" 200 2 "-" "Pulsar-Java-v2.5.2" 1
   00:02:43.812 [pulsar-web-40-2] INFO  org.eclipse.jetty.server.RequestLog - 168.63.193.104 - - [20/Dec/2021:00:02:43 +0800] "GET /admin/v2/brokers/test-cluster HTTP/1.1" 200 41 "-" "Pulsar-Java-v2.5.2" 1
   00:02:43.813 [pulsar-web-40-4] INFO  org.eclipse.jetty.server.RequestLog - 168.63.193.104 - - [20/Dec/2021:00:02:43 +0800] "GET /admin/v2/broker-stats/topics HTTP/1.1" 200 1866 "-" "Pulsar-Java-v2.5.2" 0
   00:02:43.956 [pulsar-web-40-13] INFO  org.eclipse.jetty.server.RequestLog - 168.63.193.104 - - [20/Dec/2021:00:02:43 +0800] "GET /admin/v2/clusters/test-cluster/failureDomains HTTP/1.1" 200 2 "-" "Pulsar-Java-v2.5.2" 0
   00:02:43.957 [pulsar-web-40-5] INFO  org.eclipse.jetty.server.RequestLog - 168.63.193.104 - - [20/Dec/2021:00:02:43 +0800] "GET /admin/v2/brokers/test-cluster HTTP/1.1" 200 41 "-" "Pulsar-Java-v2.5.2" 0
   00:02:43.957 [pulsar-web-40-11] INFO  org.eclipse.jetty.server.RequestLog - 168.63.193.104 - - [20/Dec/2021:00:02:43 +0800] "GET /admin/v2/broker-stats/topics HTTP/1.1" 200 1866 "-" "Pulsar-Java-v2.5.2" 0
   00:02:43.985 [pulsar-web-40-14] INFO  org.eclipse.jetty.server.RequestLog - 168.63.193.104 - - [20/Dec/2021:00:02:43 +0800] "GET /admin/v2/clusters HTTP/1.1" 200 47 "-" "Pulsar-Java-v2.5.2" 0
   00:02:43.986 [pulsar-web-40-6] INFO  org.eclipse.jetty.server.RequestLog - 168.63.193.104 - - [20/Dec/2021:00:02:43 +0800] "GET /admin/v2/clusters/test-cluster HTTP/1.1" 200 272 "-" "Pulsar-Java-v2.5.2" 0
   00:02:43.987 [pulsar-web-40-9] INFO  org.eclipse.jetty.server.RequestLog - 168.63.193.104 - - [20/Dec/2021:00:02:43 +0800] "GET /admin/v2/clusters/pulsar-cluster-test HTTP/1.1" 200 280 "-" "Pulsar-Java-v2.5.2" 1
   00:02:43.989 [pulsar-web-40-8] INFO  org.eclipse.jetty.server.RequestLog - 168.63.193.104 - - [20/Dec/2021:00:02:43 +0800] "GET /admin/v2/clusters HTTP/1.1" 200 47 "-" "Pulsar-Java-v2.5.2" 0
   00:02:43.990 [pulsar-web-40-1] INFO  org.eclipse.jetty.server.RequestLog - 168.63.193.104 - - [20/Dec/2021:00:02:43 +0800] "GET /admin/v2/clusters/test-cluster HTTP/1.1" 200 272 "-" "Pulsar-Java-v2.5.2" 1
   00:02:43.990 [pulsar-web-40-15] INFO  org.eclipse.jetty.server.RequestLog - 168.63.193.104 - - [20/Dec/2021:00:02:43 +0800] "GET /admin/v2/clusters/pulsar-cluster-test HTTP/1.1" 200 280 "-" "Pulsar-Java-v2.5.2" 0
   00:02:55.186 [pulsar-web-40-7] INFO  org.eclipse.jetty.server.RequestLog - 168.64.5.169 - - [20/Dec/2021:00:02:55 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.17.2" 1
   00:02:55.188 [prometheus-stats-41-1] INFO  org.eclipse.jetty.server.RequestLog - 168.64.5.169 - - [20/Dec/2021:00:02:55 +0800] "GET /metrics/ HTTP/1.1" 200 59127 "http://192.168.5.165:8080/metrics" "Prometheus/2.17.2" 2
   00:03:10.186 [pulsar-web-40-12] INFO  org.eclipse.jetty.server.RequestLog - 168.64.5.169 - - [20/Dec/2021:00:03:10 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.17.2" 1
   00:03:10.188 [prometheus-stats-41-1] INFO  org.eclipse.jetty.server.RequestLog - 168.64.5.169 - - [20/Dec/2021:00:03:10 +0800] "GET /metrics/ HTTP/1.1" 200 59127 "http://192.168.5.165:8080/metrics" "Prometheus/2.17.2" 2
   00:03:13.989 [ForkJoinPool.commonPool-worker-9] INFO  org.apache.pulsar.broker.cache.LocalZooKeeperCacheService - Successfully created local policies for /admin/local-policies/pulsar/test-cluster/192.168.5.165:8080 -- Optional.empty
   00:03:13.989 [pulsar-ordered-OrderedExecutor-3-0] INFO  org.apache.pulsar.broker.PulsarService - No ledger offloader configured, using NULL instance
   ```
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org