[GitHub] [pulsar] f-ld opened a new issue #5436: Topic do not always expire as expected

[GitBox <gi...@apache.org>]

f-ld opened a new issue #5436: Topic do not always expire as expected
URL: https://github.com/apache/pulsar/issues/5436
 
 
   **Describe the bug**
   I have 5 pulsar clusters with geo-replication. All running version 2.4.1.
   They are called "region1", "region2", "region3", "region4", "region5".
   Each cluster has 5 brokers / 3 bookies / 3 zookeeper / 3 global zookeeper
   
   My brokers are configured with:
   ```
       managedLedgerDefaultEnsembleSize: "2"
       managedLedgerDefaultWriteQuorum: "2"
       managedLedgerDefaultAckQuorum: "2"
       deduplicationEnabled: "false"
       defaultNumberOfNamespaceBundles: "24"
       loadBalancerSheddingEnabled: "true"
       loadBalancerAutoBundleSplitEnabled: "true"
       loadBalancerAutoUnloadSplitBundlesEnabled: "true"
       loadBalancerNamespaceMaximumBundles: "256"
       loadBalancerNamespaceBundleMaxTopics: "4000"
       loadBalancerNamespaceBundleMaxSessions: "10000"
       loadBalancerNamespaceBundleMaxMsgRate: "75000"
       loadBalancerNamespaceBundleMaxBandwidthMbytes: "100"
       ttlDurationDefaultInSeconds: "300"
       brokerDeleteInactiveTopicsEnabled: "true"
       brokerDeleteInactiveTopicsFrequencySeconds: "60"
       brokerServicePurgeInactiveFrequencyInSeconds: "30"
       subscriptionExpirationTimeMinutes: "1"
       messageExpiryCheckIntervalInMinutes: "1"
       subscriptionExpiryCheckIntervalInMinutes: "1"
   ```
   
   I have non partitioned persistent topics in a namespace "tenant/namespace" allowed in all 5 regions. Everything worked well at the beginning. I sent quite some traffic to the services using pulsar and at some point it seems a broker in one region ("region2") got some issues (got like frozen, trying to see if related to some issues like 3630).
   So re-routed all the traffic to previous system not involving pulsar and started to look to logs and stats. And I notice I have a lot of topics without any subscription that are not deleted after expiration (here from region2):
   
   ```
   pulsar-admin topics stats persistent://tenant/namespace/topic-056d26b5
   {
     "msgRateIn" : 0.0,
     "msgThroughputIn" : 0.0,
     "msgRateOut" : 0.0,
     "msgThroughputOut" : 0.0,
     "averageMsgSize" : 0.0,
     "storageSize" : 0,
     "publishers" : [ ],
     "subscriptions" : { },
     "replication" : {
       "region1" : {
         "msgRateIn" : 0.0,
         "msgThroughputIn" : 0.0,
         "msgRateOut" : 0.0,
         "msgThroughputOut" : 0.0,
         "msgRateExpired" : 0.0,
         "replicationBacklog" : 0,
         "connected" : false,
         "replicationDelayInSeconds" : 0,
         "inboundConnection" : "/10.10.3.135:34718",
         "inboundConnectedSince" : "2019-10-19T23:16:41.231Z"
       },
       "region3" : {
         "msgRateIn" : 0.0,
         "msgThroughputIn" : 0.0,
         "msgRateOut" : 0.0,
         "msgThroughputOut" : 0.0,
         "msgRateExpired" : 0.0,
         "replicationBacklog" : 0,
         "connected" : false,
         "replicationDelayInSeconds" : 0
       },
       "region4" : {
         "msgRateIn" : 0.0,
         "msgThroughputIn" : 0.0,
         "msgRateOut" : 0.0,
         "msgThroughputOut" : 0.0,
         "msgRateExpired" : 0.0,
         "replicationBacklog" : 0,
         "connected" : false,
         "replicationDelayInSeconds" : 0
       },
       "region5" : {
         "msgRateIn" : 0.0,
         "msgThroughputIn" : 0.0,
         "msgRateOut" : 0.0,
         "msgThroughputOut" : 0.0,
         "msgRateExpired" : 0.0,
         "replicationBacklog" : 0,
         "connected" : false,
         "replicationDelayInSeconds" : 0
       }
     },
     "deduplicationStatus" : "Disabled"
   }
   ```
   Indeed, those topics should have been deleted a long time ago since they have no subscription and no producer, only some inbound replication connection 
   
   **To Reproduce**
   No idea how to get there apart from stress testing a pulsar cluster with a lot of topic creation / subscription / publication and closing consumer.
   Could it be linked to the stress on topic creation / suppression ? To the fact one broker was probably frozen when topic was created ? To the fact some producers might not leave properly ?
   
   **Expected behavior**
   Before `brokerDeleteInactiveTopicsFrequencySeconds+brokerServicePurgeInactiveFrequencyInSeconds` seconds after deleting the last subscription from a topic having no producer, topic should be deleted
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services