You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/05/02 00:03:47 UTC
[GitHub] [pulsar] yangou opened a new issue #10464: Shared subscription with Earliest initial position doesn't clear backlog
yangou opened a new issue #10464:
URL: https://github.com/apache/pulsar/issues/10464
**Describe the bug**
There are chances that messages were created before the subscriptions were created. So we disabled the topic auto-deletion in the namespace. If the messages were created 4 hours ahead of time, my subscription with the initial position works as expected. However if the messages were created more than 6 hours ahead of time, after my consumer establishes the subscription, the backlog won't get cleared. Consumers are able to consume from the point of subscription, but none of the earlier messages would be delivered to consumers.
**To Reproduce**
1. Create a namespace with the below policies:
```
---
bundles:
numBundles: 64 # bundles serve the purpose of topic load balancing among brokers
backlog_quota_map:
destination_storage:
limit: 1099511627776 # 1TB, maximal backlog size of unack'ed messages
policy: producer_request_hold # keep the producer waiting/timeout when backlog reaches limit
deduplicationEnabled: false
autoTopicCreationOverride:
allowAutoTopicCreation: true
topicType: non-partitioned
autoSubscriptionCreationOverride:
allowAutoSubscriptionCreation: true
message_ttl_in_seconds: 0 # don't expire unack'ed messages
subscription_expiration_time_minutes: 0 # don't expire subscription once inactive(no connections, no consumptions, etc)
retention_policies:
retentionTimeInMinutes: 0 # don't retent the messages around once they are consumed
retentionSizeInMB: 0 # save as above
delayed_delivery_policies:
active: false # disable delayed delivery in minutely namespace
inactive_topic_policies:
deleteWhileInactive: true # enable auto-deletion on inactive topics
inactiveTopicDeleteMode: delete_when_subscriptions_caught_up # once subscroptions have caught up allow deletion on the topics
maxInactiveDurationSeconds: 216000 # wait util 60 hours after topic become inactive, this will make this namespace to have 60*60*2=7200 topics at max
```
2. publish some messages to a topic such that topic gets automatically created.
3. wait for 6 hours, create a `Shared` consumer using the `Earliest` position.
4. You won't see any messages being delivered.
5. Get the topic stats:
```
{
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesInCounter" : 74768,
"msgInCounter" : 495,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"averageMsgSize" : 0.0,
"msgChunkPublished" : false,
"storageSize" : 0,
"backlogSize" : 0,
"publishers" : [ ],
"subscriptions" : {
"scheduler" : {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"chuckedMessageRate" : 0,
"msgBacklog" : 472,
"msgBacklogNoDelayed" : 472,
"blockedSubscriptionOnUnackedMsgs" : false,
"msgDelayed" : 0,
"unackedMessages" : 0,
"type" : "Shared",
"msgRateExpired" : 0.0,
"lastExpireTimestamp" : 0,
"lastConsumedFlowTimestamp" : 1619913730283,
"lastConsumedTimestamp" : 0,
"lastAckedTimestamp" : 0,
"consumers" : [ {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"chuckedMessageRate" : 0.0,
"consumerName" : "zacqt",
"availablePermits" : 1000,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 1000,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"metadata" : { },
"connectedSince" : "2021-05-02T00:02:10.281Z",
"address" : "/10.72.94.193:56002"
} ],
"isDurable" : true,
"isReplicated" : false,
"consumersAfterMarkDeletePosition" : { }
}
},
"replication" : { },
"deduplicationStatus" : "Disabled"
}
```
**Expected behavior**
The consumer starts with the Earliest position would clear up the backlog created before the subscription was created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] yangou closed issue #10464: Shared subscription with Earliest initial position doesn't clear backlog
Posted by GitBox <gi...@apache.org>.
yangou closed issue #10464:
URL: https://github.com/apache/pulsar/issues/10464
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] yangou commented on issue #10464: Shared subscription with Earliest initial position doesn't clear backlog
Posted by GitBox <gi...@apache.org>.
yangou commented on issue #10464:
URL: https://github.com/apache/pulsar/issues/10464#issuecomment-831390145
After carefully reading this documentation again, I realized it's due to my misconfiguration.
https://pulsar.apache.org/docs/en/cookbooks-retention-expiry/#retention-policies
Closing this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] yangou commented on issue #10464: Shared subscription with Earliest initial position doesn't clear backlog
Posted by GitBox <gi...@apache.org>.
yangou commented on issue #10464:
URL: https://github.com/apache/pulsar/issues/10464#issuecomment-830712538
Tried all subscription types:
Key_shared, Shared, Failover, Exclusive.
All yield to the same behavior
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org