You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/05/06 20:53:38 UTC
[GitHub] [pulsar] yangou opened a new issue #10497: missing message on partitioned topics with delayed messages
yangou opened a new issue #10497:
URL: https://github.com/apache/pulsar/issues/10497
**Describe the bug**
Delayed messages won't get delivered on partitioned topics. Happens with both the built-in command-line consumer and golang consumer.
**To Reproduce**
Steps to reproduce the behavior:
1. Create a partitioned topic, set the namespace policies as below:
```yaml
bundles:
numBundles: 16 # bundles serve the purpose of topic load balancing among brokers
backlog_quota_map:
destination_storage:
limit: 1099511627776 # 1TB, maximal backlog size of unack'ed messages
policy: producer_request_hold # keep the producer waiting/timeout when backlog reaches limit
deduplicationEnabled: false
autoTopicCreationOverride:
allowAutoTopicCreation: false
autoSubscriptionCreationOverride:
allowAutoSubscriptionCreation: false
message_ttl_in_seconds: 0 # don't expire unconsumed messages
subscription_expiration_time_minutes: 0 # don't expire subscription once inactive(no connections, no consumptions, etc)
retention_policies:
retentionTimeInMinutes: 0 # don't retent the messages around once they are consumed
retentionSizeInMB: 0 # save as above
delayed_delivery_policies:
active: true # enable delayed delivery in now namespace
tickTime: 1000 # tick time for delayed delivery
inactive_topic_policies:
deleteWhileInactive: false # disable auto-deletion on inactive topics
```
2. Create the subscription manually with Latest position and Shared mode
3. Launch the command line consumer for 1000 messages
4. Try a couple of time, the command line consumer would stuck waiting for incoming messages.
5. Check partitioned-stats with admin CLI you could potentially get below:
6. subscription `scheduler` was from golang client, while subscription `test` was with CLI consumer. Restarting consumers would trigger pulsar broker to flush the messages
```yaml
{
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesInCounter" : 349869,
"msgInCounter" : 2906,
"bytesOutCounter" : 316008,
"msgOutCounter" : 2618,
"averageMsgSize" : 0.0,
"msgChunkPublished" : false,
"storageSize" : 349869,
"backlogSize" : 41503,
"offloadedStorageSize" : 0,
"publishers" : [ {
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"averageMsgSize" : 0.0,
"chunkedMessageRate" : 0.0,
"producerId" : 0
} ],
"subscriptions" : {
"scheduler" : {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 216457,
"msgOutCounter" : 1794,
"msgRateRedeliver" : 0.0,
"chuckedMessageRate" : 0,
"msgBacklog" : 134,
"backlogSize" : 0,
"msgBacklogNoDelayed" : 0,
"blockedSubscriptionOnUnackedMsgs" : false,
"msgDelayed" : 134,
"unackedMessages" : 0,
"msgRateExpired" : 0.0,
"totalMsgExpired" : 0,
"lastExpireTimestamp" : 0,
"lastConsumedFlowTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"lastAckedTimestamp" : 0,
"lastMarkDeleteAdvancedTimestamp" : 0,
"consumers" : [ {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 216457,
"msgOutCounter" : 1794,
"msgRateRedeliver" : 0.0,
"chuckedMessageRate" : 0.0,
"availablePermits" : 6256,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0
} ],
"isDurable" : true,
"isReplicated" : false,
"consumersAfterMarkDeletePosition" : { },
"nonContiguousDeletedMessagesRanges" : 29,
"nonContiguousDeletedMessagesRangesSerializedSize" : 2824
},
"test" : {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 99551,
"msgOutCounter" : 824,
"msgRateRedeliver" : 0.0,
"chuckedMessageRate" : 0,
"msgBacklog" : 134,
"backlogSize" : 0,
"msgBacklogNoDelayed" : 0,
"blockedSubscriptionOnUnackedMsgs" : false,
"msgDelayed" : 134,
"unackedMessages" : 0,
"msgRateExpired" : 0.0,
"totalMsgExpired" : 0,
"lastExpireTimestamp" : 0,
"lastConsumedFlowTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"lastAckedTimestamp" : 0,
"lastMarkDeleteAdvancedTimestamp" : 0,
"consumers" : [ {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 99551,
"msgOutCounter" : 824,
"msgRateRedeliver" : 0.0,
"chuckedMessageRate" : 0.0,
"availablePermits" : 63176,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0
} ],
"isDurable" : true,
"isReplicated" : false,
"consumersAfterMarkDeletePosition" : { },
"nonContiguousDeletedMessagesRanges" : 29,
"nonContiguousDeletedMessagesRangesSerializedSize" : 2670
}
},
"replication" : { },
"nonContiguousDeletedMessagesRanges" : 58,
"nonContiguousDeletedMessagesRangesSerializedSize" : 5494,
"metadata" : {
"partitions" : 64
},
"partitions" : { }
}
```
**Expected behavior**
Messages should be cleared out with restart of consumers
**Additional context**
Add any other context about the problem here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #10497: blocked consumers on partitioned topics with delayed messages and Shared subscription
Posted by GitBox <gi...@apache.org>.
codelipenghui closed issue #10497:
URL: https://github.com/apache/pulsar/issues/10497
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] RobertIndie edited a comment on issue #10497: blocked consumers on partitioned topics with delayed messages and Shared subscription
Posted by GitBox <gi...@apache.org>.
RobertIndie edited a comment on issue #10497:
URL: https://github.com/apache/pulsar/issues/10497#issuecomment-979050615
What pulsar version did you use? I was unable to reproduce the problem in 2.8.1 by following the steps above. Can this problem be reproduced stably?
Could you enable the debug log for `org.apache.pulsar.broker.delayed.InMemoryDelayedDeliveryTracker` and paste the broker log here when you can reproduce the problem? This helps debug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] RobertIndie edited a comment on issue #10497: blocked consumers on partitioned topics with delayed messages and Shared subscription
Posted by GitBox <gi...@apache.org>.
RobertIndie edited a comment on issue #10497:
URL: https://github.com/apache/pulsar/issues/10497#issuecomment-979050615
@yangou What pulsar version did you use? I was unable to reproduce the problem in 2.8.1 by following the steps above. Can this problem be reproduced stably?
Could you enable the debug log for `org.apache.pulsar.broker.delayed.InMemoryDelayedDeliveryTracker` and paste the broker log here when you can reproduce the problem? This helps debug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] RobertIndie commented on issue #10497: blocked consumers on partitioned topics with delayed messages and Shared subscription
Posted by GitBox <gi...@apache.org>.
RobertIndie commented on issue #10497:
URL: https://github.com/apache/pulsar/issues/10497#issuecomment-979050615
What pulsar version did you use? I was unable to reproduce the problem in 2.8.1 by following the steps above. Can this problem be reproduced stably?
Could you enable the debug log for `org.apache.pulsar.broker.delayed.InMemoryDelayedDeliveryTracker` and paste the broker log here when you can reproduce the problem? This helps to debug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org