You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/09/27 07:56:46 UTC
[GitHub] [pulsar] 4onni opened a new issue #12208: Consumers get stuck in Key_shard mode and delayed messages
4onni opened a new issue #12208:
URL: https://github.com/apache/pulsar/issues/12208
Pulsar version : 2.7.3
Pulsar Client version: 2.7.1
I have a topic that sends delayed messages, and consumers use Key_shard mode, and whenever consumers make changes (batch reboots or adding machines) messages start to pile up and deliveries stop. Here are the topic statistics when stuck
```json
"persistent://infra/es-channel/coupon_activity-partition-0": {
"msgRateIn": 99.32590087869627,
"msgRateOut": 0,
"msgThroughputIn": 81197.97397460767,
"msgThroughputOut": 0,
"averageMsgSize": 817.490435589125,
"storageSize": 28423651,
"publishers": [
{
"producerId": 1,
"msgRateIn": 1.3998954266917099,
"msgThroughputIn": 645.8350889355218,
"averageMsgSize": 461,
"metadata": {}
},
{
"producerId": 2,
"msgRateIn": 97.92600545200456,
"msgThroughputIn": 80552.13888567215,
"averageMsgSize": 822,
"metadata": {}
}
],
"subscriptions": {
"exp-es-sync_order_sms_send_info_sub": {
"blockedSubscriptionOnUnackedMsgs": false,
"isReplicated": false,
"msgRateOut": 0,
"msgThroughputOut": 0,
"msgRateRedeliver": 0,
"msgRateExpired": 0,
"msgBacklog": 31594,
"msgDelayed": 136,
"unackedMessages": 0,
"type": "Key_Shared",
"activeConsumerName": "",
"consumers": [
{
"blockedConsumerOnUnackedMsgs": false,
"availablePermits": 957,
"unackedMessages": 0,
"msgRateOut": 0,
"msgThroughputOut": 0,
"msgRateRedeliver": 0,
"consumerName": "exp-es-sync_consumer_10.42.58.61_0",
"metadata": {}
},
{
"blockedConsumerOnUnackedMsgs": false,
"availablePermits": 836,
"unackedMessages": 0,
"msgRateOut": 0,
"msgThroughputOut": 0,
"msgRateRedeliver": 0,
......
......
.....
```
Note: unload topic can solve the problem
Steps to reproduce the behavior:
1. a producer keeps sending delayed messages
```java
producer.newMessage().key("" + index).value("a message : " + (index))
.deliverAfter(3, TimeUnit.SECONDS)
.sendAsync()
```
2. start several consumers in Key_shard mode
```java
IntStream.range(start, end).forEach(index -> {
AtomicInteger i = new AtomicInteger();
client.newConsumer(Schema.STRING)
.topic("persistent://public/default/3p")
.subscriptionName("test_sub")
.consumerName("c_" + index)
.subscriptionType(SubscriptionType.Key_Shared)
........
.......
```
3. and then add or restart consumers
4. you'll see consumers get stuck
The expectation that no matter how consumers change, they won't get stuck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 4onni closed issue #12208: Consumers get stuck in Key_shard mode and delayed messages
Posted by GitBox <gi...@apache.org>.
4onni closed issue #12208:
URL: https://github.com/apache/pulsar/issues/12208
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 4onni commented on issue #12208: Consumers get stuck in Key_shard mode and delayed messages
Posted by GitBox <gi...@apache.org>.
4onni commented on issue #12208:
URL: https://github.com/apache/pulsar/issues/12208#issuecomment-927670581
Delayed messages can only be used with shard mode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] n3ziniuka5 commented on issue #12208: Consumers get stuck in Key_shard mode and delayed messages
Posted by GitBox <gi...@apache.org>.
n3ziniuka5 commented on issue #12208:
URL: https://github.com/apache/pulsar/issues/12208#issuecomment-930182966
We are facing the same issue at the moment. I think at a minimum, the docs could be a bit more clear. Here is what they currently say:
> Delayed message delivery only works in Shared subscription mode. In Exclusive and Failover subscription modes, the delayed message is dispatched immediately.
Exclusive and Failover modes are explicitly mentioned, but nothing says that KeyShared subscriptions will be stuck, which is quite an important detail.
But it would be nice if it was impossible to encounter this problem in the first place. Since delayed messages do have some caveats, one way to address this would be to require them to be explicitly enabled for a topic. Once this feature is enabled you would not be able to create unsupported subscription types on that topic. Without the feature enabled all delayed messages would be rejected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] n3ziniuka5 commented on issue #12208: Consumers get stuck in Key_shard mode and delayed messages
Posted by GitBox <gi...@apache.org>.
n3ziniuka5 commented on issue #12208:
URL: https://github.com/apache/pulsar/issues/12208#issuecomment-930182966
We are facing the same issue at the moment. I think at a minimum, the docs could be a bit more clear. Here is what they currently say:
> Delayed message delivery only works in Shared subscription mode. In Exclusive and Failover subscription modes, the delayed message is dispatched immediately.
Exclusive and Failover modes are explicitly mentioned, but nothing says that KeyShared subscriptions will be stuck, which is quite an important detail.
But it would be nice if it was impossible to encounter this problem in the first place. Since delayed messages do have some caveats, one way to address this would be to require them to be explicitly enabled for a topic. Once this feature is enabled you would not be able to create unsupported subscription types on that topic. Without the feature enabled all delayed messages would be rejected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org