You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "SennoYuki (via GitHub)" <gi...@apache.org> on 2024/05/15 09:54:27 UTC
[I] [Bug] Delayed Message Delivery immediately after [pulsar]
SennoYuki opened a new issue, #22716:
URL: https://github.com/apache/pulsar/issues/22716
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Read release policy
- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
### Version
OS: linux 3.10.0
Java: jdk17
pulsar: 3.2.0
### Minimal reproduce step
1. produce delayed message to a topic
2. restart topic owner broker
### What did you expect to see?
the message consume keeps normal
### What did you see instead?
The delayed message delivery immediately
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Delayed Message Delivery immediately after owner broker restart [pulsar]
Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on issue #22716:
URL: https://github.com/apache/pulsar/issues/22716#issuecomment-2115528938
I need more context:
1. What's your subscription type? `Exclusive`,`Failover`,`Shared` or `Key_Shared`?
2. Did you enable `delayedDeliveryEnabled` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Delayed Message Delivery immediately after [pulsar]
Posted by "SennoYuki (via GitHub)" <gi...@apache.org>.
SennoYuki commented on issue #22716:
URL: https://github.com/apache/pulsar/issues/22716#issuecomment-2112077561
And I had already used config
`delayedDeliveryTrackerFactoryClassName=org.apache.pulsar.broker.delayed.BucketDelayedDeliveryTrackerFactory`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Delayed Message Delivery immediately after owner broker restart [pulsar]
Posted by "SennoYuki (via GitHub)" <gi...@apache.org>.
SennoYuki commented on issue #22716:
URL: https://github.com/apache/pulsar/issues/22716#issuecomment-2116543723
> I need more context:
>
> 1. What's your subscription type? `Exclusive`,`Failover`,`Shared` or `Key_Shared`?
> 2. Did you enable `delayedDeliveryEnabled` ?
1. Shared
2. enable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Delayed Message cursor lost after topic unload [pulsar]
Posted by "SennoYuki (via GitHub)" <gi...@apache.org>.
SennoYuki commented on issue #22716:
URL: https://github.com/apache/pulsar/issues/22716#issuecomment-2119894040
@dao-jun @coderzc Any update?
Today I reproduce the problem with topic unload operation. Can you check the unload code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Delayed Message cursor lost after topic unload [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed issue #22716: [Bug] Delayed Message cursor lost after topic unload
URL: https://github.com/apache/pulsar/issues/22716
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Delayed Message cursor lost after topic unload [pulsar]
Posted by "SennoYuki (via GitHub)" <gi...@apache.org>.
SennoYuki commented on issue #22716:
URL: https://github.com/apache/pulsar/issues/22716#issuecomment-2121581784
@dao-jun
unload log
`5月 21 02:13:04 sg-prod-server-pulsar-3 pulsar[14996]: 2024-05-21T02:13:04,493+0000 [pulsar-web-42-3] INFO org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [null] Unloading topic persistent://public/default/push_al_queue
5月 21 02:13:04 sg-prod-server-pulsar-3 pulsar[14996]: 2024-05-21T02:13:04,511+0000 [metadata-store-9-1] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [21/May/2024:02:13:04 +0000] "PUT /admin/v2/persistent/public/default/push_al_queue/unload HTTP/1.1" 307 0 "-" "Pulsar-Java-v3.2.3" 21`
And metrics change
Sorry, maybe I cannot provide heap dump because of data safety
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Delayed Message cursor lost after topic unload [pulsar]
Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on issue #22716:
URL: https://github.com/apache/pulsar/issues/22716#issuecomment-2123060863
see also https://github.com/apache/pulsar/issues/20099#issuecomment-2123050102
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Delayed Message Delivery immediately after owner broker restart [pulsar]
Posted by "SennoYuki (via GitHub)" <gi...@apache.org>.
SennoYuki commented on issue #22716:
URL: https://github.com/apache/pulsar/issues/22716#issuecomment-2116768208
> I don't reproduce the issue, and I'll keep trying
>
> For determine if a subscription is durable, try run `./pulsar-admin topics stats persistent://public/default/my-topic1`, and see the `isDurable` field is true or false
It's true
` "isDurable" : true,
"isReplicated" : false,
"allowOutOfOrderDelivery" : false,
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Delayed Message Delivery immediately after owner broker restart [pulsar]
Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on issue #22716:
URL: https://github.com/apache/pulsar/issues/22716#issuecomment-2116941088
@coderzc I cannot reproduce the issue on the master branch, could you please take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Delayed Message Delivery immediately after owner broker restart [pulsar]
Posted by "SennoYuki (via GitHub)" <gi...@apache.org>.
SennoYuki commented on issue #22716:
URL: https://github.com/apache/pulsar/issues/22716#issuecomment-2116565100
And I discover my subscription cursor may not be durable. But I confirm I used durable subscription from consumer config and pulsar-admin topic stats
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Delayed Message cursor lost after topic unload [pulsar]
Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on issue #22716:
URL: https://github.com/apache/pulsar/issues/22716#issuecomment-2120536233
@SennoYuki Can you please provide broker log and heap dump?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Delayed Message cursor lost after topic unload [pulsar]
Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on issue #22716:
URL: https://github.com/apache/pulsar/issues/22716#issuecomment-2684024700
```
@FieldContext(
category = CATEGORY_STORAGE_ML,
doc = "Max number of `acknowledgment holes` that can be stored in MetadataStore.\n\n"
+ "If number of unack message range is higher than this limit then broker will persist"
+ " unacked ranges into bookkeeper to avoid additional data overhead into MetadataStore.")
private int managedLedgerMaxUnackedRangesToPersistInMetadataStore = 1000;
```
@SennoYuki
If the ack is discontinuous, Pulsar will record acknowledgment holes. However, by default, only 1000 acknowledgment holes will be persisted. Maybe you can try increasing the configuration above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Delayed Message Delivery immediately after owner broker restart [pulsar]
Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on issue #22716:
URL: https://github.com/apache/pulsar/issues/22716#issuecomment-2116907071
If `isDurable` is true, it means your cursor is durable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Delayed Message Delivery immediately after owner broker restart [pulsar]
Posted by "SennoYuki (via GitHub)" <gi...@apache.org>.
SennoYuki commented on issue #22716:
URL: https://github.com/apache/pulsar/issues/22716#issuecomment-2116565911
> I need more context:
>
> 1. What's your subscription type? `Exclusive`,`Failover`,`Shared` or `Key_Shared`?
> 2. Did you enable `delayedDeliveryEnabled` ?
If possible, maybe we can check this problem with IM software like wechat for efficiency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Delayed Message cursor lost after topic unload [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #22716:
URL: https://github.com/apache/pulsar/issues/22716#issuecomment-6381514946
Thanks @SennoYuki. This is tracked by #20099 (duplicate delayed messages after a topic unload), which you also commented on and which is still open. Closing this one in its favour so the discussion stays in one place. Please reopen if you see behaviour on a currently supported release (4.0.x, 4.2.x or 5.0.x) that #20099 does not cover.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Delayed Message cursor lost after topic unload [pulsar]
Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on issue #22716:
URL: https://github.com/apache/pulsar/issues/22716#issuecomment-2683964024
I think first of all, we need to confirm whether markDelete has been successfully persistented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Delayed Message Delivery immediately after owner broker restart [pulsar]
Posted by "SennoYuki (via GitHub)" <gi...@apache.org>.
SennoYuki commented on issue #22716:
URL: https://github.com/apache/pulsar/issues/22716#issuecomment-2116930529
> If `isDurable` is true, it means your cursor is durable
So I can't fix the situation. I can't restart the broker to tuning config because of duplicated consume happening
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Delayed Message Delivery immediately after owner broker restart [pulsar]
Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on issue #22716:
URL: https://github.com/apache/pulsar/issues/22716#issuecomment-2116672235
I don't reproduce the issue, and I'll keep trying
For determine if a subscription is durable, try run `./pulsar-admin topics stats persistent://public/default/my-topic1`, and see the `isDurable` field is true or false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Delayed Message cursor lost after topic unload [pulsar]
Posted by "SennoYuki (via GitHub)" <gi...@apache.org>.
SennoYuki commented on issue #22716:
URL: https://github.com/apache/pulsar/issues/22716#issuecomment-2121657225
https://github.com/apache/pulsar/issues/20638
Other people encountered similar problems. After cursor recovery, it seems that the cursor is simply reset to the position of the last ack submission. This will cause a large number of messages to be consumed repeatedly in the delay queue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org