You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "mawenyu (via GitHub)" <gi...@apache.org> on 2024/07/19 15:14:49 UTC
[I] [Bug] Bundle unload can cause shared consumer receive duplicate messages [pulsar]
mawenyu opened a new issue, #23056:
URL: https://github.com/apache/pulsar/issues/23056
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Read release policy
- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
### Version
3.0.4
### Minimal reproduce step
1. two node broker
2. three client process use same subscription consumer one topic
3. consumer receiveQueue 5
4. consumer cost 3 second to handle each message and then ack
5. unload topic from brokerA to brokerB
6. three client process may receive some messages that already handled by other consumer ;
### What did you expect to see?
bundle unload will not cause message duplication
### What did you see instead?
bundle unload will not cause unacked message deliver to other consumer， which will cause data duplication
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Bundle unload can cause shared consumer receive duplicate messages [pulsar]
Posted by "mawenyu (via GitHub)" <gi...@apache.org>.
mawenyu commented on issue #23056:
URL: https://github.com/apache/pulsar/issues/23056#issuecomment-2437067537
> @mawenyu Which Pulsar client version and Pulsar broker version are you using?
3.0.4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Bundle unload can cause shared consumer receive duplicate messages [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #23056:
URL: https://github.com/apache/pulsar/issues/23056#issuecomment-2277902880
@mawenyu Which Pulsar client version and Pulsar broker version are you using?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Bundle unload can cause shared consumer receive duplicate messages [pulsar]
Posted by "mawenyu (via GitHub)" <gi...@apache.org>.
mawenyu commented on issue #23056:
URL: https://github.com/apache/pulsar/issues/23056#issuecomment-2249821241
I solved this issue by modifying the consumer source code. Could you please ask the community if they would consider adopting this approach or if there are any more elegant solutions? The main modifications are:
1. In the `org.apache.pulsar.client.impl.ConsumerImpl#connectionOpened` , call the parent consumer’s `org.apache.pulsar.client.impl.MultiTopicsConsumerImpl` clearIncomingMessages(). —— This is necessary because Pulsar does not perform this cleanup , these message will cause duplication
1. In the `org.apache.pulsar.client.impl.ConsumerImpl#connectionOpened` delay the invoke of `org.apache.pulsar.client.impl.ConsumerImpl#increaseAvailablePermits` by 3 seconds and execute it asynchronously. —— This 3-second delay allows the client to acknowledge the received messages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Bundle unload can cause shared consumer receive duplicate messages [pulsar]
Posted by "hanmz (via GitHub)" <gi...@apache.org>.
hanmz commented on issue #23056:
URL: https://github.com/apache/pulsar/issues/23056#issuecomment-2242243648
The problem of message duplication is difficult to avoid. The better solution now is to implement idempotence on the client side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Bundle unload can cause shared consumer receive duplicate messages [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #23056:
URL: https://github.com/apache/pulsar/issues/23056#issuecomment-2277901801
> I solved this issue by modifying the consumer source code. Could you please ask the community if they would consider adopting this approach or if there are any more elegant solutions? The main modifications are:
>
> 1. In the `org.apache.pulsar.client.impl.ConsumerImpl#connectionOpened` , call the parent consumer’s `org.apache.pulsar.client.impl.MultiTopicsConsumerImpl` clearIncomingMessages(). —— This is necessary because Pulsar does not perform this cleanup , these message will cause duplication
@mawenyu Interesting. This seems to be related to #21767 which is a draft PR I have.
> 2. In the `org.apache.pulsar.client.impl.ConsumerImpl#connectionOpened` delay the invoke of `org.apache.pulsar.client.impl.ConsumerImpl#increaseAvailablePermits` by 3 seconds and execute it asynchronously. —— This 3-second delay allows the client to acknowledge the received messages.
Just wondering if the https://github.com/apache/pulsar/wiki/PIP-84-:-Pulsar-client:-Redeliver-command-add-epoch design or implementation has some gap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org