You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/06/10 09:52:13 UTC
[GitHub] [pulsar] xuesongxs opened a new issue #10890: The consumer receives the same message again after executing the acknowledge interface
xuesongxs opened a new issue #10890:
URL: https://github.com/apache/pulsar/issues/10890
**Describe the bug**
We found a problem with pulsar consumer ack in our test. The producer sends 1W messages. We use the consumer's acknowledge interface to submit the offset. When we start the consumer process to consume a part of the message, we kill the process and start the same consumer process again to consume the message. The message that has executed the acknowledge interface before killing will be received, It leads to the repetition of consumption.
pulsar v2.6.2
consumer code:
```
ConsumerBuilder consumerBuilder = getClient().newConsumer(Schema.STRING);
List<String> topics = new ArrayList<>();
topics.add("persistent://public/default/test-string5");
consumerBuilder.topics(topics)
.subscriptionName("consumerSych")
.subscriptionType(SubscriptionType.Shared)
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest);
Consumer<String> consumer = consumerBuilder.subscribe();
while (true) {
Message msg = consumer.receive();
consumer.acknowledge(msg);
System.out.println("ack message:" + msg.getMessageId().toString());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #10890: The consumer receives the same message again after executing the acknowledge interface
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #10890:
URL: https://github.com/apache/pulsar/issues/10890#issuecomment-860731533
@xuesongxs I'll close this issue. Please reopen if your issue didn't get resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] xuesongxs commented on issue #10890: The consumer receives the same message again after executing the acknowledge interface
Posted by GitBox <gi...@apache.org>.
xuesongxs commented on issue #10890:
URL: https://github.com/apache/pulsar/issues/10890#issuecomment-861125622
> @xuesongxs I'll close this issue. Please reopen if your issue didn't get resolved.
How to reopen the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari closed issue #10890: The consumer receives the same message again after executing the acknowledge interface
Posted by GitBox <gi...@apache.org>.
lhotari closed issue #10890:
URL: https://github.com/apache/pulsar/issues/10890
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #10890: The consumer receives the same message again after executing the acknowledge interface
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #10890:
URL: https://github.com/apache/pulsar/issues/10890#issuecomment-1058889567
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] xuesongxs commented on issue #10890: The consumer receives the same message again after executing the acknowledge interface
Posted by GitBox <gi...@apache.org>.
xuesongxs commented on issue #10890:
URL: https://github.com/apache/pulsar/issues/10890#issuecomment-861124936
> @xuesongxs I'll close this issue. Please reopen if your issue didn't get resolved.
According to the modification of acknowledsegmentatbatchindexlevelenabled=true, the problem still exists. I use the producer's synchronization interface. Even if I set enable batching (false) in the producer, and enable batching (false) can not meet the needs of the production environment, because I want to use asynchronous transmission in the production environment, after enable batching (false), the performance can not meet the requirements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #10890: The consumer receives the same message again after executing the acknowledge interface
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #10890:
URL: https://github.com/apache/pulsar/issues/10890#issuecomment-860007430
@xuesongxs There's an explanation in the manual under ["Batching"](https://pulsar.apache.org/docs/en/next/concepts-messaging/#batching):
> In Pulsar, batches are tracked and stored as single units rather than as individual messages. Consumer unbundles a batch into individual messages. However, scheduled messages (configured through the deliverAt or the deliverAfter parameter) are always sent as individual messages even batching is enabled.
>
> In general, a batch is acknowledged when all of its messages are acknowledged by a consumer. It means unexpected failures, negative acknowledgements, or acknowledgement timeouts can result in redelivery of all messages in a batch, even if some of the messages are acknowledged.
>
> To avoid redelivering acknowledged messages in a batch to the consumer, Pulsar introduces batch index acknowledgement since Pulsar 2.6.0. When batch index acknowledgement is enabled, the consumer filters out the batch index that has been acknowledged and sends the batch index acknowledgement request to the broker. The broker maintains the batch index acknowledgement status and tracks the acknowledgement status of each batch index to avoid dispatching acknowledged messages to the consumer. When all indexes of the batch message are acknowledged, the batch message is deleted.
>
> By default, batch index acknowledgement is disabled (acknowledgmentAtBatchIndexLevelEnabled=false). You can enable batch index acknowledgement by setting the acknowledgmentAtBatchIndexLevelEnabled parameter to true at the broker side. Enabling batch index acknowledgement results in more memory overheads.
>
Batching happens on the producer side. Another option for `acknowledgmentAtBatchIndexLevelEnabled=true` is to produce the messages into the topic without batching enabled. This can be achieved by setting `.enableBatching(false)` on the `ProducerBuilder`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org