You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/11/11 16:51:52 UTC
[GitHub] [pulsar] ericsyh opened a new issue #12759: [bug](indeviduallyDeleteMessage): slow rate consumer but ask in order will get indeviduallyDeleteMessage
ericsyh opened a new issue #12759:
URL: https://github.com/apache/pulsar/issues/12759
**Describe the bug**
In my current test found, if i create a producer to produce with no dispatch limit while create a consumer to consumer with slow rate (set a 1 second sleep in message receive), although the the consumer acks in order but there is indeviduallyDeleteMessage in the topic.
And the weird thing is if i remove the 1 second sleep in consumer or i move the 1 second sleep from consumer to producer, there is no indeviduallyDeleteMessage in the topics. It looks like when the consumer is much slower than the producer, this issue will come.
This is the consume code i used, i add a delay for 1 second but the ack should be in order.
```
for true {
msg, err := consumer.Receive(context.Background())
if err != nil {
log.Fatal(err)
}
fmt.Printf("Received message msgId: %#v -- content: '%s'\n",
msg.ID(), string(msg.Payload()))
consumer.Ack(msg)
time.Sleep(time.Duration(1) * time.Second)
}
```
**To Reproduce**
Steps to reproduce the behavior:
1. Start the producer without dispatch limit
2. Start the consumer with 1 second limit to consume
3. Stop the consumer after several consumes and acks are sent.
4. Check the topic stats-internal and there is some indeviduallyDeleteMessage.
**Expected behavior**
The ack is in order, there is no indeviduallyDeleteMessage in the topic.
**Screenshots**
I use golang-client to produce and consume
keep on producing and stop the consumer after several consume with acks
<img width="1440" alt="截屏2021-11-12 上午12 37 05" src="https://user-images.githubusercontent.com/10498732/141335595-f612f3e5-aa54-4319-a4f2-4bae795b4300.png">
Check the topic stats-internal and there is some indeviduallyDeleteMessage.
<img width="1293" alt="截屏2021-11-12 上午12 38 36" src="https://user-images.githubusercontent.com/10498732/141335725-3b4f5f65-faee-4b62-965d-4711ed0e8760.png">
The test code i used
[pulsar-client-go-demo.zip](https://github.com/apache/pulsar/files/7521899/pulsar-client-go-demo.zip)
**Additional context**
golang client and java client both tested with this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin edited a comment on issue #12759: [bug](indeviduallyDeleteMessage): slow rate consumer but ack in order will get indeviduallyDeleteMessage
Posted by GitBox <gi...@apache.org>.
gaozhangmin edited a comment on issue #12759:
URL: https://github.com/apache/pulsar/issues/12759#issuecomment-972727778
The further entry was acked in method `filterEntriesForConsumer`
Because message is an internal marker. i don't know why entry was acked under this condition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on issue #12759: [bug](indeviduallyDeleteMessage): slow rate consumer but ack in order will get indeviduallyDeleteMessage
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on issue #12759:
URL: https://github.com/apache/pulsar/issues/12759#issuecomment-978779327
@merlimat May be you can help explain the ` subscription.acknowledgeMessage` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on issue #12759: [bug](indeviduallyDeleteMessage): slow rate consumer but ask in order will get indeviduallyDeleteMessage
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on issue #12759:
URL: https://github.com/apache/pulsar/issues/12759#issuecomment-969978134
@ericsyh Please provide your java code . I tested with java client on pulsar 2.8 and master branch. I cannot reply this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ericsyh commented on issue #12759: [bug](indeviduallyDeleteMessage): slow rate consumer but ask in order will get indeviduallyDeleteMessage
Posted by GitBox <gi...@apache.org>.
ericsyh commented on issue #12759:
URL: https://github.com/apache/pulsar/issues/12759#issuecomment-969990878
@zymap Do you still have the Java client which can reproduce this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ericsyh closed issue #12759: [bug](indeviduallyDeleteMessage): slow rate consumer but ack in order will get indeviduallyDeleteMessage
Posted by GitBox <gi...@apache.org>.
ericsyh closed issue #12759:
URL: https://github.com/apache/pulsar/issues/12759
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ericsyh commented on issue #12759: [bug](indeviduallyDeleteMessage): slow rate consumer but ask in order will get indeviduallyDeleteMessage
Posted by GitBox <gi...@apache.org>.
ericsyh commented on issue #12759:
URL: https://github.com/apache/pulsar/issues/12759#issuecomment-970155010
@gaozhangmin PTAL. I can reproduce this issue with java consumer code.
The way i reproduce this issue:
1. Start the producer without dispatch limit and keep producing for a long time
2. Start the consumer with the consumer.java
3. Stop the consumer after several consumes and acks are sent while keep producing
4. Check the topic stats-internal and there is some indeviduallyDeleteMessage.
[consumer.java.zip](https://github.com/apache/pulsar/files/7545698/consumer.java.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin edited a comment on issue #12759: [bug](indeviduallyDeleteMessage): slow rate consumer but ack in order will get indeviduallyDeleteMessage
Posted by GitBox <gi...@apache.org>.
gaozhangmin edited a comment on issue #12759:
URL: https://github.com/apache/pulsar/issues/12759#issuecomment-972488995
@ericsyh The reason is, as the below figure shows, ack of entry 40:517 should not sent after 40:4. It's weird.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on issue #12759: [bug](indeviduallyDeleteMessage): slow rate consumer but ack in order will get indeviduallyDeleteMessage
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on issue #12759:
URL: https://github.com/apache/pulsar/issues/12759#issuecomment-972488995
@ericsyh The reason is, as the below figure shows, ack of entry 40:517 should not sent after 40:4. It's weird.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ericsyh commented on issue #12759: [bug](indeviduallyDeleteMessage): slow rate consumer but ask in order will get indeviduallyDeleteMessage
Posted by GitBox <gi...@apache.org>.
ericsyh commented on issue #12759:
URL: https://github.com/apache/pulsar/issues/12759#issuecomment-966461249
@codelipenghui @hangc0276 PTAL, this issue probably a bug and will impact the replicated subscriptions.
@zymap Also helped on it before, he can also reproduce this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] hangc0276 commented on issue #12759: [bug](indeviduallyDeleteMessage): slow rate consumer but ack in order will get indeviduallyDeleteMessage
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on issue #12759:
URL: https://github.com/apache/pulsar/issues/12759#issuecomment-980845899
This is the expected behavior.
It was introduced by https://github.com/apache/pulsar/wiki/PIP-33%3A-Replicated-subscriptions
In order to support replicated subscription stats between different pulsar clusters, it will start subscription snapshot every 1 second by default if we enable this feature in consumer subscription `replicateSubscriptionState(true)`.
The snapshot will write an internal mark into the topic message and will be filter out on message dispatch period. Thus you will see the messageId has been acked even though you haven't ack the message yet. In fact, you will never receive this message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ericsyh commented on issue #12759: [bug](indeviduallyDeleteMessage): slow rate consumer but ask in order will get indeviduallyDeleteMessage
Posted by GitBox <gi...@apache.org>.
ericsyh commented on issue #12759:
URL: https://github.com/apache/pulsar/issues/12759#issuecomment-970227493
I got it on 2.8.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ericsyh commented on issue #12759: [bug](indeviduallyDeleteMessage): slow rate consumer but ack in order will get indeviduallyDeleteMessage
Posted by GitBox <gi...@apache.org>.
ericsyh commented on issue #12759:
URL: https://github.com/apache/pulsar/issues/12759#issuecomment-973746977
@codelipenghui @hangc0276 We need some help to have a deep look on this issue.
@gaozhangmin Helped a lot on the issue reproduce and locate the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on issue #12759: [bug](indeviduallyDeleteMessage): slow rate consumer but ack in order will get indeviduallyDeleteMessage
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on issue #12759:
URL: https://github.com/apache/pulsar/issues/12759#issuecomment-972727778
The further entry ack was acked in method `filterEntriesForConsumer`
Because message is an internal marker. i don't know why entry was acked under this condition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on issue #12759: [bug](indeviduallyDeleteMessage): slow rate consumer but ack in order will get indeviduallyDeleteMessage
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on issue #12759:
URL: https://github.com/apache/pulsar/issues/12759#issuecomment-971437099
Can u test with #12343. @ericsyh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on issue #12759: [bug](indeviduallyDeleteMessage): slow rate consumer but ack in order will get indeviduallyDeleteMessage
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on issue #12759:
URL: https://github.com/apache/pulsar/issues/12759#issuecomment-973670809
@ericsyh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin removed a comment on issue #12759: [bug](indeviduallyDeleteMessage): slow rate consumer but ack in order will get indeviduallyDeleteMessage
Posted by GitBox <gi...@apache.org>.
gaozhangmin removed a comment on issue #12759:
URL: https://github.com/apache/pulsar/issues/12759#issuecomment-971437099
Can u test with #12343. @ericsyh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ericsyh commented on issue #12759: [bug](indeviduallyDeleteMessage): slow rate consumer but ack in order will get indeviduallyDeleteMessage
Posted by GitBox <gi...@apache.org>.
ericsyh commented on issue #12759:
URL: https://github.com/apache/pulsar/issues/12759#issuecomment-972553275
@gaozhangmin Yep, in my test also found that some further acks were sent before my consume. Maybe we need to check the sdk code why the further entry ack would be sent to Pulsar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ericsyh edited a comment on issue #12759: [bug](indeviduallyDeleteMessage): slow rate consumer but ask in order will get indeviduallyDeleteMessage
Posted by GitBox <gi...@apache.org>.
ericsyh edited a comment on issue #12759:
URL: https://github.com/apache/pulsar/issues/12759#issuecomment-966461249
@codelipenghui @hangc0276 PTAL, this issue probably a bug and will impact the replicated subscriptions.
@zymap Also helped on it before, he can also reproduce this issue with Java client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on issue #12759: [bug](indeviduallyDeleteMessage): slow rate consumer but ask in order will get indeviduallyDeleteMessage
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on issue #12759:
URL: https://github.com/apache/pulsar/issues/12759#issuecomment-966850442
I would like to help on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on issue #12759: [bug](indeviduallyDeleteMessage): slow rate consumer but ask in order will get indeviduallyDeleteMessage
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on issue #12759:
URL: https://github.com/apache/pulsar/issues/12759#issuecomment-970188365
> @gaozhangmin PTAL. I can reproduce this issue with java consumer code.
>
> The way i reproduce this issue:
>
> 1. Start the producer without dispatch limit and keep producing for a long time
> 2. Start the consumer with the consumer.java
> 3. Stop the consumer after several consumes and acks are sent while keep producing
> 4. Check the topic stats-internal and there is some indeviduallyDeleteMessage.
>
> [consumer.java.zip](https://github.com/apache/pulsar/files/7545698/consumer.java.zip)
On which pulsar version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org