You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/07/26 15:22:19 UTC
[GitHub] [pulsar] BewareMyPower opened a new issue, #16802: Repeated messages of shared dispatcher
BewareMyPower opened a new issue, #16802:
URL: https://github.com/apache/pulsar/issues/16802
**Describe the bug**
Recently I found the C++ tests `BasicEndToEndTest.testPatternMultiTopicsConsumerPubSub` and `BasicEndToEndTest.testpatternMultiTopicsHttpConsumerPubSub` became flaky. Eventually I found it's caused by the duplicated messages.
**To Reproduce**
First, start a standalone with latest master.
Then, run the following Java test.
```java
@Test(invocationCount = 10)
public void test() throws Exception {
@Cleanup final PulsarAdmin admin = PulsarAdmin.builder().serviceHttpUrl("http://127.0.0.1:8080").build();
final String topicBase = "persistent://public/default/my-topic-" + System.currentTimeMillis();
for (int i = 0; i < 3; i++) {
admin.topics().createPartitionedTopic(topicBase + "-" + i, i + 2);
}
@Cleanup final PulsarClient client = PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build();
final Consumer<byte[]> consumer = client.newConsumer()
.topicsPattern(topicBase + ".*")
.subscriptionName("sub")
.receiverQueueSize(10)
.subscriptionType(SubscriptionType.Shared)
.subscribe();
final int numMessages = 100;
for (int i = 0; i < 3; i++) {
final Producer<byte[]> producer = client.newProducer()
.topic(topicBase + "-" + i)
.messageRoutingMode(MessageRoutingMode.RoundRobinPartition)
.create();
for (int j = 0; j < numMessages; j++) {
producer.send(("msg-content-" + i + "-" + j).getBytes());
}
}
int numReceived = 0;
while (true) {
final Message<byte[]> msg = consumer.receive(1, TimeUnit.SECONDS);
if (msg == null) {
break;
}
numReceived++;
}
Assert.assertEquals(numReceived, numMessages * 3);
}
```
Sometimes it failed. (You can add more logs or debugged code to see the received messages of all partitions)
**Expected behavior**
The tests should always succeed.
**Screenshots**
The failure in my local env.
<img width="1478" alt="截屏2022-07-26 23 00 19" src="https://user-images.githubusercontent.com/18204803/181043015-a83b7ca4-d12d-4e6d-bc73-30b592d16ab4.png">
<img width="1463" alt="image" src="https://user-images.githubusercontent.com/18204803/181043586-2d5f805c-2222-41da-8aef-20c12bb6f458.png">
**Additional context**
Adding some logs into C++ tests could make tests more stable to pass. And the Java test looks less flaky than the C++ tests.
After applying `dispatcherDispatchMessagesInSubscriptionThread=false` in `conf/standalone.conf`, the Java test could pass even if `invocationCount = 50`.
<img width="1700" alt="image" src="https://user-images.githubusercontent.com/18204803/181045520-9394d7c2-cd88-4c6b-b052-2c8ba1a140a7.png">
From the test result, we can see the bug was introduced from https://github.com/apache/pulsar/pull/16603 that we should not run `sendMessagesToConsumers` in another thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tisonkun commented on issue #16802: Repeated messages of shared dispatcher
Posted by GitBox <gi...@apache.org>.
tisonkun commented on issue #16802:
URL: https://github.com/apache/pulsar/issues/16802#issuecomment-1208934884
```
/pulsar/pulsar-client-cpp/tests/BasicEndToEndTest.cc:2184: Failure
Expected equality of these values:
ResultTimeout
Which is: TimeOut
consumer.receive(m, 1000)
Which is: Ok
[ FAILED ] BasicEndToEndTest.testPatternMultiTopicsConsumerPubSub (1173 ms)
[----------] 1 test from BasicEndToEndTest (1173 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (1173 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] BasicEndToEndTest.testPatternMultiTopicsConsumerPubSub
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tisonkun commented on issue #16802: Repeated messages of shared dispatcher
Posted by GitBox <gi...@apache.org>.
tisonkun commented on issue #16802:
URL: https://github.com/apache/pulsar/issues/16802#issuecomment-1208989679
It seems the exception is timeout instead of unexpected result, reported at https://github.com/apache/pulsar/issues/17008.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on issue #16802: Repeated messages of shared dispatcher
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #16802:
URL: https://github.com/apache/pulsar/issues/16802#issuecomment-1196335777
I am working on a fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli closed issue #16802: Repeated messages of shared dispatcher
Posted by GitBox <gi...@apache.org>.
eolivelli closed issue #16802: Repeated messages of shared dispatcher
URL: https://github.com/apache/pulsar/issues/16802
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on issue #16802: Repeated messages of shared dispatcher
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #16802:
URL: https://github.com/apache/pulsar/issues/16802#issuecomment-1196339003
in #16603 I had to change a test that was failing in a similar way.
but the test was missing a "acknowledge(message)" so I thought that the test was wrong.
https://github.com/apache/pulsar/pull/16603/files#diff-33303e92b2d03115c1bde9ad3af39f9127f6a25270855fb5b5c2eb7c3db74e50R281
I will revert the change to the test and add your new test case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tisonkun commented on issue #16802: Repeated messages of shared dispatcher
Posted by GitBox <gi...@apache.org>.
tisonkun commented on issue #16802:
URL: https://github.com/apache/pulsar/issues/16802#issuecomment-1208934327
https://github.com/apache/pulsar/runs/7739162912?check_suite_focus=true
This can be another failure case of `BasicEndToEndTest.testPatternMultiTopicsConsumerPubSub`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org