You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/01/24 22:56:52 UTC
[GitHub] [pulsar] devinbost opened a new issue #6137: Unstable Github tests
for Pulsar
devinbost opened a new issue #6137: Unstable Github tests for Pulsar
URL: https://github.com/apache/pulsar/issues/6137
**Description**
Pulsar tests regularly succeed locally, but after submitting a Pull Request to merge changes into master, some of the tests (usually < 3 at a time) will randomly fail.
Examples include, but are not limited to:
In CI - CPP, Python Tests / cpp-tests:
- BasicEndToEndTest.testPatternEmptyUnsubscribe
- BasicEndToEndTest.testSinglePartitionRoutingPolicy
In CI - Unit - Brokers:
- org.apache.pulsar.client.api.SimpleProducerConsumerTest
- org.apache.pulsar.client.api.SimpleProducerConsumerTest.setup
- org.apache.pulsar.client.impl.BrokerClientIntegrationTest
- testUnsupportedBatchMessageConsumer
In CI - Unit - Flaky:
- org.apache.pulsar.client.kafka.test.KafkaProducerSimpleConsumerTest
- testPulsarKafkaProducerWithSerializer
- org.apache.pulsar.functions.worker.PulsarFunctionE2ESecurityTest
- testAuthorizationWithAnonymousUser
In CI - Unit - Proxy:
- org.apache.pulsar.proxy.server.ProxyParserTest
- org.apache.pulsar.proxy.server.ProxyParserTest.testRegexSubscription
Regarding `org.apache.pulsar.client.api.SimpleProducerConsumerTest.setup`, I found an interesting exception message:
```
org.apache.pulsar.client.api.SimpleProducerConsumerTest.setup(org.apache.pulsar.client.api.SimpleProducerConsumerTest)
165[ERROR] Run 1: SimpleProducerConsumerTest.setup:108->MockedPulsarServiceBaseTest.internalSetup:107->MockedPulsarServiceBaseTest.init:144->MockedPulsarServiceBaseTest.startBroker:195->MockedPulsarServiceBaseTest.startBroker:218 ยป WrongTypeOfReturnValue
```
This message suggests that there's a race condition in the testing framework (or our use of it).
Perhaps there are known concurrency bugs in some of the versions of the test libraries we are using.
There is also a risk that there are concurrency bugs in the actual framework that only appear in certain environments. If this is the case, then these bugs could result in instability for certain users in production environments.
**Expected behavior**
Tests should not randomly fail when run by Jenkins or the Github CI Action test runner after submitting a Pull Request. These random failures significantly slow down the rate of being able to merge PRs and raise the possibility of other potential risks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] devinbost commented on issue #6137: Large number of
unstable Github tests for Pulsar
Posted by GitBox <gi...@apache.org>.
devinbost commented on issue #6137: Large number of unstable Github tests for Pulsar
URL: https://github.com/apache/pulsar/issues/6137#issuecomment-581594078
I found an interesting study that examined common causes of Flaky tests: http://mir.cs.illinois.edu/~qluo2/fse14LuoHEM.pdf
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] jiazhai commented on issue #6137: Large number of unstable
Github tests for Pulsar
Posted by GitBox <gi...@apache.org>.
jiazhai commented on issue #6137: Large number of unstable Github tests for Pulsar
URL: https://github.com/apache/pulsar/issues/6137#issuecomment-581216552
Thanks @devinbost for the great help
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] devinbost commented on issue #6137: Unstable Github tests
for Pulsar
Posted by GitBox <gi...@apache.org>.
devinbost commented on issue #6137: Unstable Github tests for Pulsar
URL: https://github.com/apache/pulsar/issues/6137#issuecomment-578358463
@merlimat Do you have any theories about what could be happening? It looks like you've spent a decent amount of time in a lot of these tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] devinbost commented on issue #6137: Large number of
unstable Github tests for Pulsar
Posted by GitBox <gi...@apache.org>.
devinbost commented on issue #6137: Large number of unstable Github tests for Pulsar
URL: https://github.com/apache/pulsar/issues/6137#issuecomment-579962632
I noticed that there are many tests (including at least two of the tests above) that use this block of code that involves a Thread.sleep, which is often an anti-pattern in tests.
In MockedPulsarServiceBaseTest.java:
```
public static boolean retryStrategically(Predicate<Void> predicate, int retryCount, long intSleepTimeInMillis)
throws Exception {
for (int i = 0; i < retryCount; i++) {
if (predicate.test(null) || i == (retryCount - 1)) {
return true;
}
Thread.sleep(intSleepTimeInMillis + (intSleepTimeInMillis * i));
}
return false;
}
```
At least these two tests (referenced above) use `retryStrategically`:
org.apache.pulsar.client.kafka.test.KafkaProducerSimpleConsumerTest
- testPulsarKafkaProducerWithSerializer
org.apache.pulsar.functions.worker.PulsarFunctionE2ESecurityTest
- testAuthorizationWithAnonymousUser
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] devinbost commented on issue #6137: Unstable Github tests
for Pulsar
Posted by GitBox <gi...@apache.org>.
devinbost commented on issue #6137: Unstable Github tests for Pulsar
URL: https://github.com/apache/pulsar/issues/6137#issuecomment-578339786
Do we ever have cases where multiple threads are stubbing/verifying a shared mock?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] devinbost commented on issue #6137: Large number of
unstable Github tests for Pulsar
Posted by GitBox <gi...@apache.org>.
devinbost commented on issue #6137: Large number of unstable Github tests for Pulsar
URL: https://github.com/apache/pulsar/issues/6137#issuecomment-580133386
After discovering quite a few closed issues involving intermittent tests, I realized that almost all of them involved race conditions or concurrency issues involving shared state.
So, I'll start looking at these failing tests in greater depth individually.
I'm hoping that some common causes will emerge, but so far, it looks like each one of the solved intermittent tests has been a unique case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] devinbost commented on issue #6137: Large number of
unstable Github tests for Pulsar
Posted by GitBox <gi...@apache.org>.
devinbost commented on issue #6137: Large number of unstable Github tests for Pulsar
URL: https://github.com/apache/pulsar/issues/6137#issuecomment-578424902
@sijie I added a lot of detail about the different exceptions that are appearing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services