You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Lari Hotari <lh...@apache.org> on 2022/02/03 12:59:11 UTC
Pulsar Flaky test report 2022-02-03 for PR builds in CI
Dear Pulsar community members,
Here's a report of the flaky tests in Pulsar CI during the observation
period of 2022-01-27 to 2022-02-03 .
The full report is available as a Google Sheet,
https://docs.google.com/spreadsheets/d/165FHpHjs5fHccSsmQM4beeg6brn-zfUjcrXf6xAu4yQ
There are a lot more flaky test failures that what are seen in the report.
The report contains a subset of the test failures.
The flaky tests are observed from builds of merged PRs.
The GitHub Actions logs will be checked for builds where the SHA of the
head of the PR matches the SHA which got merged.
This ensures that all found exceptions are real flakes, since no changes
were made to the PR to make the tests pass later
so that the PR was merged successfully.
Here are the most flaky test methods:
Test method name Number of build failures due to this test
Detailed test results
org.apache.pulsar.client.api.BrokerServiceLookupTest.testModularLoadManagerSplitBundle
21 Test results
<https://github.com/lhotari/pulsar-flakes/blob/master/2022-01-27-to-2022-02-03/org.apache.pulsar.client.api.BrokerServiceLookupTest.testModularLoadManagerSplitBundle.md>
org.apache.pulsar.broker.service.PersistentTopicTest.setup 14 Test results
<https://github.com/lhotari/pulsar-flakes/blob/master/2022-01-27-to-2022-02-03/org.apache.pulsar.broker.service.PersistentTopicTest.setup.md>
org.apache.pulsar.metadata.LockManagerTest.revalidateLockOnDifferentSession
13 Test results
<https://github.com/lhotari/pulsar-flakes/blob/master/2022-01-27-to-2022-02-03/org.apache.pulsar.metadata.LockManagerTest.revalidateLockOnDifferentSession.md>
org.apache.pulsar.broker.admin.AdminApi2Test.testGetListInBundle 13 Test
results
<https://github.com/lhotari/pulsar-flakes/blob/master/2022-01-27-to-2022-02-03/org.apache.pulsar.broker.admin.AdminApi2Test.testGetListInBundle.md>
org.apache.pulsar.broker.service.RackAwareTest.testPlacement 12 Test results
<https://github.com/lhotari/pulsar-flakes/blob/master/2022-01-27-to-2022-02-03/org.apache.pulsar.broker.service.RackAwareTest.testPlacement.md>
org.apache.pulsar.metadata.ZKSessionTest.testDisconnection 12 Test results
<https://github.com/lhotari/pulsar-flakes/blob/master/2022-01-27-to-2022-02-03/org.apache.pulsar.metadata.ZKSessionTest.testDisconnection.md>
org.apache.pulsar.broker.service.persistent.PersistentTopicStreamingDispatcherTest.setup
12 Test results
<https://github.com/lhotari/pulsar-flakes/blob/master/2022-01-27-to-2022-02-03/org.apache.pulsar.broker.service.persistent.PersistentTopicStreamingDispatcherTest.setup.md>
org.apache.pulsar.client.api.BrokerServiceLookupTest.testPartitionTopicLookup
11 Test results
<https://github.com/lhotari/pulsar-flakes/blob/master/2022-01-27-to-2022-02-03/org.apache.pulsar.client.api.BrokerServiceLookupTest.testPartitionTopicLookup.md>
org.apache.pulsar.broker.service.PersistentTopicE2ETest.testBrokerConnectionStats
11 Test results
<https://github.com/lhotari/pulsar-flakes/blob/master/2022-01-27-to-2022-02-03/org.apache.pulsar.broker.service.PersistentTopicE2ETest.testBrokerConnectionStats.md>
org.apache.pulsar.broker.service.ReplicatorTest.testDoNotReplicateSystemTopic
9 Test results
<https://github.com/lhotari/pulsar-flakes/blob/master/2022-01-27-to-2022-02-03/org.apache.pulsar.broker.service.ReplicatorTest.testDoNotReplicateSystemTopic.md>
org.apache.pulsar.broker.service.BacklogQuotaManagerTest.testConsumerBacklogEvictionTimeQuota
9 Test results
<https://github.com/lhotari/pulsar-flakes/blob/master/2022-01-27-to-2022-02-03/org.apache.pulsar.broker.service.BacklogQuotaManagerTest.testConsumerBacklogEvictionTimeQuota.md>
org.apache.pulsar.broker.service.persistent.PersistentTopicStreamingDispatcherE2ETest.testBrokerConnectionStats
8 Test results
<https://github.com/lhotari/pulsar-flakes/blob/master/2022-01-27-to-2022-02-03/org.apache.pulsar.broker.service.persistent.PersistentTopicStreamingDispatcherE2ETest.testBrokerConnectionStats.md>
org.apache.pulsar.testclient.PerformanceProducerTest.testMsgKey 7 Test
results
<https://github.com/lhotari/pulsar-flakes/blob/master/2022-01-27-to-2022-02-03/org.apache.pulsar.testclient.PerformanceProducerTest.testMsgKey.md>
org.apache.pulsar.metadata.LockManagerTest.updateValue 6 Test results
<https://github.com/lhotari/pulsar-flakes/blob/master/2022-01-27-to-2022-02-03/org.apache.pulsar.metadata.LockManagerTest.updateValue.md>
org.apache.pulsar.metadata.ZKSessionTest.testSessionLost 5 Test results
<https://github.com/lhotari/pulsar-flakes/blob/master/2022-01-27-to-2022-02-03/org.apache.pulsar.metadata.ZKSessionTest.testSessionLost.md>
org.apache.pulsar.broker.service.BacklogQuotaManagerTest.testConsumerBacklogEvictionTimeQuotaWithEmptyLedger
5 Test results
<https://github.com/lhotari/pulsar-flakes/blob/master/2022-01-27-to-2022-02-03/org.apache.pulsar.broker.service.BacklogQuotaManagerTest.testConsumerBacklogEvictionTimeQuotaWithEmptyLedger.md>
org.apache.pulsar.broker.service.ReplicatorSubscriptionTest.testGetReplicatedSubscriptionStatus
5 Test results
<https://github.com/lhotari/pulsar-flakes/blob/master/2022-01-27-to-2022-02-03/org.apache.pulsar.broker.service.ReplicatorSubscriptionTest.testGetReplicatedSubscriptionStatus.md>
org.apache.pulsar.testclient.PerformanceTransactionTest.testConsumeTxnMessage
5 Test results
<https://github.com/lhotari/pulsar-flakes/blob/master/2022-01-27-to-2022-02-03/org.apache.pulsar.testclient.PerformanceTransactionTest.testConsumeTxnMessage.md>
org.apache.pulsar.functions.source.batch.BatchSourceExecutorTest.testPushLifeCycle
5 Test results
<https://github.com/lhotari/pulsar-flakes/blob/master/2022-01-27-to-2022-02-03/org.apache.pulsar.functions.source.batch.BatchSourceExecutorTest.testPushLifeCycle.md>
org.apache.pulsar.metadata.MetadataCacheTest.insertionDeletion 5 Test
results
<https://github.com/lhotari/pulsar-flakes/blob/master/2022-01-27-to-2022-02-03/org.apache.pulsar.metadata.MetadataCacheTest.insertionDeletion.md>
Markdown formatted summary reports for each test class can be accessed at
https://github.com/lhotari/pulsar-flakes/tree/master/2022-01-27-to-2022-02-03
The summary report links are now available in the Google sheet
The issue for the most flaky test
BrokerServiceLookupTest.testModularLoadManagerSplitBundle is
https://github.com/apache/pulsar/issues/13102
*We need more help in addressing the flaky tests. Please join the efforts
so that we can get CI to a more stable state. *
To coordinate the work,
1) please search for an existing issues or search for all flaky issues with
"flaky" or the test class name (without package) in the search:
https://github.com/apache/pulsar/issues?q=is%3Aopen+flaky
+sort%3Aupdated-desc
2) If there isn't an issue for a particular flaky test failure that you'd
like to fix, please create an issue using the "Flaky test" template at
https://github.com/apache/pulsar/issues/new/choose
3) Please comment on the issue that you are working on it.
We have a few active contributors working on the flaky tests, thanks for
the contributions.
I'm looking forward to more contributors joining the efforts. Please join
the #testing channel on Slack if you'd like to ask questions and tips about
reproducing flaky tests locally and how to fix them.
Sharing stories about fixing flaky tests is also helpful for sharing the
knowledge about how flaky tests can be fixed. That's also a valuable way to
contribute.
*Some flaky tests might be actual real production code bugs. Fixing
the flaky test might result in fixing a real production code bug.*
*Current contributors, please keep up the good work! *
*New contributors, you are welcome to join the efforts! You will learn
about Pulsar and its internals as a side effect. *
*If you'd love to learn Pulsar internals and Pulsar OSS development, start
by fixing flaky tests. :)*
BR, -Lari
Re: Pulsar Flaky test report 2022-02-03 for PR builds in CI
Posted by Lari Hotari <lh...@apache.org>.
I added the links to GitHub issues to the spreadsheet:
https://docs.google.com/spreadsheets/d/165FHpHjs5fHccSsmQM4beeg6brn-zfUjcrXf6xAu4yQ/edit#gid=456314619
Let's focus fixing the top 10 most flaky tests asap. Please comment on the
issue that you are working on it so that we don't unnecessarily do
duplicate work on fixing the issues.
-Lari
Test method name Reported issue
org.apache.pulsar.client.api.BrokerServiceLookupTest.testModularLoadManagerSplitBundle
<https://github.com/apache/pulsar/issues/13102>
https://github.com/apache/pulsar/issues/13102
org.apache.pulsar.broker.service.PersistentTopicTest.setup
<https://github.com/apache/pulsar/issues/13620>
https://github.com/apache/pulsar/issues/13620
org.apache.pulsar.metadata.LockManagerTest.revalidateLockOnDifferentSession
<https://github.com/apache/pulsar/issues/11690>
https://github.com/apache/pulsar/issues/11690
org.apache.pulsar.broker.admin.AdminApi2Test.testGetListInBundle
<https://github.com/apache/pulsar/issues/14105>
https://github.com/apache/pulsar/issues/14105
org.apache.pulsar.broker.service.RackAwareTest.testPlacement
<https://github.com/apache/pulsar/issues/14106>
https://github.com/apache/pulsar/issues/14106
org.apache.pulsar.metadata.ZKSessionTest.testDisconnection
<https://github.com/apache/pulsar/issues/13008>
https://github.com/apache/pulsar/issues/13008
org.apache.pulsar.broker.service.persistent.PersistentTopicStreamingDispatcherTest.setup
<https://github.com/apache/pulsar/issues/13808>
https://github.com/apache/pulsar/issues/13808
org.apache.pulsar.client.api.BrokerServiceLookupTest.testPartitionTopicLookup
<https://github.com/apache/pulsar/issues/14046>
https://github.com/apache/pulsar/issues/14046
org.apache.pulsar.broker.service.PersistentTopicE2ETest.testBrokerConnectionStats
<https://github.com/apache/pulsar/issues/10150>
https://github.com/apache/pulsar/issues/10150
org.apache.pulsar.broker.service.ReplicatorTest.testDoNotReplicateSystemTopic
<https://github.com/apache/pulsar/issues/12774>
https://github.com/apache/pulsar/issues/12774
org.apache.pulsar.broker.service.BacklogQuotaManagerTest.testConsumerBacklogEvictionTimeQuota
<https://github.com/apache/pulsar/issues/13952>
https://github.com/apache/pulsar/issues/13952
org.apache.pulsar.broker.service.persistent.PersistentTopicStreamingDispatcherE2ETest.testBrokerConnectionStats
<https://github.com/apache/pulsar/issues/10150>
https://github.com/apache/pulsar/issues/10150
org.apache.pulsar.testclient.PerformanceProducerTest.testMsgKey
<https://github.com/apache/pulsar/issues/14052>
https://github.com/apache/pulsar/issues/14052
org.apache.pulsar.metadata.LockManagerTest.updateValue
<https://github.com/apache/pulsar/issues/13663>
https://github.com/apache/pulsar/issues/13663
org.apache.pulsar.metadata.ZKSessionTest.testSessionLost
<https://github.com/apache/pulsar/issues/14107>
https://github.com/apache/pulsar/issues/14107
org.apache.pulsar.broker.service.BacklogQuotaManagerTest.testConsumerBacklogEvictionTimeQuotaWithEmptyLedger
<https://github.com/apache/pulsar/issues/14108>
https://github.com/apache/pulsar/issues/14108
org.apache.pulsar.broker.service.ReplicatorSubscriptionTest.testGetReplicatedSubscriptionStatus
<https://github.com/apache/pulsar/issues/13626>
https://github.com/apache/pulsar/issues/13626
org.apache.pulsar.testclient.PerformanceTransactionTest.testConsumeTxnMessage
<https://github.com/apache/pulsar/issues/14109>
https://github.com/apache/pulsar/issues/14109
org.apache.pulsar.functions.source.batch.BatchSourceExecutorTest.testPushLifeCycle
<https://github.com/apache/pulsar/issues/11735>
https://github.com/apache/pulsar/issues/11735
org.apache.pulsar.metadata.MetadataCacheTest.insertionDeletion
<https://github.com/apache/pulsar/issues/14110>
https://github.com/apache/pulsar/issues/14110
>