Pulsar Flaky test report 2021-09-01 for PR builds in CI

[Lari Hotari <lh...@apache.org>]

Dear Pulsar community members,

Here's a report of the flaky tests in Pulsar CI during the observation
period of 2021-08-25 - 2021-09-01 .
The full report has been also updated to a Google Sheet,
https://docs.google.com/spreadsheets/d/165FHpHjs5fHccSsmQM4beeg6brn-zfUjcrXf6xAu4yQ/edit#gid=0

Test method name Number of build failures due to this test
org.apache.pulsar.metadata.LockManagerTest.revalidateLockOnDifferentSession
94
org.apache.pulsar.metadata.LockManagerTest.revalidateLockWithinSameSession
45
org.apache.pulsar.metadata.LockManagerTest.updateValue 45
org.apache.pulsar.metadata.ZKSessionTest.testReacquireLocksAfterSessionLost
29
org.apache.pulsar.broker.stats.ManagedCursorMetricsTest.testCursorReadWriteMetrics
23
org.apache.pulsar.client.api.ClientErrorsTest.testProducerReconnect 20
org.apache.pulsar.client.api.SimpleProducerConsumerTest.testRedeliveryOfFailedMessages
19
org.apache.pulsar.client.api.DeadLetterTopicTest.testDeadLetterTopic 19
org.apache.pulsar.client.api.ClientDeduplicationFailureTest.testClientDeduplicationCorrectnessWithFailure
19
org.apache.pulsar.client.api.SimpleProducerConsumerTest.testConcurrentConsumerReceiveWhileReconnect
19
org.apache.pulsar.client.api.ClientDeduplicationFailureTest.testClientDeduplicationWithBkFailure
19
org.apache.pulsar.client.api.DeadLetterTopicTest.testDeadLetterTopicByCustomTopicName
19
org.apache.pulsar.client.api.SimpleProducerConsumerTest.testRedeliveryFailOverConsumer
19
org.apache.pulsar.client.impl.SequenceIdWithErrorTest.testCheckSequenceId 19
org.apache.pulsar.broker.stats.ManagedCursorMetricsTest.testManagedCursorMetrics
19
org.apache.pulsar.client.api.v1.V1_ProducerConsumerTest.testActiveAndInActiveConsumerEntryCacheBehavior
18
org.apache.pulsar.client.api.ConsumerBatchReceiveTest.testAsyncBatchReceivePartitionedTopic
18
org.apache.pulsar.client.api.ConsumerBatchReceiveTest.testBatchReceiveNonPartitionedTopic
18
org.apache.pulsar.client.api.ConsumerBatchReceiveTest.testBatchReceiveAndRedeliveryPartitionedTopic
18
org.apache.pulsar.client.api.ConsumerBatchReceiveTest.testBatchReceivePartitionedTopic
18
org.apache.pulsar.client.api.ConsumerBatchReceiveTest.testBatchReceiveAndRedeliveryNonPartitionedTopic
18
org.apache.pulsar.client.api.ConsumerBatchReceiveTest.testAsyncBatchReceiveNonPartitionedTopic
18
org.apache.pulsar.client.api.ConsumerBatchReceiveTest.verifyBatchSizeIsEqualToPolicyConfiguration
18
org.apache.pulsar.broker.stats.PrometheusMetricsTest.testDuplicateMetricTypeDefinitions
16
org.apache.pulsar.client.api.KeySharedSubscriptionTest.testRemoveFirstConsumer
12
observation period 2021-08-25 to 2021-09-01

Markdown formatted summary reports for each test class can be accessed at
https://github.com/lhotari/pulsar-flakes/tree/master/2021-08-25-to-2021-09-01/1630481590725
(type 't' to activate search by filename in GitHub UI, then type the test
class name to find the report)

*We need more help in addressing the flaky tests. Please join the efforts
so that we can get CI to a more stable state. *

To coordinate the work,
1) please search for an existing issues or search for all flaky issues with
"flaky" or the test class name (without package) in the search:
https://github.com/apache/pulsar/issues?q=is%3Aopen+flaky
+sort%3Aupdated-desc
2) If there isn't an issue for a particular flaky test failure that you'd
like to fix, please create an issue using the "Flaky test" template at
https://github.com/apache/pulsar/issues/new/choose
3) Please comment on the issue that you are working on it.

We have a few active contributors working on the flaky tests, thanks for
the contributions.

I'm looking forward to more contributors joining the efforts. Please join
the #testing channel on Slack if you'd like to ask questions and tips about
reproducing flaky tests locally and how to fix them.
Sharing stories about fixing flaky tests is also helpful for sharing the
knowledge about how flaky tests can be fixed. That's a valuable way to
contribute too.

*Current contributors, please keep up the good work! *
*New contributors, you are welcome to join the efforts! You will learn
about Pulsar and its internals as a side-effect. *
*If you'd love to learn Pulsar internals and Pulsar OSS development, start
by fixing flaky tests.*

BR,  -Lari