Pulsar Flaky test report 2023-10-27 to 2023-11-06 for PR builds in CI

[Lari Hotari <lh...@apache.org>]

Dear Pulsar community,

Here's a report of the flaky tests in Pulsar CI during the observation
period of 2023-10-27 to 2023-11-06.

The Pulsar CI is in fairly good shape at the moment. We have been able
to reduce flakiness, and it is not currently slowing down PR
processing significantly.

The flaky test reporting has highlighted these tests as the most flaky ones:

https://github.com/apache/pulsar/issues/21287
PersistentDispatcherFailoverConsumerTest.testAddRemoveConsumer
11 failures

https://github.com/apache/pulsar/issues/13953
PulsarDebeziumOracleSourceTest.testDebeziumOracleDbSource
11 failures

https://github.com/apache/pulsar/issues/21469
fix PR: https://github.com/apache/pulsar/pull/21479
ExtensibleLoadManagerImplTest.testCheckOwnershipAsync
6 failures

https://github.com/apache/pulsar/issues/16786
PulsarFunctionsJavaProcessTest.testJavaExclamationFunction
6 failures

https://github.com/apache/pulsar/issues/21292
BrokerServiceLookupTest.testLookupConnectionNotCloseIfGetUnloadingExOrMetadataEx
3 failures

Putting focus on fixing the most flaky tests will be helpful.

More details in this Google sheet:
https://docs.google.com/spreadsheets/d/1gtu-XrLumjBFPk9kDKcJOQfxsvIE2EiuZO7IB7ab6q0/edit

Detailed reports and flaky test reporting source:
https://github.com/lhotari/pulsar-flakes/tree/master/2023-10-27-to-2023-11-06

In addition to the flaky test reporting, there's also thread leak
reporting in the Pulsar CI build.
Thread leaks could be one source of test flakiness and that's why it
is helpful to fix thread leaks in our tests and not introduce new
thread leaks.
You can view the thread leak reports in the unit test jobs in any of
the most recently Pulsar CI builds. For example, the scheduled builds
for the
master branch can be found here:
https://github.com/apache/pulsar/actions/workflows/pulsar-ci.yaml?query=branch%3Amaster+event%3Aschedule
.
example of a thread leak report:
https://github.com/apache/pulsar/actions/runs/6784235160/job/18440670499#step:16:23
(can be viewed by clicking on the "Report detected thread leaks" in
all unit test jobs)

To coordinate the work of fixing flaky tests,

1) please search for an existing issues or search for all flaky issues with
"flaky" or the test class name (without package) in the search:
https://github.com/apache/pulsar/issues?q=is%3Aopen+flaky+sort%3Aupdated-desc

2) If there isn't an issue for a particular flaky test failure that you'd
like to fix, please create an issue using the "Flaky test" template at
https://github.com/apache/pulsar/issues/new/choose

3) Please comment on the issue that you are working on it.

Let's continue to reduce the flakiness to make contributing to Pulsar
a better experience!

-Lari