You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geode.apache.org by Mark Hanson <ha...@vmware.com> on 2020/06/19 05:06:01 UTC

[INFO] Latest test run of 200 DistributedTestOpenJDK8 passes

FYI, the build success rate was around 90% or so about two months ago.

Here are the DUnit tests that are currently failing in our tests, most likely in CI, and PR pipelines.

Please let me know if you have any questions.

Thanks,
Mark


***********************************************************************************

 Overall build success rate: 78.00000% (156 of 200)

***********************************************************************************



The following test methods see failures in more than one class.  There may be a failing *TestBase class



*.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:  12 failures :

  SerialWANPersistenceEnabledGatewaySenderDUnitTest:  8 failures (96.000% success rate)

  SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  4 failures (98.000% success rate)



*.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:  12 failures :

  ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  5 failures (97.500% success rate)

  ParallelWANPersistenceEnabledGatewaySenderDUnitTest:  7 failures (96.500% success rate)



*.testPingWrongServer:  4 failures :

  ClientServerMiscSelectorDUnitTest:  3 failures (98.500% success rate)

  ClientServerMiscDUnitTest:  1 failures (99.500% success rate)



***********************************************************************************





org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest:  8 failures (96.000% success rate)



         testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3539

         testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3526

         testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3505

         testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3435

         testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3414

         testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3391

         testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3363

         testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3359



org.apache.geode.management.MemberMXBeanDistributedTest:  2 failures (99.000% success rate)



         testBucketCount       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463

         testBucketCount       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3411



org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest:  1 failures (99.500% success rate)



         testRegisterInterestAndMakePrimaryWithFullRedundancy       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3381



org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest:  1 failures (99.500% success rate)



         testSimpleQueryOnLocator       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3516



org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest:  1 failures (99.500% success rate)



         testPingWrongServer       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3400



org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest:  3 failures (98.500% success rate)



         testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3495

         testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3470

         testReplicatedSerialPropagationHAWithGroupTransactionEvents       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3479



org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest:  1 failures (99.500% success rate)



         testDistributedDeadlockWithDLock       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3421



org.apache.geode.distributed.LocatorDUnitTest:  1 failures (99.500% success rate)



         testStartTwoLocatorsWithMultiKeystoreSSL       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3398



org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  5 failures (97.500% success rate)



         testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3531

         testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3522

         testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3456

         testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439

         testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405



org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest:  1 failures (99.500% success rate)



         clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3433



org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:  7 failures (96.500% success rate)



         testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3478

         testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463

         testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439

         testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436

         testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405

         testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3370

         testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3351



org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest:  4 failures (98.000% success rate)



         testMultipleColocatedChildPRsMissingWithSequencedStart       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3493

         testMissingColocatedChildPRDueToDelayedStart       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503

         testHierarchyOfColocatedChildPRsMissingGrandchild       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3537

         testHierarchyOfColocatedChildPRsMissingGrandchild       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3424



org.apache.geode.distributed.DistributedMemberDUnitTest:  2 failures (99.000% success rate)



         testGroupsInAllVMs       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3423

         testGroupsInAllVMs       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3374



org.apache.geode.management.JMXMBeanReconnectDUnitTest:  5 failures (97.500% success rate)



         serverMXBeansOnServerAreUnaffectedByLocatorCrash       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503

         serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436

         serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3406

         locatorHasMemberTypeMXBeansForBothLocators       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3352

         serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3457



org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  4 failures (98.000% success rate)



         testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3494

         testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3465

         testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3464

         testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3379



org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest:  3 failures (98.500% success rate)



         testPingWrongServer       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3402

         testPingWrongServer       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3380

         testPingWrongServer       https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3357


Re: Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes

Posted by Alexander Murmann <am...@vmware.com>.
Oh, I am so sorry Mario! I mistakenly was under the impression the change was related to more failures. I should have looked more carefully! Thank you for the fix! ??
________________________________
From: Mario Ivanac <ma...@est.tech>
Sent: Friday, October 23, 2020 13:39
To: Alexander Murmann <am...@vmware.com>; dev@geode.apache.org <de...@geode.apache.org>
Subject: Odg: Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes

Hi,

I am not aware that tests (GEODE-8172<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FGEODE-8172&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620044260%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3aT5yJlgvgg47aJSJ7SlfekvQ2Sl%2B%2FfWaOQi%2BI6R5YU%3D&reserved=0>, GEODE-8216<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FGEODE-8216&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620054258%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=LId8KQoqMGeRG1k3issagxNxepi5arCaofEjMVMGFzU%3D&reserved=0>) are still failing, after last modification of test (October 7).

BR,
Mario
________________________________
Šalje: Alexander Murmann <am...@vmware.com>
Poslano: 23. listopada 2020. 22:25
Prima: dev@geode.apache.org <de...@geode.apache.org>
Kopija: Mario Ivanac <ma...@est.tech>
Predmet: Re: Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes

The failing WAN-related tests are still making up the majority of flaky tests<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-develop-mass-test-run%2Fjobs%2Fcreate-mass-test-run-report%2Fbuilds%2F40&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620064252%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=2uP8sqNXyghB3vQ3LKYXPEawlioaDxo03Vsh0GgISj4%3D&reserved=0> we are observing. It's a real issue, since the noise from flaky tests can easily hide very real bugs. That's why earlier this year several people spent a few months improving the reliability of our tests.

Mario, in July you said you were investigating this. Is that still ongoing? Do you need any help?
________________________________
From: Alexander Murmann <am...@apache.org>
Sent: Tuesday, July 14, 2020 09:00
To: dev@geode.apache.org <de...@geode.apache.org>
Cc: Alexander Murmann <am...@apache.org>
Subject: Re: Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes

Thanks for looking into this, Mario!

You are probably right, that the underlying issue might have been
pre-existing and that the test is surfacing it. I am glad though that you
are investigating, because a close to 30% fail rate is a problem. Something
like this happens every once in a while and then someone has to do some
work to resolve historical problems that they hadn't planned to addres.

Thanks!

On Tue, Jul 14, 2020 at 1:06 AM Mario Ivanac <ma...@est.tech> wrote:

> Hi,
>
> after adding additional checks in failing test, now I can see that test
> are failing due to fault that some batch are distributed at stopping of GW
> sender.
> Cause of that, I suspect that this problem existed prior to this PR, but
> this PR is first to introduce test to check this.
>
> I will continue to investigate this fault, but I can not locally reproduce
> this fault, so this is slowing troubleshooting.
>
> BR,
> Mario
> ________________________________
> Šalje: Alexander Murmann <am...@apache.org>
> Poslano: 14. srpnja 2020. 1:11
> Prima: Alexander Murmann <am...@apache.org>
> Kopija: dev@geode.apache.org <de...@geode.apache.org>; Mario Ivanac
> <ma...@est.tech>
> Predmet: Re: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
>
> We continue to see these WAN tests adding a fail rate of just below 30% in
> our mass test runs
> <
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-develop-mass-test-run%2Fjobs%2Fcreate-mass-test-run-report%2Fbuilds%2F10&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&amp;sdata=STlN5I9iM8gRqsKndH6GKwuY5cVOskAgUjNgkoJvzCw%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-develop-mass-test-run%2Fjobs%2Fcreate-mass-test-run-report%2Fbuilds%2F10&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620064252%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=HIAg2oiethvYT4cu8VdDWKRj6R1vHA0UF2%2FDjtnM4so%3D&reserved=0>
> >
> .
>
> That's a very significant fail rate that impacts our ability to get our
> code committed with confidence.
>
> Can we resolve this issue? Otherwise, I think we need to consider reverting
> GEODE-7458.
>
> On Fri, Jun 19, 2020 at 3:28 PM Alexander Murmann <am...@apache.org>
> wrote:
>
> > Looking more into this, it looks like this was introduced by the changes
> > for GEODE-7458 - "Adding additional option in gfsh command "start gateway
> > sender" to control clearing of existing queues".
> >
> > That happened about a month ago, but it's inherent to those flaky tests
> > that we discover them only after a while. Nonetheless, they become paper
> > cuts that ultimately slow us down substantially if they persist.
> >
> > @Mario Ivanac If I am correct and GEODE-7458 introduced this you were the
> > one making that change. Might you be able to take a look at making that
> > test more reliable or reverting the change?
> >
> > Thank you!
> >
> > On Fri, Jun 19, 2020 at 7:57 AM Alexander Murmann <am...@apache.org>
> > wrote:
> >
> >> Thank you so much for sharing this, Mark!
> >>
> >> It looks like there is a big cluster around WAN Gateway. Is anyone
> >> already looking into the WAN issues?
> >>
> >> On Thu, Jun 18, 2020 at 10:06 PM Mark Hanson <ha...@vmware.com>
> wrote:
> >>
> >>> FYI, the build success rate was around 90% or so about two months ago.
> >>>
> >>> Here are the DUnit tests that are currently failing in our tests, most
> >>> likely in CI, and PR pipelines.
> >>>
> >>> Please let me know if you have any questions.
> >>>
> >>> Thanks,
> >>> Mark
> >>>
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>>  Overall build success rate: 78.00000% (156 of 200)
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>>
> >>>
> >>> The following test methods see failures in more than one class.  There
> >>> may be a failing *TestBase class
> >>>
> >>>
> >>>
> >>>
> *.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
> >>> 12 failures :
> >>>
> >>>   SerialWANPersistenceEnabledGatewaySenderDUnitTest:  8 failures
> >>> (96.000% success rate)
> >>>
> >>>   SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  4 failures
> >>> (98.000% success rate)
> >>>
> >>>
> >>>
> >>>
> *.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
> >>> 12 failures :
> >>>
> >>>   ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  5
> >>> failures (97.500% success rate)
> >>>
> >>>   ParallelWANPersistenceEnabledGatewaySenderDUnitTest:  7 failures
> >>> (96.500% success rate)
> >>>
> >>>
> >>>
> >>> *.testPingWrongServer:  4 failures :
> >>>
> >>>   ClientServerMiscSelectorDUnitTest:  3 failures (98.500% success rate)
> >>>
> >>>   ClientServerMiscDUnitTest:  1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest:
> >>> 8 failures (96.000% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3539&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&amp;sdata=zcUAsAVVsUprEFeDpWryPa%2F4QTf0pQxcZ8yNpzFwvtQ%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3539&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620074248%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Yx1x5SVgNs1jIB%2BlQFiJRZxQk33SH8FXWLEc1HMhsmw%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3526&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&amp;sdata=309OdZMeCRLDty3sTtCumrWt2T0b76JNA94EvwuB%2BSI%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3526&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620084241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ocCSd88Gmzy4P18HHEbVFGnedc9S7u7mu6DyQBD%2FYfU%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3505&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&amp;sdata=SWeCxshemmITfDqulXqu4gQVRPzMOXrgvf3LGcXSTUs%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3505&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620084241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=QIgLFPSvS5%2BuKQxMJrOTgxK8zUAwcI0Re%2BEHlyiK7wg%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3435&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&amp;sdata=7gzxFbHVzKgpdZ1ZU8vhDYzXzU4cxyRPxgly6dIt31k%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3435&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620094240%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=sUB5vWuucARDa3UW3%2FKclTTcKBR682TMGGxYDQL%2BbQg%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3414&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&amp;sdata=Xa8bjxPmysZluEn6HbvQzqbccJl%2B%2FsiGIhWYGeBP%2Ba4%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3414&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620104236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aW6rxW%2FcPV2qIqLr%2Bbu0zN361aztnhJUvnr4wWTmGrE%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3391&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&amp;sdata=O%2B0CoquKmvf5eT%2Ff%2FMLoXVuojjwk4IeeyI5dIdJugYg%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3391&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620114231%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=xZ9Mv5RTioZhYi2%2FAgWmEFW%2BVOXrsm5rxcMzRz8k%2FUs%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3363&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=AILkL%2Fhz3I%2FasCXa8fkTSB9mA26gmDxJ4fe%2FeIiS27g%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3363&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620114231%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Tt0JTrXhgXvmv3QCd698KjaGVB3xjhMEI%2FCvJ3Ky1pI%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3359&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=LF0Jr4d9jundS8cNJtWK%2FnRGVLBp4gsX8DUduuFbp0E%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3359&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620124231%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ziP%2FRsgaOGOPGnKawqF14LMevt0rtjAJjiRMFvbCBuQ%3D&reserved=0>
> >>>
> >>>
> >>>
> >>> org.apache.geode.management.MemberMXBeanDistributedTest:  2 failures
> >>> (99.000% success rate)
> >>>
> >>>
> >>>
> >>>          testBucketCount
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=oYLB8lcafCMgeSVcwwuaIOdKzEzgg4ypKfS4HUrMHrU%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620134221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=pZ%2BKd95p6xMmxer%2F%2FnYheiziYf9bpuz1SOHTHXc7fyQ%3D&reserved=0>
> >>>
> >>>          testBucketCount
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3411&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=O0jF6ARYoS0cIxzTNq%2BJQtHB30W45nrBj%2B5lOqu0esY%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3411&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620144215%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=4CczMenyjC2xUdA38kTtcLWeDAWMJgi%2F1HEGgIHMmd4%3D&reserved=0>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>          testRegisterInterestAndMakePrimaryWithFullRedundancy
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3381&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=la5BbwTkvNS1gFlNf3yr%2BF21L0t2z6NY6Clgwv4WGr4%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3381&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620144215%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=cZK04DW4YBb%2BcMzUmzSA%2Br4XIkOTTBwiw54cXqMgzo4%3D&reserved=0>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>          testSimpleQueryOnLocator
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3516&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=Li1jYIfu1biAC47UGz1zqJbAW6neNxgvcw1JaFeR2pk%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3516&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620154218%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=zGNco0K3djzu2%2BOY8xEjCJf5knrYNUb3TqCXj4WQpJ4%3D&reserved=0>
> >>>
> >>>
> >>>
> >>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>          testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3400&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=u7%2FQXmha26M6yX4rjRspIRDOEwQZB0NjaCxwBBkfFws%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3400&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620164209%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=2qpYFPu0E2%2FiaEQsbBYAljD%2FqT5f4wGcuwW%2BJxmMczw%3D&reserved=0>
> >>>
> >>>
> >>>
> >>> org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest:  3
> >>> failures (98.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3495&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=jA3538gndW%2BM42LYlJBMzJX9K4psvoQiYxs0761Viog%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3495&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620164209%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=oAi1vqwhmJY0hjbLTCS24VeiXmL08Hk8BvxKyYur06A%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3470&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=%2Fh6%2FPjO2f1nnZ7RzMoUnLmRM0Mq2XHlGlSblj7jlcPc%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3470&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620174205%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=x3fHa2kUK0ZkjhPAmlw7hxCJ1NwOrAYthx91%2FVTNDCw%3D&reserved=0>
> >>>
> >>>          testReplicatedSerialPropagationHAWithGroupTransactionEvents
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3479&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=ikv%2FMsJVZbPhGRndMcviXfSUsYjMqob2NAZuW4GbfMw%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3479&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620184199%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=MifO3ns8FpEA8wxaepew2oLQw4UyI42n6F7Nsa0IRKw%3D&reserved=0>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>          testDistributedDeadlockWithDLock
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3421&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=ZMgFxv%2FpfiVF4xSl5C0VtJSfCVoeA5phFRJF8IS79Lk%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3421&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620194194%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=p%2FRgPB2wj76n45OE6hb85UpWykaiwhIeOOxh2qMRphU%3D&reserved=0>
> >>>
> >>>
> >>>
> >>> org.apache.geode.distributed.LocatorDUnitTest:  1 failures (99.500%
> >>> success rate)
> >>>
> >>>
> >>>
> >>>          testStartTwoLocatorsWithMultiKeystoreSSL
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3398&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=Ro4ej1P%2BTLAN%2FM%2B8v9DVFbK%2B8cSDtCKdjeoYEIfbWgw%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3398&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620194194%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=tLccD38betJg%2F80Ep02rw8sB9DVnIg6w4OryPn6fg5U%3D&reserved=0>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
> >>> 5 failures (97.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3531&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=TAWaGwXo4t3%2FCQp3MewJKEn7p1rdQgp%2FVDWHqQxva4c%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3531&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620204191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=AuugEVllnP6wyKxt3dNyHPgzbUkRMqu%2BBmTVqcn0yjo%3D&reserved=0>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3522&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=lApVu2gdSk38MjHjZlWZhPWyQ5fOez2Sma5WmYcNE38%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3522&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620214187%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=cJBMFgDI75TzgzCjiKEFO88BxJ7azzzBWy28Bc7wUFE%3D&reserved=0>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3456&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=6gC%2BqQusVFP5JqeHkxtSvk8%2FWCRIEg90ZeiWomJ1NrE%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3456&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620224183%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=jrRKM3HtHxXCALs4NuFFvYYqdkdZv0BV3wJdOtEq1as%3D&reserved=0>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=DJ9zA4EJyrjd01eidiCznYn%2FOAVtVfzYzxsfapCGUD4%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620224183%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=QJT8cbe7%2BNwSmeUwbjPheyyNXlQ9tlLaQoiBiUOJkMg%3D&reserved=0>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=%2FvRCNF0K8V9LfYTsdDxRniadtdXVuV7cZZZsmQTydw0%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620234177%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=DGkaO1JMH1Wuy9oAzj%2FK6BgIkanuqUiBdca632wbBkY%3D&reserved=0>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>  clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3433&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=rrg9pcWvSZgJ106rjbsS8wpCBhX20er1hdNhW0qGrZ4%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3433&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620244174%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9gJbgt5nxPNmrdtMfgRY6hgMyzWQoofBRq3JuYCHqNg%3D&reserved=0>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
> >>> 7 failures (96.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3478&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=7R6DvYToK1cwdTfBK93HA9y83qw9OKd4Kj%2BbEe%2F8Yd4%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3478&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620244174%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=bUdp9ffXucXzpzVL8GNg0TR%2FfI37mnjfjY6NfyLf7hY%3D&reserved=0>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=1RXrV5Zw0kQSLDrPLu45DlWy4ZR%2B2hs7rfyk8BAFeG4%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620254173%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=S131cef%2BrDdiBOiCCquxCFNqY0vSWVGglca3sF3U7eY%3D&reserved=0>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=DJ9zA4EJyrjd01eidiCznYn%2FOAVtVfzYzxsfapCGUD4%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620264166%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Hs1z2YcYJFGDq3W4jGIzpJ%2Bj4vkdPl1vhSZTbzoEGFo%3D&reserved=0>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=vF7zJq%2BZnOIa1W6oVrfKRJlV6nFIBitzjCVPPCgc6CQ%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620274159%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=U9t4sCOmgk%2BFIMPBQQO0bLWnFlSw8i9tSgwPV1wvyNQ%3D&reserved=0>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=%2FvRCNF0K8V9LfYTsdDxRniadtdXVuV7cZZZsmQTydw0%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620274159%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5fSvQHKecSSQUEzmUX28BZJ3eH6rhX9YgX23Nk%2BcH7I%3D&reserved=0>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3370&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=WDNL8hapq0GmB%2BrxSxFYJ0c4nArV8k15m79OkwrtASY%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3370&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620284153%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Qa7QoJYcb9%2FQC0wa%2BcXULfMIxunWtP5ekGkyHVuK5ds%3D&reserved=0>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3351&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=8%2B%2FBMm9eeXPcjAL4Hj94aduYQUyDrJVgPqB5%2FSqZ4kY%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3351&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620294148%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NPUcHLyaASzyvuaQNQ0P34dieCmYnxDc4MFo15CakPE%3D&reserved=0>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest:
> >>> 4 failures (98.000% success rate)
> >>>
> >>>
> >>>
> >>>          testMultipleColocatedChildPRsMissingWithSequencedStart
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3493&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=l7RlB5fYmzwGJvYuy5QFIPWJd5mi1o5BAlCYff0XeDQ%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3493&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620294148%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3qS2Cc7w6X0zBAORj1Mp%2FCEUTCMgH%2FoPHJ3L%2Fxrn5s0%3D&reserved=0>
> >>>
> >>>          testMissingColocatedChildPRDueToDelayedStart
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=SgLx2XahBD8x2Cez3aK%2BhShvVc%2BP3SFmMfeqB3pdHLM%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620304147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=AvvWFA0Fs98ke7q%2Bf3dkO7%2FjtmS%2BXtGwtad%2Bndo99dY%3D&reserved=0>
> >>>
> >>>          testHierarchyOfColocatedChildPRsMissingGrandchild
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3537&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=3Y5qtB5P6Gz8ey%2F6eeEOOGdLptT6LVIBVWLpF4lrRxs%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3537&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620314143%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=g5gEMOt0qYxWqJCyPAvcXymWUlubjR%2FpWkn5dsw2Whk%3D&reserved=0>
> >>>
> >>>          testHierarchyOfColocatedChildPRsMissingGrandchild
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3424&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=n6qxZzVKFYqiTY7kUw%2BoQfZKE2aUvbBXs6cXAcqBZaw%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3424&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620314143%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=E6YK%2BiyiSjG3%2Fre3vOMnquGSa9B7InrBaFeQ8QRX8%2Fs%3D&reserved=0>
> >>>
> >>>
> >>>
> >>> org.apache.geode.distributed.DistributedMemberDUnitTest:  2 failures
> >>> (99.000% success rate)
> >>>
> >>>
> >>>
> >>>          testGroupsInAllVMs
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3423&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=prw%2BEBBIgOH8BWwD5SBpvVd%2BvdQSh%2FOJf8Xk6zPZ7yk%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3423&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620324140%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=TeY3rxpLjMTgATjZa9Trriams5ivBhDy4wAatK581t0%3D&reserved=0>
> >>>
> >>>          testGroupsInAllVMs
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3374&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=g2PQp%2FiMZEVjc4zSOxNhks48UTPdEAVGyhdJQueipuQ%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3374&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620334136%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=7ak%2BCVW2%2B6h%2BG2I7qEEaZX5d%2BtIYNhvHzWUXsMuH2t0%3D&reserved=0>
> >>>
> >>>
> >>>
> >>> org.apache.geode.management.JMXMBeanReconnectDUnitTest:  5 failures
> >>> (97.500% success rate)
> >>>
> >>>
> >>>
> >>>          serverMXBeansOnServerAreUnaffectedByLocatorCrash
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=zVXQeZRlv6qJGvlpOFvJIt5WDzvSAq2ohyQ1DRZ1xWE%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620344130%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=VXpU8FU%2FYMgh629IrL3kTUVn%2FrF0CyWFw9fJTGsF0eA%3D&reserved=0>
> >>>
> >>>
> >>>  serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=t5BIGh1%2BcB6SKdUviko%2FghBDtJIrmEUVMGQVXdEhHbA%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620344130%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=EfTi%2FI4Ntm79kjf%2F8a1F4kS5uP9FPSs%2BL1T4%2FdqODU0%3D&reserved=0>
> >>>
> >>>
> >>>  serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3406&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=GyvOWUusGUi48vS6wWA6GrJN3J5aAMbqeXxy2VevLXU%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3406&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620354126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=y%2BwpZcXwC32vupu%2FIq0bZp8eNis2hiKHhWIzfKsARVU%3D&reserved=0>
> >>>
> >>>          locatorHasMemberTypeMXBeansForBothLocators
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3352&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=N261EBctQXEW0FR0XTTHd4w6nmrEkj%2F%2FLd8e14m0cnw%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3352&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620364119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=58GwB5FQDtBQLO2Omhm9EitUT%2F883l3QAn0eicEx02I%3D&reserved=0>
> >>>
> >>>          serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3457&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=QNtL7hTrfxZHkLfKGx7P01m4quS6%2FhkkjyYdhA7J8bA%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3457&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620374117%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=sIwqTvAdPXy4ci9b8HxCuj0UrVH91ELYg4ycIzyILbw%3D&reserved=0>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
> >>> 4 failures (98.000% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3494&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=TPAyTUGYWUv%2FifU58XondaeGNJBXmB8sskEp0J6R8l4%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3494&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620374117%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dVdHUnV25Jup8CKlR0u6FhJRtThsECIrg4NGCjN9Dys%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3465&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=gVTwnIWWDDjOhkfyM4SzYegMuQjyaXDg7sFkzkhUyJA%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3465&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620384113%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=8fOk97yNQRmtZUnqCl5UMKE4PO%2Fg1wwnPgTOgzj6u3s%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3464&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=iErMnLzSVjZpkZO%2FJ%2BETC0QbO0neLHNlA41OkXWWBuk%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3464&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620394107%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=wZzE52SfCYo1wLJ7DrGpmsOxLVM4lxxo%2BHikn9m11vc%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3379&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=GptTVqXhyQlHTByVbEoQJtjF%2FWc4xlC2KLHHZ3qhbrw%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3379&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620394107%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=MRJoOCY1QnTyvu214tNqy6cX9r4ZpsFKmxs6Jw6YJZ8%3D&reserved=0>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest:
> >>> 3 failures (98.500% success rate)
> >>>
> >>>
> >>>
> >>>          testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3402&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3402&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4>d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=S9EbS96H9rYgoCfTtXXTVbcMZAIBZ%2BlIbQgElXKzX98%3D&amp;reserved=0
> >>>
> >>>          testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3380&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297271685&amp;sdata=4OQQFRPSwtx3QcENDyVCUsoMJd0K%2Bwl5o759cyZfsYw%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3380&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620404104%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FDqkFd5z5KBiV80FCslKd6NUCs%2BDl7EEHaKGmAEav%2BU%3D&reserved=0>
> >>>
> >>>          testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3357&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297271685&amp;sdata=XjIc4NZPwT4HlRnEL9zfWcB5AlxLiO%2B9C9jf77TJ0Dg%3D&amp;reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3357&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620414103%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=w2hTfsWj2n5HfdOxkQA%2FQltIpZfwc1JoOMHdDdq2UCw%3D&reserved=0>
> >>>
> >>>
>

Odg: Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes

Posted by Mario Ivanac <ma...@est.tech>.
Hi,

I am not aware that tests (GEODE-8172<https://issues.apache.org/jira/browse/GEODE-8172>, GEODE-8216<https://issues.apache.org/jira/browse/GEODE-8216>) are still failing, after last modification of test (October 7).

BR,
Mario
________________________________
Šalje: Alexander Murmann <am...@vmware.com>
Poslano: 23. listopada 2020. 22:25
Prima: dev@geode.apache.org <de...@geode.apache.org>
Kopija: Mario Ivanac <ma...@est.tech>
Predmet: Re: Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes

The failing WAN-related tests are still making up the majority of flaky tests<https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-mass-test-run/jobs/create-mass-test-run-report/builds/40> we are observing. It's a real issue, since the noise from flaky tests can easily hide very real bugs. That's why earlier this year several people spent a few months improving the reliability of our tests.

Mario, in July you said you were investigating this. Is that still ongoing? Do you need any help?
________________________________
From: Alexander Murmann <am...@apache.org>
Sent: Tuesday, July 14, 2020 09:00
To: dev@geode.apache.org <de...@geode.apache.org>
Cc: Alexander Murmann <am...@apache.org>
Subject: Re: Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes

Thanks for looking into this, Mario!

You are probably right, that the underlying issue might have been
pre-existing and that the test is surfacing it. I am glad though that you
are investigating, because a close to 30% fail rate is a problem. Something
like this happens every once in a while and then someone has to do some
work to resolve historical problems that they hadn't planned to addres.

Thanks!

On Tue, Jul 14, 2020 at 1:06 AM Mario Ivanac <ma...@est.tech> wrote:

> Hi,
>
> after adding additional checks in failing test, now I can see that test
> are failing due to fault that some batch are distributed at stopping of GW
> sender.
> Cause of that, I suspect that this problem existed prior to this PR, but
> this PR is first to introduce test to check this.
>
> I will continue to investigate this fault, but I can not locally reproduce
> this fault, so this is slowing troubleshooting.
>
> BR,
> Mario
> ________________________________
> Šalje: Alexander Murmann <am...@apache.org>
> Poslano: 14. srpnja 2020. 1:11
> Prima: Alexander Murmann <am...@apache.org>
> Kopija: dev@geode.apache.org <de...@geode.apache.org>; Mario Ivanac
> <ma...@est.tech>
> Predmet: Re: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
>
> We continue to see these WAN tests adding a fail rate of just below 30% in
> our mass test runs
> <
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-develop-mass-test-run%2Fjobs%2Fcreate-mass-test-run-report%2Fbuilds%2F10&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&amp;sdata=STlN5I9iM8gRqsKndH6GKwuY5cVOskAgUjNgkoJvzCw%3D&amp;reserved=0
> >
> .
>
> That's a very significant fail rate that impacts our ability to get our
> code committed with confidence.
>
> Can we resolve this issue? Otherwise, I think we need to consider reverting
> GEODE-7458.
>
> On Fri, Jun 19, 2020 at 3:28 PM Alexander Murmann <am...@apache.org>
> wrote:
>
> > Looking more into this, it looks like this was introduced by the changes
> > for GEODE-7458 - "Adding additional option in gfsh command "start gateway
> > sender" to control clearing of existing queues".
> >
> > That happened about a month ago, but it's inherent to those flaky tests
> > that we discover them only after a while. Nonetheless, they become paper
> > cuts that ultimately slow us down substantially if they persist.
> >
> > @Mario Ivanac If I am correct and GEODE-7458 introduced this you were the
> > one making that change. Might you be able to take a look at making that
> > test more reliable or reverting the change?
> >
> > Thank you!
> >
> > On Fri, Jun 19, 2020 at 7:57 AM Alexander Murmann <am...@apache.org>
> > wrote:
> >
> >> Thank you so much for sharing this, Mark!
> >>
> >> It looks like there is a big cluster around WAN Gateway. Is anyone
> >> already looking into the WAN issues?
> >>
> >> On Thu, Jun 18, 2020 at 10:06 PM Mark Hanson <ha...@vmware.com>
> wrote:
> >>
> >>> FYI, the build success rate was around 90% or so about two months ago.
> >>>
> >>> Here are the DUnit tests that are currently failing in our tests, most
> >>> likely in CI, and PR pipelines.
> >>>
> >>> Please let me know if you have any questions.
> >>>
> >>> Thanks,
> >>> Mark
> >>>
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>>  Overall build success rate: 78.00000% (156 of 200)
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>>
> >>>
> >>> The following test methods see failures in more than one class.  There
> >>> may be a failing *TestBase class
> >>>
> >>>
> >>>
> >>>
> *.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
> >>> 12 failures :
> >>>
> >>>   SerialWANPersistenceEnabledGatewaySenderDUnitTest:  8 failures
> >>> (96.000% success rate)
> >>>
> >>>   SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  4 failures
> >>> (98.000% success rate)
> >>>
> >>>
> >>>
> >>>
> *.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
> >>> 12 failures :
> >>>
> >>>   ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  5
> >>> failures (97.500% success rate)
> >>>
> >>>   ParallelWANPersistenceEnabledGatewaySenderDUnitTest:  7 failures
> >>> (96.500% success rate)
> >>>
> >>>
> >>>
> >>> *.testPingWrongServer:  4 failures :
> >>>
> >>>   ClientServerMiscSelectorDUnitTest:  3 failures (98.500% success rate)
> >>>
> >>>   ClientServerMiscDUnitTest:  1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest:
> >>> 8 failures (96.000% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3539&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&amp;sdata=zcUAsAVVsUprEFeDpWryPa%2F4QTf0pQxcZ8yNpzFwvtQ%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3526&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&amp;sdata=309OdZMeCRLDty3sTtCumrWt2T0b76JNA94EvwuB%2BSI%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3505&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&amp;sdata=SWeCxshemmITfDqulXqu4gQVRPzMOXrgvf3LGcXSTUs%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3435&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&amp;sdata=7gzxFbHVzKgpdZ1ZU8vhDYzXzU4cxyRPxgly6dIt31k%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3414&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&amp;sdata=Xa8bjxPmysZluEn6HbvQzqbccJl%2B%2FsiGIhWYGeBP%2Ba4%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3391&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&amp;sdata=O%2B0CoquKmvf5eT%2Ff%2FMLoXVuojjwk4IeeyI5dIdJugYg%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3363&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=AILkL%2Fhz3I%2FasCXa8fkTSB9mA26gmDxJ4fe%2FeIiS27g%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3359&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=LF0Jr4d9jundS8cNJtWK%2FnRGVLBp4gsX8DUduuFbp0E%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.management.MemberMXBeanDistributedTest:  2 failures
> >>> (99.000% success rate)
> >>>
> >>>
> >>>
> >>>          testBucketCount
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=oYLB8lcafCMgeSVcwwuaIOdKzEzgg4ypKfS4HUrMHrU%3D&amp;reserved=0
> >>>
> >>>          testBucketCount
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3411&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=O0jF6ARYoS0cIxzTNq%2BJQtHB30W45nrBj%2B5lOqu0esY%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>          testRegisterInterestAndMakePrimaryWithFullRedundancy
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3381&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=la5BbwTkvNS1gFlNf3yr%2BF21L0t2z6NY6Clgwv4WGr4%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>          testSimpleQueryOnLocator
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3516&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=Li1jYIfu1biAC47UGz1zqJbAW6neNxgvcw1JaFeR2pk%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>          testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3400&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=u7%2FQXmha26M6yX4rjRspIRDOEwQZB0NjaCxwBBkfFws%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest:  3
> >>> failures (98.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3495&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=jA3538gndW%2BM42LYlJBMzJX9K4psvoQiYxs0761Viog%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3470&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=%2Fh6%2FPjO2f1nnZ7RzMoUnLmRM0Mq2XHlGlSblj7jlcPc%3D&amp;reserved=0
> >>>
> >>>          testReplicatedSerialPropagationHAWithGroupTransactionEvents
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3479&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=ikv%2FMsJVZbPhGRndMcviXfSUsYjMqob2NAZuW4GbfMw%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>          testDistributedDeadlockWithDLock
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3421&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=ZMgFxv%2FpfiVF4xSl5C0VtJSfCVoeA5phFRJF8IS79Lk%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.distributed.LocatorDUnitTest:  1 failures (99.500%
> >>> success rate)
> >>>
> >>>
> >>>
> >>>          testStartTwoLocatorsWithMultiKeystoreSSL
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3398&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=Ro4ej1P%2BTLAN%2FM%2B8v9DVFbK%2B8cSDtCKdjeoYEIfbWgw%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
> >>> 5 failures (97.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3531&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=TAWaGwXo4t3%2FCQp3MewJKEn7p1rdQgp%2FVDWHqQxva4c%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3522&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=lApVu2gdSk38MjHjZlWZhPWyQ5fOez2Sma5WmYcNE38%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3456&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=6gC%2BqQusVFP5JqeHkxtSvk8%2FWCRIEg90ZeiWomJ1NrE%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=DJ9zA4EJyrjd01eidiCznYn%2FOAVtVfzYzxsfapCGUD4%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=%2FvRCNF0K8V9LfYTsdDxRniadtdXVuV7cZZZsmQTydw0%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>  clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3433&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=rrg9pcWvSZgJ106rjbsS8wpCBhX20er1hdNhW0qGrZ4%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
> >>> 7 failures (96.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3478&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=7R6DvYToK1cwdTfBK93HA9y83qw9OKd4Kj%2BbEe%2F8Yd4%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=1RXrV5Zw0kQSLDrPLu45DlWy4ZR%2B2hs7rfyk8BAFeG4%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=DJ9zA4EJyrjd01eidiCznYn%2FOAVtVfzYzxsfapCGUD4%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=vF7zJq%2BZnOIa1W6oVrfKRJlV6nFIBitzjCVPPCgc6CQ%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=%2FvRCNF0K8V9LfYTsdDxRniadtdXVuV7cZZZsmQTydw0%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3370&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=WDNL8hapq0GmB%2BrxSxFYJ0c4nArV8k15m79OkwrtASY%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3351&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=8%2B%2FBMm9eeXPcjAL4Hj94aduYQUyDrJVgPqB5%2FSqZ4kY%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest:
> >>> 4 failures (98.000% success rate)
> >>>
> >>>
> >>>
> >>>          testMultipleColocatedChildPRsMissingWithSequencedStart
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3493&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=l7RlB5fYmzwGJvYuy5QFIPWJd5mi1o5BAlCYff0XeDQ%3D&amp;reserved=0
> >>>
> >>>          testMissingColocatedChildPRDueToDelayedStart
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=SgLx2XahBD8x2Cez3aK%2BhShvVc%2BP3SFmMfeqB3pdHLM%3D&amp;reserved=0
> >>>
> >>>          testHierarchyOfColocatedChildPRsMissingGrandchild
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3537&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=3Y5qtB5P6Gz8ey%2F6eeEOOGdLptT6LVIBVWLpF4lrRxs%3D&amp;reserved=0
> >>>
> >>>          testHierarchyOfColocatedChildPRsMissingGrandchild
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3424&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=n6qxZzVKFYqiTY7kUw%2BoQfZKE2aUvbBXs6cXAcqBZaw%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.distributed.DistributedMemberDUnitTest:  2 failures
> >>> (99.000% success rate)
> >>>
> >>>
> >>>
> >>>          testGroupsInAllVMs
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3423&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=prw%2BEBBIgOH8BWwD5SBpvVd%2BvdQSh%2FOJf8Xk6zPZ7yk%3D&amp;reserved=0
> >>>
> >>>          testGroupsInAllVMs
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3374&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=g2PQp%2FiMZEVjc4zSOxNhks48UTPdEAVGyhdJQueipuQ%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.management.JMXMBeanReconnectDUnitTest:  5 failures
> >>> (97.500% success rate)
> >>>
> >>>
> >>>
> >>>          serverMXBeansOnServerAreUnaffectedByLocatorCrash
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=zVXQeZRlv6qJGvlpOFvJIt5WDzvSAq2ohyQ1DRZ1xWE%3D&amp;reserved=0
> >>>
> >>>
> >>>  serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=t5BIGh1%2BcB6SKdUviko%2FghBDtJIrmEUVMGQVXdEhHbA%3D&amp;reserved=0
> >>>
> >>>
> >>>  serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3406&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=GyvOWUusGUi48vS6wWA6GrJN3J5aAMbqeXxy2VevLXU%3D&amp;reserved=0
> >>>
> >>>          locatorHasMemberTypeMXBeansForBothLocators
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3352&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=N261EBctQXEW0FR0XTTHd4w6nmrEkj%2F%2FLd8e14m0cnw%3D&amp;reserved=0
> >>>
> >>>          serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3457&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=QNtL7hTrfxZHkLfKGx7P01m4quS6%2FhkkjyYdhA7J8bA%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
> >>> 4 failures (98.000% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3494&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=TPAyTUGYWUv%2FifU58XondaeGNJBXmB8sskEp0J6R8l4%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3465&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=gVTwnIWWDDjOhkfyM4SzYegMuQjyaXDg7sFkzkhUyJA%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3464&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=iErMnLzSVjZpkZO%2FJ%2BETC0QbO0neLHNlA41OkXWWBuk%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3379&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=GptTVqXhyQlHTByVbEoQJtjF%2FWc4xlC2KLHHZ3qhbrw%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest:
> >>> 3 failures (98.500% success rate)
> >>>
> >>>
> >>>
> >>>          testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3402&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=S9EbS96H9rYgoCfTtXXTVbcMZAIBZ%2BlIbQgElXKzX98%3D&amp;reserved=0
> >>>
> >>>          testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3380&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297271685&amp;sdata=4OQQFRPSwtx3QcENDyVCUsoMJd0K%2Bwl5o759cyZfsYw%3D&amp;reserved=0
> >>>
> >>>          testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3357&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297271685&amp;sdata=XjIc4NZPwT4HlRnEL9zfWcB5AlxLiO%2B9C9jf77TJ0Dg%3D&amp;reserved=0
> >>>
> >>>
>

Re: Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes

Posted by Alexander Murmann <am...@vmware.com>.
The failing WAN-related tests are still making up the majority of flaky tests<https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-mass-test-run/jobs/create-mass-test-run-report/builds/40> we are observing. It's a real issue, since the noise from flaky tests can easily hide very real bugs. That's why earlier this year several people spent a few months improving the reliability of our tests.

Mario, in July you said you were investigating this. Is that still ongoing? Do you need any help?
________________________________
From: Alexander Murmann <am...@apache.org>
Sent: Tuesday, July 14, 2020 09:00
To: dev@geode.apache.org <de...@geode.apache.org>
Cc: Alexander Murmann <am...@apache.org>
Subject: Re: Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes

Thanks for looking into this, Mario!

You are probably right, that the underlying issue might have been
pre-existing and that the test is surfacing it. I am glad though that you
are investigating, because a close to 30% fail rate is a problem. Something
like this happens every once in a while and then someone has to do some
work to resolve historical problems that they hadn't planned to addres.

Thanks!

On Tue, Jul 14, 2020 at 1:06 AM Mario Ivanac <ma...@est.tech> wrote:

> Hi,
>
> after adding additional checks in failing test, now I can see that test
> are failing due to fault that some batch are distributed at stopping of GW
> sender.
> Cause of that, I suspect that this problem existed prior to this PR, but
> this PR is first to introduce test to check this.
>
> I will continue to investigate this fault, but I can not locally reproduce
> this fault, so this is slowing troubleshooting.
>
> BR,
> Mario
> ________________________________
> Šalje: Alexander Murmann <am...@apache.org>
> Poslano: 14. srpnja 2020. 1:11
> Prima: Alexander Murmann <am...@apache.org>
> Kopija: dev@geode.apache.org <de...@geode.apache.org>; Mario Ivanac
> <ma...@est.tech>
> Predmet: Re: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
>
> We continue to see these WAN tests adding a fail rate of just below 30% in
> our mass test runs
> <
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-develop-mass-test-run%2Fjobs%2Fcreate-mass-test-run-report%2Fbuilds%2F10&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&amp;sdata=STlN5I9iM8gRqsKndH6GKwuY5cVOskAgUjNgkoJvzCw%3D&amp;reserved=0
> >
> .
>
> That's a very significant fail rate that impacts our ability to get our
> code committed with confidence.
>
> Can we resolve this issue? Otherwise, I think we need to consider reverting
> GEODE-7458.
>
> On Fri, Jun 19, 2020 at 3:28 PM Alexander Murmann <am...@apache.org>
> wrote:
>
> > Looking more into this, it looks like this was introduced by the changes
> > for GEODE-7458 - "Adding additional option in gfsh command "start gateway
> > sender" to control clearing of existing queues".
> >
> > That happened about a month ago, but it's inherent to those flaky tests
> > that we discover them only after a while. Nonetheless, they become paper
> > cuts that ultimately slow us down substantially if they persist.
> >
> > @Mario Ivanac If I am correct and GEODE-7458 introduced this you were the
> > one making that change. Might you be able to take a look at making that
> > test more reliable or reverting the change?
> >
> > Thank you!
> >
> > On Fri, Jun 19, 2020 at 7:57 AM Alexander Murmann <am...@apache.org>
> > wrote:
> >
> >> Thank you so much for sharing this, Mark!
> >>
> >> It looks like there is a big cluster around WAN Gateway. Is anyone
> >> already looking into the WAN issues?
> >>
> >> On Thu, Jun 18, 2020 at 10:06 PM Mark Hanson <ha...@vmware.com>
> wrote:
> >>
> >>> FYI, the build success rate was around 90% or so about two months ago.
> >>>
> >>> Here are the DUnit tests that are currently failing in our tests, most
> >>> likely in CI, and PR pipelines.
> >>>
> >>> Please let me know if you have any questions.
> >>>
> >>> Thanks,
> >>> Mark
> >>>
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>>  Overall build success rate: 78.00000% (156 of 200)
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>>
> >>>
> >>> The following test methods see failures in more than one class.  There
> >>> may be a failing *TestBase class
> >>>
> >>>
> >>>
> >>>
> *.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
> >>> 12 failures :
> >>>
> >>>   SerialWANPersistenceEnabledGatewaySenderDUnitTest:  8 failures
> >>> (96.000% success rate)
> >>>
> >>>   SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  4 failures
> >>> (98.000% success rate)
> >>>
> >>>
> >>>
> >>>
> *.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
> >>> 12 failures :
> >>>
> >>>   ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  5
> >>> failures (97.500% success rate)
> >>>
> >>>   ParallelWANPersistenceEnabledGatewaySenderDUnitTest:  7 failures
> >>> (96.500% success rate)
> >>>
> >>>
> >>>
> >>> *.testPingWrongServer:  4 failures :
> >>>
> >>>   ClientServerMiscSelectorDUnitTest:  3 failures (98.500% success rate)
> >>>
> >>>   ClientServerMiscDUnitTest:  1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest:
> >>> 8 failures (96.000% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3539&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&amp;sdata=zcUAsAVVsUprEFeDpWryPa%2F4QTf0pQxcZ8yNpzFwvtQ%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3526&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&amp;sdata=309OdZMeCRLDty3sTtCumrWt2T0b76JNA94EvwuB%2BSI%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3505&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&amp;sdata=SWeCxshemmITfDqulXqu4gQVRPzMOXrgvf3LGcXSTUs%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3435&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&amp;sdata=7gzxFbHVzKgpdZ1ZU8vhDYzXzU4cxyRPxgly6dIt31k%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3414&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&amp;sdata=Xa8bjxPmysZluEn6HbvQzqbccJl%2B%2FsiGIhWYGeBP%2Ba4%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3391&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&amp;sdata=O%2B0CoquKmvf5eT%2Ff%2FMLoXVuojjwk4IeeyI5dIdJugYg%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3363&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=AILkL%2Fhz3I%2FasCXa8fkTSB9mA26gmDxJ4fe%2FeIiS27g%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3359&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=LF0Jr4d9jundS8cNJtWK%2FnRGVLBp4gsX8DUduuFbp0E%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.management.MemberMXBeanDistributedTest:  2 failures
> >>> (99.000% success rate)
> >>>
> >>>
> >>>
> >>>          testBucketCount
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=oYLB8lcafCMgeSVcwwuaIOdKzEzgg4ypKfS4HUrMHrU%3D&amp;reserved=0
> >>>
> >>>          testBucketCount
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3411&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=O0jF6ARYoS0cIxzTNq%2BJQtHB30W45nrBj%2B5lOqu0esY%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>          testRegisterInterestAndMakePrimaryWithFullRedundancy
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3381&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=la5BbwTkvNS1gFlNf3yr%2BF21L0t2z6NY6Clgwv4WGr4%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>          testSimpleQueryOnLocator
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3516&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=Li1jYIfu1biAC47UGz1zqJbAW6neNxgvcw1JaFeR2pk%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>          testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3400&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=u7%2FQXmha26M6yX4rjRspIRDOEwQZB0NjaCxwBBkfFws%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest:  3
> >>> failures (98.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3495&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=jA3538gndW%2BM42LYlJBMzJX9K4psvoQiYxs0761Viog%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3470&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=%2Fh6%2FPjO2f1nnZ7RzMoUnLmRM0Mq2XHlGlSblj7jlcPc%3D&amp;reserved=0
> >>>
> >>>          testReplicatedSerialPropagationHAWithGroupTransactionEvents
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3479&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=ikv%2FMsJVZbPhGRndMcviXfSUsYjMqob2NAZuW4GbfMw%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>          testDistributedDeadlockWithDLock
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3421&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=ZMgFxv%2FpfiVF4xSl5C0VtJSfCVoeA5phFRJF8IS79Lk%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.distributed.LocatorDUnitTest:  1 failures (99.500%
> >>> success rate)
> >>>
> >>>
> >>>
> >>>          testStartTwoLocatorsWithMultiKeystoreSSL
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3398&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=Ro4ej1P%2BTLAN%2FM%2B8v9DVFbK%2B8cSDtCKdjeoYEIfbWgw%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
> >>> 5 failures (97.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3531&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=TAWaGwXo4t3%2FCQp3MewJKEn7p1rdQgp%2FVDWHqQxva4c%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3522&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&amp;sdata=lApVu2gdSk38MjHjZlWZhPWyQ5fOez2Sma5WmYcNE38%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3456&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=6gC%2BqQusVFP5JqeHkxtSvk8%2FWCRIEg90ZeiWomJ1NrE%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=DJ9zA4EJyrjd01eidiCznYn%2FOAVtVfzYzxsfapCGUD4%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=%2FvRCNF0K8V9LfYTsdDxRniadtdXVuV7cZZZsmQTydw0%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>  clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3433&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=rrg9pcWvSZgJ106rjbsS8wpCBhX20er1hdNhW0qGrZ4%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
> >>> 7 failures (96.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3478&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=7R6DvYToK1cwdTfBK93HA9y83qw9OKd4Kj%2BbEe%2F8Yd4%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=1RXrV5Zw0kQSLDrPLu45DlWy4ZR%2B2hs7rfyk8BAFeG4%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=DJ9zA4EJyrjd01eidiCznYn%2FOAVtVfzYzxsfapCGUD4%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=vF7zJq%2BZnOIa1W6oVrfKRJlV6nFIBitzjCVPPCgc6CQ%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=%2FvRCNF0K8V9LfYTsdDxRniadtdXVuV7cZZZsmQTydw0%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3370&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=WDNL8hapq0GmB%2BrxSxFYJ0c4nArV8k15m79OkwrtASY%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3351&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=8%2B%2FBMm9eeXPcjAL4Hj94aduYQUyDrJVgPqB5%2FSqZ4kY%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest:
> >>> 4 failures (98.000% success rate)
> >>>
> >>>
> >>>
> >>>          testMultipleColocatedChildPRsMissingWithSequencedStart
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3493&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=l7RlB5fYmzwGJvYuy5QFIPWJd5mi1o5BAlCYff0XeDQ%3D&amp;reserved=0
> >>>
> >>>          testMissingColocatedChildPRDueToDelayedStart
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=SgLx2XahBD8x2Cez3aK%2BhShvVc%2BP3SFmMfeqB3pdHLM%3D&amp;reserved=0
> >>>
> >>>          testHierarchyOfColocatedChildPRsMissingGrandchild
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3537&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=3Y5qtB5P6Gz8ey%2F6eeEOOGdLptT6LVIBVWLpF4lrRxs%3D&amp;reserved=0
> >>>
> >>>          testHierarchyOfColocatedChildPRsMissingGrandchild
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3424&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&amp;sdata=n6qxZzVKFYqiTY7kUw%2BoQfZKE2aUvbBXs6cXAcqBZaw%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.distributed.DistributedMemberDUnitTest:  2 failures
> >>> (99.000% success rate)
> >>>
> >>>
> >>>
> >>>          testGroupsInAllVMs
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3423&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=prw%2BEBBIgOH8BWwD5SBpvVd%2BvdQSh%2FOJf8Xk6zPZ7yk%3D&amp;reserved=0
> >>>
> >>>          testGroupsInAllVMs
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3374&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=g2PQp%2FiMZEVjc4zSOxNhks48UTPdEAVGyhdJQueipuQ%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.management.JMXMBeanReconnectDUnitTest:  5 failures
> >>> (97.500% success rate)
> >>>
> >>>
> >>>
> >>>          serverMXBeansOnServerAreUnaffectedByLocatorCrash
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=zVXQeZRlv6qJGvlpOFvJIt5WDzvSAq2ohyQ1DRZ1xWE%3D&amp;reserved=0
> >>>
> >>>
> >>>  serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=t5BIGh1%2BcB6SKdUviko%2FghBDtJIrmEUVMGQVXdEhHbA%3D&amp;reserved=0
> >>>
> >>>
> >>>  serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3406&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=GyvOWUusGUi48vS6wWA6GrJN3J5aAMbqeXxy2VevLXU%3D&amp;reserved=0
> >>>
> >>>          locatorHasMemberTypeMXBeansForBothLocators
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3352&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=N261EBctQXEW0FR0XTTHd4w6nmrEkj%2F%2FLd8e14m0cnw%3D&amp;reserved=0
> >>>
> >>>          serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3457&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=QNtL7hTrfxZHkLfKGx7P01m4quS6%2FhkkjyYdhA7J8bA%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
> >>> 4 failures (98.000% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3494&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=TPAyTUGYWUv%2FifU58XondaeGNJBXmB8sskEp0J6R8l4%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3465&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=gVTwnIWWDDjOhkfyM4SzYegMuQjyaXDg7sFkzkhUyJA%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3464&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=iErMnLzSVjZpkZO%2FJ%2BETC0QbO0neLHNlA41OkXWWBuk%3D&amp;reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3379&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=GptTVqXhyQlHTByVbEoQJtjF%2FWc4xlC2KLHHZ3qhbrw%3D&amp;reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest:
> >>> 3 failures (98.500% success rate)
> >>>
> >>>
> >>>
> >>>          testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3402&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&amp;sdata=S9EbS96H9rYgoCfTtXXTVbcMZAIBZ%2BlIbQgElXKzX98%3D&amp;reserved=0
> >>>
> >>>          testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3380&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297271685&amp;sdata=4OQQFRPSwtx3QcENDyVCUsoMJd0K%2Bwl5o759cyZfsYw%3D&amp;reserved=0
> >>>
> >>>          testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3357&amp;data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297271685&amp;sdata=XjIc4NZPwT4HlRnEL9zfWcB5AlxLiO%2B9C9jf77TJ0Dg%3D&amp;reserved=0
> >>>
> >>>
>

Re: Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes

Posted by Alexander Murmann <am...@apache.org>.
Thanks for looking into this, Mario!

You are probably right, that the underlying issue might have been
pre-existing and that the test is surfacing it. I am glad though that you
are investigating, because a close to 30% fail rate is a problem. Something
like this happens every once in a while and then someone has to do some
work to resolve historical problems that they hadn't planned to addres.

Thanks!

On Tue, Jul 14, 2020 at 1:06 AM Mario Ivanac <ma...@est.tech> wrote:

> Hi,
>
> after adding additional checks in failing test, now I can see that test
> are failing due to fault that some batch are distributed at stopping of GW
> sender.
> Cause of that, I suspect that this problem existed prior to this PR, but
> this PR is first to introduce test to check this.
>
> I will continue to investigate this fault, but I can not locally reproduce
> this fault, so this is slowing troubleshooting.
>
> BR,
> Mario
> ________________________________
> Šalje: Alexander Murmann <am...@apache.org>
> Poslano: 14. srpnja 2020. 1:11
> Prima: Alexander Murmann <am...@apache.org>
> Kopija: dev@geode.apache.org <de...@geode.apache.org>; Mario Ivanac
> <ma...@est.tech>
> Predmet: Re: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
>
> We continue to see these WAN tests adding a fail rate of just below 30% in
> our mass test runs
> <
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-mass-test-run/jobs/create-mass-test-run-report/builds/10
> >
> .
>
> That's a very significant fail rate that impacts our ability to get our
> code committed with confidence.
>
> Can we resolve this issue? Otherwise, I think we need to consider reverting
> GEODE-7458.
>
> On Fri, Jun 19, 2020 at 3:28 PM Alexander Murmann <am...@apache.org>
> wrote:
>
> > Looking more into this, it looks like this was introduced by the changes
> > for GEODE-7458 - "Adding additional option in gfsh command "start gateway
> > sender" to control clearing of existing queues".
> >
> > That happened about a month ago, but it's inherent to those flaky tests
> > that we discover them only after a while. Nonetheless, they become paper
> > cuts that ultimately slow us down substantially if they persist.
> >
> > @Mario Ivanac If I am correct and GEODE-7458 introduced this you were the
> > one making that change. Might you be able to take a look at making that
> > test more reliable or reverting the change?
> >
> > Thank you!
> >
> > On Fri, Jun 19, 2020 at 7:57 AM Alexander Murmann <am...@apache.org>
> > wrote:
> >
> >> Thank you so much for sharing this, Mark!
> >>
> >> It looks like there is a big cluster around WAN Gateway. Is anyone
> >> already looking into the WAN issues?
> >>
> >> On Thu, Jun 18, 2020 at 10:06 PM Mark Hanson <ha...@vmware.com>
> wrote:
> >>
> >>> FYI, the build success rate was around 90% or so about two months ago.
> >>>
> >>> Here are the DUnit tests that are currently failing in our tests, most
> >>> likely in CI, and PR pipelines.
> >>>
> >>> Please let me know if you have any questions.
> >>>
> >>> Thanks,
> >>> Mark
> >>>
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>>  Overall build success rate: 78.00000% (156 of 200)
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>>
> >>>
> >>> The following test methods see failures in more than one class.  There
> >>> may be a failing *TestBase class
> >>>
> >>>
> >>>
> >>>
> *.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
> >>> 12 failures :
> >>>
> >>>   SerialWANPersistenceEnabledGatewaySenderDUnitTest:  8 failures
> >>> (96.000% success rate)
> >>>
> >>>   SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  4 failures
> >>> (98.000% success rate)
> >>>
> >>>
> >>>
> >>>
> *.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
> >>> 12 failures :
> >>>
> >>>   ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  5
> >>> failures (97.500% success rate)
> >>>
> >>>   ParallelWANPersistenceEnabledGatewaySenderDUnitTest:  7 failures
> >>> (96.500% success rate)
> >>>
> >>>
> >>>
> >>> *.testPingWrongServer:  4 failures :
> >>>
> >>>   ClientServerMiscSelectorDUnitTest:  3 failures (98.500% success rate)
> >>>
> >>>   ClientServerMiscDUnitTest:  1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest:
> >>> 8 failures (96.000% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3539
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3526
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3505
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3435
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3414
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3391
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3363
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3359
> >>>
> >>>
> >>>
> >>> org.apache.geode.management.MemberMXBeanDistributedTest:  2 failures
> >>> (99.000% success rate)
> >>>
> >>>
> >>>
> >>>          testBucketCount
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
> >>>
> >>>          testBucketCount
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3411
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>          testRegisterInterestAndMakePrimaryWithFullRedundancy
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3381
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>          testSimpleQueryOnLocator
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3516
> >>>
> >>>
> >>>
> >>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>          testPingWrongServer
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3400
> >>>
> >>>
> >>>
> >>> org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest:  3
> >>> failures (98.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3495
> >>>
> >>>
> >>>
> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3470
> >>>
> >>>          testReplicatedSerialPropagationHAWithGroupTransactionEvents
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3479
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>          testDistributedDeadlockWithDLock
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3421
> >>>
> >>>
> >>>
> >>> org.apache.geode.distributed.LocatorDUnitTest:  1 failures (99.500%
> >>> success rate)
> >>>
> >>>
> >>>
> >>>          testStartTwoLocatorsWithMultiKeystoreSSL
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3398
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
> >>> 5 failures (97.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3531
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3522
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3456
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>  clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3433
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
> >>> 7 failures (96.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3478
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3370
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3351
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest:
> >>> 4 failures (98.000% success rate)
> >>>
> >>>
> >>>
> >>>          testMultipleColocatedChildPRsMissingWithSequencedStart
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3493
> >>>
> >>>          testMissingColocatedChildPRDueToDelayedStart
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
> >>>
> >>>          testHierarchyOfColocatedChildPRsMissingGrandchild
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3537
> >>>
> >>>          testHierarchyOfColocatedChildPRsMissingGrandchild
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3424
> >>>
> >>>
> >>>
> >>> org.apache.geode.distributed.DistributedMemberDUnitTest:  2 failures
> >>> (99.000% success rate)
> >>>
> >>>
> >>>
> >>>          testGroupsInAllVMs
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3423
> >>>
> >>>          testGroupsInAllVMs
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3374
> >>>
> >>>
> >>>
> >>> org.apache.geode.management.JMXMBeanReconnectDUnitTest:  5 failures
> >>> (97.500% success rate)
> >>>
> >>>
> >>>
> >>>          serverMXBeansOnServerAreUnaffectedByLocatorCrash
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
> >>>
> >>>
> >>>  serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
> >>>
> >>>
> >>>  serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3406
> >>>
> >>>          locatorHasMemberTypeMXBeansForBothLocators
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3352
> >>>
> >>>          serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3457
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
> >>> 4 failures (98.000% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3494
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3465
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3464
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3379
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest:
> >>> 3 failures (98.500% success rate)
> >>>
> >>>
> >>>
> >>>          testPingWrongServer
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3402
> >>>
> >>>          testPingWrongServer
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3380
> >>>
> >>>          testPingWrongServer
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3357
> >>>
> >>>
>

Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes

Posted by Mario Ivanac <ma...@est.tech>.
Hi,

after adding additional checks in failing test, now I can see that test are failing due to fault that some batch are distributed at stopping of GW sender.
Cause of that, I suspect that this problem existed prior to this PR, but this PR is first to introduce test to check this.

I will continue to investigate this fault, but I can not locally reproduce this fault, so this is slowing troubleshooting.

BR,
Mario
________________________________
Šalje: Alexander Murmann <am...@apache.org>
Poslano: 14. srpnja 2020. 1:11
Prima: Alexander Murmann <am...@apache.org>
Kopija: dev@geode.apache.org <de...@geode.apache.org>; Mario Ivanac <ma...@est.tech>
Predmet: Re: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes

We continue to see these WAN tests adding a fail rate of just below 30% in
our mass test runs
<https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-mass-test-run/jobs/create-mass-test-run-report/builds/10>
.

That's a very significant fail rate that impacts our ability to get our
code committed with confidence.

Can we resolve this issue? Otherwise, I think we need to consider reverting
GEODE-7458.

On Fri, Jun 19, 2020 at 3:28 PM Alexander Murmann <am...@apache.org>
wrote:

> Looking more into this, it looks like this was introduced by the changes
> for GEODE-7458 - "Adding additional option in gfsh command "start gateway
> sender" to control clearing of existing queues".
>
> That happened about a month ago, but it's inherent to those flaky tests
> that we discover them only after a while. Nonetheless, they become paper
> cuts that ultimately slow us down substantially if they persist.
>
> @Mario Ivanac If I am correct and GEODE-7458 introduced this you were the
> one making that change. Might you be able to take a look at making that
> test more reliable or reverting the change?
>
> Thank you!
>
> On Fri, Jun 19, 2020 at 7:57 AM Alexander Murmann <am...@apache.org>
> wrote:
>
>> Thank you so much for sharing this, Mark!
>>
>> It looks like there is a big cluster around WAN Gateway. Is anyone
>> already looking into the WAN issues?
>>
>> On Thu, Jun 18, 2020 at 10:06 PM Mark Hanson <ha...@vmware.com> wrote:
>>
>>> FYI, the build success rate was around 90% or so about two months ago.
>>>
>>> Here are the DUnit tests that are currently failing in our tests, most
>>> likely in CI, and PR pipelines.
>>>
>>> Please let me know if you have any questions.
>>>
>>> Thanks,
>>> Mark
>>>
>>>
>>>
>>> ***********************************************************************************
>>>
>>>  Overall build success rate: 78.00000% (156 of 200)
>>>
>>>
>>> ***********************************************************************************
>>>
>>>
>>>
>>> The following test methods see failures in more than one class.  There
>>> may be a failing *TestBase class
>>>
>>>
>>>
>>> *.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
>>> 12 failures :
>>>
>>>   SerialWANPersistenceEnabledGatewaySenderDUnitTest:  8 failures
>>> (96.000% success rate)
>>>
>>>   SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  4 failures
>>> (98.000% success rate)
>>>
>>>
>>>
>>> *.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
>>> 12 failures :
>>>
>>>   ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  5
>>> failures (97.500% success rate)
>>>
>>>   ParallelWANPersistenceEnabledGatewaySenderDUnitTest:  7 failures
>>> (96.500% success rate)
>>>
>>>
>>>
>>> *.testPingWrongServer:  4 failures :
>>>
>>>   ClientServerMiscSelectorDUnitTest:  3 failures (98.500% success rate)
>>>
>>>   ClientServerMiscDUnitTest:  1 failures (99.500% success rate)
>>>
>>>
>>>
>>>
>>> ***********************************************************************************
>>>
>>>
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest:
>>> 8 failures (96.000% success rate)
>>>
>>>
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3539
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3526
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3505
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3435
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3414
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3391
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3363
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3359
>>>
>>>
>>>
>>> org.apache.geode.management.MemberMXBeanDistributedTest:  2 failures
>>> (99.000% success rate)
>>>
>>>
>>>
>>>          testBucketCount
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
>>>
>>>          testBucketCount
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3411
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>>          testRegisterInterestAndMakePrimaryWithFullRedundancy
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3381
>>>
>>>
>>>
>>> org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>>          testSimpleQueryOnLocator
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3516
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>>          testPingWrongServer
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3400
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest:  3
>>> failures (98.500% success rate)
>>>
>>>
>>>
>>>
>>>  testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3495
>>>
>>>
>>>  testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3470
>>>
>>>          testReplicatedSerialPropagationHAWithGroupTransactionEvents
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3479
>>>
>>>
>>>
>>> org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>>          testDistributedDeadlockWithDLock
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3421
>>>
>>>
>>>
>>> org.apache.geode.distributed.LocatorDUnitTest:  1 failures (99.500%
>>> success rate)
>>>
>>>
>>>
>>>          testStartTwoLocatorsWithMultiKeystoreSSL
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3398
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
>>> 5 failures (97.500% success rate)
>>>
>>>
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3531
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3522
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3456
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>>
>>>  clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3433
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
>>> 7 failures (96.500% success rate)
>>>
>>>
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3478
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3370
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3351
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest:
>>> 4 failures (98.000% success rate)
>>>
>>>
>>>
>>>          testMultipleColocatedChildPRsMissingWithSequencedStart
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3493
>>>
>>>          testMissingColocatedChildPRDueToDelayedStart
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
>>>
>>>          testHierarchyOfColocatedChildPRsMissingGrandchild
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3537
>>>
>>>          testHierarchyOfColocatedChildPRsMissingGrandchild
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3424
>>>
>>>
>>>
>>> org.apache.geode.distributed.DistributedMemberDUnitTest:  2 failures
>>> (99.000% success rate)
>>>
>>>
>>>
>>>          testGroupsInAllVMs
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3423
>>>
>>>          testGroupsInAllVMs
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3374
>>>
>>>
>>>
>>> org.apache.geode.management.JMXMBeanReconnectDUnitTest:  5 failures
>>> (97.500% success rate)
>>>
>>>
>>>
>>>          serverMXBeansOnServerAreUnaffectedByLocatorCrash
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
>>>
>>>
>>>  serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
>>>
>>>
>>>  serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3406
>>>
>>>          locatorHasMemberTypeMXBeansForBothLocators
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3352
>>>
>>>          serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3457
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
>>> 4 failures (98.000% success rate)
>>>
>>>
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3494
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3465
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3464
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3379
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest:
>>> 3 failures (98.500% success rate)
>>>
>>>
>>>
>>>          testPingWrongServer
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3402
>>>
>>>          testPingWrongServer
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3380
>>>
>>>          testPingWrongServer
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3357
>>>
>>>

Re: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes

Posted by Mark Hanson <ha...@vmware.com>.
The previous statement about 30% didn't make sense to me, so I thought I would throw in this tidbit...

70 failures out 75 failures in 200 runs are caused by WAN.  

Thanks,
Mark

On 7/13/20, 4:11 PM, "Alexander Murmann" <am...@apache.org> wrote:

    We continue to see these WAN tests adding a fail rate of just below 30% in
    our mass test runs
    <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-develop-mass-test-run%2Fjobs%2Fcreate-mass-test-run-report%2Fbuilds%2F10&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835802218&amp;sdata=PZs6%2B%2F2vRL94anXVe0h6xt7U%2BwGR%2BKPbu7ZHLMr4vGQ%3D&amp;reserved=0>
    .

    That's a very significant fail rate that impacts our ability to get our
    code committed with confidence.

    Can we resolve this issue? Otherwise, I think we need to consider reverting
    GEODE-7458.

    On Fri, Jun 19, 2020 at 3:28 PM Alexander Murmann <am...@apache.org>
    wrote:

    > Looking more into this, it looks like this was introduced by the changes
    > for GEODE-7458 - "Adding additional option in gfsh command "start gateway
    > sender" to control clearing of existing queues".
    >
    > That happened about a month ago, but it's inherent to those flaky tests
    > that we discover them only after a while. Nonetheless, they become paper
    > cuts that ultimately slow us down substantially if they persist.
    >
    > @Mario Ivanac If I am correct and GEODE-7458 introduced this you were the
    > one making that change. Might you be able to take a look at making that
    > test more reliable or reverting the change?
    >
    > Thank you!
    >
    > On Fri, Jun 19, 2020 at 7:57 AM Alexander Murmann <am...@apache.org>
    > wrote:
    >
    >> Thank you so much for sharing this, Mark!
    >>
    >> It looks like there is a big cluster around WAN Gateway. Is anyone
    >> already looking into the WAN issues?
    >>
    >> On Thu, Jun 18, 2020 at 10:06 PM Mark Hanson <ha...@vmware.com> wrote:
    >>
    >>> FYI, the build success rate was around 90% or so about two months ago.
    >>>
    >>> Here are the DUnit tests that are currently failing in our tests, most
    >>> likely in CI, and PR pipelines.
    >>>
    >>> Please let me know if you have any questions.
    >>>
    >>> Thanks,
    >>> Mark
    >>>
    >>>
    >>>
    >>> ***********************************************************************************
    >>>
    >>>  Overall build success rate: 78.00000% (156 of 200)
    >>>
    >>>
    >>> ***********************************************************************************
    >>>
    >>>
    >>>
    >>> The following test methods see failures in more than one class.  There
    >>> may be a failing *TestBase class
    >>>
    >>>
    >>>
    >>> *.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
    >>> 12 failures :
    >>>
    >>>   SerialWANPersistenceEnabledGatewaySenderDUnitTest:  8 failures
    >>> (96.000% success rate)
    >>>
    >>>   SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  4 failures
    >>> (98.000% success rate)
    >>>
    >>>
    >>>
    >>> *.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
    >>> 12 failures :
    >>>
    >>>   ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  5
    >>> failures (97.500% success rate)
    >>>
    >>>   ParallelWANPersistenceEnabledGatewaySenderDUnitTest:  7 failures
    >>> (96.500% success rate)
    >>>
    >>>
    >>>
    >>> *.testPingWrongServer:  4 failures :
    >>>
    >>>   ClientServerMiscSelectorDUnitTest:  3 failures (98.500% success rate)
    >>>
    >>>   ClientServerMiscDUnitTest:  1 failures (99.500% success rate)
    >>>
    >>>
    >>>
    >>>
    >>> ***********************************************************************************
    >>>
    >>>
    >>>
    >>>
    >>>
    >>> org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest:
    >>> 8 failures (96.000% success rate)
    >>>
    >>>
    >>>
    >>>
    >>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3539&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835802218&amp;sdata=ILB4NvS0zakenbq00pfHjOZlb9pt7n60%2FpjNUV%2FUQp8%3D&amp;reserved=0
    >>>
    >>>
    >>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3526&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835802218&amp;sdata=4pejQ84CaPxlR6terHjOb4MlSDhqmiZWw5FMfshFbL0%3D&amp;reserved=0
    >>>
    >>>
    >>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3505&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835802218&amp;sdata=0sOJRUud9GWXHo3iPgSU53z9Eyi9sh7LiYSXSRYCg38%3D&amp;reserved=0
    >>>
    >>>
    >>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3435&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&amp;sdata=iJJyU6KNfWTGF3cSdgfQzCtbh%2BroLJxhVfW0HL4NVKk%3D&amp;reserved=0
    >>>
    >>>
    >>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3414&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&amp;sdata=Qd90VpFi1ckXArWAerACT8QKQi5S3DPF9O%2Fc%2FWuOras%3D&amp;reserved=0
    >>>
    >>>
    >>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3391&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&amp;sdata=sybHZb4TXpDQndMWJazxYqpC8YL2yLCKcinJomP6WP0%3D&amp;reserved=0
    >>>
    >>>
    >>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3363&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&amp;sdata=AlBfHXFJCbKYhqOrO3c7n0vFWBK3n5jNmJt3ecQFS8Q%3D&amp;reserved=0
    >>>
    >>>
    >>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3359&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&amp;sdata=0Kj7PBqxzd8M53Hh7F2r%2B9mtYj57Gfc3WxCOVPCTYVo%3D&amp;reserved=0
    >>>
    >>>
    >>>
    >>> org.apache.geode.management.MemberMXBeanDistributedTest:  2 failures
    >>> (99.000% success rate)
    >>>
    >>>
    >>>
    >>>          testBucketCount
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&amp;sdata=rE%2FYuEs6HBgpz0P3WYbe%2Bt3goceTlR%2Bs51tqzhDkFsA%3D&amp;reserved=0
    >>>
    >>>          testBucketCount
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3411&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&amp;sdata=7M7j%2BVhhYDA9QpXQWfCr9nXi9Krl1HMObVJk8D%2FjufY%3D&amp;reserved=0
    >>>
    >>>
    >>>
    >>> org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest:
    >>> 1 failures (99.500% success rate)
    >>>
    >>>
    >>>
    >>>          testRegisterInterestAndMakePrimaryWithFullRedundancy
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3381&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&amp;sdata=J0lyeA9l1b0gT4A0PR4N5KKMBySOQrg8zWCgKptE7iQ%3D&amp;reserved=0
    >>>
    >>>
    >>>
    >>> org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest:
    >>> 1 failures (99.500% success rate)
    >>>
    >>>
    >>>
    >>>          testSimpleQueryOnLocator
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3516&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&amp;sdata=FARo2AYcibOpGuFwt25N7s8rfUDddLe5QkhR4OxFJ8k%3D&amp;reserved=0
    >>>
    >>>
    >>>
    >>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest:
    >>> 1 failures (99.500% success rate)
    >>>
    >>>
    >>>
    >>>          testPingWrongServer
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3400&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&amp;sdata=ORdmb4v2Lrp2SxI8HtiToZN%2BvSSddgg4OqzFuexvrSo%3D&amp;reserved=0
    >>>
    >>>
    >>>
    >>> org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest:  3
    >>> failures (98.500% success rate)
    >>>
    >>>
    >>>
    >>>
    >>>  testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3495&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&amp;sdata=G%2Bxs28AKlk%2Fa9PjMg0X0xP8tHzfBpa8IWnqCUuzNdik%3D&amp;reserved=0
    >>>
    >>>
    >>>  testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3470&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&amp;sdata=GAr47n034NLpEAgyYMM0%2Bg%2BcJIX5wV%2BAAAnKef3OEj0%3D&amp;reserved=0
    >>>
    >>>          testReplicatedSerialPropagationHAWithGroupTransactionEvents
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3479&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&amp;sdata=z81ZJBzMqT%2F4SR3DySZgzO7GA8EnOY4jgNsqyUi6d4o%3D&amp;reserved=0
    >>>
    >>>
    >>>
    >>> org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest:
    >>> 1 failures (99.500% success rate)
    >>>
    >>>
    >>>
    >>>          testDistributedDeadlockWithDLock
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3421&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&amp;sdata=Ux54TOh1LLp9XRAjdbMK0scUn70V32Up%2FV78RjTtr8A%3D&amp;reserved=0
    >>>
    >>>
    >>>
    >>> org.apache.geode.distributed.LocatorDUnitTest:  1 failures (99.500%
    >>> success rate)
    >>>
    >>>
    >>>
    >>>          testStartTwoLocatorsWithMultiKeystoreSSL
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3398&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&amp;sdata=eic2QNLwx%2B18jVZHPfOGfA99%2B8gLaXXal2KuSBvi6s0%3D&amp;reserved=0
    >>>
    >>>
    >>>
    >>> org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
    >>> 5 failures (97.500% success rate)
    >>>
    >>>
    >>>
    >>>
    >>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3531&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&amp;sdata=3apyNPTCcAbKQ55s1RqP7c%2BiuTjYWoDHN7itNiJBr6s%3D&amp;reserved=0
    >>>
    >>>
    >>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3522&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&amp;sdata=dsdzFOC4jPXN2pKibdZR8MWCwKEJWOCv8nNSnvrhclg%3D&amp;reserved=0
    >>>
    >>>
    >>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3456&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&amp;sdata=K2%2Fo2mdpIVC4u2V12VRkpyyGWJq2vYqtm1%2Fvki8mB9A%3D&amp;reserved=0
    >>>
    >>>
    >>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&amp;sdata=Dje%2F53pJYke0d5yosB61kMB77CLZYdLfqxWoD%2BfFzZs%3D&amp;reserved=0
    >>>
    >>>
    >>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&amp;sdata=1dKgH5KcHmj7snaRO46OW8k0GPrnTottsNCWH2HdQAE%3D&amp;reserved=0
    >>>
    >>>
    >>>
    >>> org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest:
    >>> 1 failures (99.500% success rate)
    >>>
    >>>
    >>>
    >>>
    >>>  clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3433&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&amp;sdata=fH3g67sitVbMU9Ir1Exm8MclDAVqM4x8mg%2F9y7WYi6I%3D&amp;reserved=0
    >>>
    >>>
    >>>
    >>> org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
    >>> 7 failures (96.500% success rate)
    >>>
    >>>
    >>>
    >>>
    >>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3478&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&amp;sdata=9JBfKJMKPrarh2IkhTzz%2BxxQ6Mdt0J2aaqHnn7fUsNo%3D&amp;reserved=0
    >>>
    >>>
    >>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&amp;sdata=Wi%2FM0H%2FEbUoS%2FEDaHs0ybqHqc5ZJY9jhai9XvGqOaeg%3D&amp;reserved=0
    >>>
    >>>
    >>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&amp;sdata=Dje%2F53pJYke0d5yosB61kMB77CLZYdLfqxWoD%2BfFzZs%3D&amp;reserved=0
    >>>
    >>>
    >>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&amp;sdata=ASOBOjhutihuLnyEk9v%2ByTlzeorAJrYFBlWKpIq3yVA%3D&amp;reserved=0
    >>>
    >>>
    >>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&amp;sdata=1dKgH5KcHmj7snaRO46OW8k0GPrnTottsNCWH2HdQAE%3D&amp;reserved=0
    >>>
    >>>
    >>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3370&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&amp;sdata=k1QhouuiRkGwHEoTve21cd1rClD6%2F3PEO%2Fp8pQUrX7A%3D&amp;reserved=0
    >>>
    >>>
    >>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3351&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&amp;sdata=yDB750p04VE%2FuMwwjVpZ3zAtW50BlfdWUVFW205XnKw%3D&amp;reserved=0
    >>>
    >>>
    >>>
    >>> org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest:
    >>> 4 failures (98.000% success rate)
    >>>
    >>>
    >>>
    >>>          testMultipleColocatedChildPRsMissingWithSequencedStart
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3493&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&amp;sdata=4jOpZq1Q6qXgNqHYQAWBW702yhCDpUHzQPmOKk1ef5A%3D&amp;reserved=0
    >>>
    >>>          testMissingColocatedChildPRDueToDelayedStart
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&amp;sdata=P%2BZyelCzOwqc87bP9XFlIPdVYn0BwWk4jMIu0lxaq4M%3D&amp;reserved=0
    >>>
    >>>          testHierarchyOfColocatedChildPRsMissingGrandchild
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3537&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&amp;sdata=qt20QmQTmbe%2F33LtWmQsXnDWfIUB6vlf5xtE%2Bw23gG0%3D&amp;reserved=0
    >>>
    >>>          testHierarchyOfColocatedChildPRsMissingGrandchild
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3424&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&amp;sdata=2mu0xJE6BaugPHzEheTw%2Fp4nDt6xdifAH0rCnF3mBno%3D&amp;reserved=0
    >>>
    >>>
    >>>
    >>> org.apache.geode.distributed.DistributedMemberDUnitTest:  2 failures
    >>> (99.000% success rate)
    >>>
    >>>
    >>>
    >>>          testGroupsInAllVMs
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3423&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&amp;sdata=VPQRY2DJxao%2F%2Fsj3rTMBtZ1J9R7FQYR7SqY1hgQoj1U%3D&amp;reserved=0
    >>>
    >>>          testGroupsInAllVMs
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3374&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&amp;sdata=GyAnvRP4%2BVP4bYI%2BkXzrTgsVezpe0c6SKvnqiZq1p1k%3D&amp;reserved=0
    >>>
    >>>
    >>>
    >>> org.apache.geode.management.JMXMBeanReconnectDUnitTest:  5 failures
    >>> (97.500% success rate)
    >>>
    >>>
    >>>
    >>>          serverMXBeansOnServerAreUnaffectedByLocatorCrash
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&amp;sdata=P%2BZyelCzOwqc87bP9XFlIPdVYn0BwWk4jMIu0lxaq4M%3D&amp;reserved=0
    >>>
    >>>
    >>>  serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&amp;sdata=vq40%2BFWfDbH%2Fs4b%2BooGY3KBi1WV1LT%2Bh3s45BnAbq3I%3D&amp;reserved=0
    >>>
    >>>
    >>>  serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3406&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&amp;sdata=ZNwEvU4gWXhoVGVfx3vqHjHmV2%2FQHS1VOjifnY0qZCg%3D&amp;reserved=0
    >>>
    >>>          locatorHasMemberTypeMXBeansForBothLocators
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3352&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&amp;sdata=gyq3c%2FGdfmE1OATwqe39MQUfjz%2Be6crV6dTDWQUgvUM%3D&amp;reserved=0
    >>>
    >>>          serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3457&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&amp;sdata=elpsPB4B3SCHvP0yrV0CuNMM11dAUr8yROnKa3RpQus%3D&amp;reserved=0
    >>>
    >>>
    >>>
    >>> org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
    >>> 4 failures (98.000% success rate)
    >>>
    >>>
    >>>
    >>>
    >>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3494&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&amp;sdata=tm2oxvbVR%2BcL7gPOqIOqIK0FKLbJr%2BEHrB4lywt649k%3D&amp;reserved=0
    >>>
    >>>
    >>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3465&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&amp;sdata=Ho8AWWGOORaKrtMlj38WaAs38BE5nmuaA1xGC%2B9UmCU%3D&amp;reserved=0
    >>>
    >>>
    >>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3464&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&amp;sdata=QXxXqvCi7aDtvosglZ5q51fyAguonxny%2F5b%2FAFvjFL8%3D&amp;reserved=0
    >>>
    >>>
    >>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
    >>>
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3379&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835842199&amp;sdata=7JCsVA7pAmjclJo2tk1jDq%2BCeMjHR2G2G6wLSfXCAqQ%3D&amp;reserved=0
    >>>
    >>>
    >>>
    >>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest:
    >>> 3 failures (98.500% success rate)
    >>>
    >>>
    >>>
    >>>          testPingWrongServer
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3402&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835842199&amp;sdata=d9GXmfDC%2BbpY21Pgat%2FIXRaJ61d3fWjhUCqvTwvNM9w%3D&amp;reserved=0
    >>>
    >>>          testPingWrongServer
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3380&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835842199&amp;sdata=WBz4zHqtAhx%2FBOy0lHi7vJWHgYzZHKNq7pynG7gt%2Frc%3D&amp;reserved=0
    >>>
    >>>          testPingWrongServer
    >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3357&amp;data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835842199&amp;sdata=99aY8oSD%2B2EF3RgdyZj3L9I2REpk7NPqbYuzz9SdIIA%3D&amp;reserved=0
    >>>
    >>>


Re: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes

Posted by Alexander Murmann <am...@apache.org>.
We continue to see these WAN tests adding a fail rate of just below 30% in
our mass test runs
<https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-mass-test-run/jobs/create-mass-test-run-report/builds/10>
.

That's a very significant fail rate that impacts our ability to get our
code committed with confidence.

Can we resolve this issue? Otherwise, I think we need to consider reverting
GEODE-7458.

On Fri, Jun 19, 2020 at 3:28 PM Alexander Murmann <am...@apache.org>
wrote:

> Looking more into this, it looks like this was introduced by the changes
> for GEODE-7458 - "Adding additional option in gfsh command "start gateway
> sender" to control clearing of existing queues".
>
> That happened about a month ago, but it's inherent to those flaky tests
> that we discover them only after a while. Nonetheless, they become paper
> cuts that ultimately slow us down substantially if they persist.
>
> @Mario Ivanac If I am correct and GEODE-7458 introduced this you were the
> one making that change. Might you be able to take a look at making that
> test more reliable or reverting the change?
>
> Thank you!
>
> On Fri, Jun 19, 2020 at 7:57 AM Alexander Murmann <am...@apache.org>
> wrote:
>
>> Thank you so much for sharing this, Mark!
>>
>> It looks like there is a big cluster around WAN Gateway. Is anyone
>> already looking into the WAN issues?
>>
>> On Thu, Jun 18, 2020 at 10:06 PM Mark Hanson <ha...@vmware.com> wrote:
>>
>>> FYI, the build success rate was around 90% or so about two months ago.
>>>
>>> Here are the DUnit tests that are currently failing in our tests, most
>>> likely in CI, and PR pipelines.
>>>
>>> Please let me know if you have any questions.
>>>
>>> Thanks,
>>> Mark
>>>
>>>
>>>
>>> ***********************************************************************************
>>>
>>>  Overall build success rate: 78.00000% (156 of 200)
>>>
>>>
>>> ***********************************************************************************
>>>
>>>
>>>
>>> The following test methods see failures in more than one class.  There
>>> may be a failing *TestBase class
>>>
>>>
>>>
>>> *.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
>>> 12 failures :
>>>
>>>   SerialWANPersistenceEnabledGatewaySenderDUnitTest:  8 failures
>>> (96.000% success rate)
>>>
>>>   SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  4 failures
>>> (98.000% success rate)
>>>
>>>
>>>
>>> *.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
>>> 12 failures :
>>>
>>>   ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  5
>>> failures (97.500% success rate)
>>>
>>>   ParallelWANPersistenceEnabledGatewaySenderDUnitTest:  7 failures
>>> (96.500% success rate)
>>>
>>>
>>>
>>> *.testPingWrongServer:  4 failures :
>>>
>>>   ClientServerMiscSelectorDUnitTest:  3 failures (98.500% success rate)
>>>
>>>   ClientServerMiscDUnitTest:  1 failures (99.500% success rate)
>>>
>>>
>>>
>>>
>>> ***********************************************************************************
>>>
>>>
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest:
>>> 8 failures (96.000% success rate)
>>>
>>>
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3539
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3526
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3505
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3435
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3414
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3391
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3363
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3359
>>>
>>>
>>>
>>> org.apache.geode.management.MemberMXBeanDistributedTest:  2 failures
>>> (99.000% success rate)
>>>
>>>
>>>
>>>          testBucketCount
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
>>>
>>>          testBucketCount
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3411
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>>          testRegisterInterestAndMakePrimaryWithFullRedundancy
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3381
>>>
>>>
>>>
>>> org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>>          testSimpleQueryOnLocator
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3516
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>>          testPingWrongServer
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3400
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest:  3
>>> failures (98.500% success rate)
>>>
>>>
>>>
>>>
>>>  testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3495
>>>
>>>
>>>  testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3470
>>>
>>>          testReplicatedSerialPropagationHAWithGroupTransactionEvents
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3479
>>>
>>>
>>>
>>> org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>>          testDistributedDeadlockWithDLock
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3421
>>>
>>>
>>>
>>> org.apache.geode.distributed.LocatorDUnitTest:  1 failures (99.500%
>>> success rate)
>>>
>>>
>>>
>>>          testStartTwoLocatorsWithMultiKeystoreSSL
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3398
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
>>> 5 failures (97.500% success rate)
>>>
>>>
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3531
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3522
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3456
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>>
>>>  clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3433
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
>>> 7 failures (96.500% success rate)
>>>
>>>
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3478
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3370
>>>
>>>
>>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3351
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest:
>>> 4 failures (98.000% success rate)
>>>
>>>
>>>
>>>          testMultipleColocatedChildPRsMissingWithSequencedStart
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3493
>>>
>>>          testMissingColocatedChildPRDueToDelayedStart
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
>>>
>>>          testHierarchyOfColocatedChildPRsMissingGrandchild
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3537
>>>
>>>          testHierarchyOfColocatedChildPRsMissingGrandchild
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3424
>>>
>>>
>>>
>>> org.apache.geode.distributed.DistributedMemberDUnitTest:  2 failures
>>> (99.000% success rate)
>>>
>>>
>>>
>>>          testGroupsInAllVMs
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3423
>>>
>>>          testGroupsInAllVMs
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3374
>>>
>>>
>>>
>>> org.apache.geode.management.JMXMBeanReconnectDUnitTest:  5 failures
>>> (97.500% success rate)
>>>
>>>
>>>
>>>          serverMXBeansOnServerAreUnaffectedByLocatorCrash
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
>>>
>>>
>>>  serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
>>>
>>>
>>>  serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3406
>>>
>>>          locatorHasMemberTypeMXBeansForBothLocators
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3352
>>>
>>>          serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3457
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
>>> 4 failures (98.000% success rate)
>>>
>>>
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3494
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3465
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3464
>>>
>>>
>>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3379
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest:
>>> 3 failures (98.500% success rate)
>>>
>>>
>>>
>>>          testPingWrongServer
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3402
>>>
>>>          testPingWrongServer
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3380
>>>
>>>          testPingWrongServer
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3357
>>>
>>>

Re: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes

Posted by Alexander Murmann <am...@apache.org>.
Looking more into this, it looks like this was introduced by the changes
for GEODE-7458 - "Adding additional option in gfsh command "start gateway
sender" to control clearing of existing queues".

That happened about a month ago, but it's inherent to those flaky tests
that we discover them only after a while. Nonetheless, they become paper
cuts that ultimately slow us down substantially if they persist.

@Mario Ivanac If I am correct and GEODE-7458 introduced this you were the
one making that change. Might you be able to take a look at making that
test more reliable or reverting the change?

Thank you!

On Fri, Jun 19, 2020 at 7:57 AM Alexander Murmann <am...@apache.org>
wrote:

> Thank you so much for sharing this, Mark!
>
> It looks like there is a big cluster around WAN Gateway. Is anyone already
> looking into the WAN issues?
>
> On Thu, Jun 18, 2020 at 10:06 PM Mark Hanson <ha...@vmware.com> wrote:
>
>> FYI, the build success rate was around 90% or so about two months ago.
>>
>> Here are the DUnit tests that are currently failing in our tests, most
>> likely in CI, and PR pipelines.
>>
>> Please let me know if you have any questions.
>>
>> Thanks,
>> Mark
>>
>>
>>
>> ***********************************************************************************
>>
>>  Overall build success rate: 78.00000% (156 of 200)
>>
>>
>> ***********************************************************************************
>>
>>
>>
>> The following test methods see failures in more than one class.  There
>> may be a failing *TestBase class
>>
>>
>>
>> *.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
>> 12 failures :
>>
>>   SerialWANPersistenceEnabledGatewaySenderDUnitTest:  8 failures (96.000%
>> success rate)
>>
>>   SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  4 failures
>> (98.000% success rate)
>>
>>
>>
>> *.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
>> 12 failures :
>>
>>   ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  5 failures
>> (97.500% success rate)
>>
>>   ParallelWANPersistenceEnabledGatewaySenderDUnitTest:  7 failures
>> (96.500% success rate)
>>
>>
>>
>> *.testPingWrongServer:  4 failures :
>>
>>   ClientServerMiscSelectorDUnitTest:  3 failures (98.500% success rate)
>>
>>   ClientServerMiscDUnitTest:  1 failures (99.500% success rate)
>>
>>
>>
>>
>> ***********************************************************************************
>>
>>
>>
>>
>>
>> org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest:
>> 8 failures (96.000% success rate)
>>
>>
>>
>>
>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3539
>>
>>
>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3526
>>
>>
>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3505
>>
>>
>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3435
>>
>>
>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3414
>>
>>
>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3391
>>
>>
>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3363
>>
>>
>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3359
>>
>>
>>
>> org.apache.geode.management.MemberMXBeanDistributedTest:  2 failures
>> (99.000% success rate)
>>
>>
>>
>>          testBucketCount
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
>>
>>          testBucketCount
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3411
>>
>>
>>
>> org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest:
>> 1 failures (99.500% success rate)
>>
>>
>>
>>          testRegisterInterestAndMakePrimaryWithFullRedundancy
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3381
>>
>>
>>
>> org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest:
>> 1 failures (99.500% success rate)
>>
>>
>>
>>          testSimpleQueryOnLocator
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3516
>>
>>
>>
>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest:
>> 1 failures (99.500% success rate)
>>
>>
>>
>>          testPingWrongServer
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3400
>>
>>
>>
>> org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest:  3
>> failures (98.500% success rate)
>>
>>
>>
>>
>>  testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3495
>>
>>
>>  testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3470
>>
>>          testReplicatedSerialPropagationHAWithGroupTransactionEvents
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3479
>>
>>
>>
>> org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest:
>> 1 failures (99.500% success rate)
>>
>>
>>
>>          testDistributedDeadlockWithDLock
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3421
>>
>>
>>
>> org.apache.geode.distributed.LocatorDUnitTest:  1 failures (99.500%
>> success rate)
>>
>>
>>
>>          testStartTwoLocatorsWithMultiKeystoreSSL
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3398
>>
>>
>>
>> org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
>> 5 failures (97.500% success rate)
>>
>>
>>
>>
>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3531
>>
>>
>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3522
>>
>>
>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3456
>>
>>
>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
>>
>>
>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
>>
>>
>>
>> org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest:
>> 1 failures (99.500% success rate)
>>
>>
>>
>>
>>  clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3433
>>
>>
>>
>> org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
>> 7 failures (96.500% success rate)
>>
>>
>>
>>
>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3478
>>
>>
>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
>>
>>
>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
>>
>>
>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
>>
>>
>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
>>
>>
>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3370
>>
>>
>>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3351
>>
>>
>>
>> org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest:
>> 4 failures (98.000% success rate)
>>
>>
>>
>>          testMultipleColocatedChildPRsMissingWithSequencedStart
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3493
>>
>>          testMissingColocatedChildPRDueToDelayedStart
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
>>
>>          testHierarchyOfColocatedChildPRsMissingGrandchild
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3537
>>
>>          testHierarchyOfColocatedChildPRsMissingGrandchild
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3424
>>
>>
>>
>> org.apache.geode.distributed.DistributedMemberDUnitTest:  2 failures
>> (99.000% success rate)
>>
>>
>>
>>          testGroupsInAllVMs
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3423
>>
>>          testGroupsInAllVMs
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3374
>>
>>
>>
>> org.apache.geode.management.JMXMBeanReconnectDUnitTest:  5 failures
>> (97.500% success rate)
>>
>>
>>
>>          serverMXBeansOnServerAreUnaffectedByLocatorCrash
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
>>
>>          serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
>>
>>          serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3406
>>
>>          locatorHasMemberTypeMXBeansForBothLocators
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3352
>>
>>          serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3457
>>
>>
>>
>> org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
>> 4 failures (98.000% success rate)
>>
>>
>>
>>
>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3494
>>
>>
>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3465
>>
>>
>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3464
>>
>>
>>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3379
>>
>>
>>
>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest:
>> 3 failures (98.500% success rate)
>>
>>
>>
>>          testPingWrongServer
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3402
>>
>>          testPingWrongServer
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3380
>>
>>          testPingWrongServer
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3357
>>
>>

Re: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes

Posted by Alexander Murmann <am...@apache.org>.
Thank you so much for sharing this, Mark!

It looks like there is a big cluster around WAN Gateway. Is anyone already
looking into the WAN issues?

On Thu, Jun 18, 2020 at 10:06 PM Mark Hanson <ha...@vmware.com> wrote:

> FYI, the build success rate was around 90% or so about two months ago.
>
> Here are the DUnit tests that are currently failing in our tests, most
> likely in CI, and PR pipelines.
>
> Please let me know if you have any questions.
>
> Thanks,
> Mark
>
>
>
> ***********************************************************************************
>
>  Overall build success rate: 78.00000% (156 of 200)
>
>
> ***********************************************************************************
>
>
>
> The following test methods see failures in more than one class.  There may
> be a failing *TestBase class
>
>
>
> *.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
> 12 failures :
>
>   SerialWANPersistenceEnabledGatewaySenderDUnitTest:  8 failures (96.000%
> success rate)
>
>   SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  4 failures
> (98.000% success rate)
>
>
>
> *.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
> 12 failures :
>
>   ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  5 failures
> (97.500% success rate)
>
>   ParallelWANPersistenceEnabledGatewaySenderDUnitTest:  7 failures
> (96.500% success rate)
>
>
>
> *.testPingWrongServer:  4 failures :
>
>   ClientServerMiscSelectorDUnitTest:  3 failures (98.500% success rate)
>
>   ClientServerMiscDUnitTest:  1 failures (99.500% success rate)
>
>
>
>
> ***********************************************************************************
>
>
>
>
>
> org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest:
> 8 failures (96.000% success rate)
>
>
>
>
>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3539
>
>
>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3526
>
>
>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3505
>
>
>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3435
>
>
>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3414
>
>
>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3391
>
>
>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3363
>
>
>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3359
>
>
>
> org.apache.geode.management.MemberMXBeanDistributedTest:  2 failures
> (99.000% success rate)
>
>
>
>          testBucketCount
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
>
>          testBucketCount
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3411
>
>
>
> org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest:
> 1 failures (99.500% success rate)
>
>
>
>          testRegisterInterestAndMakePrimaryWithFullRedundancy
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3381
>
>
>
> org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest:
> 1 failures (99.500% success rate)
>
>
>
>          testSimpleQueryOnLocator
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3516
>
>
>
> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest:  1
> failures (99.500% success rate)
>
>
>
>          testPingWrongServer
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3400
>
>
>
> org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest:  3
> failures (98.500% success rate)
>
>
>
>
>  testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3495
>
>
>  testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3470
>
>          testReplicatedSerialPropagationHAWithGroupTransactionEvents
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3479
>
>
>
> org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest:
> 1 failures (99.500% success rate)
>
>
>
>          testDistributedDeadlockWithDLock
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3421
>
>
>
> org.apache.geode.distributed.LocatorDUnitTest:  1 failures (99.500%
> success rate)
>
>
>
>          testStartTwoLocatorsWithMultiKeystoreSSL
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3398
>
>
>
> org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
> 5 failures (97.500% success rate)
>
>
>
>
>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3531
>
>
>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3522
>
>
>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3456
>
>
>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
>
>
>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
>
>
>
> org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest:
> 1 failures (99.500% success rate)
>
>
>
>
>  clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3433
>
>
>
> org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
> 7 failures (96.500% success rate)
>
>
>
>
>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3478
>
>
>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
>
>
>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
>
>
>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
>
>
>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
>
>
>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3370
>
>
>  testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3351
>
>
>
> org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest:
> 4 failures (98.000% success rate)
>
>
>
>          testMultipleColocatedChildPRsMissingWithSequencedStart
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3493
>
>          testMissingColocatedChildPRDueToDelayedStart
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
>
>          testHierarchyOfColocatedChildPRsMissingGrandchild
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3537
>
>          testHierarchyOfColocatedChildPRsMissingGrandchild
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3424
>
>
>
> org.apache.geode.distributed.DistributedMemberDUnitTest:  2 failures
> (99.000% success rate)
>
>
>
>          testGroupsInAllVMs
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3423
>
>          testGroupsInAllVMs
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3374
>
>
>
> org.apache.geode.management.JMXMBeanReconnectDUnitTest:  5 failures
> (97.500% success rate)
>
>
>
>          serverMXBeansOnServerAreUnaffectedByLocatorCrash
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
>
>          serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
>
>          serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3406
>
>          locatorHasMemberTypeMXBeansForBothLocators
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3352
>
>          serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3457
>
>
>
> org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
> 4 failures (98.000% success rate)
>
>
>
>
>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3494
>
>
>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3465
>
>
>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3464
>
>
>  testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3379
>
>
>
> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest:
> 3 failures (98.500% success rate)
>
>
>
>          testPingWrongServer
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3402
>
>          testPingWrongServer
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3380
>
>          testPingWrongServer
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3357
>
>