You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geode.apache.org by Mark Hanson <ha...@vmware.com> on 2020/06/19 05:06:01 UTC
[INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
FYI, the build success rate was around 90% or so about two months ago.
Here are the DUnit tests that are currently failing in our tests, most likely in CI, and PR pipelines.
Please let me know if you have any questions.
Thanks,
Mark
***********************************************************************************
Overall build success rate: 78.00000% (156 of 200)
***********************************************************************************
The following test methods see failures in more than one class. There may be a failing *TestBase class
*.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived: 12 failures :
SerialWANPersistenceEnabledGatewaySenderDUnitTest: 8 failures (96.000% success rate)
SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 4 failures (98.000% success rate)
*.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived: 12 failures :
ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 5 failures (97.500% success rate)
ParallelWANPersistenceEnabledGatewaySenderDUnitTest: 7 failures (96.500% success rate)
*.testPingWrongServer: 4 failures :
ClientServerMiscSelectorDUnitTest: 3 failures (98.500% success rate)
ClientServerMiscDUnitTest: 1 failures (99.500% success rate)
***********************************************************************************
org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest: 8 failures (96.000% success rate)
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3539
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3526
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3505
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3435
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3414
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3391
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3363
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3359
org.apache.geode.management.MemberMXBeanDistributedTest: 2 failures (99.000% success rate)
testBucketCount https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
testBucketCount https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3411
org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest: 1 failures (99.500% success rate)
testRegisterInterestAndMakePrimaryWithFullRedundancy https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3381
org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest: 1 failures (99.500% success rate)
testSimpleQueryOnLocator https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3516
org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest: 1 failures (99.500% success rate)
testPingWrongServer https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3400
org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest: 3 failures (98.500% success rate)
testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3495
testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3470
testReplicatedSerialPropagationHAWithGroupTransactionEvents https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3479
org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest: 1 failures (99.500% success rate)
testDistributedDeadlockWithDLock https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3421
org.apache.geode.distributed.LocatorDUnitTest: 1 failures (99.500% success rate)
testStartTwoLocatorsWithMultiKeystoreSSL https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3398
org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 5 failures (97.500% success rate)
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3531
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3522
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3456
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest: 1 failures (99.500% success rate)
clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3433
org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest: 7 failures (96.500% success rate)
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3478
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3370
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3351
org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest: 4 failures (98.000% success rate)
testMultipleColocatedChildPRsMissingWithSequencedStart https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3493
testMissingColocatedChildPRDueToDelayedStart https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
testHierarchyOfColocatedChildPRsMissingGrandchild https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3537
testHierarchyOfColocatedChildPRsMissingGrandchild https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3424
org.apache.geode.distributed.DistributedMemberDUnitTest: 2 failures (99.000% success rate)
testGroupsInAllVMs https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3423
testGroupsInAllVMs https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3374
org.apache.geode.management.JMXMBeanReconnectDUnitTest: 5 failures (97.500% success rate)
serverMXBeansOnServerAreUnaffectedByLocatorCrash https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3406
locatorHasMemberTypeMXBeansForBothLocators https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3352
serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3457
org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 4 failures (98.000% success rate)
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3494
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3465
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3464
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3379
org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest: 3 failures (98.500% success rate)
testPingWrongServer https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3402
testPingWrongServer https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3380
testPingWrongServer https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3357
Re: Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
Posted by Alexander Murmann <am...@vmware.com>.
Oh, I am so sorry Mario! I mistakenly was under the impression the change was related to more failures. I should have looked more carefully! Thank you for the fix! ??
________________________________
From: Mario Ivanac <ma...@est.tech>
Sent: Friday, October 23, 2020 13:39
To: Alexander Murmann <am...@vmware.com>; dev@geode.apache.org <de...@geode.apache.org>
Subject: Odg: Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
Hi,
I am not aware that tests (GEODE-8172<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FGEODE-8172&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620044260%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3aT5yJlgvgg47aJSJ7SlfekvQ2Sl%2B%2FfWaOQi%2BI6R5YU%3D&reserved=0>, GEODE-8216<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FGEODE-8216&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620054258%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=LId8KQoqMGeRG1k3issagxNxepi5arCaofEjMVMGFzU%3D&reserved=0>) are still failing, after last modification of test (October 7).
BR,
Mario
________________________________
Šalje: Alexander Murmann <am...@vmware.com>
Poslano: 23. listopada 2020. 22:25
Prima: dev@geode.apache.org <de...@geode.apache.org>
Kopija: Mario Ivanac <ma...@est.tech>
Predmet: Re: Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
The failing WAN-related tests are still making up the majority of flaky tests<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-develop-mass-test-run%2Fjobs%2Fcreate-mass-test-run-report%2Fbuilds%2F40&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620064252%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=2uP8sqNXyghB3vQ3LKYXPEawlioaDxo03Vsh0GgISj4%3D&reserved=0> we are observing. It's a real issue, since the noise from flaky tests can easily hide very real bugs. That's why earlier this year several people spent a few months improving the reliability of our tests.
Mario, in July you said you were investigating this. Is that still ongoing? Do you need any help?
________________________________
From: Alexander Murmann <am...@apache.org>
Sent: Tuesday, July 14, 2020 09:00
To: dev@geode.apache.org <de...@geode.apache.org>
Cc: Alexander Murmann <am...@apache.org>
Subject: Re: Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
Thanks for looking into this, Mario!
You are probably right, that the underlying issue might have been
pre-existing and that the test is surfacing it. I am glad though that you
are investigating, because a close to 30% fail rate is a problem. Something
like this happens every once in a while and then someone has to do some
work to resolve historical problems that they hadn't planned to addres.
Thanks!
On Tue, Jul 14, 2020 at 1:06 AM Mario Ivanac <ma...@est.tech> wrote:
> Hi,
>
> after adding additional checks in failing test, now I can see that test
> are failing due to fault that some batch are distributed at stopping of GW
> sender.
> Cause of that, I suspect that this problem existed prior to this PR, but
> this PR is first to introduce test to check this.
>
> I will continue to investigate this fault, but I can not locally reproduce
> this fault, so this is slowing troubleshooting.
>
> BR,
> Mario
> ________________________________
> Šalje: Alexander Murmann <am...@apache.org>
> Poslano: 14. srpnja 2020. 1:11
> Prima: Alexander Murmann <am...@apache.org>
> Kopija: dev@geode.apache.org <de...@geode.apache.org>; Mario Ivanac
> <ma...@est.tech>
> Predmet: Re: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
>
> We continue to see these WAN tests adding a fail rate of just below 30% in
> our mass test runs
> <
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-develop-mass-test-run%2Fjobs%2Fcreate-mass-test-run-report%2Fbuilds%2F10&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=STlN5I9iM8gRqsKndH6GKwuY5cVOskAgUjNgkoJvzCw%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-develop-mass-test-run%2Fjobs%2Fcreate-mass-test-run-report%2Fbuilds%2F10&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620064252%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=HIAg2oiethvYT4cu8VdDWKRj6R1vHA0UF2%2FDjtnM4so%3D&reserved=0>
> >
> .
>
> That's a very significant fail rate that impacts our ability to get our
> code committed with confidence.
>
> Can we resolve this issue? Otherwise, I think we need to consider reverting
> GEODE-7458.
>
> On Fri, Jun 19, 2020 at 3:28 PM Alexander Murmann <am...@apache.org>
> wrote:
>
> > Looking more into this, it looks like this was introduced by the changes
> > for GEODE-7458 - "Adding additional option in gfsh command "start gateway
> > sender" to control clearing of existing queues".
> >
> > That happened about a month ago, but it's inherent to those flaky tests
> > that we discover them only after a while. Nonetheless, they become paper
> > cuts that ultimately slow us down substantially if they persist.
> >
> > @Mario Ivanac If I am correct and GEODE-7458 introduced this you were the
> > one making that change. Might you be able to take a look at making that
> > test more reliable or reverting the change?
> >
> > Thank you!
> >
> > On Fri, Jun 19, 2020 at 7:57 AM Alexander Murmann <am...@apache.org>
> > wrote:
> >
> >> Thank you so much for sharing this, Mark!
> >>
> >> It looks like there is a big cluster around WAN Gateway. Is anyone
> >> already looking into the WAN issues?
> >>
> >> On Thu, Jun 18, 2020 at 10:06 PM Mark Hanson <ha...@vmware.com>
> wrote:
> >>
> >>> FYI, the build success rate was around 90% or so about two months ago.
> >>>
> >>> Here are the DUnit tests that are currently failing in our tests, most
> >>> likely in CI, and PR pipelines.
> >>>
> >>> Please let me know if you have any questions.
> >>>
> >>> Thanks,
> >>> Mark
> >>>
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>> Overall build success rate: 78.00000% (156 of 200)
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>>
> >>>
> >>> The following test methods see failures in more than one class. There
> >>> may be a failing *TestBase class
> >>>
> >>>
> >>>
> >>>
> *.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
> >>> 12 failures :
> >>>
> >>> SerialWANPersistenceEnabledGatewaySenderDUnitTest: 8 failures
> >>> (96.000% success rate)
> >>>
> >>> SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 4 failures
> >>> (98.000% success rate)
> >>>
> >>>
> >>>
> >>>
> *.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
> >>> 12 failures :
> >>>
> >>> ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 5
> >>> failures (97.500% success rate)
> >>>
> >>> ParallelWANPersistenceEnabledGatewaySenderDUnitTest: 7 failures
> >>> (96.500% success rate)
> >>>
> >>>
> >>>
> >>> *.testPingWrongServer: 4 failures :
> >>>
> >>> ClientServerMiscSelectorDUnitTest: 3 failures (98.500% success rate)
> >>>
> >>> ClientServerMiscDUnitTest: 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest:
> >>> 8 failures (96.000% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3539&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=zcUAsAVVsUprEFeDpWryPa%2F4QTf0pQxcZ8yNpzFwvtQ%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3539&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620074248%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Yx1x5SVgNs1jIB%2BlQFiJRZxQk33SH8FXWLEc1HMhsmw%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3526&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=309OdZMeCRLDty3sTtCumrWt2T0b76JNA94EvwuB%2BSI%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3526&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620084241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ocCSd88Gmzy4P18HHEbVFGnedc9S7u7mu6DyQBD%2FYfU%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3505&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=SWeCxshemmITfDqulXqu4gQVRPzMOXrgvf3LGcXSTUs%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3505&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620084241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=QIgLFPSvS5%2BuKQxMJrOTgxK8zUAwcI0Re%2BEHlyiK7wg%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3435&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=7gzxFbHVzKgpdZ1ZU8vhDYzXzU4cxyRPxgly6dIt31k%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3435&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620094240%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=sUB5vWuucARDa3UW3%2FKclTTcKBR682TMGGxYDQL%2BbQg%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3414&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=Xa8bjxPmysZluEn6HbvQzqbccJl%2B%2FsiGIhWYGeBP%2Ba4%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3414&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620104236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aW6rxW%2FcPV2qIqLr%2Bbu0zN361aztnhJUvnr4wWTmGrE%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3391&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=O%2B0CoquKmvf5eT%2Ff%2FMLoXVuojjwk4IeeyI5dIdJugYg%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3391&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620114231%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=xZ9Mv5RTioZhYi2%2FAgWmEFW%2BVOXrsm5rxcMzRz8k%2FUs%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3363&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=AILkL%2Fhz3I%2FasCXa8fkTSB9mA26gmDxJ4fe%2FeIiS27g%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3363&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620114231%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Tt0JTrXhgXvmv3QCd698KjaGVB3xjhMEI%2FCvJ3Ky1pI%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3359&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=LF0Jr4d9jundS8cNJtWK%2FnRGVLBp4gsX8DUduuFbp0E%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3359&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620124231%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ziP%2FRsgaOGOPGnKawqF14LMevt0rtjAJjiRMFvbCBuQ%3D&reserved=0>
> >>>
> >>>
> >>>
> >>> org.apache.geode.management.MemberMXBeanDistributedTest: 2 failures
> >>> (99.000% success rate)
> >>>
> >>>
> >>>
> >>> testBucketCount
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=oYLB8lcafCMgeSVcwwuaIOdKzEzgg4ypKfS4HUrMHrU%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620134221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=pZ%2BKd95p6xMmxer%2F%2FnYheiziYf9bpuz1SOHTHXc7fyQ%3D&reserved=0>
> >>>
> >>> testBucketCount
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3411&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=O0jF6ARYoS0cIxzTNq%2BJQtHB30W45nrBj%2B5lOqu0esY%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3411&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620144215%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=4CczMenyjC2xUdA38kTtcLWeDAWMJgi%2F1HEGgIHMmd4%3D&reserved=0>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>> testRegisterInterestAndMakePrimaryWithFullRedundancy
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3381&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=la5BbwTkvNS1gFlNf3yr%2BF21L0t2z6NY6Clgwv4WGr4%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3381&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620144215%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=cZK04DW4YBb%2BcMzUmzSA%2Br4XIkOTTBwiw54cXqMgzo4%3D&reserved=0>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>> testSimpleQueryOnLocator
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3516&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=Li1jYIfu1biAC47UGz1zqJbAW6neNxgvcw1JaFeR2pk%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3516&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620154218%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=zGNco0K3djzu2%2BOY8xEjCJf5knrYNUb3TqCXj4WQpJ4%3D&reserved=0>
> >>>
> >>>
> >>>
> >>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>> testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3400&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=u7%2FQXmha26M6yX4rjRspIRDOEwQZB0NjaCxwBBkfFws%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3400&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620164209%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=2qpYFPu0E2%2FiaEQsbBYAljD%2FqT5f4wGcuwW%2BJxmMczw%3D&reserved=0>
> >>>
> >>>
> >>>
> >>> org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest: 3
> >>> failures (98.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3495&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=jA3538gndW%2BM42LYlJBMzJX9K4psvoQiYxs0761Viog%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3495&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620164209%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=oAi1vqwhmJY0hjbLTCS24VeiXmL08Hk8BvxKyYur06A%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3470&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=%2Fh6%2FPjO2f1nnZ7RzMoUnLmRM0Mq2XHlGlSblj7jlcPc%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3470&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620174205%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=x3fHa2kUK0ZkjhPAmlw7hxCJ1NwOrAYthx91%2FVTNDCw%3D&reserved=0>
> >>>
> >>> testReplicatedSerialPropagationHAWithGroupTransactionEvents
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3479&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=ikv%2FMsJVZbPhGRndMcviXfSUsYjMqob2NAZuW4GbfMw%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3479&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620184199%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=MifO3ns8FpEA8wxaepew2oLQw4UyI42n6F7Nsa0IRKw%3D&reserved=0>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>> testDistributedDeadlockWithDLock
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3421&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=ZMgFxv%2FpfiVF4xSl5C0VtJSfCVoeA5phFRJF8IS79Lk%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3421&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620194194%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=p%2FRgPB2wj76n45OE6hb85UpWykaiwhIeOOxh2qMRphU%3D&reserved=0>
> >>>
> >>>
> >>>
> >>> org.apache.geode.distributed.LocatorDUnitTest: 1 failures (99.500%
> >>> success rate)
> >>>
> >>>
> >>>
> >>> testStartTwoLocatorsWithMultiKeystoreSSL
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3398&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=Ro4ej1P%2BTLAN%2FM%2B8v9DVFbK%2B8cSDtCKdjeoYEIfbWgw%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3398&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620194194%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=tLccD38betJg%2F80Ep02rw8sB9DVnIg6w4OryPn6fg5U%3D&reserved=0>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
> >>> 5 failures (97.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3531&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=TAWaGwXo4t3%2FCQp3MewJKEn7p1rdQgp%2FVDWHqQxva4c%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3531&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620204191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=AuugEVllnP6wyKxt3dNyHPgzbUkRMqu%2BBmTVqcn0yjo%3D&reserved=0>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3522&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=lApVu2gdSk38MjHjZlWZhPWyQ5fOez2Sma5WmYcNE38%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3522&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620214187%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=cJBMFgDI75TzgzCjiKEFO88BxJ7azzzBWy28Bc7wUFE%3D&reserved=0>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3456&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=6gC%2BqQusVFP5JqeHkxtSvk8%2FWCRIEg90ZeiWomJ1NrE%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3456&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620224183%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=jrRKM3HtHxXCALs4NuFFvYYqdkdZv0BV3wJdOtEq1as%3D&reserved=0>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=DJ9zA4EJyrjd01eidiCznYn%2FOAVtVfzYzxsfapCGUD4%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620224183%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=QJT8cbe7%2BNwSmeUwbjPheyyNXlQ9tlLaQoiBiUOJkMg%3D&reserved=0>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=%2FvRCNF0K8V9LfYTsdDxRniadtdXVuV7cZZZsmQTydw0%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620234177%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=DGkaO1JMH1Wuy9oAzj%2FK6BgIkanuqUiBdca632wbBkY%3D&reserved=0>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>> clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3433&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=rrg9pcWvSZgJ106rjbsS8wpCBhX20er1hdNhW0qGrZ4%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3433&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620244174%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9gJbgt5nxPNmrdtMfgRY6hgMyzWQoofBRq3JuYCHqNg%3D&reserved=0>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
> >>> 7 failures (96.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3478&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=7R6DvYToK1cwdTfBK93HA9y83qw9OKd4Kj%2BbEe%2F8Yd4%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3478&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620244174%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=bUdp9ffXucXzpzVL8GNg0TR%2FfI37mnjfjY6NfyLf7hY%3D&reserved=0>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=1RXrV5Zw0kQSLDrPLu45DlWy4ZR%2B2hs7rfyk8BAFeG4%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620254173%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=S131cef%2BrDdiBOiCCquxCFNqY0vSWVGglca3sF3U7eY%3D&reserved=0>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=DJ9zA4EJyrjd01eidiCznYn%2FOAVtVfzYzxsfapCGUD4%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620264166%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Hs1z2YcYJFGDq3W4jGIzpJ%2Bj4vkdPl1vhSZTbzoEGFo%3D&reserved=0>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=vF7zJq%2BZnOIa1W6oVrfKRJlV6nFIBitzjCVPPCgc6CQ%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620274159%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=U9t4sCOmgk%2BFIMPBQQO0bLWnFlSw8i9tSgwPV1wvyNQ%3D&reserved=0>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=%2FvRCNF0K8V9LfYTsdDxRniadtdXVuV7cZZZsmQTydw0%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620274159%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5fSvQHKecSSQUEzmUX28BZJ3eH6rhX9YgX23Nk%2BcH7I%3D&reserved=0>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3370&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=WDNL8hapq0GmB%2BrxSxFYJ0c4nArV8k15m79OkwrtASY%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3370&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620284153%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Qa7QoJYcb9%2FQC0wa%2BcXULfMIxunWtP5ekGkyHVuK5ds%3D&reserved=0>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3351&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=8%2B%2FBMm9eeXPcjAL4Hj94aduYQUyDrJVgPqB5%2FSqZ4kY%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3351&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620294148%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NPUcHLyaASzyvuaQNQ0P34dieCmYnxDc4MFo15CakPE%3D&reserved=0>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest:
> >>> 4 failures (98.000% success rate)
> >>>
> >>>
> >>>
> >>> testMultipleColocatedChildPRsMissingWithSequencedStart
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3493&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=l7RlB5fYmzwGJvYuy5QFIPWJd5mi1o5BAlCYff0XeDQ%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3493&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620294148%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3qS2Cc7w6X0zBAORj1Mp%2FCEUTCMgH%2FoPHJ3L%2Fxrn5s0%3D&reserved=0>
> >>>
> >>> testMissingColocatedChildPRDueToDelayedStart
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=SgLx2XahBD8x2Cez3aK%2BhShvVc%2BP3SFmMfeqB3pdHLM%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620304147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=AvvWFA0Fs98ke7q%2Bf3dkO7%2FjtmS%2BXtGwtad%2Bndo99dY%3D&reserved=0>
> >>>
> >>> testHierarchyOfColocatedChildPRsMissingGrandchild
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3537&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=3Y5qtB5P6Gz8ey%2F6eeEOOGdLptT6LVIBVWLpF4lrRxs%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3537&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620314143%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=g5gEMOt0qYxWqJCyPAvcXymWUlubjR%2FpWkn5dsw2Whk%3D&reserved=0>
> >>>
> >>> testHierarchyOfColocatedChildPRsMissingGrandchild
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3424&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=n6qxZzVKFYqiTY7kUw%2BoQfZKE2aUvbBXs6cXAcqBZaw%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3424&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620314143%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=E6YK%2BiyiSjG3%2Fre3vOMnquGSa9B7InrBaFeQ8QRX8%2Fs%3D&reserved=0>
> >>>
> >>>
> >>>
> >>> org.apache.geode.distributed.DistributedMemberDUnitTest: 2 failures
> >>> (99.000% success rate)
> >>>
> >>>
> >>>
> >>> testGroupsInAllVMs
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3423&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=prw%2BEBBIgOH8BWwD5SBpvVd%2BvdQSh%2FOJf8Xk6zPZ7yk%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3423&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620324140%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=TeY3rxpLjMTgATjZa9Trriams5ivBhDy4wAatK581t0%3D&reserved=0>
> >>>
> >>> testGroupsInAllVMs
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3374&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=g2PQp%2FiMZEVjc4zSOxNhks48UTPdEAVGyhdJQueipuQ%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3374&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620334136%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=7ak%2BCVW2%2B6h%2BG2I7qEEaZX5d%2BtIYNhvHzWUXsMuH2t0%3D&reserved=0>
> >>>
> >>>
> >>>
> >>> org.apache.geode.management.JMXMBeanReconnectDUnitTest: 5 failures
> >>> (97.500% success rate)
> >>>
> >>>
> >>>
> >>> serverMXBeansOnServerAreUnaffectedByLocatorCrash
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=zVXQeZRlv6qJGvlpOFvJIt5WDzvSAq2ohyQ1DRZ1xWE%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620344130%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=VXpU8FU%2FYMgh629IrL3kTUVn%2FrF0CyWFw9fJTGsF0eA%3D&reserved=0>
> >>>
> >>>
> >>> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=t5BIGh1%2BcB6SKdUviko%2FghBDtJIrmEUVMGQVXdEhHbA%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620344130%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=EfTi%2FI4Ntm79kjf%2F8a1F4kS5uP9FPSs%2BL1T4%2FdqODU0%3D&reserved=0>
> >>>
> >>>
> >>> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3406&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=GyvOWUusGUi48vS6wWA6GrJN3J5aAMbqeXxy2VevLXU%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3406&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620354126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=y%2BwpZcXwC32vupu%2FIq0bZp8eNis2hiKHhWIzfKsARVU%3D&reserved=0>
> >>>
> >>> locatorHasMemberTypeMXBeansForBothLocators
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3352&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=N261EBctQXEW0FR0XTTHd4w6nmrEkj%2F%2FLd8e14m0cnw%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3352&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620364119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=58GwB5FQDtBQLO2Omhm9EitUT%2F883l3QAn0eicEx02I%3D&reserved=0>
> >>>
> >>> serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3457&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=QNtL7hTrfxZHkLfKGx7P01m4quS6%2FhkkjyYdhA7J8bA%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3457&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620374117%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=sIwqTvAdPXy4ci9b8HxCuj0UrVH91ELYg4ycIzyILbw%3D&reserved=0>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
> >>> 4 failures (98.000% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3494&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=TPAyTUGYWUv%2FifU58XondaeGNJBXmB8sskEp0J6R8l4%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3494&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620374117%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dVdHUnV25Jup8CKlR0u6FhJRtThsECIrg4NGCjN9Dys%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3465&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=gVTwnIWWDDjOhkfyM4SzYegMuQjyaXDg7sFkzkhUyJA%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3465&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620384113%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=8fOk97yNQRmtZUnqCl5UMKE4PO%2Fg1wwnPgTOgzj6u3s%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3464&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=iErMnLzSVjZpkZO%2FJ%2BETC0QbO0neLHNlA41OkXWWBuk%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3464&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620394107%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=wZzE52SfCYo1wLJ7DrGpmsOxLVM4lxxo%2BHikn9m11vc%3D&reserved=0>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3379&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=GptTVqXhyQlHTByVbEoQJtjF%2FWc4xlC2KLHHZ3qhbrw%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3379&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620394107%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=MRJoOCY1QnTyvu214tNqy6cX9r4ZpsFKmxs6Jw6YJZ8%3D&reserved=0>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest:
> >>> 3 failures (98.500% success rate)
> >>>
> >>>
> >>>
> >>> testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3402&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3402&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4>d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=S9EbS96H9rYgoCfTtXXTVbcMZAIBZ%2BlIbQgElXKzX98%3D&reserved=0
> >>>
> >>> testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3380&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297271685&sdata=4OQQFRPSwtx3QcENDyVCUsoMJd0K%2Bwl5o759cyZfsYw%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3380&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620404104%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FDqkFd5z5KBiV80FCslKd6NUCs%2BDl7EEHaKGmAEav%2BU%3D&reserved=0>
> >>>
> >>> testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3357&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297271685&sdata=XjIc4NZPwT4HlRnEL9zfWcB5AlxLiO%2B9C9jf77TJ0Dg%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3357&data=04%7C01%7Camurmann%40vmware.com%7Cb725e3ba5535457e372e08d87793b4da%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637390823620414103%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=w2hTfsWj2n5HfdOxkQA%2FQltIpZfwc1JoOMHdDdq2UCw%3D&reserved=0>
> >>>
> >>>
>
Odg: Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8
passes
Posted by Mario Ivanac <ma...@est.tech>.
Hi,
I am not aware that tests (GEODE-8172<https://issues.apache.org/jira/browse/GEODE-8172>, GEODE-8216<https://issues.apache.org/jira/browse/GEODE-8216>) are still failing, after last modification of test (October 7).
BR,
Mario
________________________________
Šalje: Alexander Murmann <am...@vmware.com>
Poslano: 23. listopada 2020. 22:25
Prima: dev@geode.apache.org <de...@geode.apache.org>
Kopija: Mario Ivanac <ma...@est.tech>
Predmet: Re: Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
The failing WAN-related tests are still making up the majority of flaky tests<https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-mass-test-run/jobs/create-mass-test-run-report/builds/40> we are observing. It's a real issue, since the noise from flaky tests can easily hide very real bugs. That's why earlier this year several people spent a few months improving the reliability of our tests.
Mario, in July you said you were investigating this. Is that still ongoing? Do you need any help?
________________________________
From: Alexander Murmann <am...@apache.org>
Sent: Tuesday, July 14, 2020 09:00
To: dev@geode.apache.org <de...@geode.apache.org>
Cc: Alexander Murmann <am...@apache.org>
Subject: Re: Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
Thanks for looking into this, Mario!
You are probably right, that the underlying issue might have been
pre-existing and that the test is surfacing it. I am glad though that you
are investigating, because a close to 30% fail rate is a problem. Something
like this happens every once in a while and then someone has to do some
work to resolve historical problems that they hadn't planned to addres.
Thanks!
On Tue, Jul 14, 2020 at 1:06 AM Mario Ivanac <ma...@est.tech> wrote:
> Hi,
>
> after adding additional checks in failing test, now I can see that test
> are failing due to fault that some batch are distributed at stopping of GW
> sender.
> Cause of that, I suspect that this problem existed prior to this PR, but
> this PR is first to introduce test to check this.
>
> I will continue to investigate this fault, but I can not locally reproduce
> this fault, so this is slowing troubleshooting.
>
> BR,
> Mario
> ________________________________
> Šalje: Alexander Murmann <am...@apache.org>
> Poslano: 14. srpnja 2020. 1:11
> Prima: Alexander Murmann <am...@apache.org>
> Kopija: dev@geode.apache.org <de...@geode.apache.org>; Mario Ivanac
> <ma...@est.tech>
> Predmet: Re: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
>
> We continue to see these WAN tests adding a fail rate of just below 30% in
> our mass test runs
> <
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-develop-mass-test-run%2Fjobs%2Fcreate-mass-test-run-report%2Fbuilds%2F10&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=STlN5I9iM8gRqsKndH6GKwuY5cVOskAgUjNgkoJvzCw%3D&reserved=0
> >
> .
>
> That's a very significant fail rate that impacts our ability to get our
> code committed with confidence.
>
> Can we resolve this issue? Otherwise, I think we need to consider reverting
> GEODE-7458.
>
> On Fri, Jun 19, 2020 at 3:28 PM Alexander Murmann <am...@apache.org>
> wrote:
>
> > Looking more into this, it looks like this was introduced by the changes
> > for GEODE-7458 - "Adding additional option in gfsh command "start gateway
> > sender" to control clearing of existing queues".
> >
> > That happened about a month ago, but it's inherent to those flaky tests
> > that we discover them only after a while. Nonetheless, they become paper
> > cuts that ultimately slow us down substantially if they persist.
> >
> > @Mario Ivanac If I am correct and GEODE-7458 introduced this you were the
> > one making that change. Might you be able to take a look at making that
> > test more reliable or reverting the change?
> >
> > Thank you!
> >
> > On Fri, Jun 19, 2020 at 7:57 AM Alexander Murmann <am...@apache.org>
> > wrote:
> >
> >> Thank you so much for sharing this, Mark!
> >>
> >> It looks like there is a big cluster around WAN Gateway. Is anyone
> >> already looking into the WAN issues?
> >>
> >> On Thu, Jun 18, 2020 at 10:06 PM Mark Hanson <ha...@vmware.com>
> wrote:
> >>
> >>> FYI, the build success rate was around 90% or so about two months ago.
> >>>
> >>> Here are the DUnit tests that are currently failing in our tests, most
> >>> likely in CI, and PR pipelines.
> >>>
> >>> Please let me know if you have any questions.
> >>>
> >>> Thanks,
> >>> Mark
> >>>
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>> Overall build success rate: 78.00000% (156 of 200)
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>>
> >>>
> >>> The following test methods see failures in more than one class. There
> >>> may be a failing *TestBase class
> >>>
> >>>
> >>>
> >>>
> *.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
> >>> 12 failures :
> >>>
> >>> SerialWANPersistenceEnabledGatewaySenderDUnitTest: 8 failures
> >>> (96.000% success rate)
> >>>
> >>> SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 4 failures
> >>> (98.000% success rate)
> >>>
> >>>
> >>>
> >>>
> *.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
> >>> 12 failures :
> >>>
> >>> ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 5
> >>> failures (97.500% success rate)
> >>>
> >>> ParallelWANPersistenceEnabledGatewaySenderDUnitTest: 7 failures
> >>> (96.500% success rate)
> >>>
> >>>
> >>>
> >>> *.testPingWrongServer: 4 failures :
> >>>
> >>> ClientServerMiscSelectorDUnitTest: 3 failures (98.500% success rate)
> >>>
> >>> ClientServerMiscDUnitTest: 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest:
> >>> 8 failures (96.000% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3539&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=zcUAsAVVsUprEFeDpWryPa%2F4QTf0pQxcZ8yNpzFwvtQ%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3526&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=309OdZMeCRLDty3sTtCumrWt2T0b76JNA94EvwuB%2BSI%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3505&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=SWeCxshemmITfDqulXqu4gQVRPzMOXrgvf3LGcXSTUs%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3435&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=7gzxFbHVzKgpdZ1ZU8vhDYzXzU4cxyRPxgly6dIt31k%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3414&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=Xa8bjxPmysZluEn6HbvQzqbccJl%2B%2FsiGIhWYGeBP%2Ba4%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3391&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=O%2B0CoquKmvf5eT%2Ff%2FMLoXVuojjwk4IeeyI5dIdJugYg%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3363&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=AILkL%2Fhz3I%2FasCXa8fkTSB9mA26gmDxJ4fe%2FeIiS27g%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3359&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=LF0Jr4d9jundS8cNJtWK%2FnRGVLBp4gsX8DUduuFbp0E%3D&reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.management.MemberMXBeanDistributedTest: 2 failures
> >>> (99.000% success rate)
> >>>
> >>>
> >>>
> >>> testBucketCount
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=oYLB8lcafCMgeSVcwwuaIOdKzEzgg4ypKfS4HUrMHrU%3D&reserved=0
> >>>
> >>> testBucketCount
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3411&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=O0jF6ARYoS0cIxzTNq%2BJQtHB30W45nrBj%2B5lOqu0esY%3D&reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>> testRegisterInterestAndMakePrimaryWithFullRedundancy
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3381&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=la5BbwTkvNS1gFlNf3yr%2BF21L0t2z6NY6Clgwv4WGr4%3D&reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>> testSimpleQueryOnLocator
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3516&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=Li1jYIfu1biAC47UGz1zqJbAW6neNxgvcw1JaFeR2pk%3D&reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>> testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3400&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=u7%2FQXmha26M6yX4rjRspIRDOEwQZB0NjaCxwBBkfFws%3D&reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest: 3
> >>> failures (98.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3495&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=jA3538gndW%2BM42LYlJBMzJX9K4psvoQiYxs0761Viog%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3470&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=%2Fh6%2FPjO2f1nnZ7RzMoUnLmRM0Mq2XHlGlSblj7jlcPc%3D&reserved=0
> >>>
> >>> testReplicatedSerialPropagationHAWithGroupTransactionEvents
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3479&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=ikv%2FMsJVZbPhGRndMcviXfSUsYjMqob2NAZuW4GbfMw%3D&reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>> testDistributedDeadlockWithDLock
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3421&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=ZMgFxv%2FpfiVF4xSl5C0VtJSfCVoeA5phFRJF8IS79Lk%3D&reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.distributed.LocatorDUnitTest: 1 failures (99.500%
> >>> success rate)
> >>>
> >>>
> >>>
> >>> testStartTwoLocatorsWithMultiKeystoreSSL
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3398&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=Ro4ej1P%2BTLAN%2FM%2B8v9DVFbK%2B8cSDtCKdjeoYEIfbWgw%3D&reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
> >>> 5 failures (97.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3531&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=TAWaGwXo4t3%2FCQp3MewJKEn7p1rdQgp%2FVDWHqQxva4c%3D&reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3522&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=lApVu2gdSk38MjHjZlWZhPWyQ5fOez2Sma5WmYcNE38%3D&reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3456&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=6gC%2BqQusVFP5JqeHkxtSvk8%2FWCRIEg90ZeiWomJ1NrE%3D&reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=DJ9zA4EJyrjd01eidiCznYn%2FOAVtVfzYzxsfapCGUD4%3D&reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=%2FvRCNF0K8V9LfYTsdDxRniadtdXVuV7cZZZsmQTydw0%3D&reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>> clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3433&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=rrg9pcWvSZgJ106rjbsS8wpCBhX20er1hdNhW0qGrZ4%3D&reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
> >>> 7 failures (96.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3478&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=7R6DvYToK1cwdTfBK93HA9y83qw9OKd4Kj%2BbEe%2F8Yd4%3D&reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=1RXrV5Zw0kQSLDrPLu45DlWy4ZR%2B2hs7rfyk8BAFeG4%3D&reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=DJ9zA4EJyrjd01eidiCznYn%2FOAVtVfzYzxsfapCGUD4%3D&reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=vF7zJq%2BZnOIa1W6oVrfKRJlV6nFIBitzjCVPPCgc6CQ%3D&reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=%2FvRCNF0K8V9LfYTsdDxRniadtdXVuV7cZZZsmQTydw0%3D&reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3370&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=WDNL8hapq0GmB%2BrxSxFYJ0c4nArV8k15m79OkwrtASY%3D&reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3351&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=8%2B%2FBMm9eeXPcjAL4Hj94aduYQUyDrJVgPqB5%2FSqZ4kY%3D&reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest:
> >>> 4 failures (98.000% success rate)
> >>>
> >>>
> >>>
> >>> testMultipleColocatedChildPRsMissingWithSequencedStart
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3493&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=l7RlB5fYmzwGJvYuy5QFIPWJd5mi1o5BAlCYff0XeDQ%3D&reserved=0
> >>>
> >>> testMissingColocatedChildPRDueToDelayedStart
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=SgLx2XahBD8x2Cez3aK%2BhShvVc%2BP3SFmMfeqB3pdHLM%3D&reserved=0
> >>>
> >>> testHierarchyOfColocatedChildPRsMissingGrandchild
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3537&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=3Y5qtB5P6Gz8ey%2F6eeEOOGdLptT6LVIBVWLpF4lrRxs%3D&reserved=0
> >>>
> >>> testHierarchyOfColocatedChildPRsMissingGrandchild
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3424&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=n6qxZzVKFYqiTY7kUw%2BoQfZKE2aUvbBXs6cXAcqBZaw%3D&reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.distributed.DistributedMemberDUnitTest: 2 failures
> >>> (99.000% success rate)
> >>>
> >>>
> >>>
> >>> testGroupsInAllVMs
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3423&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=prw%2BEBBIgOH8BWwD5SBpvVd%2BvdQSh%2FOJf8Xk6zPZ7yk%3D&reserved=0
> >>>
> >>> testGroupsInAllVMs
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3374&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=g2PQp%2FiMZEVjc4zSOxNhks48UTPdEAVGyhdJQueipuQ%3D&reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.management.JMXMBeanReconnectDUnitTest: 5 failures
> >>> (97.500% success rate)
> >>>
> >>>
> >>>
> >>> serverMXBeansOnServerAreUnaffectedByLocatorCrash
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=zVXQeZRlv6qJGvlpOFvJIt5WDzvSAq2ohyQ1DRZ1xWE%3D&reserved=0
> >>>
> >>>
> >>> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=t5BIGh1%2BcB6SKdUviko%2FghBDtJIrmEUVMGQVXdEhHbA%3D&reserved=0
> >>>
> >>>
> >>> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3406&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=GyvOWUusGUi48vS6wWA6GrJN3J5aAMbqeXxy2VevLXU%3D&reserved=0
> >>>
> >>> locatorHasMemberTypeMXBeansForBothLocators
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3352&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=N261EBctQXEW0FR0XTTHd4w6nmrEkj%2F%2FLd8e14m0cnw%3D&reserved=0
> >>>
> >>> serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3457&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=QNtL7hTrfxZHkLfKGx7P01m4quS6%2FhkkjyYdhA7J8bA%3D&reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
> >>> 4 failures (98.000% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3494&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=TPAyTUGYWUv%2FifU58XondaeGNJBXmB8sskEp0J6R8l4%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3465&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=gVTwnIWWDDjOhkfyM4SzYegMuQjyaXDg7sFkzkhUyJA%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3464&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=iErMnLzSVjZpkZO%2FJ%2BETC0QbO0neLHNlA41OkXWWBuk%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3379&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=GptTVqXhyQlHTByVbEoQJtjF%2FWc4xlC2KLHHZ3qhbrw%3D&reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest:
> >>> 3 failures (98.500% success rate)
> >>>
> >>>
> >>>
> >>> testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3402&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=S9EbS96H9rYgoCfTtXXTVbcMZAIBZ%2BlIbQgElXKzX98%3D&reserved=0
> >>>
> >>> testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3380&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297271685&sdata=4OQQFRPSwtx3QcENDyVCUsoMJd0K%2Bwl5o759cyZfsYw%3D&reserved=0
> >>>
> >>> testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3357&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297271685&sdata=XjIc4NZPwT4HlRnEL9zfWcB5AlxLiO%2B9C9jf77TJ0Dg%3D&reserved=0
> >>>
> >>>
>
Re: Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
Posted by Alexander Murmann <am...@vmware.com>.
The failing WAN-related tests are still making up the majority of flaky tests<https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-mass-test-run/jobs/create-mass-test-run-report/builds/40> we are observing. It's a real issue, since the noise from flaky tests can easily hide very real bugs. That's why earlier this year several people spent a few months improving the reliability of our tests.
Mario, in July you said you were investigating this. Is that still ongoing? Do you need any help?
________________________________
From: Alexander Murmann <am...@apache.org>
Sent: Tuesday, July 14, 2020 09:00
To: dev@geode.apache.org <de...@geode.apache.org>
Cc: Alexander Murmann <am...@apache.org>
Subject: Re: Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
Thanks for looking into this, Mario!
You are probably right, that the underlying issue might have been
pre-existing and that the test is surfacing it. I am glad though that you
are investigating, because a close to 30% fail rate is a problem. Something
like this happens every once in a while and then someone has to do some
work to resolve historical problems that they hadn't planned to addres.
Thanks!
On Tue, Jul 14, 2020 at 1:06 AM Mario Ivanac <ma...@est.tech> wrote:
> Hi,
>
> after adding additional checks in failing test, now I can see that test
> are failing due to fault that some batch are distributed at stopping of GW
> sender.
> Cause of that, I suspect that this problem existed prior to this PR, but
> this PR is first to introduce test to check this.
>
> I will continue to investigate this fault, but I can not locally reproduce
> this fault, so this is slowing troubleshooting.
>
> BR,
> Mario
> ________________________________
> Šalje: Alexander Murmann <am...@apache.org>
> Poslano: 14. srpnja 2020. 1:11
> Prima: Alexander Murmann <am...@apache.org>
> Kopija: dev@geode.apache.org <de...@geode.apache.org>; Mario Ivanac
> <ma...@est.tech>
> Predmet: Re: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
>
> We continue to see these WAN tests adding a fail rate of just below 30% in
> our mass test runs
> <
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-develop-mass-test-run%2Fjobs%2Fcreate-mass-test-run-report%2Fbuilds%2F10&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=STlN5I9iM8gRqsKndH6GKwuY5cVOskAgUjNgkoJvzCw%3D&reserved=0
> >
> .
>
> That's a very significant fail rate that impacts our ability to get our
> code committed with confidence.
>
> Can we resolve this issue? Otherwise, I think we need to consider reverting
> GEODE-7458.
>
> On Fri, Jun 19, 2020 at 3:28 PM Alexander Murmann <am...@apache.org>
> wrote:
>
> > Looking more into this, it looks like this was introduced by the changes
> > for GEODE-7458 - "Adding additional option in gfsh command "start gateway
> > sender" to control clearing of existing queues".
> >
> > That happened about a month ago, but it's inherent to those flaky tests
> > that we discover them only after a while. Nonetheless, they become paper
> > cuts that ultimately slow us down substantially if they persist.
> >
> > @Mario Ivanac If I am correct and GEODE-7458 introduced this you were the
> > one making that change. Might you be able to take a look at making that
> > test more reliable or reverting the change?
> >
> > Thank you!
> >
> > On Fri, Jun 19, 2020 at 7:57 AM Alexander Murmann <am...@apache.org>
> > wrote:
> >
> >> Thank you so much for sharing this, Mark!
> >>
> >> It looks like there is a big cluster around WAN Gateway. Is anyone
> >> already looking into the WAN issues?
> >>
> >> On Thu, Jun 18, 2020 at 10:06 PM Mark Hanson <ha...@vmware.com>
> wrote:
> >>
> >>> FYI, the build success rate was around 90% or so about two months ago.
> >>>
> >>> Here are the DUnit tests that are currently failing in our tests, most
> >>> likely in CI, and PR pipelines.
> >>>
> >>> Please let me know if you have any questions.
> >>>
> >>> Thanks,
> >>> Mark
> >>>
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>> Overall build success rate: 78.00000% (156 of 200)
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>>
> >>>
> >>> The following test methods see failures in more than one class. There
> >>> may be a failing *TestBase class
> >>>
> >>>
> >>>
> >>>
> *.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
> >>> 12 failures :
> >>>
> >>> SerialWANPersistenceEnabledGatewaySenderDUnitTest: 8 failures
> >>> (96.000% success rate)
> >>>
> >>> SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 4 failures
> >>> (98.000% success rate)
> >>>
> >>>
> >>>
> >>>
> *.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
> >>> 12 failures :
> >>>
> >>> ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 5
> >>> failures (97.500% success rate)
> >>>
> >>> ParallelWANPersistenceEnabledGatewaySenderDUnitTest: 7 failures
> >>> (96.500% success rate)
> >>>
> >>>
> >>>
> >>> *.testPingWrongServer: 4 failures :
> >>>
> >>> ClientServerMiscSelectorDUnitTest: 3 failures (98.500% success rate)
> >>>
> >>> ClientServerMiscDUnitTest: 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest:
> >>> 8 failures (96.000% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3539&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=zcUAsAVVsUprEFeDpWryPa%2F4QTf0pQxcZ8yNpzFwvtQ%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3526&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=309OdZMeCRLDty3sTtCumrWt2T0b76JNA94EvwuB%2BSI%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3505&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=SWeCxshemmITfDqulXqu4gQVRPzMOXrgvf3LGcXSTUs%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3435&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=7gzxFbHVzKgpdZ1ZU8vhDYzXzU4cxyRPxgly6dIt31k%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3414&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=Xa8bjxPmysZluEn6HbvQzqbccJl%2B%2FsiGIhWYGeBP%2Ba4%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3391&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=O%2B0CoquKmvf5eT%2Ff%2FMLoXVuojjwk4IeeyI5dIdJugYg%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3363&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=AILkL%2Fhz3I%2FasCXa8fkTSB9mA26gmDxJ4fe%2FeIiS27g%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3359&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=LF0Jr4d9jundS8cNJtWK%2FnRGVLBp4gsX8DUduuFbp0E%3D&reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.management.MemberMXBeanDistributedTest: 2 failures
> >>> (99.000% success rate)
> >>>
> >>>
> >>>
> >>> testBucketCount
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=oYLB8lcafCMgeSVcwwuaIOdKzEzgg4ypKfS4HUrMHrU%3D&reserved=0
> >>>
> >>> testBucketCount
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3411&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=O0jF6ARYoS0cIxzTNq%2BJQtHB30W45nrBj%2B5lOqu0esY%3D&reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>> testRegisterInterestAndMakePrimaryWithFullRedundancy
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3381&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=la5BbwTkvNS1gFlNf3yr%2BF21L0t2z6NY6Clgwv4WGr4%3D&reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>> testSimpleQueryOnLocator
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3516&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=Li1jYIfu1biAC47UGz1zqJbAW6neNxgvcw1JaFeR2pk%3D&reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>> testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3400&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=u7%2FQXmha26M6yX4rjRspIRDOEwQZB0NjaCxwBBkfFws%3D&reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest: 3
> >>> failures (98.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3495&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=jA3538gndW%2BM42LYlJBMzJX9K4psvoQiYxs0761Viog%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3470&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=%2Fh6%2FPjO2f1nnZ7RzMoUnLmRM0Mq2XHlGlSblj7jlcPc%3D&reserved=0
> >>>
> >>> testReplicatedSerialPropagationHAWithGroupTransactionEvents
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3479&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=ikv%2FMsJVZbPhGRndMcviXfSUsYjMqob2NAZuW4GbfMw%3D&reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>> testDistributedDeadlockWithDLock
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3421&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=ZMgFxv%2FpfiVF4xSl5C0VtJSfCVoeA5phFRJF8IS79Lk%3D&reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.distributed.LocatorDUnitTest: 1 failures (99.500%
> >>> success rate)
> >>>
> >>>
> >>>
> >>> testStartTwoLocatorsWithMultiKeystoreSSL
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3398&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=Ro4ej1P%2BTLAN%2FM%2B8v9DVFbK%2B8cSDtCKdjeoYEIfbWgw%3D&reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
> >>> 5 failures (97.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3531&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=TAWaGwXo4t3%2FCQp3MewJKEn7p1rdQgp%2FVDWHqQxva4c%3D&reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3522&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=lApVu2gdSk38MjHjZlWZhPWyQ5fOez2Sma5WmYcNE38%3D&reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3456&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=6gC%2BqQusVFP5JqeHkxtSvk8%2FWCRIEg90ZeiWomJ1NrE%3D&reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=DJ9zA4EJyrjd01eidiCznYn%2FOAVtVfzYzxsfapCGUD4%3D&reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=%2FvRCNF0K8V9LfYTsdDxRniadtdXVuV7cZZZsmQTydw0%3D&reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>> clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3433&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=rrg9pcWvSZgJ106rjbsS8wpCBhX20er1hdNhW0qGrZ4%3D&reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
> >>> 7 failures (96.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3478&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=7R6DvYToK1cwdTfBK93HA9y83qw9OKd4Kj%2BbEe%2F8Yd4%3D&reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=1RXrV5Zw0kQSLDrPLu45DlWy4ZR%2B2hs7rfyk8BAFeG4%3D&reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=DJ9zA4EJyrjd01eidiCznYn%2FOAVtVfzYzxsfapCGUD4%3D&reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=vF7zJq%2BZnOIa1W6oVrfKRJlV6nFIBitzjCVPPCgc6CQ%3D&reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=%2FvRCNF0K8V9LfYTsdDxRniadtdXVuV7cZZZsmQTydw0%3D&reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3370&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=WDNL8hapq0GmB%2BrxSxFYJ0c4nArV8k15m79OkwrtASY%3D&reserved=0
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3351&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=8%2B%2FBMm9eeXPcjAL4Hj94aduYQUyDrJVgPqB5%2FSqZ4kY%3D&reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest:
> >>> 4 failures (98.000% success rate)
> >>>
> >>>
> >>>
> >>> testMultipleColocatedChildPRsMissingWithSequencedStart
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3493&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=l7RlB5fYmzwGJvYuy5QFIPWJd5mi1o5BAlCYff0XeDQ%3D&reserved=0
> >>>
> >>> testMissingColocatedChildPRDueToDelayedStart
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=SgLx2XahBD8x2Cez3aK%2BhShvVc%2BP3SFmMfeqB3pdHLM%3D&reserved=0
> >>>
> >>> testHierarchyOfColocatedChildPRsMissingGrandchild
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3537&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=3Y5qtB5P6Gz8ey%2F6eeEOOGdLptT6LVIBVWLpF4lrRxs%3D&reserved=0
> >>>
> >>> testHierarchyOfColocatedChildPRsMissingGrandchild
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3424&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=n6qxZzVKFYqiTY7kUw%2BoQfZKE2aUvbBXs6cXAcqBZaw%3D&reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.distributed.DistributedMemberDUnitTest: 2 failures
> >>> (99.000% success rate)
> >>>
> >>>
> >>>
> >>> testGroupsInAllVMs
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3423&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=prw%2BEBBIgOH8BWwD5SBpvVd%2BvdQSh%2FOJf8Xk6zPZ7yk%3D&reserved=0
> >>>
> >>> testGroupsInAllVMs
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3374&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=g2PQp%2FiMZEVjc4zSOxNhks48UTPdEAVGyhdJQueipuQ%3D&reserved=0
> >>>
> >>>
> >>>
> >>> org.apache.geode.management.JMXMBeanReconnectDUnitTest: 5 failures
> >>> (97.500% success rate)
> >>>
> >>>
> >>>
> >>> serverMXBeansOnServerAreUnaffectedByLocatorCrash
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=zVXQeZRlv6qJGvlpOFvJIt5WDzvSAq2ohyQ1DRZ1xWE%3D&reserved=0
> >>>
> >>>
> >>> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=t5BIGh1%2BcB6SKdUviko%2FghBDtJIrmEUVMGQVXdEhHbA%3D&reserved=0
> >>>
> >>>
> >>> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3406&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=GyvOWUusGUi48vS6wWA6GrJN3J5aAMbqeXxy2VevLXU%3D&reserved=0
> >>>
> >>> locatorHasMemberTypeMXBeansForBothLocators
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3352&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=N261EBctQXEW0FR0XTTHd4w6nmrEkj%2F%2FLd8e14m0cnw%3D&reserved=0
> >>>
> >>> serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3457&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=QNtL7hTrfxZHkLfKGx7P01m4quS6%2FhkkjyYdhA7J8bA%3D&reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
> >>> 4 failures (98.000% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3494&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=TPAyTUGYWUv%2FifU58XondaeGNJBXmB8sskEp0J6R8l4%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3465&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=gVTwnIWWDDjOhkfyM4SzYegMuQjyaXDg7sFkzkhUyJA%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3464&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=iErMnLzSVjZpkZO%2FJ%2BETC0QbO0neLHNlA41OkXWWBuk%3D&reserved=0
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3379&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=GptTVqXhyQlHTByVbEoQJtjF%2FWc4xlC2KLHHZ3qhbrw%3D&reserved=0
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest:
> >>> 3 failures (98.500% success rate)
> >>>
> >>>
> >>>
> >>> testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3402&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=S9EbS96H9rYgoCfTtXXTVbcMZAIBZ%2BlIbQgElXKzX98%3D&reserved=0
> >>>
> >>> testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3380&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297271685&sdata=4OQQFRPSwtx3QcENDyVCUsoMJd0K%2Bwl5o759cyZfsYw%3D&reserved=0
> >>>
> >>> testPingWrongServer
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3357&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297271685&sdata=XjIc4NZPwT4HlRnEL9zfWcB5AlxLiO%2B9C9jf77TJ0Dg%3D&reserved=0
> >>>
> >>>
>
Re: Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
Posted by Alexander Murmann <am...@apache.org>.
Thanks for looking into this, Mario!
You are probably right, that the underlying issue might have been
pre-existing and that the test is surfacing it. I am glad though that you
are investigating, because a close to 30% fail rate is a problem. Something
like this happens every once in a while and then someone has to do some
work to resolve historical problems that they hadn't planned to addres.
Thanks!
On Tue, Jul 14, 2020 at 1:06 AM Mario Ivanac <ma...@est.tech> wrote:
> Hi,
>
> after adding additional checks in failing test, now I can see that test
> are failing due to fault that some batch are distributed at stopping of GW
> sender.
> Cause of that, I suspect that this problem existed prior to this PR, but
> this PR is first to introduce test to check this.
>
> I will continue to investigate this fault, but I can not locally reproduce
> this fault, so this is slowing troubleshooting.
>
> BR,
> Mario
> ________________________________
> Šalje: Alexander Murmann <am...@apache.org>
> Poslano: 14. srpnja 2020. 1:11
> Prima: Alexander Murmann <am...@apache.org>
> Kopija: dev@geode.apache.org <de...@geode.apache.org>; Mario Ivanac
> <ma...@est.tech>
> Predmet: Re: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
>
> We continue to see these WAN tests adding a fail rate of just below 30% in
> our mass test runs
> <
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-mass-test-run/jobs/create-mass-test-run-report/builds/10
> >
> .
>
> That's a very significant fail rate that impacts our ability to get our
> code committed with confidence.
>
> Can we resolve this issue? Otherwise, I think we need to consider reverting
> GEODE-7458.
>
> On Fri, Jun 19, 2020 at 3:28 PM Alexander Murmann <am...@apache.org>
> wrote:
>
> > Looking more into this, it looks like this was introduced by the changes
> > for GEODE-7458 - "Adding additional option in gfsh command "start gateway
> > sender" to control clearing of existing queues".
> >
> > That happened about a month ago, but it's inherent to those flaky tests
> > that we discover them only after a while. Nonetheless, they become paper
> > cuts that ultimately slow us down substantially if they persist.
> >
> > @Mario Ivanac If I am correct and GEODE-7458 introduced this you were the
> > one making that change. Might you be able to take a look at making that
> > test more reliable or reverting the change?
> >
> > Thank you!
> >
> > On Fri, Jun 19, 2020 at 7:57 AM Alexander Murmann <am...@apache.org>
> > wrote:
> >
> >> Thank you so much for sharing this, Mark!
> >>
> >> It looks like there is a big cluster around WAN Gateway. Is anyone
> >> already looking into the WAN issues?
> >>
> >> On Thu, Jun 18, 2020 at 10:06 PM Mark Hanson <ha...@vmware.com>
> wrote:
> >>
> >>> FYI, the build success rate was around 90% or so about two months ago.
> >>>
> >>> Here are the DUnit tests that are currently failing in our tests, most
> >>> likely in CI, and PR pipelines.
> >>>
> >>> Please let me know if you have any questions.
> >>>
> >>> Thanks,
> >>> Mark
> >>>
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>> Overall build success rate: 78.00000% (156 of 200)
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>>
> >>>
> >>> The following test methods see failures in more than one class. There
> >>> may be a failing *TestBase class
> >>>
> >>>
> >>>
> >>>
> *.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
> >>> 12 failures :
> >>>
> >>> SerialWANPersistenceEnabledGatewaySenderDUnitTest: 8 failures
> >>> (96.000% success rate)
> >>>
> >>> SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 4 failures
> >>> (98.000% success rate)
> >>>
> >>>
> >>>
> >>>
> *.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
> >>> 12 failures :
> >>>
> >>> ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 5
> >>> failures (97.500% success rate)
> >>>
> >>> ParallelWANPersistenceEnabledGatewaySenderDUnitTest: 7 failures
> >>> (96.500% success rate)
> >>>
> >>>
> >>>
> >>> *.testPingWrongServer: 4 failures :
> >>>
> >>> ClientServerMiscSelectorDUnitTest: 3 failures (98.500% success rate)
> >>>
> >>> ClientServerMiscDUnitTest: 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> ***********************************************************************************
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest:
> >>> 8 failures (96.000% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3539
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3526
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3505
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3435
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3414
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3391
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3363
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3359
> >>>
> >>>
> >>>
> >>> org.apache.geode.management.MemberMXBeanDistributedTest: 2 failures
> >>> (99.000% success rate)
> >>>
> >>>
> >>>
> >>> testBucketCount
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
> >>>
> >>> testBucketCount
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3411
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>> testRegisterInterestAndMakePrimaryWithFullRedundancy
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3381
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>> testSimpleQueryOnLocator
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3516
> >>>
> >>>
> >>>
> >>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>> testPingWrongServer
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3400
> >>>
> >>>
> >>>
> >>> org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest: 3
> >>> failures (98.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3495
> >>>
> >>>
> >>>
> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3470
> >>>
> >>> testReplicatedSerialPropagationHAWithGroupTransactionEvents
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3479
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>> testDistributedDeadlockWithDLock
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3421
> >>>
> >>>
> >>>
> >>> org.apache.geode.distributed.LocatorDUnitTest: 1 failures (99.500%
> >>> success rate)
> >>>
> >>>
> >>>
> >>> testStartTwoLocatorsWithMultiKeystoreSSL
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3398
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
> >>> 5 failures (97.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3531
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3522
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3456
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest:
> >>> 1 failures (99.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>> clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3433
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
> >>> 7 failures (96.500% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3478
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3370
> >>>
> >>>
> >>>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3351
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest:
> >>> 4 failures (98.000% success rate)
> >>>
> >>>
> >>>
> >>> testMultipleColocatedChildPRsMissingWithSequencedStart
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3493
> >>>
> >>> testMissingColocatedChildPRDueToDelayedStart
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
> >>>
> >>> testHierarchyOfColocatedChildPRsMissingGrandchild
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3537
> >>>
> >>> testHierarchyOfColocatedChildPRsMissingGrandchild
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3424
> >>>
> >>>
> >>>
> >>> org.apache.geode.distributed.DistributedMemberDUnitTest: 2 failures
> >>> (99.000% success rate)
> >>>
> >>>
> >>>
> >>> testGroupsInAllVMs
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3423
> >>>
> >>> testGroupsInAllVMs
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3374
> >>>
> >>>
> >>>
> >>> org.apache.geode.management.JMXMBeanReconnectDUnitTest: 5 failures
> >>> (97.500% success rate)
> >>>
> >>>
> >>>
> >>> serverMXBeansOnServerAreUnaffectedByLocatorCrash
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
> >>>
> >>>
> >>> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
> >>>
> >>>
> >>> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3406
> >>>
> >>> locatorHasMemberTypeMXBeansForBothLocators
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3352
> >>>
> >>> serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3457
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
> >>> 4 failures (98.000% success rate)
> >>>
> >>>
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3494
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3465
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3464
> >>>
> >>>
> >>>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
> >>>
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3379
> >>>
> >>>
> >>>
> >>>
> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest:
> >>> 3 failures (98.500% success rate)
> >>>
> >>>
> >>>
> >>> testPingWrongServer
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3402
> >>>
> >>> testPingWrongServer
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3380
> >>>
> >>> testPingWrongServer
> >>>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3357
> >>>
> >>>
>
Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
Posted by Mario Ivanac <ma...@est.tech>.
Hi,
after adding additional checks in failing test, now I can see that test are failing due to fault that some batch are distributed at stopping of GW sender.
Cause of that, I suspect that this problem existed prior to this PR, but this PR is first to introduce test to check this.
I will continue to investigate this fault, but I can not locally reproduce this fault, so this is slowing troubleshooting.
BR,
Mario
________________________________
Šalje: Alexander Murmann <am...@apache.org>
Poslano: 14. srpnja 2020. 1:11
Prima: Alexander Murmann <am...@apache.org>
Kopija: dev@geode.apache.org <de...@geode.apache.org>; Mario Ivanac <ma...@est.tech>
Predmet: Re: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
We continue to see these WAN tests adding a fail rate of just below 30% in
our mass test runs
<https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-mass-test-run/jobs/create-mass-test-run-report/builds/10>
.
That's a very significant fail rate that impacts our ability to get our
code committed with confidence.
Can we resolve this issue? Otherwise, I think we need to consider reverting
GEODE-7458.
On Fri, Jun 19, 2020 at 3:28 PM Alexander Murmann <am...@apache.org>
wrote:
> Looking more into this, it looks like this was introduced by the changes
> for GEODE-7458 - "Adding additional option in gfsh command "start gateway
> sender" to control clearing of existing queues".
>
> That happened about a month ago, but it's inherent to those flaky tests
> that we discover them only after a while. Nonetheless, they become paper
> cuts that ultimately slow us down substantially if they persist.
>
> @Mario Ivanac If I am correct and GEODE-7458 introduced this you were the
> one making that change. Might you be able to take a look at making that
> test more reliable or reverting the change?
>
> Thank you!
>
> On Fri, Jun 19, 2020 at 7:57 AM Alexander Murmann <am...@apache.org>
> wrote:
>
>> Thank you so much for sharing this, Mark!
>>
>> It looks like there is a big cluster around WAN Gateway. Is anyone
>> already looking into the WAN issues?
>>
>> On Thu, Jun 18, 2020 at 10:06 PM Mark Hanson <ha...@vmware.com> wrote:
>>
>>> FYI, the build success rate was around 90% or so about two months ago.
>>>
>>> Here are the DUnit tests that are currently failing in our tests, most
>>> likely in CI, and PR pipelines.
>>>
>>> Please let me know if you have any questions.
>>>
>>> Thanks,
>>> Mark
>>>
>>>
>>>
>>> ***********************************************************************************
>>>
>>> Overall build success rate: 78.00000% (156 of 200)
>>>
>>>
>>> ***********************************************************************************
>>>
>>>
>>>
>>> The following test methods see failures in more than one class. There
>>> may be a failing *TestBase class
>>>
>>>
>>>
>>> *.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
>>> 12 failures :
>>>
>>> SerialWANPersistenceEnabledGatewaySenderDUnitTest: 8 failures
>>> (96.000% success rate)
>>>
>>> SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 4 failures
>>> (98.000% success rate)
>>>
>>>
>>>
>>> *.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
>>> 12 failures :
>>>
>>> ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 5
>>> failures (97.500% success rate)
>>>
>>> ParallelWANPersistenceEnabledGatewaySenderDUnitTest: 7 failures
>>> (96.500% success rate)
>>>
>>>
>>>
>>> *.testPingWrongServer: 4 failures :
>>>
>>> ClientServerMiscSelectorDUnitTest: 3 failures (98.500% success rate)
>>>
>>> ClientServerMiscDUnitTest: 1 failures (99.500% success rate)
>>>
>>>
>>>
>>>
>>> ***********************************************************************************
>>>
>>>
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest:
>>> 8 failures (96.000% success rate)
>>>
>>>
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3539
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3526
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3505
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3435
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3414
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3391
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3363
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3359
>>>
>>>
>>>
>>> org.apache.geode.management.MemberMXBeanDistributedTest: 2 failures
>>> (99.000% success rate)
>>>
>>>
>>>
>>> testBucketCount
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
>>>
>>> testBucketCount
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3411
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>> testRegisterInterestAndMakePrimaryWithFullRedundancy
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3381
>>>
>>>
>>>
>>> org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>> testSimpleQueryOnLocator
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3516
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>> testPingWrongServer
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3400
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest: 3
>>> failures (98.500% success rate)
>>>
>>>
>>>
>>>
>>> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3495
>>>
>>>
>>> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3470
>>>
>>> testReplicatedSerialPropagationHAWithGroupTransactionEvents
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3479
>>>
>>>
>>>
>>> org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>> testDistributedDeadlockWithDLock
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3421
>>>
>>>
>>>
>>> org.apache.geode.distributed.LocatorDUnitTest: 1 failures (99.500%
>>> success rate)
>>>
>>>
>>>
>>> testStartTwoLocatorsWithMultiKeystoreSSL
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3398
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
>>> 5 failures (97.500% success rate)
>>>
>>>
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3531
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3522
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3456
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>>
>>> clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3433
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
>>> 7 failures (96.500% success rate)
>>>
>>>
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3478
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3370
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3351
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest:
>>> 4 failures (98.000% success rate)
>>>
>>>
>>>
>>> testMultipleColocatedChildPRsMissingWithSequencedStart
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3493
>>>
>>> testMissingColocatedChildPRDueToDelayedStart
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
>>>
>>> testHierarchyOfColocatedChildPRsMissingGrandchild
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3537
>>>
>>> testHierarchyOfColocatedChildPRsMissingGrandchild
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3424
>>>
>>>
>>>
>>> org.apache.geode.distributed.DistributedMemberDUnitTest: 2 failures
>>> (99.000% success rate)
>>>
>>>
>>>
>>> testGroupsInAllVMs
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3423
>>>
>>> testGroupsInAllVMs
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3374
>>>
>>>
>>>
>>> org.apache.geode.management.JMXMBeanReconnectDUnitTest: 5 failures
>>> (97.500% success rate)
>>>
>>>
>>>
>>> serverMXBeansOnServerAreUnaffectedByLocatorCrash
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
>>>
>>>
>>> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
>>>
>>>
>>> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3406
>>>
>>> locatorHasMemberTypeMXBeansForBothLocators
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3352
>>>
>>> serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3457
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
>>> 4 failures (98.000% success rate)
>>>
>>>
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3494
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3465
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3464
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3379
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest:
>>> 3 failures (98.500% success rate)
>>>
>>>
>>>
>>> testPingWrongServer
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3402
>>>
>>> testPingWrongServer
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3380
>>>
>>> testPingWrongServer
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3357
>>>
>>>
Re: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
Posted by Mark Hanson <ha...@vmware.com>.
The previous statement about 30% didn't make sense to me, so I thought I would throw in this tidbit...
70 failures out 75 failures in 200 runs are caused by WAN.
Thanks,
Mark
On 7/13/20, 4:11 PM, "Alexander Murmann" <am...@apache.org> wrote:
We continue to see these WAN tests adding a fail rate of just below 30% in
our mass test runs
<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-develop-mass-test-run%2Fjobs%2Fcreate-mass-test-run-report%2Fbuilds%2F10&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835802218&sdata=PZs6%2B%2F2vRL94anXVe0h6xt7U%2BwGR%2BKPbu7ZHLMr4vGQ%3D&reserved=0>
.
That's a very significant fail rate that impacts our ability to get our
code committed with confidence.
Can we resolve this issue? Otherwise, I think we need to consider reverting
GEODE-7458.
On Fri, Jun 19, 2020 at 3:28 PM Alexander Murmann <am...@apache.org>
wrote:
> Looking more into this, it looks like this was introduced by the changes
> for GEODE-7458 - "Adding additional option in gfsh command "start gateway
> sender" to control clearing of existing queues".
>
> That happened about a month ago, but it's inherent to those flaky tests
> that we discover them only after a while. Nonetheless, they become paper
> cuts that ultimately slow us down substantially if they persist.
>
> @Mario Ivanac If I am correct and GEODE-7458 introduced this you were the
> one making that change. Might you be able to take a look at making that
> test more reliable or reverting the change?
>
> Thank you!
>
> On Fri, Jun 19, 2020 at 7:57 AM Alexander Murmann <am...@apache.org>
> wrote:
>
>> Thank you so much for sharing this, Mark!
>>
>> It looks like there is a big cluster around WAN Gateway. Is anyone
>> already looking into the WAN issues?
>>
>> On Thu, Jun 18, 2020 at 10:06 PM Mark Hanson <ha...@vmware.com> wrote:
>>
>>> FYI, the build success rate was around 90% or so about two months ago.
>>>
>>> Here are the DUnit tests that are currently failing in our tests, most
>>> likely in CI, and PR pipelines.
>>>
>>> Please let me know if you have any questions.
>>>
>>> Thanks,
>>> Mark
>>>
>>>
>>>
>>> ***********************************************************************************
>>>
>>> Overall build success rate: 78.00000% (156 of 200)
>>>
>>>
>>> ***********************************************************************************
>>>
>>>
>>>
>>> The following test methods see failures in more than one class. There
>>> may be a failing *TestBase class
>>>
>>>
>>>
>>> *.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
>>> 12 failures :
>>>
>>> SerialWANPersistenceEnabledGatewaySenderDUnitTest: 8 failures
>>> (96.000% success rate)
>>>
>>> SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 4 failures
>>> (98.000% success rate)
>>>
>>>
>>>
>>> *.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
>>> 12 failures :
>>>
>>> ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 5
>>> failures (97.500% success rate)
>>>
>>> ParallelWANPersistenceEnabledGatewaySenderDUnitTest: 7 failures
>>> (96.500% success rate)
>>>
>>>
>>>
>>> *.testPingWrongServer: 4 failures :
>>>
>>> ClientServerMiscSelectorDUnitTest: 3 failures (98.500% success rate)
>>>
>>> ClientServerMiscDUnitTest: 1 failures (99.500% success rate)
>>>
>>>
>>>
>>>
>>> ***********************************************************************************
>>>
>>>
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest:
>>> 8 failures (96.000% success rate)
>>>
>>>
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3539&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835802218&sdata=ILB4NvS0zakenbq00pfHjOZlb9pt7n60%2FpjNUV%2FUQp8%3D&reserved=0
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3526&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835802218&sdata=4pejQ84CaPxlR6terHjOb4MlSDhqmiZWw5FMfshFbL0%3D&reserved=0
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3505&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835802218&sdata=0sOJRUud9GWXHo3iPgSU53z9Eyi9sh7LiYSXSRYCg38%3D&reserved=0
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3435&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=iJJyU6KNfWTGF3cSdgfQzCtbh%2BroLJxhVfW0HL4NVKk%3D&reserved=0
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3414&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=Qd90VpFi1ckXArWAerACT8QKQi5S3DPF9O%2Fc%2FWuOras%3D&reserved=0
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3391&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=sybHZb4TXpDQndMWJazxYqpC8YL2yLCKcinJomP6WP0%3D&reserved=0
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3363&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=AlBfHXFJCbKYhqOrO3c7n0vFWBK3n5jNmJt3ecQFS8Q%3D&reserved=0
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3359&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=0Kj7PBqxzd8M53Hh7F2r%2B9mtYj57Gfc3WxCOVPCTYVo%3D&reserved=0
>>>
>>>
>>>
>>> org.apache.geode.management.MemberMXBeanDistributedTest: 2 failures
>>> (99.000% success rate)
>>>
>>>
>>>
>>> testBucketCount
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=rE%2FYuEs6HBgpz0P3WYbe%2Bt3goceTlR%2Bs51tqzhDkFsA%3D&reserved=0
>>>
>>> testBucketCount
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3411&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=7M7j%2BVhhYDA9QpXQWfCr9nXi9Krl1HMObVJk8D%2FjufY%3D&reserved=0
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>> testRegisterInterestAndMakePrimaryWithFullRedundancy
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3381&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=J0lyeA9l1b0gT4A0PR4N5KKMBySOQrg8zWCgKptE7iQ%3D&reserved=0
>>>
>>>
>>>
>>> org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>> testSimpleQueryOnLocator
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3516&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=FARo2AYcibOpGuFwt25N7s8rfUDddLe5QkhR4OxFJ8k%3D&reserved=0
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>> testPingWrongServer
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3400&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=ORdmb4v2Lrp2SxI8HtiToZN%2BvSSddgg4OqzFuexvrSo%3D&reserved=0
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest: 3
>>> failures (98.500% success rate)
>>>
>>>
>>>
>>>
>>> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3495&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=G%2Bxs28AKlk%2Fa9PjMg0X0xP8tHzfBpa8IWnqCUuzNdik%3D&reserved=0
>>>
>>>
>>> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3470&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=GAr47n034NLpEAgyYMM0%2Bg%2BcJIX5wV%2BAAAnKef3OEj0%3D&reserved=0
>>>
>>> testReplicatedSerialPropagationHAWithGroupTransactionEvents
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3479&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=z81ZJBzMqT%2F4SR3DySZgzO7GA8EnOY4jgNsqyUi6d4o%3D&reserved=0
>>>
>>>
>>>
>>> org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>> testDistributedDeadlockWithDLock
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3421&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=Ux54TOh1LLp9XRAjdbMK0scUn70V32Up%2FV78RjTtr8A%3D&reserved=0
>>>
>>>
>>>
>>> org.apache.geode.distributed.LocatorDUnitTest: 1 failures (99.500%
>>> success rate)
>>>
>>>
>>>
>>> testStartTwoLocatorsWithMultiKeystoreSSL
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3398&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=eic2QNLwx%2B18jVZHPfOGfA99%2B8gLaXXal2KuSBvi6s0%3D&reserved=0
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
>>> 5 failures (97.500% success rate)
>>>
>>>
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3531&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=3apyNPTCcAbKQ55s1RqP7c%2BiuTjYWoDHN7itNiJBr6s%3D&reserved=0
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3522&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=dsdzFOC4jPXN2pKibdZR8MWCwKEJWOCv8nNSnvrhclg%3D&reserved=0
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3456&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=K2%2Fo2mdpIVC4u2V12VRkpyyGWJq2vYqtm1%2Fvki8mB9A%3D&reserved=0
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=Dje%2F53pJYke0d5yosB61kMB77CLZYdLfqxWoD%2BfFzZs%3D&reserved=0
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=1dKgH5KcHmj7snaRO46OW8k0GPrnTottsNCWH2HdQAE%3D&reserved=0
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>>
>>> clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3433&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=fH3g67sitVbMU9Ir1Exm8MclDAVqM4x8mg%2F9y7WYi6I%3D&reserved=0
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
>>> 7 failures (96.500% success rate)
>>>
>>>
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3478&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=9JBfKJMKPrarh2IkhTzz%2BxxQ6Mdt0J2aaqHnn7fUsNo%3D&reserved=0
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=Wi%2FM0H%2FEbUoS%2FEDaHs0ybqHqc5ZJY9jhai9XvGqOaeg%3D&reserved=0
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=Dje%2F53pJYke0d5yosB61kMB77CLZYdLfqxWoD%2BfFzZs%3D&reserved=0
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=ASOBOjhutihuLnyEk9v%2ByTlzeorAJrYFBlWKpIq3yVA%3D&reserved=0
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=1dKgH5KcHmj7snaRO46OW8k0GPrnTottsNCWH2HdQAE%3D&reserved=0
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3370&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=k1QhouuiRkGwHEoTve21cd1rClD6%2F3PEO%2Fp8pQUrX7A%3D&reserved=0
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3351&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=yDB750p04VE%2FuMwwjVpZ3zAtW50BlfdWUVFW205XnKw%3D&reserved=0
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest:
>>> 4 failures (98.000% success rate)
>>>
>>>
>>>
>>> testMultipleColocatedChildPRsMissingWithSequencedStart
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3493&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=4jOpZq1Q6qXgNqHYQAWBW702yhCDpUHzQPmOKk1ef5A%3D&reserved=0
>>>
>>> testMissingColocatedChildPRDueToDelayedStart
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=P%2BZyelCzOwqc87bP9XFlIPdVYn0BwWk4jMIu0lxaq4M%3D&reserved=0
>>>
>>> testHierarchyOfColocatedChildPRsMissingGrandchild
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3537&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=qt20QmQTmbe%2F33LtWmQsXnDWfIUB6vlf5xtE%2Bw23gG0%3D&reserved=0
>>>
>>> testHierarchyOfColocatedChildPRsMissingGrandchild
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3424&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=2mu0xJE6BaugPHzEheTw%2Fp4nDt6xdifAH0rCnF3mBno%3D&reserved=0
>>>
>>>
>>>
>>> org.apache.geode.distributed.DistributedMemberDUnitTest: 2 failures
>>> (99.000% success rate)
>>>
>>>
>>>
>>> testGroupsInAllVMs
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3423&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=VPQRY2DJxao%2F%2Fsj3rTMBtZ1J9R7FQYR7SqY1hgQoj1U%3D&reserved=0
>>>
>>> testGroupsInAllVMs
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3374&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=GyAnvRP4%2BVP4bYI%2BkXzrTgsVezpe0c6SKvnqiZq1p1k%3D&reserved=0
>>>
>>>
>>>
>>> org.apache.geode.management.JMXMBeanReconnectDUnitTest: 5 failures
>>> (97.500% success rate)
>>>
>>>
>>>
>>> serverMXBeansOnServerAreUnaffectedByLocatorCrash
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=P%2BZyelCzOwqc87bP9XFlIPdVYn0BwWk4jMIu0lxaq4M%3D&reserved=0
>>>
>>>
>>> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=vq40%2BFWfDbH%2Fs4b%2BooGY3KBi1WV1LT%2Bh3s45BnAbq3I%3D&reserved=0
>>>
>>>
>>> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3406&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=ZNwEvU4gWXhoVGVfx3vqHjHmV2%2FQHS1VOjifnY0qZCg%3D&reserved=0
>>>
>>> locatorHasMemberTypeMXBeansForBothLocators
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3352&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=gyq3c%2FGdfmE1OATwqe39MQUfjz%2Be6crV6dTDWQUgvUM%3D&reserved=0
>>>
>>> serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3457&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=elpsPB4B3SCHvP0yrV0CuNMM11dAUr8yROnKa3RpQus%3D&reserved=0
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
>>> 4 failures (98.000% success rate)
>>>
>>>
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3494&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=tm2oxvbVR%2BcL7gPOqIOqIK0FKLbJr%2BEHrB4lywt649k%3D&reserved=0
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3465&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=Ho8AWWGOORaKrtMlj38WaAs38BE5nmuaA1xGC%2B9UmCU%3D&reserved=0
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3464&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=QXxXqvCi7aDtvosglZ5q51fyAguonxny%2F5b%2FAFvjFL8%3D&reserved=0
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3379&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835842199&sdata=7JCsVA7pAmjclJo2tk1jDq%2BCeMjHR2G2G6wLSfXCAqQ%3D&reserved=0
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest:
>>> 3 failures (98.500% success rate)
>>>
>>>
>>>
>>> testPingWrongServer
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3402&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835842199&sdata=d9GXmfDC%2BbpY21Pgat%2FIXRaJ61d3fWjhUCqvTwvNM9w%3D&reserved=0
>>>
>>> testPingWrongServer
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3380&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835842199&sdata=WBz4zHqtAhx%2FBOy0lHi7vJWHgYzZHKNq7pynG7gt%2Frc%3D&reserved=0
>>>
>>> testPingWrongServer
>>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3357&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835842199&sdata=99aY8oSD%2B2EF3RgdyZj3L9I2REpk7NPqbYuzz9SdIIA%3D&reserved=0
>>>
>>>
Re: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
Posted by Alexander Murmann <am...@apache.org>.
We continue to see these WAN tests adding a fail rate of just below 30% in
our mass test runs
<https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-mass-test-run/jobs/create-mass-test-run-report/builds/10>
.
That's a very significant fail rate that impacts our ability to get our
code committed with confidence.
Can we resolve this issue? Otherwise, I think we need to consider reverting
GEODE-7458.
On Fri, Jun 19, 2020 at 3:28 PM Alexander Murmann <am...@apache.org>
wrote:
> Looking more into this, it looks like this was introduced by the changes
> for GEODE-7458 - "Adding additional option in gfsh command "start gateway
> sender" to control clearing of existing queues".
>
> That happened about a month ago, but it's inherent to those flaky tests
> that we discover them only after a while. Nonetheless, they become paper
> cuts that ultimately slow us down substantially if they persist.
>
> @Mario Ivanac If I am correct and GEODE-7458 introduced this you were the
> one making that change. Might you be able to take a look at making that
> test more reliable or reverting the change?
>
> Thank you!
>
> On Fri, Jun 19, 2020 at 7:57 AM Alexander Murmann <am...@apache.org>
> wrote:
>
>> Thank you so much for sharing this, Mark!
>>
>> It looks like there is a big cluster around WAN Gateway. Is anyone
>> already looking into the WAN issues?
>>
>> On Thu, Jun 18, 2020 at 10:06 PM Mark Hanson <ha...@vmware.com> wrote:
>>
>>> FYI, the build success rate was around 90% or so about two months ago.
>>>
>>> Here are the DUnit tests that are currently failing in our tests, most
>>> likely in CI, and PR pipelines.
>>>
>>> Please let me know if you have any questions.
>>>
>>> Thanks,
>>> Mark
>>>
>>>
>>>
>>> ***********************************************************************************
>>>
>>> Overall build success rate: 78.00000% (156 of 200)
>>>
>>>
>>> ***********************************************************************************
>>>
>>>
>>>
>>> The following test methods see failures in more than one class. There
>>> may be a failing *TestBase class
>>>
>>>
>>>
>>> *.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
>>> 12 failures :
>>>
>>> SerialWANPersistenceEnabledGatewaySenderDUnitTest: 8 failures
>>> (96.000% success rate)
>>>
>>> SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 4 failures
>>> (98.000% success rate)
>>>
>>>
>>>
>>> *.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
>>> 12 failures :
>>>
>>> ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 5
>>> failures (97.500% success rate)
>>>
>>> ParallelWANPersistenceEnabledGatewaySenderDUnitTest: 7 failures
>>> (96.500% success rate)
>>>
>>>
>>>
>>> *.testPingWrongServer: 4 failures :
>>>
>>> ClientServerMiscSelectorDUnitTest: 3 failures (98.500% success rate)
>>>
>>> ClientServerMiscDUnitTest: 1 failures (99.500% success rate)
>>>
>>>
>>>
>>>
>>> ***********************************************************************************
>>>
>>>
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest:
>>> 8 failures (96.000% success rate)
>>>
>>>
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3539
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3526
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3505
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3435
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3414
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3391
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3363
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3359
>>>
>>>
>>>
>>> org.apache.geode.management.MemberMXBeanDistributedTest: 2 failures
>>> (99.000% success rate)
>>>
>>>
>>>
>>> testBucketCount
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
>>>
>>> testBucketCount
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3411
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>> testRegisterInterestAndMakePrimaryWithFullRedundancy
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3381
>>>
>>>
>>>
>>> org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>> testSimpleQueryOnLocator
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3516
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>> testPingWrongServer
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3400
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest: 3
>>> failures (98.500% success rate)
>>>
>>>
>>>
>>>
>>> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3495
>>>
>>>
>>> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3470
>>>
>>> testReplicatedSerialPropagationHAWithGroupTransactionEvents
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3479
>>>
>>>
>>>
>>> org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>> testDistributedDeadlockWithDLock
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3421
>>>
>>>
>>>
>>> org.apache.geode.distributed.LocatorDUnitTest: 1 failures (99.500%
>>> success rate)
>>>
>>>
>>>
>>> testStartTwoLocatorsWithMultiKeystoreSSL
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3398
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
>>> 5 failures (97.500% success rate)
>>>
>>>
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3531
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3522
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3456
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>>
>>> clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3433
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
>>> 7 failures (96.500% success rate)
>>>
>>>
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3478
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3370
>>>
>>>
>>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3351
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest:
>>> 4 failures (98.000% success rate)
>>>
>>>
>>>
>>> testMultipleColocatedChildPRsMissingWithSequencedStart
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3493
>>>
>>> testMissingColocatedChildPRDueToDelayedStart
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
>>>
>>> testHierarchyOfColocatedChildPRsMissingGrandchild
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3537
>>>
>>> testHierarchyOfColocatedChildPRsMissingGrandchild
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3424
>>>
>>>
>>>
>>> org.apache.geode.distributed.DistributedMemberDUnitTest: 2 failures
>>> (99.000% success rate)
>>>
>>>
>>>
>>> testGroupsInAllVMs
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3423
>>>
>>> testGroupsInAllVMs
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3374
>>>
>>>
>>>
>>> org.apache.geode.management.JMXMBeanReconnectDUnitTest: 5 failures
>>> (97.500% success rate)
>>>
>>>
>>>
>>> serverMXBeansOnServerAreUnaffectedByLocatorCrash
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
>>>
>>>
>>> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
>>>
>>>
>>> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3406
>>>
>>> locatorHasMemberTypeMXBeansForBothLocators
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3352
>>>
>>> serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3457
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
>>> 4 failures (98.000% success rate)
>>>
>>>
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3494
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3465
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3464
>>>
>>>
>>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3379
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest:
>>> 3 failures (98.500% success rate)
>>>
>>>
>>>
>>> testPingWrongServer
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3402
>>>
>>> testPingWrongServer
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3380
>>>
>>> testPingWrongServer
>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3357
>>>
>>>
Re: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
Posted by Alexander Murmann <am...@apache.org>.
Looking more into this, it looks like this was introduced by the changes
for GEODE-7458 - "Adding additional option in gfsh command "start gateway
sender" to control clearing of existing queues".
That happened about a month ago, but it's inherent to those flaky tests
that we discover them only after a while. Nonetheless, they become paper
cuts that ultimately slow us down substantially if they persist.
@Mario Ivanac If I am correct and GEODE-7458 introduced this you were the
one making that change. Might you be able to take a look at making that
test more reliable or reverting the change?
Thank you!
On Fri, Jun 19, 2020 at 7:57 AM Alexander Murmann <am...@apache.org>
wrote:
> Thank you so much for sharing this, Mark!
>
> It looks like there is a big cluster around WAN Gateway. Is anyone already
> looking into the WAN issues?
>
> On Thu, Jun 18, 2020 at 10:06 PM Mark Hanson <ha...@vmware.com> wrote:
>
>> FYI, the build success rate was around 90% or so about two months ago.
>>
>> Here are the DUnit tests that are currently failing in our tests, most
>> likely in CI, and PR pipelines.
>>
>> Please let me know if you have any questions.
>>
>> Thanks,
>> Mark
>>
>>
>>
>> ***********************************************************************************
>>
>> Overall build success rate: 78.00000% (156 of 200)
>>
>>
>> ***********************************************************************************
>>
>>
>>
>> The following test methods see failures in more than one class. There
>> may be a failing *TestBase class
>>
>>
>>
>> *.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
>> 12 failures :
>>
>> SerialWANPersistenceEnabledGatewaySenderDUnitTest: 8 failures (96.000%
>> success rate)
>>
>> SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 4 failures
>> (98.000% success rate)
>>
>>
>>
>> *.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
>> 12 failures :
>>
>> ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 5 failures
>> (97.500% success rate)
>>
>> ParallelWANPersistenceEnabledGatewaySenderDUnitTest: 7 failures
>> (96.500% success rate)
>>
>>
>>
>> *.testPingWrongServer: 4 failures :
>>
>> ClientServerMiscSelectorDUnitTest: 3 failures (98.500% success rate)
>>
>> ClientServerMiscDUnitTest: 1 failures (99.500% success rate)
>>
>>
>>
>>
>> ***********************************************************************************
>>
>>
>>
>>
>>
>> org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest:
>> 8 failures (96.000% success rate)
>>
>>
>>
>>
>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3539
>>
>>
>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3526
>>
>>
>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3505
>>
>>
>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3435
>>
>>
>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3414
>>
>>
>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3391
>>
>>
>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3363
>>
>>
>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3359
>>
>>
>>
>> org.apache.geode.management.MemberMXBeanDistributedTest: 2 failures
>> (99.000% success rate)
>>
>>
>>
>> testBucketCount
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
>>
>> testBucketCount
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3411
>>
>>
>>
>> org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest:
>> 1 failures (99.500% success rate)
>>
>>
>>
>> testRegisterInterestAndMakePrimaryWithFullRedundancy
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3381
>>
>>
>>
>> org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest:
>> 1 failures (99.500% success rate)
>>
>>
>>
>> testSimpleQueryOnLocator
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3516
>>
>>
>>
>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest:
>> 1 failures (99.500% success rate)
>>
>>
>>
>> testPingWrongServer
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3400
>>
>>
>>
>> org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest: 3
>> failures (98.500% success rate)
>>
>>
>>
>>
>> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3495
>>
>>
>> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3470
>>
>> testReplicatedSerialPropagationHAWithGroupTransactionEvents
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3479
>>
>>
>>
>> org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest:
>> 1 failures (99.500% success rate)
>>
>>
>>
>> testDistributedDeadlockWithDLock
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3421
>>
>>
>>
>> org.apache.geode.distributed.LocatorDUnitTest: 1 failures (99.500%
>> success rate)
>>
>>
>>
>> testStartTwoLocatorsWithMultiKeystoreSSL
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3398
>>
>>
>>
>> org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
>> 5 failures (97.500% success rate)
>>
>>
>>
>>
>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3531
>>
>>
>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3522
>>
>>
>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3456
>>
>>
>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
>>
>>
>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
>>
>>
>>
>> org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest:
>> 1 failures (99.500% success rate)
>>
>>
>>
>>
>> clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3433
>>
>>
>>
>> org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
>> 7 failures (96.500% success rate)
>>
>>
>>
>>
>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3478
>>
>>
>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
>>
>>
>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
>>
>>
>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
>>
>>
>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
>>
>>
>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3370
>>
>>
>> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3351
>>
>>
>>
>> org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest:
>> 4 failures (98.000% success rate)
>>
>>
>>
>> testMultipleColocatedChildPRsMissingWithSequencedStart
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3493
>>
>> testMissingColocatedChildPRDueToDelayedStart
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
>>
>> testHierarchyOfColocatedChildPRsMissingGrandchild
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3537
>>
>> testHierarchyOfColocatedChildPRsMissingGrandchild
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3424
>>
>>
>>
>> org.apache.geode.distributed.DistributedMemberDUnitTest: 2 failures
>> (99.000% success rate)
>>
>>
>>
>> testGroupsInAllVMs
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3423
>>
>> testGroupsInAllVMs
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3374
>>
>>
>>
>> org.apache.geode.management.JMXMBeanReconnectDUnitTest: 5 failures
>> (97.500% success rate)
>>
>>
>>
>> serverMXBeansOnServerAreUnaffectedByLocatorCrash
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
>>
>> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
>>
>> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3406
>>
>> locatorHasMemberTypeMXBeansForBothLocators
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3352
>>
>> serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3457
>>
>>
>>
>> org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
>> 4 failures (98.000% success rate)
>>
>>
>>
>>
>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3494
>>
>>
>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3465
>>
>>
>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3464
>>
>>
>> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3379
>>
>>
>>
>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest:
>> 3 failures (98.500% success rate)
>>
>>
>>
>> testPingWrongServer
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3402
>>
>> testPingWrongServer
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3380
>>
>> testPingWrongServer
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3357
>>
>>
Re: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes
Posted by Alexander Murmann <am...@apache.org>.
Thank you so much for sharing this, Mark!
It looks like there is a big cluster around WAN Gateway. Is anyone already
looking into the WAN issues?
On Thu, Jun 18, 2020 at 10:06 PM Mark Hanson <ha...@vmware.com> wrote:
> FYI, the build success rate was around 90% or so about two months ago.
>
> Here are the DUnit tests that are currently failing in our tests, most
> likely in CI, and PR pipelines.
>
> Please let me know if you have any questions.
>
> Thanks,
> Mark
>
>
>
> ***********************************************************************************
>
> Overall build success rate: 78.00000% (156 of 200)
>
>
> ***********************************************************************************
>
>
>
> The following test methods see failures in more than one class. There may
> be a failing *TestBase class
>
>
>
> *.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
> 12 failures :
>
> SerialWANPersistenceEnabledGatewaySenderDUnitTest: 8 failures (96.000%
> success rate)
>
> SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 4 failures
> (98.000% success rate)
>
>
>
> *.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
> 12 failures :
>
> ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 5 failures
> (97.500% success rate)
>
> ParallelWANPersistenceEnabledGatewaySenderDUnitTest: 7 failures
> (96.500% success rate)
>
>
>
> *.testPingWrongServer: 4 failures :
>
> ClientServerMiscSelectorDUnitTest: 3 failures (98.500% success rate)
>
> ClientServerMiscDUnitTest: 1 failures (99.500% success rate)
>
>
>
>
> ***********************************************************************************
>
>
>
>
>
> org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest:
> 8 failures (96.000% success rate)
>
>
>
>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3539
>
>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3526
>
>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3505
>
>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3435
>
>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3414
>
>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3391
>
>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3363
>
>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3359
>
>
>
> org.apache.geode.management.MemberMXBeanDistributedTest: 2 failures
> (99.000% success rate)
>
>
>
> testBucketCount
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
>
> testBucketCount
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3411
>
>
>
> org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest:
> 1 failures (99.500% success rate)
>
>
>
> testRegisterInterestAndMakePrimaryWithFullRedundancy
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3381
>
>
>
> org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest:
> 1 failures (99.500% success rate)
>
>
>
> testSimpleQueryOnLocator
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3516
>
>
>
> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest: 1
> failures (99.500% success rate)
>
>
>
> testPingWrongServer
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3400
>
>
>
> org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest: 3
> failures (98.500% success rate)
>
>
>
>
> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3495
>
>
> testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3470
>
> testReplicatedSerialPropagationHAWithGroupTransactionEvents
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3479
>
>
>
> org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest:
> 1 failures (99.500% success rate)
>
>
>
> testDistributedDeadlockWithDLock
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3421
>
>
>
> org.apache.geode.distributed.LocatorDUnitTest: 1 failures (99.500%
> success rate)
>
>
>
> testStartTwoLocatorsWithMultiKeystoreSSL
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3398
>
>
>
> org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
> 5 failures (97.500% success rate)
>
>
>
>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3531
>
>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3522
>
>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3456
>
>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
>
>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
>
>
>
> org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest:
> 1 failures (99.500% success rate)
>
>
>
>
> clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3433
>
>
>
> org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
> 7 failures (96.500% success rate)
>
>
>
>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3478
>
>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3463
>
>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3439
>
>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
>
>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3405
>
>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3370
>
>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3351
>
>
>
> org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest:
> 4 failures (98.000% success rate)
>
>
>
> testMultipleColocatedChildPRsMissingWithSequencedStart
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3493
>
> testMissingColocatedChildPRDueToDelayedStart
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
>
> testHierarchyOfColocatedChildPRsMissingGrandchild
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3537
>
> testHierarchyOfColocatedChildPRsMissingGrandchild
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3424
>
>
>
> org.apache.geode.distributed.DistributedMemberDUnitTest: 2 failures
> (99.000% success rate)
>
>
>
> testGroupsInAllVMs
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3423
>
> testGroupsInAllVMs
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3374
>
>
>
> org.apache.geode.management.JMXMBeanReconnectDUnitTest: 5 failures
> (97.500% success rate)
>
>
>
> serverMXBeansOnServerAreUnaffectedByLocatorCrash
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3503
>
> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3436
>
> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3406
>
> locatorHasMemberTypeMXBeansForBothLocators
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3352
>
> serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3457
>
>
>
> org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
> 4 failures (98.000% success rate)
>
>
>
>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3494
>
>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3465
>
>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3464
>
>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3379
>
>
>
> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest:
> 3 failures (98.500% success rate)
>
>
>
> testPingWrongServer
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3402
>
> testPingWrongServer
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3380
>
> testPingWrongServer
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3357
>
>