You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geode.apache.org by Mark Hanson <ha...@vmware.com> on 2020/06/11 05:59:15 UTC
[INFO] Distributed Test runs in bulk results.
Hello All,
I have been doing bulk test runs of DistributedTestOpenJDK8, in this case over 200. Here is a simplified report to kind of help you see what I am seeing and I think everybody sees with random failures as part of the PR process.
It is very easy to cause failures like this by not knowing what is running asynchronous and Geode is a complex system or introducing timing constraints that may not hold up in the system e.g. waiting 5 seconds for a test result that could take longer unbeknownst you.
All of that said, here are the results. There are tickets already open for most if not all of these issues.
Please let me know how often you all would like to see these reports…
Thanks,
Mark
***********************************************************************************
Overall build success rate: 84.00000%
The following test methods see failures in more than one class. There may be a failing *TestBase class
*.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived: 18 failures :
ParallelWANPersistenceEnabledGatewaySenderDUnitTest: 7 failures (96.889% success rate)
ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 11 failures (95.111% success rate)
*.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived: 4 failures :
SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 3 failures (98.667% success rate)
SerialWANPersistenceEnabledGatewaySenderDUnitTest: 1 failures (99.556% success rate)
***********************************************************************************
org.apache.geode.management.MemberMXBeanDistributedTest: 3 failures (98.667% success rate)
testBucketCount https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3247
testBucketCount https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3241
testBucketCount https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3199
org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest: 7 failures (96.889% success rate)
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3335
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3331
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3294
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3285
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3218
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3180
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3156
org.apache.geode.internal.cache.partitioned.PersistentPartitionedRegionDistributedTest: 1 failures (99.556% success rate)
testCacheCloseDuringBucketMoveDoesntCauseDataLoss https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3267
org.apache.geode.cache.management.MemoryThresholdsOffHeapDUnitTest: 1 failures (99.556% success rate)
testDistributedRegionClientPutRejection https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3319
org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 3 failures (98.667% success rate)
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3239
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3148
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3127
org.apache.geode.test.dunit.tests.AsyncInvocationTimeoutDistributedTest: 3 failures (98.667% success rate)
await_callable_timeout_includesStackTraceAsCause https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3288
await_callable_timeout_includesStackTraceAsCause https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3242
await_callable_timeout_includesStackTraceAsCause https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3177
org.apache.geode.internal.cache.tier.sockets.ReliableMessagingDUnitTest: 1 failures (99.556% success rate)
testPeriodicAckSendByClientPrimaryFailover https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3304
org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 11 failures (95.111% success rate)
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3309
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3305
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3279
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3273
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3257
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3210
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3181
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3166
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3158
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3137
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3130
org.apache.geode.management.JMXMBeanReconnectDUnitTest: 7 failures (96.889% success rate)
serverHasMemberTypeMXBeans https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3151
serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3319
serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3298
serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3285
serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3193
serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3325
locatorsHaveMemberTypeMXBeansForServer https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3288
org.apache.geode.management.internal.cli.commands.DescribeClientCommandDUnitTest: 1 failures (99.556% success rate)
describeClient https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3218
org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest: 1 failures (99.556% success rate)
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3225
org.apache.geode.cache30.DiskDistributedNoAckSyncOverflowRegionDUnitTest: 1 failures (99.556% success rate)
testRegionIdleDestroy https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3310
RE: [INFO] Distributed Test runs in bulk results.
Posted by Alberto Bustamante Reyes <al...@est.tech>.
I think a report like this is very useful to have real data about which flaky tests fail more often.
It would be great if a report like this were automatically generated and updated after each CI execution. In a project I was working before a similar report was implemented and it was very useful for developers to check if a test case was failing in the past, and also to identify which were the flaky test cases that we should try to fix first.
________________________________
De: Mark Hanson <ha...@vmware.com>
Enviado: jueves, 11 de junio de 2020 7:59
Para: dev@geode.apache.org <de...@geode.apache.org>
Asunto: [INFO] Distributed Test runs in bulk results.
Hello All,
I have been doing bulk test runs of DistributedTestOpenJDK8, in this case over 200. Here is a simplified report to kind of help you see what I am seeing and I think everybody sees with random failures as part of the PR process.
It is very easy to cause failures like this by not knowing what is running asynchronous and Geode is a complex system or introducing timing constraints that may not hold up in the system e.g. waiting 5 seconds for a test result that could take longer unbeknownst you.
All of that said, here are the results. There are tickets already open for most if not all of these issues.
Please let me know how often you all would like to see these reports…
Thanks,
Mark
***********************************************************************************
Overall build success rate: 84.00000%
The following test methods see failures in more than one class. There may be a failing *TestBase class
*.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived: 18 failures :
ParallelWANPersistenceEnabledGatewaySenderDUnitTest: 7 failures (96.889% success rate)
ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 11 failures (95.111% success rate)
*.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived: 4 failures :
SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 3 failures (98.667% success rate)
SerialWANPersistenceEnabledGatewaySenderDUnitTest: 1 failures (99.556% success rate)
***********************************************************************************
org.apache.geode.management.MemberMXBeanDistributedTest: 3 failures (98.667% success rate)
testBucketCount https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3247
testBucketCount https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3241
testBucketCount https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3199
org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest: 7 failures (96.889% success rate)
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3335
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3331
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3294
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3285
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3218
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3180
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3156
org.apache.geode.internal.cache.partitioned.PersistentPartitionedRegionDistributedTest: 1 failures (99.556% success rate)
testCacheCloseDuringBucketMoveDoesntCauseDataLoss https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3267
org.apache.geode.cache.management.MemoryThresholdsOffHeapDUnitTest: 1 failures (99.556% success rate)
testDistributedRegionClientPutRejection https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3319
org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 3 failures (98.667% success rate)
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3239
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3148
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3127
org.apache.geode.test.dunit.tests.AsyncInvocationTimeoutDistributedTest: 3 failures (98.667% success rate)
await_callable_timeout_includesStackTraceAsCause https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3288
await_callable_timeout_includesStackTraceAsCause https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3242
await_callable_timeout_includesStackTraceAsCause https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3177
org.apache.geode.internal.cache.tier.sockets.ReliableMessagingDUnitTest: 1 failures (99.556% success rate)
testPeriodicAckSendByClientPrimaryFailover https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3304
org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 11 failures (95.111% success rate)
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3309
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3305
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3279
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3273
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3257
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3210
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3181
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3166
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3158
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3137
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3130
org.apache.geode.management.JMXMBeanReconnectDUnitTest: 7 failures (96.889% success rate)
serverHasMemberTypeMXBeans https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3151
serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3319
serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3298
serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3285
serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3193
serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3325
locatorsHaveMemberTypeMXBeansForServer https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3288
org.apache.geode.management.internal.cli.commands.DescribeClientCommandDUnitTest: 1 failures (99.556% success rate)
describeClient https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3218
org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest: 1 failures (99.556% success rate)
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3225
org.apache.geode.cache30.DiskDistributedNoAckSyncOverflowRegionDUnitTest: 1 failures (99.556% success rate)
testRegionIdleDestroy https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3310
Re: [INFO] Distributed Test runs in bulk results.
Posted by Anthony Baker <ba...@vmware.com>.
Thanks Mark! Can you highlight any surprises in these results compared to previous runs?
Anthony
> On Jun 10, 2020, at 10:59 PM, Mark Hanson <ha...@vmware.com> wrote:
>
> Hello All,
>
> I have been doing bulk test runs of DistributedTestOpenJDK8, in this case over 200. Here is a simplified report to kind of help you see what I am seeing and I think everybody sees with random failures as part of the PR process.
>
> It is very easy to cause failures like this by not knowing what is running asynchronous and Geode is a complex system or introducing timing constraints that may not hold up in the system e.g. waiting 5 seconds for a test result that could take longer unbeknownst you.
>
> All of that said, here are the results. There are tickets already open for most if not all of these issues.
>
> Please let me know how often you all would like to see these reports…
>
> Thanks,
> Mark
>
>
> ***********************************************************************************
> Overall build success rate: 84.00000%
>
>
> The following test methods see failures in more than one class. There may be a failing *TestBase class
>
> *.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived: 18 failures :
> ParallelWANPersistenceEnabledGatewaySenderDUnitTest: 7 failures (96.889% success rate)
> ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 11 failures (95.111% success rate)
>
> *.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived: 4 failures :
> SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 3 failures (98.667% success rate)
> SerialWANPersistenceEnabledGatewaySenderDUnitTest: 1 failures (99.556% success rate)
>
> ***********************************************************************************
>
>
> org.apache.geode.management.MemberMXBeanDistributedTest: 3 failures (98.667% success rate)
>
> testBucketCount https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3247&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730317409&sdata=HRdBnvmthfuddRqVjC8T9EzMufHQ1959MIAcPzJINm0%3D&reserved=0
> testBucketCount https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3241&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404&sdata=DxqiSO%2BbdLgvEND5SdqiWUu8vpKGhNsOvKhWjFydxDA%3D&reserved=0
> testBucketCount https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3199&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404&sdata=ZhrecVfpX9T5NJMFsuxhU0QQbx3LwJx3QjVzK3EvXbo%3D&reserved=0
>
> org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest: 7 failures (96.889% success rate)
>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3335&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404&sdata=NzB40k6LwF5cj5D20XL7CyOO7eGAQHuWe0MUzfDKn%2BM%3D&reserved=0
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3331&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404&sdata=%2F4x18mauNSUWf5wng3AU%2B0RNjrMoa0M9X5hfohbf47I%3D&reserved=0
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3294&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404&sdata=O7rOO971KGc2nUS3I6yAFlKDnmcTEJPIiLDmuAw%2B4Q0%3D&reserved=0
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3285&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404&sdata=CT5x%2BVpOX0pGf7o8dUPvrplysZYHBSACwzc4EKvdUWk%3D&reserved=0
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3218&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404&sdata=X7MUpSDdrbLXzQxSlvymWfwnpPPkl6Sau8J%2FbMp%2BLL0%3D&reserved=0
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3180&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404&sdata=%2Bu9ai5DBb2LjNEoOo3wb0RMjNq20drTXTKRKeA%2F2ep0%3D&reserved=0
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3156&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404&sdata=ksd4BSwELeEbGXVDbhLuaueFX%2Fj5POdNRXceJ%2Fa7azs%3D&reserved=0
>
> org.apache.geode.internal.cache.partitioned.PersistentPartitionedRegionDistributedTest: 1 failures (99.556% success rate)
>
> testCacheCloseDuringBucketMoveDoesntCauseDataLoss https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3267&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404&sdata=ATJif7SqryvcTo0CLr%2Fk%2BJZLryT9Tajn4OTkrNn5sak%3D&reserved=0
>
> org.apache.geode.cache.management.MemoryThresholdsOffHeapDUnitTest: 1 failures (99.556% success rate)
>
> testDistributedRegionClientPutRejection https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3319&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404&sdata=ZItyH1OYkKl44BBm%2B%2BmxUwIoScNNOd5NyY0YcDGHtcE%3D&reserved=0
>
> org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 3 failures (98.667% success rate)
>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3239&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404&sdata=4eRedK180yq%2BPckaRlsaJOnSZaAajy9pEi0wZ0TOuYs%3D&reserved=0
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3148&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404&sdata=i94UhFrk%2FM8Ux%2FBxIcJPhOxY26V6RV979H%2FEaZyzVX0%3D&reserved=0
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3127&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404&sdata=KqUP2M46dTJbNm8FWzqIeTZecURE%2FQigflk8Q5bFiRs%3D&reserved=0
>
> org.apache.geode.test.dunit.tests.AsyncInvocationTimeoutDistributedTest: 3 failures (98.667% success rate)
>
> await_callable_timeout_includesStackTraceAsCause https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3288&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404&sdata=p02SGgv6dsZgbiMpMNWHqmLJ5EkYKaYbbjjAq1r9PGI%3D&reserved=0
> await_callable_timeout_includesStackTraceAsCause https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3242&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404&sdata=E%2Flcmv6%2Byzs5eJhZ%2F1c%2FTeA4T7lEHLc%2Fl2FAgzciORs%3D&reserved=0
> await_callable_timeout_includesStackTraceAsCause https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3177&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404&sdata=kgBiyFoSKjbdjQjKARejsFVnQtF2OxilMiBfNStsVhQ%3D&reserved=0
>
> org.apache.geode.internal.cache.tier.sockets.ReliableMessagingDUnitTest: 1 failures (99.556% success rate)
>
> testPeriodicAckSendByClientPrimaryFailover https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3304&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730337401&sdata=%2FCchFNzNiQmgNM2xndmr7tyvZm86InOMaUYVYdXsKyQ%3D&reserved=0
>
> org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 11 failures (95.111% success rate)
>
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3309&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730337401&sdata=FMT%2BgqH4eyWr5%2F56p5jIJYcuCErNeb4Pd9%2BjI7FAj%2BI%3D&reserved=0
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3305&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730337401&sdata=JF8hkquIFHQReFjbZnBYxzTsy6dGlke1ohWNUzfngZY%3D&reserved=0
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3279&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730337401&sdata=DUHycn09u3LJ3F%2FQWNrQ85k%2BTlWlvEPP1Ggq5P6xHQs%3D&reserved=0
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3273&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730337401&sdata=2VkMiFmvILYayn7Urn1J7KNaZHfSUCBJRMTD92vJ7jQ%3D&reserved=0
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3257&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730337401&sdata=kfGe5LxqA1Qsad4KdtHqim%2B3rvmNMW%2FYFpaqLghdERc%3D&reserved=0
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3210&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730337401&sdata=kgG5tEbclpXWcJALvXgL1zkzcXtjuEQn1dBNMxx3xCo%3D&reserved=0
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3181&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730337401&sdata=PplBqiNYrrrkv8BXtX4YQxFjBpX0wt2XJJxHrFoz7FI%3D&reserved=0
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3166&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730337401&sdata=Skv9Vj2OsKHtmhHX6pVeoERnk1fZiPbrdHN5FkvrQkQ%3D&reserved=0
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3158&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730337401&sdata=IE8bamIZCHMzFBYCFkKP8Row4htW0zsPoLX%2FQaaiSpI%3D&reserved=0
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3137&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730337401&sdata=TLLRiobZjL0WjutigOFy2Ql783GlStuTMJJPU6nu7VY%3D&reserved=0
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3130&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730337401&sdata=X5l70G8cD3Ek4fZdrQtQpmnKzoVmoLoBYo%2Bjsbe0WMs%3D&reserved=0
>
> org.apache.geode.management.JMXMBeanReconnectDUnitTest: 7 failures (96.889% success rate)
>
> serverHasMemberTypeMXBeans https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3151&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730337401&sdata=2%2BQORdwtsqcRX0Xt7HIDx8A4%2FHCOVZ8W1oMebL9w9l4%3D&reserved=0
> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3319&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730337401&sdata=Vs%2FdPvmPiXdThiLA%2FvS1mszoqEultDnDfimFhLLP8HY%3D&reserved=0
> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3298&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730337401&sdata=oA1RDrtfp1MCCvGWq0lEAgrmrITqpI5es7q3LG4MR1I%3D&reserved=0
> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3285&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730337401&sdata=2hlxfLjVWNd6icLoT68Jd1GUO7lkUimAoZtH%2BjUOM4E%3D&reserved=0
> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3193&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730347388&sdata=Y85zMb7Zg2dJL6QIIMFOYzaJAT3Qkh0w2UTu48ZuqfA%3D&reserved=0
> serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3325&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730347388&sdata=%2FBEF7pVAIQ5zNv1gMq%2BRw%2FYHAMNOEZw9LXXEk5rdhu0%3D&reserved=0
> locatorsHaveMemberTypeMXBeansForServer https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3288&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730347388&sdata=Pg762iU3uTeT4G2N%2BvUVk7A8JDTpRUUqLhisoHYsaTA%3D&reserved=0
>
> org.apache.geode.management.internal.cli.commands.DescribeClientCommandDUnitTest: 1 failures (99.556% success rate)
>
> describeClient https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3218&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730347388&sdata=AfbqZb12R%2FHT5e0NdwdiA2%2BSWhlqWO%2Fx4%2FeFAK9kDyE%3D&reserved=0
>
> org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest: 1 failures (99.556% success rate)
>
> testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3225&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730347388&sdata=i2l83QtQ%2BMRjC5wRzLh9MMTIMQrbxAeeU7uaIJl1Ocs%3D&reserved=0
>
> org.apache.geode.cache30.DiskDistributedNoAckSyncOverflowRegionDUnitTest: 1 failures (99.556% success rate)
>
> testRegionIdleDestroy https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3310&data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730347388&sdata=jSTTAN1JuHmUyCOdPo0TYEHLA8Q%2Bo5GSqP9Q1GyRQ8Y%3D&reserved=0
>