This week's mass-test run results

[Alexander Murmann <am...@apache.org>]

Hi everyone,

Just like Mark did two weeks ago, I'd like to bring some attention to our
mass test runs. These now run in the pipeline
<https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-mass-test-run>
once a week. The results should typically be available on Tuesdays.

*Context*
Some context since we only recently started running these in our pipeline:
In the past we've seen the number of flaky tests, in particular among our
DUnite tests, grow over time. This would eventually lead to death by
thousand paper cuts when trying to merge a PR. The increased noise in our
test suite also gives room for new signals about actual bugs to hide in. In
past efforts to combat the flakiness of our test suite we have found that
running the tests many times against a non-broken SHA gave us great data to
identify which tests contribute most flakiness to the test suite.

*How does this week look?*
This week
<https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-mass-test-run/jobs/create-mass-test-run-report/builds/8>
doesn't look much different than last week. We got a 78% pass-rate

The test with the single highest fail rate
remains org.apache.geode.management.JMXMBeanReconnectDUnitTest with a fail
rate of 4.5%. I believe Kirk had been working on this for a prolonged time.
Kirk, is this still on your radar?

The other big issue remains a cluster of issues around WAN Gateway that
caused 18 of the total 44 failures we saw in our 200 runs
I believe the WAN test issues got introduced fairly recently by GEODE-7458.

If we got just the JMXMBean issue and the WAN tests addressed, our fail
rate should go back to well above 90% which would be a big win

Re: This week's mass-test run results

[Kirk Lund <kl...@apache.org>]

Yes, I'm still working on JMXMBeanReconnectDUnitTest. The test fails
intermittently due to a couple product bugs that the test found. I've filed
separate tickets for these bugs and I expect the test's flakiness to be
fixed when the bugs have been addressed.

-Kirk

On Tue, Jun 30, 2020 at 2:57 PM Alexander Murmann <am...@apache.org>
wrote:

> Hi everyone,
>
> Just like Mark did two weeks ago, I'd like to bring some attention to our
> mass test runs. These now run in the pipeline
> <
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-mass-test-run
> >
> once a week. The results should typically be available on Tuesdays.
>
> *Context*
> Some context since we only recently started running these in our pipeline:
> In the past we've seen the number of flaky tests, in particular among our
> DUnite tests, grow over time. This would eventually lead to death by
> thousand paper cuts when trying to merge a PR. The increased noise in our
> test suite also gives room for new signals about actual bugs to hide in. In
> past efforts to combat the flakiness of our test suite we have found that
> running the tests many times against a non-broken SHA gave us great data to
> identify which tests contribute most flakiness to the test suite.
>
> *How does this week look?*
> This week
> <
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-mass-test-run/jobs/create-mass-test-run-report/builds/8
> >
> doesn't look much different than last week. We got a 78% pass-rate
>
> The test with the single highest fail rate
> remains org.apache.geode.management.JMXMBeanReconnectDUnitTest with a fail
> rate of 4.5%. I believe Kirk had been working on this for a prolonged time.
> Kirk, is this still on your radar?
>
> The other big issue remains a cluster of issues around WAN Gateway that
> caused 18 of the total 44 failures we saw in our 200 runs
> I believe the WAN test issues got introduced fairly recently by GEODE-7458.
>
> If we got just the JMXMBean issue and the WAN tests addressed, our fail
> rate should go back to well above 90% which would be a big win
>