16 days without a green build

[Roman Shaposhnik <ro...@shaposhnik.org>]

Hi!

it seems that we're running into various spurious test
failures. What's the plan to get it back to green again?

Thanks,
Roman.

Re: 16 days without a green build

[Kirk Lund <ki...@gmail.com>]

I've created GEODE-218 "Change distributedTest task to fork every 1 test
case" for review and group discussion.

-Kirk


On Thu, Aug 13, 2015 at 5:29 PM, Kirk Lund <ki...@gmail.com> wrote:

> For spurious failures, the problematic tests are predominantly dunit tests
> which are actually end-to-end tests involving 6 jvms. End-to-end tests are
> notorious for having reliability issues. This is magnified by reusing jvms
> from one test case to the next. Eventually, one of the 6 jvms hits a
> problem: 1) a race condition (writting a good distributed test is tricky)
> or 2) a long stop-the-world GC pause which causes a test to fail with a
> timeout or pause long enough to hit a more rare race condition. Since the
> jvms are reused, a test failure is also very likely to pollute the jvm
> which can in turn cause subsequent test failure(s).
>
> The best overall way to improve the reliability of these dunit tests is to
> fork brand-new jvms for every dunit test case which would force each test
> case to run in isolation. I think changing build.gradle distributedTest
> task from "forkEvery 30" to "forkEvery 1" would help greatly. I'll file a
> JIRA ticket for forking the dunit jvms every test and propose it on dev.
>
> Updating dunit to use junit 4 syntax would help a little by allowing us to
> use some newer features and rules that are suitable to integration or
> end-to-end tests. This is already under way with GEODE-217.
>
> End-to-end tests are important but have some issues:
> 1) spurious failures and reliability problems are common
> 2) excessive redundant code coverage -- they tend to execute the same 10%
> of the code base 90% of the time
> 3) long execution time
>
> I would be remiss if I didn't use this opportunity to encourage the Geode
> dev community to refocus on writing true unit tests with Mockito or JMock
> while "moderating" the creation of new dunit (end-to-end) tests.
>
> We have our automated developer tests categorized as UnitTest,
> IntegrationTest and DistributedTest. True unit tests written using Mockito
> or JMock do result in meaningful, worthwhile tests both as "tests" and as
> design tools even for a code base such as Geode. These unit tests no less
> important than integration tests or end-to-end tests (both are still
> important and all three provide different types of complementary test
> coverage).
>
> Lastly a list of my favorite books (so far) on unit testing:
>
> Working Effectively with Legacy Code by Michael Feathers
> Practical Unit Testing with JUnit and Mockito by Tomek Kaczanowski
> Effective Unit Testing by Lasse Koskela
> Growing Object-Oriented Software, Guided by Tests by Steve Freeman and Nat
> Pryce
>
> Working Effectively with Legacy Code in particular covers best practices
> for breaking dependencies so you can isolate a class for testing it.
>
> -Kirk
>
> On Thu, Aug 13, 2015 at 12:55 PM, Roman Shaposhnik <ro...@shaposhnik.org>
> wrote:
>
>> Hi!
>>
>> it seems that we're running into various spurious test
>> failures. What's the plan to get it back to green again?
>>
>> Thanks,
>> Roman.
>>
>
>

Re: 16 days without a green build

[Kirk Lund <ki...@gmail.com>]

For spurious failures, the problematic tests are predominantly dunit tests
which are actually end-to-end tests involving 6 jvms. End-to-end tests are
notorious for having reliability issues. This is magnified by reusing jvms
from one test case to the next. Eventually, one of the 6 jvms hits a
problem: 1) a race condition (writting a good distributed test is tricky)
or 2) a long stop-the-world GC pause which causes a test to fail with a
timeout or pause long enough to hit a more rare race condition. Since the
jvms are reused, a test failure is also very likely to pollute the jvm
which can in turn cause subsequent test failure(s).

The best overall way to improve the reliability of these dunit tests is to
fork brand-new jvms for every dunit test case which would force each test
case to run in isolation. I think changing build.gradle distributedTest
task from "forkEvery 30" to "forkEvery 1" would help greatly. I'll file a
JIRA ticket for forking the dunit jvms every test and propose it on dev.

Updating dunit to use junit 4 syntax would help a little by allowing us to
use some newer features and rules that are suitable to integration or
end-to-end tests. This is already under way with GEODE-217.

End-to-end tests are important but have some issues:
1) spurious failures and reliability problems are common
2) excessive redundant code coverage -- they tend to execute the same 10%
of the code base 90% of the time
3) long execution time

I would be remiss if I didn't use this opportunity to encourage the Geode
dev community to refocus on writing true unit tests with Mockito or JMock
while "moderating" the creation of new dunit (end-to-end) tests.

We have our automated developer tests categorized as UnitTest,
IntegrationTest and DistributedTest. True unit tests written using Mockito
or JMock do result in meaningful, worthwhile tests both as "tests" and as
design tools even for a code base such as Geode. These unit tests no less
important than integration tests or end-to-end tests (both are still
important and all three provide different types of complementary test
coverage).

Lastly a list of my favorite books (so far) on unit testing:

Working Effectively with Legacy Code by Michael Feathers
Practical Unit Testing with JUnit and Mockito by Tomek Kaczanowski
Effective Unit Testing by Lasse Koskela
Growing Object-Oriented Software, Guided by Tests by Steve Freeman and Nat
Pryce

Working Effectively with Legacy Code in particular covers best practices
for breaking dependencies so you can isolate a class for testing it.

-Kirk

On Thu, Aug 13, 2015 at 12:55 PM, Roman Shaposhnik <ro...@shaposhnik.org>
wrote:

> Hi!
>
> it seems that we're running into various spurious test
> failures. What's the plan to get it back to green again?
>
> Thanks,
> Roman.
>