Flaky tests and CI

[Paul Merlin <pa...@nosphere.org>]

Gang,

We have some flaky tests.

We know about Metrics and Scheduler tests.
Both are time-related.

For Metrics, maybe a simple retry strategy would do. WDYT Niclas?

The Scheduler library situation is not good.
I'm tempted to simply disable the tests until a rewrite occurs.
 
And the new kid on the block is the Rest library.
Tests started to fail from time to time with root cause of "Address
already in use".
The tests starts the rest server on port 8192 that may be used by
something else, on o.a.builds it is sometime used by something, it
appears to be more frequent recently.
The tests should be adapted to use FreePortFinder from testsupport.

Cheers

Re: Flaky tests and CI

[Paul Merlin <pa...@nosphere.org>]

Niclas Hedhman a �crit :
> I agree.

Metrics tests will now be retried 3 times up until success.

> I am even at the point of killing Scheduler completely for now, since there
> are critical bugs in it, and scheduling might be better done with external
> trigger.

Yeah, let's ditch it!

> Rest tests, sure go ahead. I also think that it need a major overhaul,
> since none of us are familiar with the library and can't support it.

Hum, I won't dive into this and do any overhaul :)
But, I added a configuration to the "admin" server so we can set the port.
And the tests now use a dynamically allowed free port.
This should prevent the CI builds to fail intermitently and send useless
emails to dev@.

Cheers


> On Jun 15, 2016 01:31, "Paul Merlin" <pa...@nosphere.org> wrote
>> Gang,
>>
>> We have some flaky tests.
>>
>> We know about Metrics and Scheduler tests.
>> Both are time-related.
>>
>> For Metrics, maybe a simple retry strategy would do. WDYT Niclas?
>>
>> The Scheduler library situation is not good.
>> I'm tempted to simply disable the tests until a rewrite occurs.
>>
>> And the new kid on the block is the Rest library.
>> Tests started to fail from time to time with root cause of "Address
>> already in use".
>> The tests starts the rest server on port 8192 that may be used by
>> something else, on o.a.builds it is sometime used by something, it
>> appears to be more frequent recently.
>> The tests should be adapted to use FreePortFinder from testsupport.
>>
>> Cheers
>>
>>
>

Re: Flaky tests and CI

[Niclas Hedhman <he...@gmail.com>]

I agree.
I am even at the point of killing Scheduler completely for now, since there
are critical bugs in it, and scheduling might be better done with external
trigger.

Rest tests, sure go ahead. I also think that it need a major overhaul,
since none of us are familiar with the library and can't support it.

Niclas
On Jun 15, 2016 01:31, "Paul Merlin" <pa...@nosphere.org> wrote:

> Gang,
>
> We have some flaky tests.
>
> We know about Metrics and Scheduler tests.
> Both are time-related.
>
> For Metrics, maybe a simple retry strategy would do. WDYT Niclas?
>
> The Scheduler library situation is not good.
> I'm tempted to simply disable the tests until a rewrite occurs.
>
> And the new kid on the block is the Rest library.
> Tests started to fail from time to time with root cause of "Address
> already in use".
> The tests starts the rest server on port 8192 that may be used by
> something else, on o.a.builds it is sometime used by something, it
> appears to be more frequent recently.
> The tests should be adapted to use FreePortFinder from testsupport.
>
> Cheers
>
>