Re: Disabling flaky tests

[Eric Norman <er...@gmail.com>]

Personally, I'm not a big fan of hiding flaky/failing tests since it tends
to remove some of the motivation to stabilize/fix them in a timely manner.

That's my 2 cents.

Regards,
Eric

On Fri, May 31, 2013 at 12:14 PM, Robert Munteanu <ro...@apache.org>wrote:

> Hi,
>
> It seems that the ErrorHandlingTest fails sporadically when run inside a
> full maven build. I've tried locating the root cause for a couple of
> hours but failed. For this test, and for future flaky/failing tests, I
> suggest that we
>
> 1. Create an issue for the failing test
> 2. Disable the test and mark it with the issue key
> 3. Re-enable the test when it is stable/passing ( which may be
> considerably later than step 2)
> 4. Close the issue after the test is re-enabled
>
> This has the advantage of keeping the build green and making it easier
> to find regressions since a failing or unstable build will actually mean
> something.
>
> What do you think?
>
> Robert
>
>

Re: Disabling flaky tests

[Bertrand Delacretaz <bd...@apache.org>]

On Tue, Jun 4, 2013 at 6:37 PM, Carsten Ziegeler <cz...@apache.org> wrote:
> ...I would agree if the jenkins we have would be stable; i have the feeling
> that a lot of build failures on that instance are simply because of some
> problems of the build server itself...

There were issues in the last few weeks but
https://builds.apache.org/view/S-Z/view/Sling/ looks better now IMO.

Builds have been failing there in the last few days for rational
reasons - I just fixed SLING-2903 which broke the build yesterday
(shame on me ;-)

We also have or had tests that fail semi-randomly, like SLING-2818
which looks good now since I added retries. We have quite a lot of
asynchronous setup points in Sling, which means our tests must
sometimes wait for things to settle after setting up scripts of
configs, that complicates things a bit.

-Bertrand

Re: Disabling flaky tests

[Carsten Ziegeler <cz...@apache.org>]

I would agree if the jenkins we have would be stable; i have the feeling
that a lot of build failures on that instance are simply because of some
problems of the build server itself. I tried to chased too many times such
issues. For example, right now, every day we get build errors reported in
some modules while at the next day they pass - without any changes at all.

So I think the first step is to have a stable build env

But I'm fine with any direction as long as we fix known bugs before a
release

Carsten


2013/6/4 Jeff Young <je...@adobe.com>

> I used to agree as well, but my opinion is now more nuanced.  I've
> experienced projects where a test keeps failing day after day, and after a
> while developers stop looking at the test results with the same level of
> discipline.
>
> Perhaps Sling is small enough (and the developers are pro-active enough)
> that this isn't an issue.  But it certainly is on some other, larger, more
> disperse projects (such as CQ).  In those, moving a failing test to an
> issue (which can be assigned to an individual) can produce better results
> than everyone simply getting used to the build being red.
>
> Cheers,
> Jeff.
>
>
> > -----Original Message-----
> > From: Carsten Ziegeler [mailto:cziegeler@apache.org]
> > Sent: 03 June 2013 07:01
> > To: dev@sling.apache.org
> > Subject: Re: Disabling flaky tests
> >
> > I agree as well, especially for the error handling as this is partially
> not
> > a problem of the test but really a bug in Sling - we have an issue for
> > that, it just needs to be done :)
> >
> > Carsten
> >
> >
> > 2013/6/3 Felix Meschberger <fm...@adobe.com>
> >
> > > I agree here: Disabling the test and having an issue keeps the build
> green
> > > but bears the danger of forgetting about it ...
> > >
> > > Regards
> > > Felix
> > >
> > > Am 02.06.2013 um 16:04 schrieb Eric Norman:
> > >
> > > > Personally, I'm not a big fan of hiding flaky/failing tests since it
> > > tends
> > > > to remove some of the motivation to stabilize/fix them in a timely
> > > manner.
> > > >
> > > > That's my 2 cents.
> > > >
> > > > Regards,
> > > > Eric
> > > >
> > > > On Fri, May 31, 2013 at 12:14 PM, Robert Munteanu <
> rombert@apache.org
> > > >wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> It seems that the ErrorHandlingTest fails sporadically when run
> inside a
> > > >> full maven build. I've tried locating the root cause for a couple of
> > > >> hours but failed. For this test, and for future flaky/failing
> tests, I
> > > >> suggest that we
> > > >>
> > > >> 1. Create an issue for the failing test
> > > >> 2. Disable the test and mark it with the issue key
> > > >> 3. Re-enable the test when it is stable/passing ( which may be
> > > >> considerably later than step 2)
> > > >> 4. Close the issue after the test is re-enabled
> > > >>
> > > >> This has the advantage of keeping the build green and making it
> easier
> > > >> to find regressions since a failing or unstable build will actually
> mean
> > > >> something.
> > > >>
> > > >> What do you think?
> > > >>
> > > >> Robert
> > > >>
> > > >>
> > >
> > >
> >
> >
> > --
> > Carsten Ziegeler
> > cziegeler@apache.org
>



-- 
Carsten Ziegeler
cziegeler@apache.org

RE: Disabling flaky tests

[Jeff Young <je...@adobe.com>]

I used to agree as well, but my opinion is now more nuanced.  I've experienced projects where a test keeps failing day after day, and after a while developers stop looking at the test results with the same level of discipline.

Perhaps Sling is small enough (and the developers are pro-active enough) that this isn't an issue.  But it certainly is on some other, larger, more disperse projects (such as CQ).  In those, moving a failing test to an issue (which can be assigned to an individual) can produce better results than everyone simply getting used to the build being red.

Cheers,
Jeff.


> -----Original Message-----
> From: Carsten Ziegeler [mailto:cziegeler@apache.org]
> Sent: 03 June 2013 07:01
> To: dev@sling.apache.org
> Subject: Re: Disabling flaky tests
> 
> I agree as well, especially for the error handling as this is partially not
> a problem of the test but really a bug in Sling - we have an issue for
> that, it just needs to be done :)
> 
> Carsten
> 
> 
> 2013/6/3 Felix Meschberger <fm...@adobe.com>
> 
> > I agree here: Disabling the test and having an issue keeps the build green
> > but bears the danger of forgetting about it ...
> >
> > Regards
> > Felix
> >
> > Am 02.06.2013 um 16:04 schrieb Eric Norman:
> >
> > > Personally, I'm not a big fan of hiding flaky/failing tests since it
> > tends
> > > to remove some of the motivation to stabilize/fix them in a timely
> > manner.
> > >
> > > That's my 2 cents.
> > >
> > > Regards,
> > > Eric
> > >
> > > On Fri, May 31, 2013 at 12:14 PM, Robert Munteanu <rombert@apache.org
> > >wrote:
> > >
> > >> Hi,
> > >>
> > >> It seems that the ErrorHandlingTest fails sporadically when run inside a
> > >> full maven build. I've tried locating the root cause for a couple of
> > >> hours but failed. For this test, and for future flaky/failing tests, I
> > >> suggest that we
> > >>
> > >> 1. Create an issue for the failing test
> > >> 2. Disable the test and mark it with the issue key
> > >> 3. Re-enable the test when it is stable/passing ( which may be
> > >> considerably later than step 2)
> > >> 4. Close the issue after the test is re-enabled
> > >>
> > >> This has the advantage of keeping the build green and making it easier
> > >> to find regressions since a failing or unstable build will actually mean
> > >> something.
> > >>
> > >> What do you think?
> > >>
> > >> Robert
> > >>
> > >>
> >
> >
> 
> 
> --
> Carsten Ziegeler
> cziegeler@apache.org

Re: Disabling flaky tests

[Robert Munteanu <ro...@apache.org>]

Hm, my proposal isn't winning any popularity contests, so I won't go
through with it :-)

Re: Disabling flaky tests

[Carsten Ziegeler <cz...@apache.org>]

I agree as well, especially for the error handling as this is partially not
a problem of the test but really a bug in Sling - we have an issue for
that, it just needs to be done :)

Carsten


2013/6/3 Felix Meschberger <fm...@adobe.com>

> I agree here: Disabling the test and having an issue keeps the build green
> but bears the danger of forgetting about it ...
>
> Regards
> Felix
>
> Am 02.06.2013 um 16:04 schrieb Eric Norman:
>
> > Personally, I'm not a big fan of hiding flaky/failing tests since it
> tends
> > to remove some of the motivation to stabilize/fix them in a timely
> manner.
> >
> > That's my 2 cents.
> >
> > Regards,
> > Eric
> >
> > On Fri, May 31, 2013 at 12:14 PM, Robert Munteanu <rombert@apache.org
> >wrote:
> >
> >> Hi,
> >>
> >> It seems that the ErrorHandlingTest fails sporadically when run inside a
> >> full maven build. I've tried locating the root cause for a couple of
> >> hours but failed. For this test, and for future flaky/failing tests, I
> >> suggest that we
> >>
> >> 1. Create an issue for the failing test
> >> 2. Disable the test and mark it with the issue key
> >> 3. Re-enable the test when it is stable/passing ( which may be
> >> considerably later than step 2)
> >> 4. Close the issue after the test is re-enabled
> >>
> >> This has the advantage of keeping the build green and making it easier
> >> to find regressions since a failing or unstable build will actually mean
> >> something.
> >>
> >> What do you think?
> >>
> >> Robert
> >>
> >>
>
>


-- 
Carsten Ziegeler
cziegeler@apache.org

Re: Disabling flaky tests

[Felix Meschberger <fm...@adobe.com>]

I agree here: Disabling the test and having an issue keeps the build green but bears the danger of forgetting about it ...

Regards
Felix

Am 02.06.2013 um 16:04 schrieb Eric Norman:

> Personally, I'm not a big fan of hiding flaky/failing tests since it tends
> to remove some of the motivation to stabilize/fix them in a timely manner.
> 
> That's my 2 cents.
> 
> Regards,
> Eric
> 
> On Fri, May 31, 2013 at 12:14 PM, Robert Munteanu <ro...@apache.org>wrote:
> 
>> Hi,
>> 
>> It seems that the ErrorHandlingTest fails sporadically when run inside a
>> full maven build. I've tried locating the root cause for a couple of
>> hours but failed. For this test, and for future flaky/failing tests, I
>> suggest that we
>> 
>> 1. Create an issue for the failing test
>> 2. Disable the test and mark it with the issue key
>> 3. Re-enable the test when it is stable/passing ( which may be
>> considerably later than step 2)
>> 4. Close the issue after the test is re-enabled
>> 
>> This has the advantage of keeping the build green and making it easier
>> to find regressions since a failing or unstable build will actually mean
>> something.
>> 
>> What do you think?
>> 
>> Robert
>> 
>>