You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Apekshit Sharma <ap...@cloudera.com> on 2016/05/16 23:38:35 UTC
Smart Flaky Handler
This mail is to introduce the work to tackle the flaky tests in our build.
*Why is it important?*
- Our build history sucks, last 175 post-commit runs failed. We need to
make it useful.
- To better understand our code’s testing status, more importantly it’s
weak points.
- We know those 2-3 tests which keep failing every now and then, but not
those ~10 nasty ones which fail like 1 out of 50 times, and screw our build.
- This isn’t something that can be done manually on a daily basis. We need
automation.
*Changes made so far:*
Code changes: HBASE-15839
<https://issues.apache.org/jira/browse/HBASE-15839> (Umbrella issue)
*Jenkins changes:*
[Diagram link:
https://issues.apache.org/jira/secure/attachment/12804292/Screen%20Shot%202016-05-16%20at%204.02.46%20PM.png
]
*(new job) HBase-Find-Flaky-Tests*: Gets test reports of recent builds of
post-commit job (TRUNK_matrix) and HBase-Flaky-Tests job (see below) to
find flaky tests. Frequency of run determines how fast we catch test
regressions. So if we run it every 4 hours, any test which started failing
in post-commit job (TRUNK_matrix) in last 4 hour will be blacklisted.
*(new job) HBase-Flaky-Tests*: This job runs only the flaky tests. The aim
is to run this job back-to-back to collect as many runs as we can. Higher
the run rate, the better will be our system at catching the flaky tests. We
currently run it hourly. so we’ll be able to keep track of flaky tests with
~5% failure rate or more.
*Post-commit (TRUNK_matrix) and pre-commit jobs*: Exclude these flaky tests.
*So what if a bad commit makes a good test bad?*
Since the test is not bad, it’ll run in next post-commit and will fail.
Next run of HBase-Find-Flaky-Tests will pick it up and blacklist it.
Blacklisting will help keep the post-commit job and more importantly
pre-commit job clean, a problem we face quite often.
*Are we just tucking away are shit?*
Nope, this will help us:
- first, Maintain a list of bad test (we lack that today).
- second, make our build greener to the point that a failed/red build is
something we worry about seriously.
Once we are confident that the system is working fine, we’ll setup up
HBase-Find-Flaky-Tests job to send reports to dev@hbase so that devs know
about the bad tests. If it remains hidden somewhere in a jenkins job’s
archive, it’s unlike that we’ll actively work on getting them fixed :).
I'll keep posting further updates on this thread.
-- Appy
Re: Smart Flaky Handler
Posted by Apekshit Sharma <ap...@cloudera.com>.
>
> Should we change the includes and excludes lists so they have a file type ending?
> .txt? Then I could open them easily in the browser. Currently I have to
> download them.
Clicking on 'view' next to the filename in artifacts list will show the
contents directly.
Whats the '**/' about? Is it supposed to have opening/closing versions?
There are two files includes and excludes because the two maven flags
(-Dtest and -Dtest.exclude.pattern) to include and exclude tests to be run
require different format. In surefire 2.19.1, this wouldn't have been
required since the two flags, surefire.excludesFile and
surefire.includesFile, can use same file containing list of tests. We tried
updating to 2.19.1, but started seeing lots of timeouts so reverted back.
Two files isn't pretty, but works with what we have.
What do I need to do to get it wired up for branch-1.1
Haven't thought about it fully since i was more focused on making it robust
first (see the coming changes below). But it may be simply
- setting up another job like HBase-Find-Flaky-Tests[1] which builds flaky
list for 1.1
- then based on branch, we can source the list of flaky tests from
appropriate Find-Flaky-Tests job
any suggestion on how to make people aware of the tests being flaky?
Once it's complete, maybe set it up to send mails to dev@hbase?
Although it's good now, there are few changes i have in mind which can make
it dramatically better.
Next changes:
1. Change the report-flakies.py script to also report timed-out tests.
2. Currently we use last 25 builds of HBase-Flaky-Tests [2] to see
'flakyness' of tests. I want to increase it to something like 50 or 100.
The idea is, it's much harder to detect tests which fail 1% of the time
than those which fail 20% time. But 25 such test (which fail 1%) will make
our builds fail >25% of time.
So one of the main goal is to bring down runtime of this job. I'll be doing
that by assigning appropriate timeouts to these bad tests individually, so
that they fail fast.
3. Last but most important change. Run all tests (good and flakies)
everytime and use "-fn" (fail never) option in maven builds. Then instead
of deciding success/failure based on maven's status, self analyze the test
results and decide pass/fail taking into account flaky tests list. This
might be get tricky and will have to see how things fit with yetus.
[1] https://builds.apache.org/job/HBASE-Find-Flaky-Tests/
[2] https://builds.apache.org/job/HBase-Flaky-Tests/
On Fri, May 20, 2016 at 1:17 PM, Matteo Bertozzi <th...@gmail.com>
wrote:
> any suggestion on how to make people aware of the tests being flaky?
>
> for example I will have never notice the procedure test being flaky if was
> not for stack posting the list here.
> so, maybe a weekly digest in the dev-list with the list of flaky will get
> more audience than having people go into the job.
>
> also, I was thinking about how do I notice if I broke something when I post
> a patch.
> since we exclude the flakys from the run, there is no way I can notice I
> broke something from QA.
> maybe we can add a section in QA that runs the flaky ones and tells you
> "those are failed but may be flaky"
> and at least can look if the failures are related to the patch or is just
> flaky.
>
> On Fri, May 20, 2016 at 11:03 AM, Nick Dimiduk <nd...@gmail.com> wrote:
>
> > Nice work Appy! What do I need to do to get it wired up for branch-1.1?
> >
> > On Fri, May 20, 2016 at 9:25 AM, Stack <st...@duboce.net> wrote:
> >
> > > The system seems to be working nicely Appy. We are getting green
> > precommit
> > > builds for the first time in ages.
> > >
> > > Should we change the includes and excludes lists so they have a file
> type
> > > ending? .txt? Then I could open them easily in the browser. Currently I
> > > have to download them.
> > >
> > > Includes are tests that are currently considered 'flakey'?
> > >
> > >
> > >
> >
> TestGenerateDelegationToken,TestMobCompactor,TestRegionServerMetrics,TestAcidGuarantees,TestMasterReplication,TestRowProcessorEndpoint,TestAsyncLogRolling,DynamicLogicExpressionSuite,TestMasterFailoverWithProcedures,TestChoreService,TestScannerHeartbeatMessages,TestWALProcedureStore,TestRegionMergeTransactionOnCluster,TestSaslFanOutOneBlockAsyncDFSOutput,TestReplicationEndpointWithMultipleWAL
> > >
> > > We have a nice list.
> > >
> > > Excludes are:
> > >
> > >
> > >
> >
> **/TestGenerateDelegationToken.java,**/TestMobCompactor.java,**/TestRegionServerMetrics.java,**/TestAcidGuarantees.java,**/TestMasterReplication.java,**/TestRowProcessorEndpoint.java,**/TestAsyncLogRolling.java,**/DynamicLogicExpressionSuite.java,**/TestMasterFailoverWithProcedures.java,**/TestChoreService.java,**/TestScannerHeartbeatMessages.java,**/TestWALProcedureStore.java,**/TestRegionMergeTransactionOnCluster.java,**/TestSaslFanOutOneBlockAsyncDFSOutput.java,**/TestReplicationEndpointWithMultipleWAL.java,
> > >
> > > Whats the '**/' about? Is it supposed to have opening/closing versions?
> > >
> > > Thanks boss,
> > > St.
> > >
> > >
> > >
> > > On Mon, May 16, 2016 at 4:45 PM, Stack <st...@duboce.net> wrote:
> > >
> > > > Sweet!
> > > >
> > > > On Mon, May 16, 2016 at 4:38 PM, Apekshit Sharma <ap...@cloudera.com>
> > > > wrote:
> > > >
> > > >> This mail is to introduce the work to tackle the flaky tests in our
> > > build.
> > > >>
> > > >> *Why is it important?*
> > > >> - Our build history sucks, last 175 post-commit runs failed. We need
> > to
> > > >> make it useful.
> > > >> - To better understand our code’s testing status, more importantly
> > it’s
> > > >> weak points.
> > > >> - We know those 2-3 tests which keep failing every now and then, but
> > not
> > > >> those ~10 nasty ones which fail like 1 out of 50 times, and screw
> our
> > > build.
> > > >> - This isn’t something that can be done manually on a daily basis.
> We
> > > >> need automation.
> > > >>
> > > >> *Changes made so far:*
> > > >> Code changes: HBASE-15839
> > > >> <https://issues.apache.org/jira/browse/HBASE-15839> (Umbrella
> issue)
> > > >>
> > > >> *Jenkins changes:*
> > > >>
> > > >>
> > > >> [Diagram link:
> > > >>
> > >
> >
> https://issues.apache.org/jira/secure/attachment/12804292/Screen%20Shot%202016-05-16%20at%204.02.46%20PM.png
> > > >> ]
> > > >>
> > > >> *(new job) HBase-Find-Flaky-Tests*: Gets test reports of recent
> builds
> > > >> of post-commit job (TRUNK_matrix) and HBase-Flaky-Tests job (see
> > below)
> > > to
> > > >> find flaky tests. Frequency of run determines how fast we catch test
> > > >> regressions. So if we run it every 4 hours, any test which started
> > > failing
> > > >> in post-commit job (TRUNK_matrix) in last 4 hour will be
> blacklisted.
> > > >>
> > > >> *(new job) HBase-Flaky-Tests*: This job runs only the flaky tests.
> The
> > > >> aim is to run this job back-to-back to collect as many runs as we
> can.
> > > >> Higher the run rate, the better will be our system at catching the
> > flaky
> > > >> tests. We currently run it hourly. so we’ll be able to keep track of
> > > flaky
> > > >> tests with ~5% failure rate or more.
> > > >>
> > > >> *Post-commit (TRUNK_matrix) and pre-commit jobs*: Exclude these
> flaky
> > > >> tests.
> > > >>
> > > >>
> > > >> *So what if a bad commit makes a good test bad?*
> > > >> Since the test is not bad, it’ll run in next post-commit and will
> > fail.
> > > >> Next run of HBase-Find-Flaky-Tests will pick it up and blacklist
> it.
> > > >> Blacklisting will help keep the post-commit job and more importantly
> > > >> pre-commit job clean, a problem we face quite often.
> > > >>
> > > >> *Are we just tucking away are shit?*
> > > >> Nope, this will help us:
> > > >> - first, Maintain a list of bad test (we lack that today).
> > > >> - second, make our build greener to the point that a failed/red
> build
> > is
> > > >> something we worry about seriously.
> > > >>
> > > >> Once we are confident that the system is working fine, we’ll setup
> up
> > > >> HBase-Find-Flaky-Tests job to send reports to dev@hbase so that
> devs
> > > >> know about the bad tests. If it remains hidden somewhere in a
> jenkins
> > > job’s
> > > >> archive, it’s unlike that we’ll actively work on getting them fixed
> > :).
> > > >>
> > > >> I'll keep posting further updates on this thread.
> > > >>
> > > >> -- Appy
> > > >>
> > > >
> > > >
> > >
> >
>
--
Regards
Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
650-963-6311
Re: Smart Flaky Handler
Posted by Apekshit Sharma <ap...@cloudera.com>.
Here's the flaky dashboard: http://hbase.x10host.com/flaky-tests/
I had to create an externally hosted web page because html rendered by
Jenkins looks ugly.
https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html
.
On Mon, May 30, 2016 at 12:18 PM, Apekshit Sharma <ap...@cloudera.com> wrote:
> I see, thanks for explaining it to me Dima.
>
> On Mon, May 30, 2016 at 10:18 AM, Dima Spivak <ds...@cloudera.com>
> wrote:
>
>> Running the install phase of the lifecycle will run integration tests
>> (since that phase comes before), so that's by design. Pass the
>> -DskipITs option to prevent that from happening.
>>
>> -Dima
>>
>> On Monday, May 30, 2016, Apekshit Sharma <ap...@cloudera.com> wrote:
>>
>> > Okay, so it was because mvn install -Dtest=foo always runs the
>> integration
>> > tests in addition to test foo.
>> > Whereas if I do mvn test -Dtest=foo, it only runs the test foo.
>> > This seems bug to me. Or is it by choice? If latter, what's the
>> rationale?
>> >
>> > On Mon, May 30, 2016 at 2:37 AM, Apekshit Sharma <appy@cloudera.com
>> > <javascript:;>> wrote:
>> >
>> > > Just a heads up, you might see flaky tests reappear in precommits
>> because
>> > > I had to reset the flaky list. Somehow integration tests started
>> showing
>> > up
>> > > in the list which is weird since they don't run as part of
>> trunk_matrix,
>> > > and were screwing up Flaky-Tests build so had to flush them out.
>> > >
>> > > On Sun, May 22, 2016 at 8:19 PM, Todd Lipcon <todd@cloudera.com
>> > <javascript:;>> wrote:
>> > >
>> > >> On Sun, May 22, 2016 at 10:12 AM, Stack <stack@duboce.net
>> > <javascript:;>> wrote:
>> > >>
>> > >> > On Fri, May 20, 2016 at 3:43 PM, Todd Lipcon <todd@cloudera.com
>> > <javascript:;>> wrote:
>> > >> >
>> > >> > > On Fri, May 20, 2016 at 1:17 PM, Matteo Bertozzi <
>> > >> > theo.bertozzi@gmail.com <javascript:;>>
>> > >> > > wrote:
>> > >> > >
>> > >> > > > any suggestion on how to make people aware of the tests being
>> > flaky?
>> > >> > > >
>> > >> > >
>> > >> > > You guys might consider doing something like what we do for
>> Apache
>> > >> Kudu
>> > >> > > (incubating):
>> > >> > >
>> > >> > > http://dist-test.cloudera.org:8080/ has a dashboard (driven from
>> > our
>> > >> > > flaky-tests job) which shows the percent flakiness of each test,
>> as
>> > >> well
>> > >> > as
>> > >> > > a breakdown of pass/fail rates by revision. We don't
>> automatically
>> > >> email
>> > >> > > these to the list or anything, currently, but would be pretty
>> easy
>> > to
>> > >> set
>> > >> > > up a cron job to do so.
>> > >> > >
>> > >> > > The dashboard is very helpful for prioritizing the de-flaking of
>> the
>> > >> > worst
>> > >> > > offenders, and also useful to quickly drill down and grab failure
>> > logs
>> > >> > from
>> > >> > > the flaky tests themselves.
>> > >> > >
>> > >> > >
>> > >> > Would you suggest copy/paste of your current setup (a python daemon
>> > and
>> > >> a
>> > >> > db instance IIRC)?
>> > >> >
>> > >>
>> > >> Sure, you're welcome (and encouraged) to borrow/steal it. If you make
>> > some
>> > >> improvements, please let us know, though, so we can merge them back
>> to
>> > our
>> > >> copy
>> > >>
>> > >> The code for the server is here:
>> > >>
>> > >>
>> >
>> https://github.com/apache/incubator-kudu/blob/master/build-support/test_result_server.py
>> > >> The only bit that's kudu-specific is the 'parse_test_failure' module
>> -
>> > it
>> > >> has some heuristics to try to pull out the failure error message from
>> > our
>> > >> tests, but could easily be left out.
>> > >>
>> > >> -Todd
>> > >> --
>> > >> Todd Lipcon
>> > >> Software Engineer, Cloudera
>> > >>
>> > >
>> > >
>> > >
>> > > --
>> > >
>> > > Regards
>> > >
>> > > Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California
>> |
>> > > 650-963-6311
>> > >
>> >
>> >
>> >
>> > --
>> >
>> > Regards
>> >
>> > Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
>> > 650-963-6311
>> >
>>
>
>
>
> --
>
> Regards
>
> Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
> 650-963-6311
>
--
Regards
Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
650-963-6311
Re: Smart Flaky Handler
Posted by Apekshit Sharma <ap...@cloudera.com>.
I see, thanks for explaining it to me Dima.
On Mon, May 30, 2016 at 10:18 AM, Dima Spivak <ds...@cloudera.com> wrote:
> Running the install phase of the lifecycle will run integration tests
> (since that phase comes before), so that's by design. Pass the
> -DskipITs option to prevent that from happening.
>
> -Dima
>
> On Monday, May 30, 2016, Apekshit Sharma <ap...@cloudera.com> wrote:
>
> > Okay, so it was because mvn install -Dtest=foo always runs the
> integration
> > tests in addition to test foo.
> > Whereas if I do mvn test -Dtest=foo, it only runs the test foo.
> > This seems bug to me. Or is it by choice? If latter, what's the
> rationale?
> >
> > On Mon, May 30, 2016 at 2:37 AM, Apekshit Sharma <appy@cloudera.com
> > <javascript:;>> wrote:
> >
> > > Just a heads up, you might see flaky tests reappear in precommits
> because
> > > I had to reset the flaky list. Somehow integration tests started
> showing
> > up
> > > in the list which is weird since they don't run as part of
> trunk_matrix,
> > > and were screwing up Flaky-Tests build so had to flush them out.
> > >
> > > On Sun, May 22, 2016 at 8:19 PM, Todd Lipcon <todd@cloudera.com
> > <javascript:;>> wrote:
> > >
> > >> On Sun, May 22, 2016 at 10:12 AM, Stack <stack@duboce.net
> > <javascript:;>> wrote:
> > >>
> > >> > On Fri, May 20, 2016 at 3:43 PM, Todd Lipcon <todd@cloudera.com
> > <javascript:;>> wrote:
> > >> >
> > >> > > On Fri, May 20, 2016 at 1:17 PM, Matteo Bertozzi <
> > >> > theo.bertozzi@gmail.com <javascript:;>>
> > >> > > wrote:
> > >> > >
> > >> > > > any suggestion on how to make people aware of the tests being
> > flaky?
> > >> > > >
> > >> > >
> > >> > > You guys might consider doing something like what we do for Apache
> > >> Kudu
> > >> > > (incubating):
> > >> > >
> > >> > > http://dist-test.cloudera.org:8080/ has a dashboard (driven from
> > our
> > >> > > flaky-tests job) which shows the percent flakiness of each test,
> as
> > >> well
> > >> > as
> > >> > > a breakdown of pass/fail rates by revision. We don't automatically
> > >> email
> > >> > > these to the list or anything, currently, but would be pretty easy
> > to
> > >> set
> > >> > > up a cron job to do so.
> > >> > >
> > >> > > The dashboard is very helpful for prioritizing the de-flaking of
> the
> > >> > worst
> > >> > > offenders, and also useful to quickly drill down and grab failure
> > logs
> > >> > from
> > >> > > the flaky tests themselves.
> > >> > >
> > >> > >
> > >> > Would you suggest copy/paste of your current setup (a python daemon
> > and
> > >> a
> > >> > db instance IIRC)?
> > >> >
> > >>
> > >> Sure, you're welcome (and encouraged) to borrow/steal it. If you make
> > some
> > >> improvements, please let us know, though, so we can merge them back to
> > our
> > >> copy
> > >>
> > >> The code for the server is here:
> > >>
> > >>
> >
> https://github.com/apache/incubator-kudu/blob/master/build-support/test_result_server.py
> > >> The only bit that's kudu-specific is the 'parse_test_failure' module -
> > it
> > >> has some heuristics to try to pull out the failure error message from
> > our
> > >> tests, but could easily be left out.
> > >>
> > >> -Todd
> > >> --
> > >> Todd Lipcon
> > >> Software Engineer, Cloudera
> > >>
> > >
> > >
> > >
> > > --
> > >
> > > Regards
> > >
> > > Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
> > > 650-963-6311
> > >
> >
> >
> >
> > --
> >
> > Regards
> >
> > Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
> > 650-963-6311
> >
>
--
Regards
Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
650-963-6311
Re: Smart Flaky Handler
Posted by Dima Spivak <ds...@cloudera.com>.
Running the install phase of the lifecycle will run integration tests
(since that phase comes before), so that's by design. Pass the
-DskipITs option to prevent that from happening.
-Dima
On Monday, May 30, 2016, Apekshit Sharma <ap...@cloudera.com> wrote:
> Okay, so it was because mvn install -Dtest=foo always runs the integration
> tests in addition to test foo.
> Whereas if I do mvn test -Dtest=foo, it only runs the test foo.
> This seems bug to me. Or is it by choice? If latter, what's the rationale?
>
> On Mon, May 30, 2016 at 2:37 AM, Apekshit Sharma <appy@cloudera.com
> <javascript:;>> wrote:
>
> > Just a heads up, you might see flaky tests reappear in precommits because
> > I had to reset the flaky list. Somehow integration tests started showing
> up
> > in the list which is weird since they don't run as part of trunk_matrix,
> > and were screwing up Flaky-Tests build so had to flush them out.
> >
> > On Sun, May 22, 2016 at 8:19 PM, Todd Lipcon <todd@cloudera.com
> <javascript:;>> wrote:
> >
> >> On Sun, May 22, 2016 at 10:12 AM, Stack <stack@duboce.net
> <javascript:;>> wrote:
> >>
> >> > On Fri, May 20, 2016 at 3:43 PM, Todd Lipcon <todd@cloudera.com
> <javascript:;>> wrote:
> >> >
> >> > > On Fri, May 20, 2016 at 1:17 PM, Matteo Bertozzi <
> >> > theo.bertozzi@gmail.com <javascript:;>>
> >> > > wrote:
> >> > >
> >> > > > any suggestion on how to make people aware of the tests being
> flaky?
> >> > > >
> >> > >
> >> > > You guys might consider doing something like what we do for Apache
> >> Kudu
> >> > > (incubating):
> >> > >
> >> > > http://dist-test.cloudera.org:8080/ has a dashboard (driven from
> our
> >> > > flaky-tests job) which shows the percent flakiness of each test, as
> >> well
> >> > as
> >> > > a breakdown of pass/fail rates by revision. We don't automatically
> >> email
> >> > > these to the list or anything, currently, but would be pretty easy
> to
> >> set
> >> > > up a cron job to do so.
> >> > >
> >> > > The dashboard is very helpful for prioritizing the de-flaking of the
> >> > worst
> >> > > offenders, and also useful to quickly drill down and grab failure
> logs
> >> > from
> >> > > the flaky tests themselves.
> >> > >
> >> > >
> >> > Would you suggest copy/paste of your current setup (a python daemon
> and
> >> a
> >> > db instance IIRC)?
> >> >
> >>
> >> Sure, you're welcome (and encouraged) to borrow/steal it. If you make
> some
> >> improvements, please let us know, though, so we can merge them back to
> our
> >> copy
> >>
> >> The code for the server is here:
> >>
> >>
> https://github.com/apache/incubator-kudu/blob/master/build-support/test_result_server.py
> >> The only bit that's kudu-specific is the 'parse_test_failure' module -
> it
> >> has some heuristics to try to pull out the failure error message from
> our
> >> tests, but could easily be left out.
> >>
> >> -Todd
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >>
> >
> >
> >
> > --
> >
> > Regards
> >
> > Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
> > 650-963-6311
> >
>
>
>
> --
>
> Regards
>
> Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
> 650-963-6311
>
Re: Smart Flaky Handler
Posted by Apekshit Sharma <ap...@cloudera.com>.
Okay, so it was because mvn install -Dtest=foo always runs the integration
tests in addition to test foo.
Whereas if I do mvn test -Dtest=foo, it only runs the test foo.
This seems bug to me. Or is it by choice? If latter, what's the rationale?
On Mon, May 30, 2016 at 2:37 AM, Apekshit Sharma <ap...@cloudera.com> wrote:
> Just a heads up, you might see flaky tests reappear in precommits because
> I had to reset the flaky list. Somehow integration tests started showing up
> in the list which is weird since they don't run as part of trunk_matrix,
> and were screwing up Flaky-Tests build so had to flush them out.
>
> On Sun, May 22, 2016 at 8:19 PM, Todd Lipcon <to...@cloudera.com> wrote:
>
>> On Sun, May 22, 2016 at 10:12 AM, Stack <st...@duboce.net> wrote:
>>
>> > On Fri, May 20, 2016 at 3:43 PM, Todd Lipcon <to...@cloudera.com> wrote:
>> >
>> > > On Fri, May 20, 2016 at 1:17 PM, Matteo Bertozzi <
>> > theo.bertozzi@gmail.com>
>> > > wrote:
>> > >
>> > > > any suggestion on how to make people aware of the tests being flaky?
>> > > >
>> > >
>> > > You guys might consider doing something like what we do for Apache
>> Kudu
>> > > (incubating):
>> > >
>> > > http://dist-test.cloudera.org:8080/ has a dashboard (driven from our
>> > > flaky-tests job) which shows the percent flakiness of each test, as
>> well
>> > as
>> > > a breakdown of pass/fail rates by revision. We don't automatically
>> email
>> > > these to the list or anything, currently, but would be pretty easy to
>> set
>> > > up a cron job to do so.
>> > >
>> > > The dashboard is very helpful for prioritizing the de-flaking of the
>> > worst
>> > > offenders, and also useful to quickly drill down and grab failure logs
>> > from
>> > > the flaky tests themselves.
>> > >
>> > >
>> > Would you suggest copy/paste of your current setup (a python daemon and
>> a
>> > db instance IIRC)?
>> >
>>
>> Sure, you're welcome (and encouraged) to borrow/steal it. If you make some
>> improvements, please let us know, though, so we can merge them back to our
>> copy
>>
>> The code for the server is here:
>>
>> https://github.com/apache/incubator-kudu/blob/master/build-support/test_result_server.py
>> The only bit that's kudu-specific is the 'parse_test_failure' module - it
>> has some heuristics to try to pull out the failure error message from our
>> tests, but could easily be left out.
>>
>> -Todd
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>
>
> --
>
> Regards
>
> Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
> 650-963-6311
>
--
Regards
Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
650-963-6311
Re: Smart Flaky Handler
Posted by Apekshit Sharma <ap...@cloudera.com>.
Just a heads up, you might see flaky tests reappear in precommits because I
had to reset the flaky list. Somehow integration tests started showing up
in the list which is weird since they don't run as part of trunk_matrix,
and were screwing up Flaky-Tests build so had to flush them out.
On Sun, May 22, 2016 at 8:19 PM, Todd Lipcon <to...@cloudera.com> wrote:
> On Sun, May 22, 2016 at 10:12 AM, Stack <st...@duboce.net> wrote:
>
> > On Fri, May 20, 2016 at 3:43 PM, Todd Lipcon <to...@cloudera.com> wrote:
> >
> > > On Fri, May 20, 2016 at 1:17 PM, Matteo Bertozzi <
> > theo.bertozzi@gmail.com>
> > > wrote:
> > >
> > > > any suggestion on how to make people aware of the tests being flaky?
> > > >
> > >
> > > You guys might consider doing something like what we do for Apache Kudu
> > > (incubating):
> > >
> > > http://dist-test.cloudera.org:8080/ has a dashboard (driven from our
> > > flaky-tests job) which shows the percent flakiness of each test, as
> well
> > as
> > > a breakdown of pass/fail rates by revision. We don't automatically
> email
> > > these to the list or anything, currently, but would be pretty easy to
> set
> > > up a cron job to do so.
> > >
> > > The dashboard is very helpful for prioritizing the de-flaking of the
> > worst
> > > offenders, and also useful to quickly drill down and grab failure logs
> > from
> > > the flaky tests themselves.
> > >
> > >
> > Would you suggest copy/paste of your current setup (a python daemon and a
> > db instance IIRC)?
> >
>
> Sure, you're welcome (and encouraged) to borrow/steal it. If you make some
> improvements, please let us know, though, so we can merge them back to our
> copy
>
> The code for the server is here:
>
> https://github.com/apache/incubator-kudu/blob/master/build-support/test_result_server.py
> The only bit that's kudu-specific is the 'parse_test_failure' module - it
> has some heuristics to try to pull out the failure error message from our
> tests, but could easily be left out.
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
--
Regards
Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
650-963-6311
Re: Smart Flaky Handler
Posted by Todd Lipcon <to...@cloudera.com>.
On Sun, May 22, 2016 at 10:12 AM, Stack <st...@duboce.net> wrote:
> On Fri, May 20, 2016 at 3:43 PM, Todd Lipcon <to...@cloudera.com> wrote:
>
> > On Fri, May 20, 2016 at 1:17 PM, Matteo Bertozzi <
> theo.bertozzi@gmail.com>
> > wrote:
> >
> > > any suggestion on how to make people aware of the tests being flaky?
> > >
> >
> > You guys might consider doing something like what we do for Apache Kudu
> > (incubating):
> >
> > http://dist-test.cloudera.org:8080/ has a dashboard (driven from our
> > flaky-tests job) which shows the percent flakiness of each test, as well
> as
> > a breakdown of pass/fail rates by revision. We don't automatically email
> > these to the list or anything, currently, but would be pretty easy to set
> > up a cron job to do so.
> >
> > The dashboard is very helpful for prioritizing the de-flaking of the
> worst
> > offenders, and also useful to quickly drill down and grab failure logs
> from
> > the flaky tests themselves.
> >
> >
> Would you suggest copy/paste of your current setup (a python daemon and a
> db instance IIRC)?
>
Sure, you're welcome (and encouraged) to borrow/steal it. If you make some
improvements, please let us know, though, so we can merge them back to our
copy
The code for the server is here:
https://github.com/apache/incubator-kudu/blob/master/build-support/test_result_server.py
The only bit that's kudu-specific is the 'parse_test_failure' module - it
has some heuristics to try to pull out the failure error message from our
tests, but could easily be left out.
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
Re: Smart Flaky Handler
Posted by Stack <st...@duboce.net>.
On Fri, May 20, 2016 at 3:43 PM, Todd Lipcon <to...@cloudera.com> wrote:
> On Fri, May 20, 2016 at 1:17 PM, Matteo Bertozzi <th...@gmail.com>
> wrote:
>
> > any suggestion on how to make people aware of the tests being flaky?
> >
>
> You guys might consider doing something like what we do for Apache Kudu
> (incubating):
>
> http://dist-test.cloudera.org:8080/ has a dashboard (driven from our
> flaky-tests job) which shows the percent flakiness of each test, as well as
> a breakdown of pass/fail rates by revision. We don't automatically email
> these to the list or anything, currently, but would be pretty easy to set
> up a cron job to do so.
>
> The dashboard is very helpful for prioritizing the de-flaking of the worst
> offenders, and also useful to quickly drill down and grab failure logs from
> the flaky tests themselves.
>
>
Would you suggest copy/paste of your current setup (a python daemon and a
db instance IIRC)?
Thanks Todd,
St.Ack
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: Smart Flaky Handler
Posted by Todd Lipcon <to...@cloudera.com>.
On Fri, May 20, 2016 at 4:03 PM, Apekshit Sharma <ap...@cloudera.com> wrote:
> That's a nice dashboard! I like the spark lines in the last column.
> Let me add creating a dashboard to my todos.
>
> I see denominator of ~400 in 2 day failure rate column. Do you really run
> the flaky tests about 200 times a day?
> Would be hard to get that kind of resource upstream :).
>
Yep, we do. It's not so hard - 200 times a day is only once every 7
minutes. We basically just run the flaky-tests job in a loop, and when
there aren't that many tests that are flaky, they only take a few minutes
total to run. For Kudu in particular, we also run four build configurations
in parallel, so it's really only 50 times per day (x 4 configs). Overall
cost on GCE spot instances is pretty cheap (couple hundred bucks a month)
-Todd
> On Fri, May 20, 2016 at 3:43 PM, Todd Lipcon <to...@cloudera.com> wrote:
>
> > On Fri, May 20, 2016 at 1:17 PM, Matteo Bertozzi <
> theo.bertozzi@gmail.com>
> > wrote:
> >
> > > any suggestion on how to make people aware of the tests being flaky?
> > >
> >
> > You guys might consider doing something like what we do for Apache Kudu
> > (incubating):
> >
> > http://dist-test.cloudera.org:8080/ has a dashboard (driven from our
> > flaky-tests job) which shows the percent flakiness of each test, as well
> as
> > a breakdown of pass/fail rates by revision. We don't automatically email
> > these to the list or anything, currently, but would be pretty easy to set
> > up a cron job to do so.
> >
> > The dashboard is very helpful for prioritizing the de-flaking of the
> worst
> > offenders, and also useful to quickly drill down and grab failure logs
> from
> > the flaky tests themselves.
> >
> > -Todd
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>
>
>
> --
>
> Regards
>
> Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
> 650-963-6311
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Smart Flaky Handler
Posted by Apekshit Sharma <ap...@cloudera.com>.
That's a nice dashboard! I like the spark lines in the last column.
Let me add creating a dashboard to my todos.
I see denominator of ~400 in 2 day failure rate column. Do you really run
the flaky tests about 200 times a day?
Would be hard to get that kind of resource upstream :).
On Fri, May 20, 2016 at 3:43 PM, Todd Lipcon <to...@cloudera.com> wrote:
> On Fri, May 20, 2016 at 1:17 PM, Matteo Bertozzi <th...@gmail.com>
> wrote:
>
> > any suggestion on how to make people aware of the tests being flaky?
> >
>
> You guys might consider doing something like what we do for Apache Kudu
> (incubating):
>
> http://dist-test.cloudera.org:8080/ has a dashboard (driven from our
> flaky-tests job) which shows the percent flakiness of each test, as well as
> a breakdown of pass/fail rates by revision. We don't automatically email
> these to the list or anything, currently, but would be pretty easy to set
> up a cron job to do so.
>
> The dashboard is very helpful for prioritizing the de-flaking of the worst
> offenders, and also useful to quickly drill down and grab failure logs from
> the flaky tests themselves.
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
--
Regards
Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
650-963-6311
Re: Smart Flaky Handler
Posted by Todd Lipcon <to...@cloudera.com>.
On Fri, May 20, 2016 at 1:17 PM, Matteo Bertozzi <th...@gmail.com>
wrote:
> any suggestion on how to make people aware of the tests being flaky?
>
You guys might consider doing something like what we do for Apache Kudu
(incubating):
http://dist-test.cloudera.org:8080/ has a dashboard (driven from our
flaky-tests job) which shows the percent flakiness of each test, as well as
a breakdown of pass/fail rates by revision. We don't automatically email
these to the list or anything, currently, but would be pretty easy to set
up a cron job to do so.
The dashboard is very helpful for prioritizing the de-flaking of the worst
offenders, and also useful to quickly drill down and grab failure logs from
the flaky tests themselves.
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
Re: Smart Flaky Handler
Posted by Matteo Bertozzi <th...@gmail.com>.
any suggestion on how to make people aware of the tests being flaky?
for example I will have never notice the procedure test being flaky if was
not for stack posting the list here.
so, maybe a weekly digest in the dev-list with the list of flaky will get
more audience than having people go into the job.
also, I was thinking about how do I notice if I broke something when I post
a patch.
since we exclude the flakys from the run, there is no way I can notice I
broke something from QA.
maybe we can add a section in QA that runs the flaky ones and tells you
"those are failed but may be flaky"
and at least can look if the failures are related to the patch or is just
flaky.
On Fri, May 20, 2016 at 11:03 AM, Nick Dimiduk <nd...@gmail.com> wrote:
> Nice work Appy! What do I need to do to get it wired up for branch-1.1?
>
> On Fri, May 20, 2016 at 9:25 AM, Stack <st...@duboce.net> wrote:
>
> > The system seems to be working nicely Appy. We are getting green
> precommit
> > builds for the first time in ages.
> >
> > Should we change the includes and excludes lists so they have a file type
> > ending? .txt? Then I could open them easily in the browser. Currently I
> > have to download them.
> >
> > Includes are tests that are currently considered 'flakey'?
> >
> >
> >
> TestGenerateDelegationToken,TestMobCompactor,TestRegionServerMetrics,TestAcidGuarantees,TestMasterReplication,TestRowProcessorEndpoint,TestAsyncLogRolling,DynamicLogicExpressionSuite,TestMasterFailoverWithProcedures,TestChoreService,TestScannerHeartbeatMessages,TestWALProcedureStore,TestRegionMergeTransactionOnCluster,TestSaslFanOutOneBlockAsyncDFSOutput,TestReplicationEndpointWithMultipleWAL
> >
> > We have a nice list.
> >
> > Excludes are:
> >
> >
> >
> **/TestGenerateDelegationToken.java,**/TestMobCompactor.java,**/TestRegionServerMetrics.java,**/TestAcidGuarantees.java,**/TestMasterReplication.java,**/TestRowProcessorEndpoint.java,**/TestAsyncLogRolling.java,**/DynamicLogicExpressionSuite.java,**/TestMasterFailoverWithProcedures.java,**/TestChoreService.java,**/TestScannerHeartbeatMessages.java,**/TestWALProcedureStore.java,**/TestRegionMergeTransactionOnCluster.java,**/TestSaslFanOutOneBlockAsyncDFSOutput.java,**/TestReplicationEndpointWithMultipleWAL.java,
> >
> > Whats the '**/' about? Is it supposed to have opening/closing versions?
> >
> > Thanks boss,
> > St.
> >
> >
> >
> > On Mon, May 16, 2016 at 4:45 PM, Stack <st...@duboce.net> wrote:
> >
> > > Sweet!
> > >
> > > On Mon, May 16, 2016 at 4:38 PM, Apekshit Sharma <ap...@cloudera.com>
> > > wrote:
> > >
> > >> This mail is to introduce the work to tackle the flaky tests in our
> > build.
> > >>
> > >> *Why is it important?*
> > >> - Our build history sucks, last 175 post-commit runs failed. We need
> to
> > >> make it useful.
> > >> - To better understand our code’s testing status, more importantly
> it’s
> > >> weak points.
> > >> - We know those 2-3 tests which keep failing every now and then, but
> not
> > >> those ~10 nasty ones which fail like 1 out of 50 times, and screw our
> > build.
> > >> - This isn’t something that can be done manually on a daily basis. We
> > >> need automation.
> > >>
> > >> *Changes made so far:*
> > >> Code changes: HBASE-15839
> > >> <https://issues.apache.org/jira/browse/HBASE-15839> (Umbrella issue)
> > >>
> > >> *Jenkins changes:*
> > >>
> > >>
> > >> [Diagram link:
> > >>
> >
> https://issues.apache.org/jira/secure/attachment/12804292/Screen%20Shot%202016-05-16%20at%204.02.46%20PM.png
> > >> ]
> > >>
> > >> *(new job) HBase-Find-Flaky-Tests*: Gets test reports of recent builds
> > >> of post-commit job (TRUNK_matrix) and HBase-Flaky-Tests job (see
> below)
> > to
> > >> find flaky tests. Frequency of run determines how fast we catch test
> > >> regressions. So if we run it every 4 hours, any test which started
> > failing
> > >> in post-commit job (TRUNK_matrix) in last 4 hour will be blacklisted.
> > >>
> > >> *(new job) HBase-Flaky-Tests*: This job runs only the flaky tests. The
> > >> aim is to run this job back-to-back to collect as many runs as we can.
> > >> Higher the run rate, the better will be our system at catching the
> flaky
> > >> tests. We currently run it hourly. so we’ll be able to keep track of
> > flaky
> > >> tests with ~5% failure rate or more.
> > >>
> > >> *Post-commit (TRUNK_matrix) and pre-commit jobs*: Exclude these flaky
> > >> tests.
> > >>
> > >>
> > >> *So what if a bad commit makes a good test bad?*
> > >> Since the test is not bad, it’ll run in next post-commit and will
> fail.
> > >> Next run of HBase-Find-Flaky-Tests will pick it up and blacklist it.
> > >> Blacklisting will help keep the post-commit job and more importantly
> > >> pre-commit job clean, a problem we face quite often.
> > >>
> > >> *Are we just tucking away are shit?*
> > >> Nope, this will help us:
> > >> - first, Maintain a list of bad test (we lack that today).
> > >> - second, make our build greener to the point that a failed/red build
> is
> > >> something we worry about seriously.
> > >>
> > >> Once we are confident that the system is working fine, we’ll setup up
> > >> HBase-Find-Flaky-Tests job to send reports to dev@hbase so that devs
> > >> know about the bad tests. If it remains hidden somewhere in a jenkins
> > job’s
> > >> archive, it’s unlike that we’ll actively work on getting them fixed
> :).
> > >>
> > >> I'll keep posting further updates on this thread.
> > >>
> > >> -- Appy
> > >>
> > >
> > >
> >
>
Re: Smart Flaky Handler
Posted by Nick Dimiduk <nd...@gmail.com>.
Nice work Appy! What do I need to do to get it wired up for branch-1.1?
On Fri, May 20, 2016 at 9:25 AM, Stack <st...@duboce.net> wrote:
> The system seems to be working nicely Appy. We are getting green precommit
> builds for the first time in ages.
>
> Should we change the includes and excludes lists so they have a file type
> ending? .txt? Then I could open them easily in the browser. Currently I
> have to download them.
>
> Includes are tests that are currently considered 'flakey'?
>
>
> TestGenerateDelegationToken,TestMobCompactor,TestRegionServerMetrics,TestAcidGuarantees,TestMasterReplication,TestRowProcessorEndpoint,TestAsyncLogRolling,DynamicLogicExpressionSuite,TestMasterFailoverWithProcedures,TestChoreService,TestScannerHeartbeatMessages,TestWALProcedureStore,TestRegionMergeTransactionOnCluster,TestSaslFanOutOneBlockAsyncDFSOutput,TestReplicationEndpointWithMultipleWAL
>
> We have a nice list.
>
> Excludes are:
>
>
> **/TestGenerateDelegationToken.java,**/TestMobCompactor.java,**/TestRegionServerMetrics.java,**/TestAcidGuarantees.java,**/TestMasterReplication.java,**/TestRowProcessorEndpoint.java,**/TestAsyncLogRolling.java,**/DynamicLogicExpressionSuite.java,**/TestMasterFailoverWithProcedures.java,**/TestChoreService.java,**/TestScannerHeartbeatMessages.java,**/TestWALProcedureStore.java,**/TestRegionMergeTransactionOnCluster.java,**/TestSaslFanOutOneBlockAsyncDFSOutput.java,**/TestReplicationEndpointWithMultipleWAL.java,
>
> Whats the '**/' about? Is it supposed to have opening/closing versions?
>
> Thanks boss,
> St.
>
>
>
> On Mon, May 16, 2016 at 4:45 PM, Stack <st...@duboce.net> wrote:
>
> > Sweet!
> >
> > On Mon, May 16, 2016 at 4:38 PM, Apekshit Sharma <ap...@cloudera.com>
> > wrote:
> >
> >> This mail is to introduce the work to tackle the flaky tests in our
> build.
> >>
> >> *Why is it important?*
> >> - Our build history sucks, last 175 post-commit runs failed. We need to
> >> make it useful.
> >> - To better understand our code’s testing status, more importantly it’s
> >> weak points.
> >> - We know those 2-3 tests which keep failing every now and then, but not
> >> those ~10 nasty ones which fail like 1 out of 50 times, and screw our
> build.
> >> - This isn’t something that can be done manually on a daily basis. We
> >> need automation.
> >>
> >> *Changes made so far:*
> >> Code changes: HBASE-15839
> >> <https://issues.apache.org/jira/browse/HBASE-15839> (Umbrella issue)
> >>
> >> *Jenkins changes:*
> >>
> >>
> >> [Diagram link:
> >>
> https://issues.apache.org/jira/secure/attachment/12804292/Screen%20Shot%202016-05-16%20at%204.02.46%20PM.png
> >> ]
> >>
> >> *(new job) HBase-Find-Flaky-Tests*: Gets test reports of recent builds
> >> of post-commit job (TRUNK_matrix) and HBase-Flaky-Tests job (see below)
> to
> >> find flaky tests. Frequency of run determines how fast we catch test
> >> regressions. So if we run it every 4 hours, any test which started
> failing
> >> in post-commit job (TRUNK_matrix) in last 4 hour will be blacklisted.
> >>
> >> *(new job) HBase-Flaky-Tests*: This job runs only the flaky tests. The
> >> aim is to run this job back-to-back to collect as many runs as we can.
> >> Higher the run rate, the better will be our system at catching the flaky
> >> tests. We currently run it hourly. so we’ll be able to keep track of
> flaky
> >> tests with ~5% failure rate or more.
> >>
> >> *Post-commit (TRUNK_matrix) and pre-commit jobs*: Exclude these flaky
> >> tests.
> >>
> >>
> >> *So what if a bad commit makes a good test bad?*
> >> Since the test is not bad, it’ll run in next post-commit and will fail.
> >> Next run of HBase-Find-Flaky-Tests will pick it up and blacklist it.
> >> Blacklisting will help keep the post-commit job and more importantly
> >> pre-commit job clean, a problem we face quite often.
> >>
> >> *Are we just tucking away are shit?*
> >> Nope, this will help us:
> >> - first, Maintain a list of bad test (we lack that today).
> >> - second, make our build greener to the point that a failed/red build is
> >> something we worry about seriously.
> >>
> >> Once we are confident that the system is working fine, we’ll setup up
> >> HBase-Find-Flaky-Tests job to send reports to dev@hbase so that devs
> >> know about the bad tests. If it remains hidden somewhere in a jenkins
> job’s
> >> archive, it’s unlike that we’ll actively work on getting them fixed :).
> >>
> >> I'll keep posting further updates on this thread.
> >>
> >> -- Appy
> >>
> >
> >
>
Re: Smart Flaky Handler
Posted by Stack <st...@duboce.net>.
The system seems to be working nicely Appy. We are getting green precommit
builds for the first time in ages.
Should we change the includes and excludes lists so they have a file type
ending? .txt? Then I could open them easily in the browser. Currently I
have to download them.
Includes are tests that are currently considered 'flakey'?
TestGenerateDelegationToken,TestMobCompactor,TestRegionServerMetrics,TestAcidGuarantees,TestMasterReplication,TestRowProcessorEndpoint,TestAsyncLogRolling,DynamicLogicExpressionSuite,TestMasterFailoverWithProcedures,TestChoreService,TestScannerHeartbeatMessages,TestWALProcedureStore,TestRegionMergeTransactionOnCluster,TestSaslFanOutOneBlockAsyncDFSOutput,TestReplicationEndpointWithMultipleWAL
We have a nice list.
Excludes are:
**/TestGenerateDelegationToken.java,**/TestMobCompactor.java,**/TestRegionServerMetrics.java,**/TestAcidGuarantees.java,**/TestMasterReplication.java,**/TestRowProcessorEndpoint.java,**/TestAsyncLogRolling.java,**/DynamicLogicExpressionSuite.java,**/TestMasterFailoverWithProcedures.java,**/TestChoreService.java,**/TestScannerHeartbeatMessages.java,**/TestWALProcedureStore.java,**/TestRegionMergeTransactionOnCluster.java,**/TestSaslFanOutOneBlockAsyncDFSOutput.java,**/TestReplicationEndpointWithMultipleWAL.java,
Whats the '**/' about? Is it supposed to have opening/closing versions?
Thanks boss,
St.
On Mon, May 16, 2016 at 4:45 PM, Stack <st...@duboce.net> wrote:
> Sweet!
>
> On Mon, May 16, 2016 at 4:38 PM, Apekshit Sharma <ap...@cloudera.com>
> wrote:
>
>> This mail is to introduce the work to tackle the flaky tests in our build.
>>
>> *Why is it important?*
>> - Our build history sucks, last 175 post-commit runs failed. We need to
>> make it useful.
>> - To better understand our code’s testing status, more importantly it’s
>> weak points.
>> - We know those 2-3 tests which keep failing every now and then, but not
>> those ~10 nasty ones which fail like 1 out of 50 times, and screw our build.
>> - This isn’t something that can be done manually on a daily basis. We
>> need automation.
>>
>> *Changes made so far:*
>> Code changes: HBASE-15839
>> <https://issues.apache.org/jira/browse/HBASE-15839> (Umbrella issue)
>>
>> *Jenkins changes:*
>>
>>
>> [Diagram link:
>> https://issues.apache.org/jira/secure/attachment/12804292/Screen%20Shot%202016-05-16%20at%204.02.46%20PM.png
>> ]
>>
>> *(new job) HBase-Find-Flaky-Tests*: Gets test reports of recent builds
>> of post-commit job (TRUNK_matrix) and HBase-Flaky-Tests job (see below) to
>> find flaky tests. Frequency of run determines how fast we catch test
>> regressions. So if we run it every 4 hours, any test which started failing
>> in post-commit job (TRUNK_matrix) in last 4 hour will be blacklisted.
>>
>> *(new job) HBase-Flaky-Tests*: This job runs only the flaky tests. The
>> aim is to run this job back-to-back to collect as many runs as we can.
>> Higher the run rate, the better will be our system at catching the flaky
>> tests. We currently run it hourly. so we’ll be able to keep track of flaky
>> tests with ~5% failure rate or more.
>>
>> *Post-commit (TRUNK_matrix) and pre-commit jobs*: Exclude these flaky
>> tests.
>>
>>
>> *So what if a bad commit makes a good test bad?*
>> Since the test is not bad, it’ll run in next post-commit and will fail.
>> Next run of HBase-Find-Flaky-Tests will pick it up and blacklist it.
>> Blacklisting will help keep the post-commit job and more importantly
>> pre-commit job clean, a problem we face quite often.
>>
>> *Are we just tucking away are shit?*
>> Nope, this will help us:
>> - first, Maintain a list of bad test (we lack that today).
>> - second, make our build greener to the point that a failed/red build is
>> something we worry about seriously.
>>
>> Once we are confident that the system is working fine, we’ll setup up
>> HBase-Find-Flaky-Tests job to send reports to dev@hbase so that devs
>> know about the bad tests. If it remains hidden somewhere in a jenkins job’s
>> archive, it’s unlike that we’ll actively work on getting them fixed :).
>>
>> I'll keep posting further updates on this thread.
>>
>> -- Appy
>>
>
>
Re: Smart Flaky Handler
Posted by Stack <st...@duboce.net>.
Sweet!
On Mon, May 16, 2016 at 4:38 PM, Apekshit Sharma <ap...@cloudera.com> wrote:
> This mail is to introduce the work to tackle the flaky tests in our build.
>
> *Why is it important?*
> - Our build history sucks, last 175 post-commit runs failed. We need to
> make it useful.
> - To better understand our code’s testing status, more importantly it’s
> weak points.
> - We know those 2-3 tests which keep failing every now and then, but not
> those ~10 nasty ones which fail like 1 out of 50 times, and screw our build.
> - This isn’t something that can be done manually on a daily basis. We need
> automation.
>
> *Changes made so far:*
> Code changes: HBASE-15839
> <https://issues.apache.org/jira/browse/HBASE-15839> (Umbrella issue)
>
> *Jenkins changes:*
>
>
> [Diagram link:
> https://issues.apache.org/jira/secure/attachment/12804292/Screen%20Shot%202016-05-16%20at%204.02.46%20PM.png
> ]
>
> *(new job) HBase-Find-Flaky-Tests*: Gets test reports of recent builds of
> post-commit job (TRUNK_matrix) and HBase-Flaky-Tests job (see below) to
> find flaky tests. Frequency of run determines how fast we catch test
> regressions. So if we run it every 4 hours, any test which started failing
> in post-commit job (TRUNK_matrix) in last 4 hour will be blacklisted.
>
> *(new job) HBase-Flaky-Tests*: This job runs only the flaky tests. The
> aim is to run this job back-to-back to collect as many runs as we can.
> Higher the run rate, the better will be our system at catching the flaky
> tests. We currently run it hourly. so we’ll be able to keep track of flaky
> tests with ~5% failure rate or more.
>
> *Post-commit (TRUNK_matrix) and pre-commit jobs*: Exclude these flaky
> tests.
>
>
> *So what if a bad commit makes a good test bad?*
> Since the test is not bad, it’ll run in next post-commit and will fail.
> Next run of HBase-Find-Flaky-Tests will pick it up and blacklist it.
> Blacklisting will help keep the post-commit job and more importantly
> pre-commit job clean, a problem we face quite often.
>
> *Are we just tucking away are shit?*
> Nope, this will help us:
> - first, Maintain a list of bad test (we lack that today).
> - second, make our build greener to the point that a failed/red build is
> something we worry about seriously.
>
> Once we are confident that the system is working fine, we’ll setup up
> HBase-Find-Flaky-Tests job to send reports to dev@hbase so that devs know
> about the bad tests. If it remains hidden somewhere in a jenkins job’s
> archive, it’s unlike that we’ll actively work on getting them fixed :).
>
> I'll keep posting further updates on this thread.
>
> -- Appy
>