You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Michael Han <ha...@cloudera.com> on 2017/04/12 02:33:05 UTC
Introduce ZooKeeper flaky test dashboard
tl;dr
https://builds.apache.org/job/ZooKeeper-Find-Flaky-Tests/lastSuccessfulBuild/artifact/report.html
We all know quality is important to this project. One of major advantage of
ZooKeeper comparing to other similar solutions is its quality: solid,
stable, and well tested. Unit tests are playing an important role for ZK's
quality so we need take these tests seriously. In particular for failed
tests, it's easy to blame they are failed because of flaky but usually
there are bugs (sometimes very subtle ones) behind the failures.
Inspired by some good work done on HBase project, I've borrowed and tweaked
their script so it applies to ZooKeeper project. The dashboard is now up
and a Jenkins job is running on a daily base to monitor our builds and pick
up flaky tests. I hope this dashboard can help increase the visibility and
raise awareness of the quality of unit tests, and also help developers
track and diagnose failures.
Any feedback will be appreciated, and of course, contributions are very
welcome.
--
Cheers
Michael.
Re: Introduce ZooKeeper flaky test dashboard
Posted by Michael Han <ha...@cloudera.com>.
Yes, it's trivial to add the job success rate along with other statistic
information that the Jenkins job exposes.
On Wed, Apr 12, 2017 at 3:56 PM, Patrick Hunt <ph...@apache.org> wrote:
> Great stuff, thanks Michael. Is there a way to indicate, for each of the
> jobs, how many times the job failed vs the number of times it was run
> during the time period? I often see that when one test fails it will cause
> others to fail...
>
> Patrick
>
> On Tue, Apr 11, 2017 at 8:07 PM, Edward Ribeiro <ed...@gmail.com>
> wrote:
>
> > Very cool feature! Congratulations for initiative! :D
> >
> > Em 11 de abr de 2017 11:48 PM, "Jordan Zimmerman" <
> > jordan@jordanzimmerman.com> escreveu:
> >
> > > Nice work - we could really use this on Curator. I'll be stealing this
> > > soon ;D
> > >
> > > > On Apr 11, 2017, at 9:33 PM, Michael Han <ha...@cloudera.com> wrote:
> > > >
> > > > tl;dr
> > > > https://builds.apache.org/job/ZooKeeper-Find-Flaky-Tests/
> > > lastSuccessfulBuild/artifact/report.html
> > > >
> > > >
> > > > We all know quality is important to this project. One of major
> > advantage
> > > of
> > > > ZooKeeper comparing to other similar solutions is its quality: solid,
> > > > stable, and well tested. Unit tests are playing an important role for
> > > ZK's
> > > > quality so we need take these tests seriously. In particular for
> failed
> > > > tests, it's easy to blame they are failed because of flaky but
> usually
> > > > there are bugs (sometimes very subtle ones) behind the failures.
> > > >
> > > > Inspired by some good work done on HBase project, I've borrowed and
> > > tweaked
> > > > their script so it applies to ZooKeeper project. The dashboard is now
> > up
> > > > and a Jenkins job is running on a daily base to monitor our builds
> and
> > > pick
> > > > up flaky tests. I hope this dashboard can help increase the
> visibility
> > > and
> > > > raise awareness of the quality of unit tests, and also help
> developers
> > > > track and diagnose failures.
> > > >
> > > > Any feedback will be appreciated, and of course, contributions are
> very
> > > > welcome.
> > > >
> > > > --
> > > > Cheers
> > > > Michael.
> > >
> > >
> >
>
--
Cheers
Michael.
Re: Introduce ZooKeeper flaky test dashboard
Posted by Patrick Hunt <ph...@apache.org>.
Great stuff, thanks Michael. Is there a way to indicate, for each of the
jobs, how many times the job failed vs the number of times it was run
during the time period? I often see that when one test fails it will cause
others to fail...
Patrick
On Tue, Apr 11, 2017 at 8:07 PM, Edward Ribeiro <ed...@gmail.com>
wrote:
> Very cool feature! Congratulations for initiative! :D
>
> Em 11 de abr de 2017 11:48 PM, "Jordan Zimmerman" <
> jordan@jordanzimmerman.com> escreveu:
>
> > Nice work - we could really use this on Curator. I'll be stealing this
> > soon ;D
> >
> > > On Apr 11, 2017, at 9:33 PM, Michael Han <ha...@cloudera.com> wrote:
> > >
> > > tl;dr
> > > https://builds.apache.org/job/ZooKeeper-Find-Flaky-Tests/
> > lastSuccessfulBuild/artifact/report.html
> > >
> > >
> > > We all know quality is important to this project. One of major
> advantage
> > of
> > > ZooKeeper comparing to other similar solutions is its quality: solid,
> > > stable, and well tested. Unit tests are playing an important role for
> > ZK's
> > > quality so we need take these tests seriously. In particular for failed
> > > tests, it's easy to blame they are failed because of flaky but usually
> > > there are bugs (sometimes very subtle ones) behind the failures.
> > >
> > > Inspired by some good work done on HBase project, I've borrowed and
> > tweaked
> > > their script so it applies to ZooKeeper project. The dashboard is now
> up
> > > and a Jenkins job is running on a daily base to monitor our builds and
> > pick
> > > up flaky tests. I hope this dashboard can help increase the visibility
> > and
> > > raise awareness of the quality of unit tests, and also help developers
> > > track and diagnose failures.
> > >
> > > Any feedback will be appreciated, and of course, contributions are very
> > > welcome.
> > >
> > > --
> > > Cheers
> > > Michael.
> >
> >
>
Re: Introduce ZooKeeper flaky test dashboard
Posted by Edward Ribeiro <ed...@gmail.com>.
Very cool feature! Congratulations for initiative! :D
Em 11 de abr de 2017 11:48 PM, "Jordan Zimmerman" <
jordan@jordanzimmerman.com> escreveu:
> Nice work - we could really use this on Curator. I'll be stealing this
> soon ;D
>
> > On Apr 11, 2017, at 9:33 PM, Michael Han <ha...@cloudera.com> wrote:
> >
> > tl;dr
> > https://builds.apache.org/job/ZooKeeper-Find-Flaky-Tests/
> lastSuccessfulBuild/artifact/report.html
> >
> >
> > We all know quality is important to this project. One of major advantage
> of
> > ZooKeeper comparing to other similar solutions is its quality: solid,
> > stable, and well tested. Unit tests are playing an important role for
> ZK's
> > quality so we need take these tests seriously. In particular for failed
> > tests, it's easy to blame they are failed because of flaky but usually
> > there are bugs (sometimes very subtle ones) behind the failures.
> >
> > Inspired by some good work done on HBase project, I've borrowed and
> tweaked
> > their script so it applies to ZooKeeper project. The dashboard is now up
> > and a Jenkins job is running on a daily base to monitor our builds and
> pick
> > up flaky tests. I hope this dashboard can help increase the visibility
> and
> > raise awareness of the quality of unit tests, and also help developers
> > track and diagnose failures.
> >
> > Any feedback will be appreciated, and of course, contributions are very
> > welcome.
> >
> > --
> > Cheers
> > Michael.
>
>
Re: Introduce ZooKeeper flaky test dashboard
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
Nice work - we could really use this on Curator. I'll be stealing this soon ;D
> On Apr 11, 2017, at 9:33 PM, Michael Han <ha...@cloudera.com> wrote:
>
> tl;dr
> https://builds.apache.org/job/ZooKeeper-Find-Flaky-Tests/lastSuccessfulBuild/artifact/report.html
>
>
> We all know quality is important to this project. One of major advantage of
> ZooKeeper comparing to other similar solutions is its quality: solid,
> stable, and well tested. Unit tests are playing an important role for ZK's
> quality so we need take these tests seriously. In particular for failed
> tests, it's easy to blame they are failed because of flaky but usually
> there are bugs (sometimes very subtle ones) behind the failures.
>
> Inspired by some good work done on HBase project, I've borrowed and tweaked
> their script so it applies to ZooKeeper project. The dashboard is now up
> and a Jenkins job is running on a daily base to monitor our builds and pick
> up flaky tests. I hope this dashboard can help increase the visibility and
> raise awareness of the quality of unit tests, and also help developers
> track and diagnose failures.
>
> Any feedback will be appreciated, and of course, contributions are very
> welcome.
>
> --
> Cheers
> Michael.
Re: Introduce ZooKeeper flaky test dashboard
Posted by Camille Fournier <ca...@apache.org>.
Awesome
On Apr 11, 2017 10:33 PM, "Michael Han" <ha...@cloudera.com> wrote:
> tl;dr
> https://builds.apache.org/job/ZooKeeper-Find-Flaky-Tests/
> lastSuccessfulBuild/artifact/report.html
>
>
> We all know quality is important to this project. One of major advantage of
> ZooKeeper comparing to other similar solutions is its quality: solid,
> stable, and well tested. Unit tests are playing an important role for ZK's
> quality so we need take these tests seriously. In particular for failed
> tests, it's easy to blame they are failed because of flaky but usually
> there are bugs (sometimes very subtle ones) behind the failures.
>
> Inspired by some good work done on HBase project, I've borrowed and tweaked
> their script so it applies to ZooKeeper project. The dashboard is now up
> and a Jenkins job is running on a daily base to monitor our builds and pick
> up flaky tests. I hope this dashboard can help increase the visibility and
> raise awareness of the quality of unit tests, and also help developers
> track and diagnose failures.
>
> Any feedback will be appreciated, and of course, contributions are very
> welcome.
>
> --
> Cheers
> Michael.
>