You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Apekshit Sharma <ap...@cloudera.com> on 2016/04/01 19:01:36 UTC

Re: Better management of flakies

So who's setting up the jobs? I don't have perms to do it.

On Wed, Mar 30, 2016 at 11:24 AM, Apekshit Sharma <ap...@cloudera.com> wrote:

> So tackling the two parts individually:
> 1) Detection : A good automatic flaky detector will no doubt save manual
> effort. The one mentioned by Matteo seems like a good one which we can hook
> up with post-commit job. I see a minor problem though, we'll should use
> about 20 runs at least, but the rate of execution of this post-commit
> <https://builds.apache.org/view/All/job/HBase-Trunk_matrix/> job for
> trunk is too low which means we'll get stale information from the tool. One
> way around that would be running the post-commit job back-to-back? It can
> than trigger another job in the end which will use the tool to recompute
> flakies.
>
> 2) Handling: Ignoring flakies in main build and having separate job just
> for flakies. We can run it often enough and get some stats on flaky-ness
> rate of each test using the same tool as above.
>
> However, I feel that we should keep it super simple in the start. Just
> have a manual list and a separate job for flakies. It it works out fine, we
> can add the automatic detection and statistics part later.
>
> - Appy
>



-- 

Regards

Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
650-963-6311

Re: Better management of flakies

Posted by Apekshit Sharma <ap...@cloudera.com>.
https://issues.apache.org/jira/browse/HBASE-15651

On Fri, Apr 1, 2016 at 1:49 PM, Stack <st...@duboce.net> wrote:

> On Fri, Apr 1, 2016 at 10:05 AM, Sean Busbey <bu...@cloudera.com> wrote:
> ....
>
> > Maybe better if we ensure everything needed to run our jobs is scripts
> > in dev-support so that the commiter-only change can just be moving
> > jenkins to execute them.
> >
> >
> I like this idea Sean.
> St.Ack
>
>
>
> > On Fri, Apr 1, 2016 at 12:01 PM, Apekshit Sharma <ap...@cloudera.com>
> > wrote:
> > > So who's setting up the jobs? I don't have perms to do it.
> > >
> > > On Wed, Mar 30, 2016 at 11:24 AM, Apekshit Sharma <ap...@cloudera.com>
> > wrote:
> > >
> > >> So tackling the two parts individually:
> > >> 1) Detection : A good automatic flaky detector will no doubt save
> manual
> > >> effort. The one mentioned by Matteo seems like a good one which we can
> > hook
> > >> up with post-commit job. I see a minor problem though, we'll should
> use
> > >> about 20 runs at least, but the rate of execution of this post-commit
> > >> <https://builds.apache.org/view/All/job/HBase-Trunk_matrix/> job for
> > >> trunk is too low which means we'll get stale information from the
> tool.
> > One
> > >> way around that would be running the post-commit job back-to-back? It
> > can
> > >> than trigger another job in the end which will use the tool to
> recompute
> > >> flakies.
> > >>
> > >> 2) Handling: Ignoring flakies in main build and having separate job
> just
> > >> for flakies. We can run it often enough and get some stats on
> flaky-ness
> > >> rate of each test using the same tool as above.
> > >>
> > >> However, I feel that we should keep it super simple in the start. Just
> > >> have a manual list and a separate job for flakies. It it works out
> > fine, we
> > >> can add the automatic detection and statistics part later.
> > >>
> > >> - Appy
> > >>
> > >
> > >
> > >
> > > --
> > >
> > > Regards
> > >
> > > Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
> > > 650-963-6311
> >
> >
> >
> > --
> > busbey
> >
>



-- 

Regards

Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
650-963-6311

Re: Better management of flakies

Posted by Stack <st...@duboce.net>.
On Fri, Apr 1, 2016 at 10:05 AM, Sean Busbey <bu...@cloudera.com> wrote:
....

> Maybe better if we ensure everything needed to run our jobs is scripts
> in dev-support so that the commiter-only change can just be moving
> jenkins to execute them.
>
>
I like this idea Sean.
St.Ack



> On Fri, Apr 1, 2016 at 12:01 PM, Apekshit Sharma <ap...@cloudera.com>
> wrote:
> > So who's setting up the jobs? I don't have perms to do it.
> >
> > On Wed, Mar 30, 2016 at 11:24 AM, Apekshit Sharma <ap...@cloudera.com>
> wrote:
> >
> >> So tackling the two parts individually:
> >> 1) Detection : A good automatic flaky detector will no doubt save manual
> >> effort. The one mentioned by Matteo seems like a good one which we can
> hook
> >> up with post-commit job. I see a minor problem though, we'll should use
> >> about 20 runs at least, but the rate of execution of this post-commit
> >> <https://builds.apache.org/view/All/job/HBase-Trunk_matrix/> job for
> >> trunk is too low which means we'll get stale information from the tool.
> One
> >> way around that would be running the post-commit job back-to-back? It
> can
> >> than trigger another job in the end which will use the tool to recompute
> >> flakies.
> >>
> >> 2) Handling: Ignoring flakies in main build and having separate job just
> >> for flakies. We can run it often enough and get some stats on flaky-ness
> >> rate of each test using the same tool as above.
> >>
> >> However, I feel that we should keep it super simple in the start. Just
> >> have a manual list and a separate job for flakies. It it works out
> fine, we
> >> can add the automatic detection and statistics part later.
> >>
> >> - Appy
> >>
> >
> >
> >
> > --
> >
> > Regards
> >
> > Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
> > 650-963-6311
>
>
>
> --
> busbey
>

Re: Better management of flakies

Posted by Sean Busbey <bu...@cloudera.com>.
If you create a jira that describes what we need to implement, I'll
find time to put in changes.

Maybe better if we ensure everything needed to run our jobs is scripts
in dev-support so that the commiter-only change can just be moving
jenkins to execute them.

On Fri, Apr 1, 2016 at 12:01 PM, Apekshit Sharma <ap...@cloudera.com> wrote:
> So who's setting up the jobs? I don't have perms to do it.
>
> On Wed, Mar 30, 2016 at 11:24 AM, Apekshit Sharma <ap...@cloudera.com> wrote:
>
>> So tackling the two parts individually:
>> 1) Detection : A good automatic flaky detector will no doubt save manual
>> effort. The one mentioned by Matteo seems like a good one which we can hook
>> up with post-commit job. I see a minor problem though, we'll should use
>> about 20 runs at least, but the rate of execution of this post-commit
>> <https://builds.apache.org/view/All/job/HBase-Trunk_matrix/> job for
>> trunk is too low which means we'll get stale information from the tool. One
>> way around that would be running the post-commit job back-to-back? It can
>> than trigger another job in the end which will use the tool to recompute
>> flakies.
>>
>> 2) Handling: Ignoring flakies in main build and having separate job just
>> for flakies. We can run it often enough and get some stats on flaky-ness
>> rate of each test using the same tool as above.
>>
>> However, I feel that we should keep it super simple in the start. Just
>> have a manual list and a separate job for flakies. It it works out fine, we
>> can add the automatic detection and statistics part later.
>>
>> - Appy
>>
>
>
>
> --
>
> Regards
>
> Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
> 650-963-6311



-- 
busbey