You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mxnet.apache.org by Meghna Baijal <me...@gmail.com> on 2017/09/12 00:50:42 UTC
MXNet: Run PR builds on Apache Jenkins only after the commit is
reviewed
Hi All,
We would like to initiate a change in the way the PR builds are being triggered. At the moment, every time a Pull Request is created, a build gets triggered on Jenkins. Additional builds also get triggered due to changes to the same PR.
Too many PR builds leads to resource starvation and very long queues and long build times. Hence we would like to add some checks where a human reviewer manually marks it to something like “ok to build” before a PR build is triggered.
Do you think this approach would be helpful and we should move forward with it?
Thanks,
Meghna Baijal
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is reviewed
Posted by Henri Yandell <ba...@apache.org>.
Side note:
There is no "we". The below should say something like "I would like to
propose...", or "As part of my work at <Corporation>, I would like to
propose..." if it's coming from a team at your day job and you don't want
to hog all the glory :)
If it truly is "we", then the email should end with the "we" that the email
is from. For example a release announcement from me would say "We are proud
to release" and would end with "Hen, on behalf of the Apache MXNet project".
Hen
On Mon, Sep 11, 2017 at 17:50 Meghna Baijal <me...@gmail.com>
wrote:
> Hi All,
> We would like to initiate a change in the way the PR builds are being
> triggered. At the moment, every time a Pull Request is created, a build
> gets triggered on Jenkins. Additional builds also get triggered due to
> changes to the same PR.
> Too many PR builds leads to resource starvation and very long queues and
> long build times. Hence we would like to add some checks where a human
> reviewer manually marks it to something like “ok to build” before a PR
> build is triggered.
>
> Do you think this approach would be helpful and we should move forward
> with it?
>
> Thanks,
> Meghna Baijal
>
>
>
>
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is reviewed
Posted by Chris Olivier <cj...@gmail.com>.
So are engineers.
On Mon, Sep 11, 2017 at 8:35 PM shiwen hu <ya...@gmail.com> wrote:
> Incredibuild is expensive
>
> 2017-09-12 11:32 GMT+08:00 Chris Olivier <cj...@gmail.com>:
>
> > In addition, there is still the possibility of Incredibuild to speed up
> > builds...
> >
> > On Mon, Sep 11, 2017 at 8:31 PM Chris Olivier <cj...@gmail.com>
> > wrote:
> >
> > > I would also prefer to throw more machines at it rather than restrict
> > > builds. I think that we should revisit the load when CI is functioning
> > > again for awhile and the backlog is caught up.
> > >
> > >
> > > On Mon, Sep 11, 2017 at 8:01 PM Nan Zhu <zh...@gmail.com>
> wrote:
> > >
> > >> +1 and recommend Jenkins-GitHub plugin with which committers/(accounts
> > >> with assigned permissions) can trigger Jenkins build with
> > >>
> > >> "Test this please, Jenkins!"
> > >>
> > >> "Retest this please, Jenkins!"
> > >>
> > >> And accounts in the list can trigger build automatically when
> submitting
> > >> a PR
> > >>
> > >>
> > >> https://wiki.jenkins.io/plugins/servlet/mobile?
> > contentId=37749162#content/view/37749162
> > >>
> > >> Best,
> > >>
> > >> Nan
> > >> ________________________________
> > >> From: workcrow@gmail.com <wo...@gmail.com> on behalf of Tianqi
> Chen
> > <
> > >> tqchen@cs.washington.edu>
> > >> Sent: Monday, September 11, 2017 7:54:05 PM
> > >> To: dev@mxnet.incubator.apache.org
> > >> Subject: Re: MXNet: Run PR builds on Apache Jenkins only after the
> > commit
> > >> is reviewed
> > >>
> > >> I agree that have the CI is useful, at least make sure that lint stage
> > is
> > >> done.
> > >>
> > >> Tianqi
> > >>
> > >> On Mon, Sep 11, 2017 at 7:49 PM, Hagay Lupesko <lu...@gmail.com>
> > wrote:
> > >>
> > >> > Build success or failure is a great feedback mechanism, of equal
> > >> importance
> > >> > to code review. Do we really want to delay it until another Dev
> gives
> > a
> > >> > thumbs up? It feels like a step backwards from automation.
> > >> >
> > >> > If our problem is resource constraint, can't we address it by
> throwing
> > >> more
> > >> > instances into the pool?
> > >> >
> > >> > On Sep 11, 2017 6:28 PM, "shiwen hu" <ya...@gmail.com> wrote:
> > >> >
> > >> > > Jenkins can be set to automatically cancel old builds, but I'm not
> > >> sure
> > >> > if
> > >> > > it's already been set
> > >> > >
> > >> > > 2017-09-12 9:19 GMT+08:00 Chris Olivier <cj...@gmail.com>:
> > >> > >
> > >> > > > Is the load artificially high because there's such a backlog due
> > to
> > >> > other
> > >> > > > reasons? Many may be triggering trivial changes to kick off
> > another
> > >> > build
> > >> > > > attempt (I have).
> > >> > > >
> > >> > > > Do new PR changes actually stop the old build or do those go to
> > >> > > completion?
> > >> > > > I realize they show on the PR as a new build has started, but
> are
> > >> the
> > >> > old
> > >> > > > builds/tests always interrupted?
> > >> > > >
> > >> > > >
> > >> > > > On Mon, Sep 11, 2017 at 6:12 PM Naveen Swamy <
> mnnaveen@gmail.com>
> > >> > wrote:
> > >> > > >
> > >> > > > > +1
> > >> > > > >
> > >> > > > > Even a small change in the PR is initiating a new build, I
> think
> > >> this
> > >> > > is
> > >> > > > > needless and not serving any good purpose until a reviewer has
> > >> looked
> > >> > > > into
> > >> > > > > the PR. This is also adding unnecessary load on the mxnet
> build
> > >> > > pipeline
> > >> > > > > and slaves.
> > >> > > > >
> > >> > > > > Thanks, Naveen
> > >> > > > >
> > >> > > > >
> > >> > > > > On Mon, Sep 11, 2017 at 5:50 PM, Meghna Baijal <
> > >> > > > meghnabaijal2017@gmail.com
> > >> > > > > >
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Hi All,
> > >> > > > > > We would like to initiate a change in the way the PR builds
> > are
> > >> > being
> > >> > > > > > triggered. At the moment, every time a Pull Request is
> > created,
> > >> a
> > >> > > build
> > >> > > > > > gets triggered on Jenkins. Additional builds also get
> > triggered
> > >> due
> > >> > > to
> > >> > > > > > changes to the same PR.
> > >> > > > > > Too many PR builds leads to resource starvation and very
> long
> > >> > queues
> > >> > > > and
> > >> > > > > > long build times. Hence we would like to add some checks
> > where a
> > >> > > human
> > >> > > > > > reviewer manually marks it to something like “ok to build”
> > >> before a
> > >> > > PR
> > >> > > > > > build is triggered.
> > >> > > > > >
> > >> > > > > > Do you think this approach would be helpful and we should
> move
> > >> > > forward
> > >> > > > > > with it?
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > > Meghna Baijal
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is reviewed
Posted by shiwen hu <ya...@gmail.com>.
Incredibuild is expensive
2017-09-12 11:32 GMT+08:00 Chris Olivier <cj...@gmail.com>:
> In addition, there is still the possibility of Incredibuild to speed up
> builds...
>
> On Mon, Sep 11, 2017 at 8:31 PM Chris Olivier <cj...@gmail.com>
> wrote:
>
> > I would also prefer to throw more machines at it rather than restrict
> > builds. I think that we should revisit the load when CI is functioning
> > again for awhile and the backlog is caught up.
> >
> >
> > On Mon, Sep 11, 2017 at 8:01 PM Nan Zhu <zh...@gmail.com> wrote:
> >
> >> +1 and recommend Jenkins-GitHub plugin with which committers/(accounts
> >> with assigned permissions) can trigger Jenkins build with
> >>
> >> "Test this please, Jenkins!"
> >>
> >> "Retest this please, Jenkins!"
> >>
> >> And accounts in the list can trigger build automatically when submitting
> >> a PR
> >>
> >>
> >> https://wiki.jenkins.io/plugins/servlet/mobile?
> contentId=37749162#content/view/37749162
> >>
> >> Best,
> >>
> >> Nan
> >> ________________________________
> >> From: workcrow@gmail.com <wo...@gmail.com> on behalf of Tianqi Chen
> <
> >> tqchen@cs.washington.edu>
> >> Sent: Monday, September 11, 2017 7:54:05 PM
> >> To: dev@mxnet.incubator.apache.org
> >> Subject: Re: MXNet: Run PR builds on Apache Jenkins only after the
> commit
> >> is reviewed
> >>
> >> I agree that have the CI is useful, at least make sure that lint stage
> is
> >> done.
> >>
> >> Tianqi
> >>
> >> On Mon, Sep 11, 2017 at 7:49 PM, Hagay Lupesko <lu...@gmail.com>
> wrote:
> >>
> >> > Build success or failure is a great feedback mechanism, of equal
> >> importance
> >> > to code review. Do we really want to delay it until another Dev gives
> a
> >> > thumbs up? It feels like a step backwards from automation.
> >> >
> >> > If our problem is resource constraint, can't we address it by throwing
> >> more
> >> > instances into the pool?
> >> >
> >> > On Sep 11, 2017 6:28 PM, "shiwen hu" <ya...@gmail.com> wrote:
> >> >
> >> > > Jenkins can be set to automatically cancel old builds, but I'm not
> >> sure
> >> > if
> >> > > it's already been set
> >> > >
> >> > > 2017-09-12 9:19 GMT+08:00 Chris Olivier <cj...@gmail.com>:
> >> > >
> >> > > > Is the load artificially high because there's such a backlog due
> to
> >> > other
> >> > > > reasons? Many may be triggering trivial changes to kick off
> another
> >> > build
> >> > > > attempt (I have).
> >> > > >
> >> > > > Do new PR changes actually stop the old build or do those go to
> >> > > completion?
> >> > > > I realize they show on the PR as a new build has started, but are
> >> the
> >> > old
> >> > > > builds/tests always interrupted?
> >> > > >
> >> > > >
> >> > > > On Mon, Sep 11, 2017 at 6:12 PM Naveen Swamy <mn...@gmail.com>
> >> > wrote:
> >> > > >
> >> > > > > +1
> >> > > > >
> >> > > > > Even a small change in the PR is initiating a new build, I think
> >> this
> >> > > is
> >> > > > > needless and not serving any good purpose until a reviewer has
> >> looked
> >> > > > into
> >> > > > > the PR. This is also adding unnecessary load on the mxnet build
> >> > > pipeline
> >> > > > > and slaves.
> >> > > > >
> >> > > > > Thanks, Naveen
> >> > > > >
> >> > > > >
> >> > > > > On Mon, Sep 11, 2017 at 5:50 PM, Meghna Baijal <
> >> > > > meghnabaijal2017@gmail.com
> >> > > > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi All,
> >> > > > > > We would like to initiate a change in the way the PR builds
> are
> >> > being
> >> > > > > > triggered. At the moment, every time a Pull Request is
> created,
> >> a
> >> > > build
> >> > > > > > gets triggered on Jenkins. Additional builds also get
> triggered
> >> due
> >> > > to
> >> > > > > > changes to the same PR.
> >> > > > > > Too many PR builds leads to resource starvation and very long
> >> > queues
> >> > > > and
> >> > > > > > long build times. Hence we would like to add some checks
> where a
> >> > > human
> >> > > > > > reviewer manually marks it to something like “ok to build”
> >> before a
> >> > > PR
> >> > > > > > build is triggered.
> >> > > > > >
> >> > > > > > Do you think this approach would be helpful and we should move
> >> > > forward
> >> > > > > > with it?
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Meghna Baijal
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is reviewed
Posted by Chris Olivier <cj...@gmail.com>.
In addition, there is still the possibility of Incredibuild to speed up
builds...
On Mon, Sep 11, 2017 at 8:31 PM Chris Olivier <cj...@gmail.com> wrote:
> I would also prefer to throw more machines at it rather than restrict
> builds. I think that we should revisit the load when CI is functioning
> again for awhile and the backlog is caught up.
>
>
> On Mon, Sep 11, 2017 at 8:01 PM Nan Zhu <zh...@gmail.com> wrote:
>
>> +1 and recommend Jenkins-GitHub plugin with which committers/(accounts
>> with assigned permissions) can trigger Jenkins build with
>>
>> "Test this please, Jenkins!"
>>
>> "Retest this please, Jenkins!"
>>
>> And accounts in the list can trigger build automatically when submitting
>> a PR
>>
>>
>> https://wiki.jenkins.io/plugins/servlet/mobile?contentId=37749162#content/view/37749162
>>
>> Best,
>>
>> Nan
>> ________________________________
>> From: workcrow@gmail.com <wo...@gmail.com> on behalf of Tianqi Chen <
>> tqchen@cs.washington.edu>
>> Sent: Monday, September 11, 2017 7:54:05 PM
>> To: dev@mxnet.incubator.apache.org
>> Subject: Re: MXNet: Run PR builds on Apache Jenkins only after the commit
>> is reviewed
>>
>> I agree that have the CI is useful, at least make sure that lint stage is
>> done.
>>
>> Tianqi
>>
>> On Mon, Sep 11, 2017 at 7:49 PM, Hagay Lupesko <lu...@gmail.com> wrote:
>>
>> > Build success or failure is a great feedback mechanism, of equal
>> importance
>> > to code review. Do we really want to delay it until another Dev gives a
>> > thumbs up? It feels like a step backwards from automation.
>> >
>> > If our problem is resource constraint, can't we address it by throwing
>> more
>> > instances into the pool?
>> >
>> > On Sep 11, 2017 6:28 PM, "shiwen hu" <ya...@gmail.com> wrote:
>> >
>> > > Jenkins can be set to automatically cancel old builds, but I'm not
>> sure
>> > if
>> > > it's already been set
>> > >
>> > > 2017-09-12 9:19 GMT+08:00 Chris Olivier <cj...@gmail.com>:
>> > >
>> > > > Is the load artificially high because there's such a backlog due to
>> > other
>> > > > reasons? Many may be triggering trivial changes to kick off another
>> > build
>> > > > attempt (I have).
>> > > >
>> > > > Do new PR changes actually stop the old build or do those go to
>> > > completion?
>> > > > I realize they show on the PR as a new build has started, but are
>> the
>> > old
>> > > > builds/tests always interrupted?
>> > > >
>> > > >
>> > > > On Mon, Sep 11, 2017 at 6:12 PM Naveen Swamy <mn...@gmail.com>
>> > wrote:
>> > > >
>> > > > > +1
>> > > > >
>> > > > > Even a small change in the PR is initiating a new build, I think
>> this
>> > > is
>> > > > > needless and not serving any good purpose until a reviewer has
>> looked
>> > > > into
>> > > > > the PR. This is also adding unnecessary load on the mxnet build
>> > > pipeline
>> > > > > and slaves.
>> > > > >
>> > > > > Thanks, Naveen
>> > > > >
>> > > > >
>> > > > > On Mon, Sep 11, 2017 at 5:50 PM, Meghna Baijal <
>> > > > meghnabaijal2017@gmail.com
>> > > > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Hi All,
>> > > > > > We would like to initiate a change in the way the PR builds are
>> > being
>> > > > > > triggered. At the moment, every time a Pull Request is created,
>> a
>> > > build
>> > > > > > gets triggered on Jenkins. Additional builds also get triggered
>> due
>> > > to
>> > > > > > changes to the same PR.
>> > > > > > Too many PR builds leads to resource starvation and very long
>> > queues
>> > > > and
>> > > > > > long build times. Hence we would like to add some checks where a
>> > > human
>> > > > > > reviewer manually marks it to something like “ok to build”
>> before a
>> > > PR
>> > > > > > build is triggered.
>> > > > > >
>> > > > > > Do you think this approach would be helpful and we should move
>> > > forward
>> > > > > > with it?
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Meghna Baijal
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is reviewed
Posted by Chris Olivier <cj...@gmail.com>.
I would also prefer to throw more machines at it rather than restrict
builds. I think that we should revisit the load when CI is functioning
again for awhile and the backlog is caught up.
On Mon, Sep 11, 2017 at 8:01 PM Nan Zhu <zh...@gmail.com> wrote:
> +1 and recommend Jenkins-GitHub plugin with which committers/(accounts
> with assigned permissions) can trigger Jenkins build with
>
> "Test this please, Jenkins!"
>
> "Retest this please, Jenkins!"
>
> And accounts in the list can trigger build automatically when submitting a
> PR
>
>
> https://wiki.jenkins.io/plugins/servlet/mobile?contentId=37749162#content/view/37749162
>
> Best,
>
> Nan
> ________________________________
> From: workcrow@gmail.com <wo...@gmail.com> on behalf of Tianqi Chen <
> tqchen@cs.washington.edu>
> Sent: Monday, September 11, 2017 7:54:05 PM
> To: dev@mxnet.incubator.apache.org
> Subject: Re: MXNet: Run PR builds on Apache Jenkins only after the commit
> is reviewed
>
> I agree that have the CI is useful, at least make sure that lint stage is
> done.
>
> Tianqi
>
> On Mon, Sep 11, 2017 at 7:49 PM, Hagay Lupesko <lu...@gmail.com> wrote:
>
> > Build success or failure is a great feedback mechanism, of equal
> importance
> > to code review. Do we really want to delay it until another Dev gives a
> > thumbs up? It feels like a step backwards from automation.
> >
> > If our problem is resource constraint, can't we address it by throwing
> more
> > instances into the pool?
> >
> > On Sep 11, 2017 6:28 PM, "shiwen hu" <ya...@gmail.com> wrote:
> >
> > > Jenkins can be set to automatically cancel old builds, but I'm not sure
> > if
> > > it's already been set
> > >
> > > 2017-09-12 9:19 GMT+08:00 Chris Olivier <cj...@gmail.com>:
> > >
> > > > Is the load artificially high because there's such a backlog due to
> > other
> > > > reasons? Many may be triggering trivial changes to kick off another
> > build
> > > > attempt (I have).
> > > >
> > > > Do new PR changes actually stop the old build or do those go to
> > > completion?
> > > > I realize they show on the PR as a new build has started, but are the
> > old
> > > > builds/tests always interrupted?
> > > >
> > > >
> > > > On Mon, Sep 11, 2017 at 6:12 PM Naveen Swamy <mn...@gmail.com>
> > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Even a small change in the PR is initiating a new build, I think
> this
> > > is
> > > > > needless and not serving any good purpose until a reviewer has
> looked
> > > > into
> > > > > the PR. This is also adding unnecessary load on the mxnet build
> > > pipeline
> > > > > and slaves.
> > > > >
> > > > > Thanks, Naveen
> > > > >
> > > > >
> > > > > On Mon, Sep 11, 2017 at 5:50 PM, Meghna Baijal <
> > > > meghnabaijal2017@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > > We would like to initiate a change in the way the PR builds are
> > being
> > > > > > triggered. At the moment, every time a Pull Request is created, a
> > > build
> > > > > > gets triggered on Jenkins. Additional builds also get triggered
> due
> > > to
> > > > > > changes to the same PR.
> > > > > > Too many PR builds leads to resource starvation and very long
> > queues
> > > > and
> > > > > > long build times. Hence we would like to add some checks where a
> > > human
> > > > > > reviewer manually marks it to something like “ok to build”
> before a
> > > PR
> > > > > > build is triggered.
> > > > > >
> > > > > > Do you think this approach would be helpful and we should move
> > > forward
> > > > > > with it?
> > > > > >
> > > > > > Thanks,
> > > > > > Meghna Baijal
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is
reviewed
Posted by Nan Zhu <zh...@gmail.com>.
+1 and recommend Jenkins-GitHub plugin with which committers/(accounts with assigned permissions) can trigger Jenkins build with
"Test this please, Jenkins!"
"Retest this please, Jenkins!"
And accounts in the list can trigger build automatically when submitting a PR
https://wiki.jenkins.io/plugins/servlet/mobile?contentId=37749162#content/view/37749162
Best,
Nan
________________________________
From: workcrow@gmail.com <wo...@gmail.com> on behalf of Tianqi Chen <tq...@cs.washington.edu>
Sent: Monday, September 11, 2017 7:54:05 PM
To: dev@mxnet.incubator.apache.org
Subject: Re: MXNet: Run PR builds on Apache Jenkins only after the commit is reviewed
I agree that have the CI is useful, at least make sure that lint stage is
done.
Tianqi
On Mon, Sep 11, 2017 at 7:49 PM, Hagay Lupesko <lu...@gmail.com> wrote:
> Build success or failure is a great feedback mechanism, of equal importance
> to code review. Do we really want to delay it until another Dev gives a
> thumbs up? It feels like a step backwards from automation.
>
> If our problem is resource constraint, can't we address it by throwing more
> instances into the pool?
>
> On Sep 11, 2017 6:28 PM, "shiwen hu" <ya...@gmail.com> wrote:
>
> > Jenkins can be set to automatically cancel old builds, but I'm not sure
> if
> > it's already been set
> >
> > 2017-09-12 9:19 GMT+08:00 Chris Olivier <cj...@gmail.com>:
> >
> > > Is the load artificially high because there's such a backlog due to
> other
> > > reasons? Many may be triggering trivial changes to kick off another
> build
> > > attempt (I have).
> > >
> > > Do new PR changes actually stop the old build or do those go to
> > completion?
> > > I realize they show on the PR as a new build has started, but are the
> old
> > > builds/tests always interrupted?
> > >
> > >
> > > On Mon, Sep 11, 2017 at 6:12 PM Naveen Swamy <mn...@gmail.com>
> wrote:
> > >
> > > > +1
> > > >
> > > > Even a small change in the PR is initiating a new build, I think this
> > is
> > > > needless and not serving any good purpose until a reviewer has looked
> > > into
> > > > the PR. This is also adding unnecessary load on the mxnet build
> > pipeline
> > > > and slaves.
> > > >
> > > > Thanks, Naveen
> > > >
> > > >
> > > > On Mon, Sep 11, 2017 at 5:50 PM, Meghna Baijal <
> > > meghnabaijal2017@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > > We would like to initiate a change in the way the PR builds are
> being
> > > > > triggered. At the moment, every time a Pull Request is created, a
> > build
> > > > > gets triggered on Jenkins. Additional builds also get triggered due
> > to
> > > > > changes to the same PR.
> > > > > Too many PR builds leads to resource starvation and very long
> queues
> > > and
> > > > > long build times. Hence we would like to add some checks where a
> > human
> > > > > reviewer manually marks it to something like “ok to build” before a
> > PR
> > > > > build is triggered.
> > > > >
> > > > > Do you think this approach would be helpful and we should move
> > forward
> > > > > with it?
> > > > >
> > > > > Thanks,
> > > > > Meghna Baijal
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is reviewed
Posted by shiwen hu <ya...@gmail.com>.
In addition, the recent CI congestion probably because of the windows CI
failure.
2017-09-12 10:54 GMT+08:00 Tianqi Chen <tq...@cs.washington.edu>:
> I agree that have the CI is useful, at least make sure that lint stage is
> done.
>
> Tianqi
>
> On Mon, Sep 11, 2017 at 7:49 PM, Hagay Lupesko <lu...@gmail.com> wrote:
>
> > Build success or failure is a great feedback mechanism, of equal
> importance
> > to code review. Do we really want to delay it until another Dev gives a
> > thumbs up? It feels like a step backwards from automation.
> >
> > If our problem is resource constraint, can't we address it by throwing
> more
> > instances into the pool?
> >
> > On Sep 11, 2017 6:28 PM, "shiwen hu" <ya...@gmail.com> wrote:
> >
> > > Jenkins can be set to automatically cancel old builds, but I'm not sure
> > if
> > > it's already been set
> > >
> > > 2017-09-12 9:19 GMT+08:00 Chris Olivier <cj...@gmail.com>:
> > >
> > > > Is the load artificially high because there's such a backlog due to
> > other
> > > > reasons? Many may be triggering trivial changes to kick off another
> > build
> > > > attempt (I have).
> > > >
> > > > Do new PR changes actually stop the old build or do those go to
> > > completion?
> > > > I realize they show on the PR as a new build has started, but are the
> > old
> > > > builds/tests always interrupted?
> > > >
> > > >
> > > > On Mon, Sep 11, 2017 at 6:12 PM Naveen Swamy <mn...@gmail.com>
> > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Even a small change in the PR is initiating a new build, I think
> this
> > > is
> > > > > needless and not serving any good purpose until a reviewer has
> looked
> > > > into
> > > > > the PR. This is also adding unnecessary load on the mxnet build
> > > pipeline
> > > > > and slaves.
> > > > >
> > > > > Thanks, Naveen
> > > > >
> > > > >
> > > > > On Mon, Sep 11, 2017 at 5:50 PM, Meghna Baijal <
> > > > meghnabaijal2017@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > > We would like to initiate a change in the way the PR builds are
> > being
> > > > > > triggered. At the moment, every time a Pull Request is created, a
> > > build
> > > > > > gets triggered on Jenkins. Additional builds also get triggered
> due
> > > to
> > > > > > changes to the same PR.
> > > > > > Too many PR builds leads to resource starvation and very long
> > queues
> > > > and
> > > > > > long build times. Hence we would like to add some checks where a
> > > human
> > > > > > reviewer manually marks it to something like “ok to build”
> before a
> > > PR
> > > > > > build is triggered.
> > > > > >
> > > > > > Do you think this approach would be helpful and we should move
> > > forward
> > > > > > with it?
> > > > > >
> > > > > > Thanks,
> > > > > > Meghna Baijal
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is reviewed
Posted by Tianqi Chen <tq...@cs.washington.edu>.
I agree that have the CI is useful, at least make sure that lint stage is
done.
Tianqi
On Mon, Sep 11, 2017 at 7:49 PM, Hagay Lupesko <lu...@gmail.com> wrote:
> Build success or failure is a great feedback mechanism, of equal importance
> to code review. Do we really want to delay it until another Dev gives a
> thumbs up? It feels like a step backwards from automation.
>
> If our problem is resource constraint, can't we address it by throwing more
> instances into the pool?
>
> On Sep 11, 2017 6:28 PM, "shiwen hu" <ya...@gmail.com> wrote:
>
> > Jenkins can be set to automatically cancel old builds, but I'm not sure
> if
> > it's already been set
> >
> > 2017-09-12 9:19 GMT+08:00 Chris Olivier <cj...@gmail.com>:
> >
> > > Is the load artificially high because there's such a backlog due to
> other
> > > reasons? Many may be triggering trivial changes to kick off another
> build
> > > attempt (I have).
> > >
> > > Do new PR changes actually stop the old build or do those go to
> > completion?
> > > I realize they show on the PR as a new build has started, but are the
> old
> > > builds/tests always interrupted?
> > >
> > >
> > > On Mon, Sep 11, 2017 at 6:12 PM Naveen Swamy <mn...@gmail.com>
> wrote:
> > >
> > > > +1
> > > >
> > > > Even a small change in the PR is initiating a new build, I think this
> > is
> > > > needless and not serving any good purpose until a reviewer has looked
> > > into
> > > > the PR. This is also adding unnecessary load on the mxnet build
> > pipeline
> > > > and slaves.
> > > >
> > > > Thanks, Naveen
> > > >
> > > >
> > > > On Mon, Sep 11, 2017 at 5:50 PM, Meghna Baijal <
> > > meghnabaijal2017@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > > We would like to initiate a change in the way the PR builds are
> being
> > > > > triggered. At the moment, every time a Pull Request is created, a
> > build
> > > > > gets triggered on Jenkins. Additional builds also get triggered due
> > to
> > > > > changes to the same PR.
> > > > > Too many PR builds leads to resource starvation and very long
> queues
> > > and
> > > > > long build times. Hence we would like to add some checks where a
> > human
> > > > > reviewer manually marks it to something like “ok to build” before a
> > PR
> > > > > build is triggered.
> > > > >
> > > > > Do you think this approach would be helpful and we should move
> > forward
> > > > > with it?
> > > > >
> > > > > Thanks,
> > > > > Meghna Baijal
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is reviewed
Posted by Hagay Lupesko <lu...@gmail.com>.
Build success or failure is a great feedback mechanism, of equal importance
to code review. Do we really want to delay it until another Dev gives a
thumbs up? It feels like a step backwards from automation.
If our problem is resource constraint, can't we address it by throwing more
instances into the pool?
On Sep 11, 2017 6:28 PM, "shiwen hu" <ya...@gmail.com> wrote:
> Jenkins can be set to automatically cancel old builds, but I'm not sure if
> it's already been set
>
> 2017-09-12 9:19 GMT+08:00 Chris Olivier <cj...@gmail.com>:
>
> > Is the load artificially high because there's such a backlog due to other
> > reasons? Many may be triggering trivial changes to kick off another build
> > attempt (I have).
> >
> > Do new PR changes actually stop the old build or do those go to
> completion?
> > I realize they show on the PR as a new build has started, but are the old
> > builds/tests always interrupted?
> >
> >
> > On Mon, Sep 11, 2017 at 6:12 PM Naveen Swamy <mn...@gmail.com> wrote:
> >
> > > +1
> > >
> > > Even a small change in the PR is initiating a new build, I think this
> is
> > > needless and not serving any good purpose until a reviewer has looked
> > into
> > > the PR. This is also adding unnecessary load on the mxnet build
> pipeline
> > > and slaves.
> > >
> > > Thanks, Naveen
> > >
> > >
> > > On Mon, Sep 11, 2017 at 5:50 PM, Meghna Baijal <
> > meghnabaijal2017@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hi All,
> > > > We would like to initiate a change in the way the PR builds are being
> > > > triggered. At the moment, every time a Pull Request is created, a
> build
> > > > gets triggered on Jenkins. Additional builds also get triggered due
> to
> > > > changes to the same PR.
> > > > Too many PR builds leads to resource starvation and very long queues
> > and
> > > > long build times. Hence we would like to add some checks where a
> human
> > > > reviewer manually marks it to something like “ok to build” before a
> PR
> > > > build is triggered.
> > > >
> > > > Do you think this approach would be helpful and we should move
> forward
> > > > with it?
> > > >
> > > > Thanks,
> > > > Meghna Baijal
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is reviewed
Posted by shiwen hu <ya...@gmail.com>.
Jenkins can be set to automatically cancel old builds, but I'm not sure if
it's already been set
2017-09-12 9:19 GMT+08:00 Chris Olivier <cj...@gmail.com>:
> Is the load artificially high because there's such a backlog due to other
> reasons? Many may be triggering trivial changes to kick off another build
> attempt (I have).
>
> Do new PR changes actually stop the old build or do those go to completion?
> I realize they show on the PR as a new build has started, but are the old
> builds/tests always interrupted?
>
>
> On Mon, Sep 11, 2017 at 6:12 PM Naveen Swamy <mn...@gmail.com> wrote:
>
> > +1
> >
> > Even a small change in the PR is initiating a new build, I think this is
> > needless and not serving any good purpose until a reviewer has looked
> into
> > the PR. This is also adding unnecessary load on the mxnet build pipeline
> > and slaves.
> >
> > Thanks, Naveen
> >
> >
> > On Mon, Sep 11, 2017 at 5:50 PM, Meghna Baijal <
> meghnabaijal2017@gmail.com
> > >
> > wrote:
> >
> > > Hi All,
> > > We would like to initiate a change in the way the PR builds are being
> > > triggered. At the moment, every time a Pull Request is created, a build
> > > gets triggered on Jenkins. Additional builds also get triggered due to
> > > changes to the same PR.
> > > Too many PR builds leads to resource starvation and very long queues
> and
> > > long build times. Hence we would like to add some checks where a human
> > > reviewer manually marks it to something like “ok to build” before a PR
> > > build is triggered.
> > >
> > > Do you think this approach would be helpful and we should move forward
> > > with it?
> > >
> > > Thanks,
> > > Meghna Baijal
> > >
> > >
> > >
> > >
> >
>
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is reviewed
Posted by Chris Olivier <cj...@gmail.com>.
Is the load artificially high because there's such a backlog due to other
reasons? Many may be triggering trivial changes to kick off another build
attempt (I have).
Do new PR changes actually stop the old build or do those go to completion?
I realize they show on the PR as a new build has started, but are the old
builds/tests always interrupted?
On Mon, Sep 11, 2017 at 6:12 PM Naveen Swamy <mn...@gmail.com> wrote:
> +1
>
> Even a small change in the PR is initiating a new build, I think this is
> needless and not serving any good purpose until a reviewer has looked into
> the PR. This is also adding unnecessary load on the mxnet build pipeline
> and slaves.
>
> Thanks, Naveen
>
>
> On Mon, Sep 11, 2017 at 5:50 PM, Meghna Baijal <meghnabaijal2017@gmail.com
> >
> wrote:
>
> > Hi All,
> > We would like to initiate a change in the way the PR builds are being
> > triggered. At the moment, every time a Pull Request is created, a build
> > gets triggered on Jenkins. Additional builds also get triggered due to
> > changes to the same PR.
> > Too many PR builds leads to resource starvation and very long queues and
> > long build times. Hence we would like to add some checks where a human
> > reviewer manually marks it to something like “ok to build” before a PR
> > build is triggered.
> >
> > Do you think this approach would be helpful and we should move forward
> > with it?
> >
> > Thanks,
> > Meghna Baijal
> >
> >
> >
> >
>
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is reviewed
Posted by Naveen Swamy <mn...@gmail.com>.
+1
Even a small change in the PR is initiating a new build, I think this is
needless and not serving any good purpose until a reviewer has looked into
the PR. This is also adding unnecessary load on the mxnet build pipeline
and slaves.
Thanks, Naveen
On Mon, Sep 11, 2017 at 5:50 PM, Meghna Baijal <me...@gmail.com>
wrote:
> Hi All,
> We would like to initiate a change in the way the PR builds are being
> triggered. At the moment, every time a Pull Request is created, a build
> gets triggered on Jenkins. Additional builds also get triggered due to
> changes to the same PR.
> Too many PR builds leads to resource starvation and very long queues and
> long build times. Hence we would like to add some checks where a human
> reviewer manually marks it to something like “ok to build” before a PR
> build is triggered.
>
> Do you think this approach would be helpful and we should move forward
> with it?
>
> Thanks,
> Meghna Baijal
>
>
>
>
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is reviewed
Posted by Madan Jampani <ma...@gmail.com>.
I agree with Naveen. This is not a one-way door. We can refine and improve
the process as needed in the future.
I don't see this as case of who works harder (machines or humans). PRs
always involve a manual step. All we are saying is lets do the full
verification when someone reviewed the changes and feels it is in a good
enough shape to be merged.
On Wed, Sep 13, 2017 at 9:56 AM, Naveen Swamy <mn...@gmail.com> wrote:
> The current number of build steps for a successful build is around 30 each
> requiring to run on a Jenkins executor(running on one instance), there are
> plans to add additional tests(more build steps) for additional platforms
> such as MAC OSX, IOT devices. I don't think it is practical nor necessary
> to run the entire pipeline on every PR change(even if you cancel the old
> build) before someone has looked into it. Please see the attached image
> that shows the complete set of build steps. Each step not a stage in the
> pipeline needs an instance, running on average to 2.5 hours. We have
> allocated 20 instances running for the build.
>
> [image: Inline image 1]
>
>
> While we optimize our builds to better handle changes, become smarter in
> what builds to kick off, As a compromise, I suggest we modify the PR builds
> to run the following steps in sequence:
> 1. Sanity Checks(lint, Coverity, etc.,)
> 2. CPU Build for Linux
> 3. Python CPU Unit Tests on Linux
>
> On a PR review and request(label or comment) run the existing pipeline.
>
> Also, we can revisit and refine this again after we have optimized the
> current build pipeline.
>
> Let me know if you still have concerns, we can chat offline as well to
> understand better.
>
> Thanks, Naveen
>
>
>
>
>
> On Tue, Sep 12, 2017 at 11:17 PM, Kumar, Gautam <ga...@amazon.com> wrote:
>
>> Agreed with Bhavin.
>>
>> One point which is being mentioned here is abort the previous build if a
>> new change is being published on same PR.
>> Which can be achievable by just changing the job configuration “Cancel
>> build on update” under Trigger setup.
>> https://github.com/jenkinsci/ghprb-plugin/issues/379 is the git hub
>> discussion page for same.
>>
>> -Gautam
>>
>>
>> On 9/12/17, 9:43 PM, "Lin, Haibin" <ha...@amazon.com> wrote:
>>
>> Also +1 on this.
>>
>> Haibin
>>
>> On 9/12/17, 9:28 PM, "Chris Olivier" <cj...@gmail.com> wrote:
>>
>> +1 to this
>>
>> On Tue, Sep 12, 2017 at 8:48 PM Bhavin Thaker <
>> bhavinthaker@gmail.com>
>> wrote:
>>
>> > My vote is to make the CI build and test process lightweight
>> and efficient
>> > and not involve a human to give a go-ahead for a PR build.
>> >
>> > There are multiple ways to engineer a stable and efficient CI
>> system,
>> > (already discussed on this email thread), including canceling
>> previously
>> > running build for a particular PR before invoking a new build,
>> using
>> > Incredibuild, adding more machines to CI, etc.
>> >
>> > In short, as we scale to many engineers working on MXNet around
>> the globe
>> > in different time-zones, precious human involvement should be
>> minimal and
>> > be used judiciously only for critical things like code-review
>> and only
>> > after sufficient amount of sanity build-tests have passed.
>> >
>> > Let the machines work harder for humans and not the other way
>> around.
>> >
>> > Bhavin Thaker.
>> >
>> > On Tue, Sep 12, 2017 at 12:20 PM Chris Olivier <
>> cjolivier01@gmail.com>
>> > wrote:
>> >
>> > > The majority of these iterations is to trigger a build on the
>> broken CI
>> > in
>> > > hopes it will finally pass...
>> > >
>> > > On Tue, Sep 12, 2017 at 11:15 AM Madan Jampani <
>> madan.jampani@gmail.com>
>> > > wrote:
>> > >
>> > > > +1
>> > > > I second only running sanity test (lint) until manual
>> approval.
>> > > >
>> > > > On Tue, Sep 12, 2017 at 11:05 AM, Naveen Swamy <
>> mnnaveen@gmail.com>
>> > > wrote:
>> > > >
>> > > > > Just to be clear, the proposal is not to remove the PR
>> build. It's
>> > only
>> > > > to
>> > > > > delay the PR build until a reviewer has looked at it and
>> marks it
>> > > > Approved
>> > > > > or adds a Label to build. Once it's approved and PR build
>> succeeds a
>> > > > > reviewer/committer can see the build result and merge to
>> the master.
>> > > > >
>> > > > > I don't mean to pick on the author of this PR(Chris), I
>> am just using
>> > > > this
>> > > > > build to support my argument.
>> > > > > https://builds.apache.org/view/Incubator%20Projects/job/
>> > > > > incubator-mxnet/view/change-requests/job/PR-7577/
>> > > > > This has gone through 74 iterations, I understand not
>> all of them
>> > are
>> > > > due
>> > > > > to changes and some of them are due to build instability
>> and some
>> > dummy
>> > > > > commits to getting the PR build to pass. However, the PR
>> has been
>> > under
>> > > > > review and changes being made for the last several days.
>> I don't
>> > think
>> > > it
>> > > > > warrants a build trigger for every new change. Each build
>> on average
>> > > > takes
>> > > > > about 2 hours running on all platforms.
>> > > > >
>> > > > > Probably we can run a lean down version of the current PR
>> build where
>> > > we
>> > > > > have sanity-test(lint)->build on linux(cpu)->unit test on
>> linux(cpu)
>> > ?
>> > > > >
>> > > > >
>> > > > > Thanks, Naveen
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Tue, Sep 12, 2017 at 4:27 AM, Joern Kottmann <
>> kottmann@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Not sure how it works with jenkins, but other CI serves
>> can look at
>> > > > > > the commit message and skip the CI run based on certain
>> commands in
>> > > > > > it.
>> > > > > >
>> > > > > > Might make sense for small changes such as
>> documentation updates,
>> > > half
>> > > > > > done PRs, etc.
>> > > > > >
>> > > > > > Jörn
>> > > > > >
>> > > > > > On Tue, Sep 12, 2017 at 11:17 AM, Larroy, Pedro <
>> > pllarroy@amazon.de>
>> > > > > > wrote:
>> > > > > > > Hi
>> > > > > > >
>> > > > > > > I would like to integrate our CI system for devices
>> to make sure
>> > > PRs
>> > > > > > build on ARM / android etc. Who has admin rights on the
>> repository
>> > so
>> > > > we
>> > > > > > can install the necessary hooks to trigger our builds?
>> > > > > > >
>> > > > > > >
>> > > > > > > Kind regards.
>> > > > > > > --
>> > > > > > >
>> > > > > > > Pedro
>> > > > > > >
>> > > > > > > On 12/09/17 02:50, "Meghna Baijal" <
>> meghnabaijal2017@gmail.com>
>> > > > wrote:
>> > > > > > >
>> > > > > > > Hi All,
>> > > > > > > We would like to initiate a change in the way the
>> PR builds
>> > are
>> > > > > > being triggered. At the moment, every time a Pull
>> Request is
>> > > created, a
>> > > > > > build gets triggered on Jenkins. Additional builds also
>> get
>> > triggered
>> > > > due
>> > > > > > to changes to the same PR.
>> > > > > > > Too many PR builds leads to resource starvation
>> and very long
>> > > > > queues
>> > > > > > and long build times. Hence we would like to add some
>> checks where
>> > a
>> > > > > human
>> > > > > > reviewer manually marks it to something like “ok to
>> build” before a
>> > > PR
>> > > > > > build is triggered.
>> > > > > > >
>> > > > > > > Do you think this approach would be helpful and
>> we should
>> > move
>> > > > > > forward with it?
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Meghna Baijal
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > Amazon Development Center Germany GmbH
>> > > > > > > Berlin - Dresden - Aachen
>> > > > > > > main office: Krausenstr. 38
>> <https://maps.google.com/?q=Krausenstr.+38&entry=gmail&source=g>, 10117
>> Berlin
>> > > > > > > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian
>> Schlaeger
>> > > > > > > Ust-ID: DE289237879
>> > > > > > > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>>
>>
>>
>>
>
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is reviewed
Posted by Naveen Swamy <mn...@gmail.com>.
The current number of build steps for a successful build is around 30 each
requiring to run on a Jenkins executor(running on one instance), there are
plans to add additional tests(more build steps) for additional platforms
such as MAC OSX, IOT devices. I don't think it is practical nor necessary
to run the entire pipeline on every PR change(even if you cancel the old
build) before someone has looked into it. Please see the attached image
that shows the complete set of build steps. Each step not a stage in the
pipeline needs an instance, running on average to 2.5 hours. We have
allocated 20 instances running for the build.
[image: Inline image 1]
While we optimize our builds to better handle changes, become smarter in
what builds to kick off, As a compromise, I suggest we modify the PR builds
to run the following steps in sequence:
1. Sanity Checks(lint, Coverity, etc.,)
2. CPU Build for Linux
3. Python CPU Unit Tests on Linux
On a PR review and request(label or comment) run the existing pipeline.
Also, we can revisit and refine this again after we have optimized the
current build pipeline.
Let me know if you still have concerns, we can chat offline as well to
understand better.
Thanks, Naveen
On Tue, Sep 12, 2017 at 11:17 PM, Kumar, Gautam <ga...@amazon.com> wrote:
> Agreed with Bhavin.
>
> One point which is being mentioned here is abort the previous build if a
> new change is being published on same PR.
> Which can be achievable by just changing the job configuration “Cancel
> build on update” under Trigger setup.
> https://github.com/jenkinsci/ghprb-plugin/issues/379 is the git hub
> discussion page for same.
>
> -Gautam
>
>
> On 9/12/17, 9:43 PM, "Lin, Haibin" <ha...@amazon.com> wrote:
>
> Also +1 on this.
>
> Haibin
>
> On 9/12/17, 9:28 PM, "Chris Olivier" <cj...@gmail.com> wrote:
>
> +1 to this
>
> On Tue, Sep 12, 2017 at 8:48 PM Bhavin Thaker <
> bhavinthaker@gmail.com>
> wrote:
>
> > My vote is to make the CI build and test process lightweight and
> efficient
> > and not involve a human to give a go-ahead for a PR build.
> >
> > There are multiple ways to engineer a stable and efficient CI
> system,
> > (already discussed on this email thread), including canceling
> previously
> > running build for a particular PR before invoking a new build,
> using
> > Incredibuild, adding more machines to CI, etc.
> >
> > In short, as we scale to many engineers working on MXNet around
> the globe
> > in different time-zones, precious human involvement should be
> minimal and
> > be used judiciously only for critical things like code-review
> and only
> > after sufficient amount of sanity build-tests have passed.
> >
> > Let the machines work harder for humans and not the other way
> around.
> >
> > Bhavin Thaker.
> >
> > On Tue, Sep 12, 2017 at 12:20 PM Chris Olivier <
> cjolivier01@gmail.com>
> > wrote:
> >
> > > The majority of these iterations is to trigger a build on the
> broken CI
> > in
> > > hopes it will finally pass...
> > >
> > > On Tue, Sep 12, 2017 at 11:15 AM Madan Jampani <
> madan.jampani@gmail.com>
> > > wrote:
> > >
> > > > +1
> > > > I second only running sanity test (lint) until manual
> approval.
> > > >
> > > > On Tue, Sep 12, 2017 at 11:05 AM, Naveen Swamy <
> mnnaveen@gmail.com>
> > > wrote:
> > > >
> > > > > Just to be clear, the proposal is not to remove the PR
> build. It's
> > only
> > > > to
> > > > > delay the PR build until a reviewer has looked at it and
> marks it
> > > > Approved
> > > > > or adds a Label to build. Once it's approved and PR build
> succeeds a
> > > > > reviewer/committer can see the build result and merge to
> the master.
> > > > >
> > > > > I don't mean to pick on the author of this PR(Chris), I am
> just using
> > > > this
> > > > > build to support my argument.
> > > > > https://builds.apache.org/view/Incubator%20Projects/job/
> > > > > incubator-mxnet/view/change-requests/job/PR-7577/
> > > > > This has gone through 74 iterations, I understand not all
> of them
> > are
> > > > due
> > > > > to changes and some of them are due to build instability
> and some
> > dummy
> > > > > commits to getting the PR build to pass. However, the PR
> has been
> > under
> > > > > review and changes being made for the last several days. I
> don't
> > think
> > > it
> > > > > warrants a build trigger for every new change. Each build
> on average
> > > > takes
> > > > > about 2 hours running on all platforms.
> > > > >
> > > > > Probably we can run a lean down version of the current PR
> build where
> > > we
> > > > > have sanity-test(lint)->build on linux(cpu)->unit test on
> linux(cpu)
> > ?
> > > > >
> > > > >
> > > > > Thanks, Naveen
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Sep 12, 2017 at 4:27 AM, Joern Kottmann <
> kottmann@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Not sure how it works with jenkins, but other CI serves
> can look at
> > > > > > the commit message and skip the CI run based on certain
> commands in
> > > > > > it.
> > > > > >
> > > > > > Might make sense for small changes such as documentation
> updates,
> > > half
> > > > > > done PRs, etc.
> > > > > >
> > > > > > Jörn
> > > > > >
> > > > > > On Tue, Sep 12, 2017 at 11:17 AM, Larroy, Pedro <
> > pllarroy@amazon.de>
> > > > > > wrote:
> > > > > > > Hi
> > > > > > >
> > > > > > > I would like to integrate our CI system for devices to
> make sure
> > > PRs
> > > > > > build on ARM / android etc. Who has admin rights on the
> repository
> > so
> > > > we
> > > > > > can install the necessary hooks to trigger our builds?
> > > > > > >
> > > > > > >
> > > > > > > Kind regards.
> > > > > > > --
> > > > > > >
> > > > > > > Pedro
> > > > > > >
> > > > > > > On 12/09/17 02:50, "Meghna Baijal" <
> meghnabaijal2017@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > Hi All,
> > > > > > > We would like to initiate a change in the way the
> PR builds
> > are
> > > > > > being triggered. At the moment, every time a Pull
> Request is
> > > created, a
> > > > > > build gets triggered on Jenkins. Additional builds also
> get
> > triggered
> > > > due
> > > > > > to changes to the same PR.
> > > > > > > Too many PR builds leads to resource starvation
> and very long
> > > > > queues
> > > > > > and long build times. Hence we would like to add some
> checks where
> > a
> > > > > human
> > > > > > reviewer manually marks it to something like “ok to
> build” before a
> > > PR
> > > > > > build is triggered.
> > > > > > >
> > > > > > > Do you think this approach would be helpful and we
> should
> > move
> > > > > > forward with it?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Meghna Baijal
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Amazon Development Center Germany GmbH
> > > > > > > Berlin - Dresden - Aachen
> > > > > > > main office: Krausenstr. 38, 10117 Berlin
> > > > > > > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian
> Schlaeger
> > > > > > > Ust-ID: DE289237879
> > > > > > > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
>
>
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is
reviewed
Posted by "Kumar, Gautam" <ga...@amazon.com>.
Agreed with Bhavin.
One point which is being mentioned here is abort the previous build if a new change is being published on same PR.
Which can be achievable by just changing the job configuration “Cancel build on update” under Trigger setup.
https://github.com/jenkinsci/ghprb-plugin/issues/379 is the git hub discussion page for same.
-Gautam
On 9/12/17, 9:43 PM, "Lin, Haibin" <ha...@amazon.com> wrote:
Also +1 on this.
Haibin
On 9/12/17, 9:28 PM, "Chris Olivier" <cj...@gmail.com> wrote:
+1 to this
On Tue, Sep 12, 2017 at 8:48 PM Bhavin Thaker <bh...@gmail.com>
wrote:
> My vote is to make the CI build and test process lightweight and efficient
> and not involve a human to give a go-ahead for a PR build.
>
> There are multiple ways to engineer a stable and efficient CI system,
> (already discussed on this email thread), including canceling previously
> running build for a particular PR before invoking a new build, using
> Incredibuild, adding more machines to CI, etc.
>
> In short, as we scale to many engineers working on MXNet around the globe
> in different time-zones, precious human involvement should be minimal and
> be used judiciously only for critical things like code-review and only
> after sufficient amount of sanity build-tests have passed.
>
> Let the machines work harder for humans and not the other way around.
>
> Bhavin Thaker.
>
> On Tue, Sep 12, 2017 at 12:20 PM Chris Olivier <cj...@gmail.com>
> wrote:
>
> > The majority of these iterations is to trigger a build on the broken CI
> in
> > hopes it will finally pass...
> >
> > On Tue, Sep 12, 2017 at 11:15 AM Madan Jampani <ma...@gmail.com>
> > wrote:
> >
> > > +1
> > > I second only running sanity test (lint) until manual approval.
> > >
> > > On Tue, Sep 12, 2017 at 11:05 AM, Naveen Swamy <mn...@gmail.com>
> > wrote:
> > >
> > > > Just to be clear, the proposal is not to remove the PR build. It's
> only
> > > to
> > > > delay the PR build until a reviewer has looked at it and marks it
> > > Approved
> > > > or adds a Label to build. Once it's approved and PR build succeeds a
> > > > reviewer/committer can see the build result and merge to the master.
> > > >
> > > > I don't mean to pick on the author of this PR(Chris), I am just using
> > > this
> > > > build to support my argument.
> > > > https://builds.apache.org/view/Incubator%20Projects/job/
> > > > incubator-mxnet/view/change-requests/job/PR-7577/
> > > > This has gone through 74 iterations, I understand not all of them
> are
> > > due
> > > > to changes and some of them are due to build instability and some
> dummy
> > > > commits to getting the PR build to pass. However, the PR has been
> under
> > > > review and changes being made for the last several days. I don't
> think
> > it
> > > > warrants a build trigger for every new change. Each build on average
> > > takes
> > > > about 2 hours running on all platforms.
> > > >
> > > > Probably we can run a lean down version of the current PR build where
> > we
> > > > have sanity-test(lint)->build on linux(cpu)->unit test on linux(cpu)
> ?
> > > >
> > > >
> > > > Thanks, Naveen
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Sep 12, 2017 at 4:27 AM, Joern Kottmann <ko...@gmail.com>
> > > > wrote:
> > > >
> > > > > Not sure how it works with jenkins, but other CI serves can look at
> > > > > the commit message and skip the CI run based on certain commands in
> > > > > it.
> > > > >
> > > > > Might make sense for small changes such as documentation updates,
> > half
> > > > > done PRs, etc.
> > > > >
> > > > > Jörn
> > > > >
> > > > > On Tue, Sep 12, 2017 at 11:17 AM, Larroy, Pedro <
> pllarroy@amazon.de>
> > > > > wrote:
> > > > > > Hi
> > > > > >
> > > > > > I would like to integrate our CI system for devices to make sure
> > PRs
> > > > > build on ARM / android etc. Who has admin rights on the repository
> so
> > > we
> > > > > can install the necessary hooks to trigger our builds?
> > > > > >
> > > > > >
> > > > > > Kind regards.
> > > > > > --
> > > > > >
> > > > > > Pedro
> > > > > >
> > > > > > On 12/09/17 02:50, "Meghna Baijal" <me...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > Hi All,
> > > > > > We would like to initiate a change in the way the PR builds
> are
> > > > > being triggered. At the moment, every time a Pull Request is
> > created, a
> > > > > build gets triggered on Jenkins. Additional builds also get
> triggered
> > > due
> > > > > to changes to the same PR.
> > > > > > Too many PR builds leads to resource starvation and very long
> > > > queues
> > > > > and long build times. Hence we would like to add some checks where
> a
> > > > human
> > > > > reviewer manually marks it to something like “ok to build” before a
> > PR
> > > > > build is triggered.
> > > > > >
> > > > > > Do you think this approach would be helpful and we should
> move
> > > > > forward with it?
> > > > > >
> > > > > > Thanks,
> > > > > > Meghna Baijal
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Amazon Development Center Germany GmbH
> > > > > > Berlin - Dresden - Aachen
> > > > > > main office: Krausenstr. 38, 10117 Berlin
> > > > > > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
> > > > > > Ust-ID: DE289237879
> > > > > > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
> > > > >
> > > >
> > >
> >
>
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is
reviewed
Posted by "Lin, Haibin" <ha...@amazon.com>.
Also +1 on this.
Haibin
On 9/12/17, 9:28 PM, "Chris Olivier" <cj...@gmail.com> wrote:
+1 to this
On Tue, Sep 12, 2017 at 8:48 PM Bhavin Thaker <bh...@gmail.com>
wrote:
> My vote is to make the CI build and test process lightweight and efficient
> and not involve a human to give a go-ahead for a PR build.
>
> There are multiple ways to engineer a stable and efficient CI system,
> (already discussed on this email thread), including canceling previously
> running build for a particular PR before invoking a new build, using
> Incredibuild, adding more machines to CI, etc.
>
> In short, as we scale to many engineers working on MXNet around the globe
> in different time-zones, precious human involvement should be minimal and
> be used judiciously only for critical things like code-review and only
> after sufficient amount of sanity build-tests have passed.
>
> Let the machines work harder for humans and not the other way around.
>
> Bhavin Thaker.
>
> On Tue, Sep 12, 2017 at 12:20 PM Chris Olivier <cj...@gmail.com>
> wrote:
>
> > The majority of these iterations is to trigger a build on the broken CI
> in
> > hopes it will finally pass...
> >
> > On Tue, Sep 12, 2017 at 11:15 AM Madan Jampani <ma...@gmail.com>
> > wrote:
> >
> > > +1
> > > I second only running sanity test (lint) until manual approval.
> > >
> > > On Tue, Sep 12, 2017 at 11:05 AM, Naveen Swamy <mn...@gmail.com>
> > wrote:
> > >
> > > > Just to be clear, the proposal is not to remove the PR build. It's
> only
> > > to
> > > > delay the PR build until a reviewer has looked at it and marks it
> > > Approved
> > > > or adds a Label to build. Once it's approved and PR build succeeds a
> > > > reviewer/committer can see the build result and merge to the master.
> > > >
> > > > I don't mean to pick on the author of this PR(Chris), I am just using
> > > this
> > > > build to support my argument.
> > > > https://builds.apache.org/view/Incubator%20Projects/job/
> > > > incubator-mxnet/view/change-requests/job/PR-7577/
> > > > This has gone through 74 iterations, I understand not all of them
> are
> > > due
> > > > to changes and some of them are due to build instability and some
> dummy
> > > > commits to getting the PR build to pass. However, the PR has been
> under
> > > > review and changes being made for the last several days. I don't
> think
> > it
> > > > warrants a build trigger for every new change. Each build on average
> > > takes
> > > > about 2 hours running on all platforms.
> > > >
> > > > Probably we can run a lean down version of the current PR build where
> > we
> > > > have sanity-test(lint)->build on linux(cpu)->unit test on linux(cpu)
> ?
> > > >
> > > >
> > > > Thanks, Naveen
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Sep 12, 2017 at 4:27 AM, Joern Kottmann <ko...@gmail.com>
> > > > wrote:
> > > >
> > > > > Not sure how it works with jenkins, but other CI serves can look at
> > > > > the commit message and skip the CI run based on certain commands in
> > > > > it.
> > > > >
> > > > > Might make sense for small changes such as documentation updates,
> > half
> > > > > done PRs, etc.
> > > > >
> > > > > Jörn
> > > > >
> > > > > On Tue, Sep 12, 2017 at 11:17 AM, Larroy, Pedro <
> pllarroy@amazon.de>
> > > > > wrote:
> > > > > > Hi
> > > > > >
> > > > > > I would like to integrate our CI system for devices to make sure
> > PRs
> > > > > build on ARM / android etc. Who has admin rights on the repository
> so
> > > we
> > > > > can install the necessary hooks to trigger our builds?
> > > > > >
> > > > > >
> > > > > > Kind regards.
> > > > > > --
> > > > > >
> > > > > > Pedro
> > > > > >
> > > > > > On 12/09/17 02:50, "Meghna Baijal" <me...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > Hi All,
> > > > > > We would like to initiate a change in the way the PR builds
> are
> > > > > being triggered. At the moment, every time a Pull Request is
> > created, a
> > > > > build gets triggered on Jenkins. Additional builds also get
> triggered
> > > due
> > > > > to changes to the same PR.
> > > > > > Too many PR builds leads to resource starvation and very long
> > > > queues
> > > > > and long build times. Hence we would like to add some checks where
> a
> > > > human
> > > > > reviewer manually marks it to something like “ok to build” before a
> > PR
> > > > > build is triggered.
> > > > > >
> > > > > > Do you think this approach would be helpful and we should
> move
> > > > > forward with it?
> > > > > >
> > > > > > Thanks,
> > > > > > Meghna Baijal
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Amazon Development Center Germany GmbH
> > > > > > Berlin - Dresden - Aachen
> > > > > > main office: Krausenstr. 38, 10117 Berlin
> > > > > > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
> > > > > > Ust-ID: DE289237879
> > > > > > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
> > > > >
> > > >
> > >
> >
>
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is reviewed
Posted by Chris Olivier <cj...@gmail.com>.
+1 to this
On Tue, Sep 12, 2017 at 8:48 PM Bhavin Thaker <bh...@gmail.com>
wrote:
> My vote is to make the CI build and test process lightweight and efficient
> and not involve a human to give a go-ahead for a PR build.
>
> There are multiple ways to engineer a stable and efficient CI system,
> (already discussed on this email thread), including canceling previously
> running build for a particular PR before invoking a new build, using
> Incredibuild, adding more machines to CI, etc.
>
> In short, as we scale to many engineers working on MXNet around the globe
> in different time-zones, precious human involvement should be minimal and
> be used judiciously only for critical things like code-review and only
> after sufficient amount of sanity build-tests have passed.
>
> Let the machines work harder for humans and not the other way around.
>
> Bhavin Thaker.
>
> On Tue, Sep 12, 2017 at 12:20 PM Chris Olivier <cj...@gmail.com>
> wrote:
>
> > The majority of these iterations is to trigger a build on the broken CI
> in
> > hopes it will finally pass...
> >
> > On Tue, Sep 12, 2017 at 11:15 AM Madan Jampani <ma...@gmail.com>
> > wrote:
> >
> > > +1
> > > I second only running sanity test (lint) until manual approval.
> > >
> > > On Tue, Sep 12, 2017 at 11:05 AM, Naveen Swamy <mn...@gmail.com>
> > wrote:
> > >
> > > > Just to be clear, the proposal is not to remove the PR build. It's
> only
> > > to
> > > > delay the PR build until a reviewer has looked at it and marks it
> > > Approved
> > > > or adds a Label to build. Once it's approved and PR build succeeds a
> > > > reviewer/committer can see the build result and merge to the master.
> > > >
> > > > I don't mean to pick on the author of this PR(Chris), I am just using
> > > this
> > > > build to support my argument.
> > > > https://builds.apache.org/view/Incubator%20Projects/job/
> > > > incubator-mxnet/view/change-requests/job/PR-7577/
> > > > This has gone through 74 iterations, I understand not all of them
> are
> > > due
> > > > to changes and some of them are due to build instability and some
> dummy
> > > > commits to getting the PR build to pass. However, the PR has been
> under
> > > > review and changes being made for the last several days. I don't
> think
> > it
> > > > warrants a build trigger for every new change. Each build on average
> > > takes
> > > > about 2 hours running on all platforms.
> > > >
> > > > Probably we can run a lean down version of the current PR build where
> > we
> > > > have sanity-test(lint)->build on linux(cpu)->unit test on linux(cpu)
> ?
> > > >
> > > >
> > > > Thanks, Naveen
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Sep 12, 2017 at 4:27 AM, Joern Kottmann <ko...@gmail.com>
> > > > wrote:
> > > >
> > > > > Not sure how it works with jenkins, but other CI serves can look at
> > > > > the commit message and skip the CI run based on certain commands in
> > > > > it.
> > > > >
> > > > > Might make sense for small changes such as documentation updates,
> > half
> > > > > done PRs, etc.
> > > > >
> > > > > Jörn
> > > > >
> > > > > On Tue, Sep 12, 2017 at 11:17 AM, Larroy, Pedro <
> pllarroy@amazon.de>
> > > > > wrote:
> > > > > > Hi
> > > > > >
> > > > > > I would like to integrate our CI system for devices to make sure
> > PRs
> > > > > build on ARM / android etc. Who has admin rights on the repository
> so
> > > we
> > > > > can install the necessary hooks to trigger our builds?
> > > > > >
> > > > > >
> > > > > > Kind regards.
> > > > > > --
> > > > > >
> > > > > > Pedro
> > > > > >
> > > > > > On 12/09/17 02:50, "Meghna Baijal" <me...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > Hi All,
> > > > > > We would like to initiate a change in the way the PR builds
> are
> > > > > being triggered. At the moment, every time a Pull Request is
> > created, a
> > > > > build gets triggered on Jenkins. Additional builds also get
> triggered
> > > due
> > > > > to changes to the same PR.
> > > > > > Too many PR builds leads to resource starvation and very long
> > > > queues
> > > > > and long build times. Hence we would like to add some checks where
> a
> > > > human
> > > > > reviewer manually marks it to something like “ok to build” before a
> > PR
> > > > > build is triggered.
> > > > > >
> > > > > > Do you think this approach would be helpful and we should
> move
> > > > > forward with it?
> > > > > >
> > > > > > Thanks,
> > > > > > Meghna Baijal
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Amazon Development Center Germany GmbH
> > > > > > Berlin - Dresden - Aachen
> > > > > > main office: Krausenstr. 38, 10117 Berlin
> > > > > > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
> > > > > > Ust-ID: DE289237879
> > > > > > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
> > > > >
> > > >
> > >
> >
>
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is reviewed
Posted by Bhavin Thaker <bh...@gmail.com>.
My vote is to make the CI build and test process lightweight and efficient
and not involve a human to give a go-ahead for a PR build.
There are multiple ways to engineer a stable and efficient CI system,
(already discussed on this email thread), including canceling previously
running build for a particular PR before invoking a new build, using
Incredibuild, adding more machines to CI, etc.
In short, as we scale to many engineers working on MXNet around the globe
in different time-zones, precious human involvement should be minimal and
be used judiciously only for critical things like code-review and only
after sufficient amount of sanity build-tests have passed.
Let the machines work harder for humans and not the other way around.
Bhavin Thaker.
On Tue, Sep 12, 2017 at 12:20 PM Chris Olivier <cj...@gmail.com>
wrote:
> The majority of these iterations is to trigger a build on the broken CI in
> hopes it will finally pass...
>
> On Tue, Sep 12, 2017 at 11:15 AM Madan Jampani <ma...@gmail.com>
> wrote:
>
> > +1
> > I second only running sanity test (lint) until manual approval.
> >
> > On Tue, Sep 12, 2017 at 11:05 AM, Naveen Swamy <mn...@gmail.com>
> wrote:
> >
> > > Just to be clear, the proposal is not to remove the PR build. It's only
> > to
> > > delay the PR build until a reviewer has looked at it and marks it
> > Approved
> > > or adds a Label to build. Once it's approved and PR build succeeds a
> > > reviewer/committer can see the build result and merge to the master.
> > >
> > > I don't mean to pick on the author of this PR(Chris), I am just using
> > this
> > > build to support my argument.
> > > https://builds.apache.org/view/Incubator%20Projects/job/
> > > incubator-mxnet/view/change-requests/job/PR-7577/
> > > This has gone through 74 iterations, I understand not all of them are
> > due
> > > to changes and some of them are due to build instability and some dummy
> > > commits to getting the PR build to pass. However, the PR has been under
> > > review and changes being made for the last several days. I don't think
> it
> > > warrants a build trigger for every new change. Each build on average
> > takes
> > > about 2 hours running on all platforms.
> > >
> > > Probably we can run a lean down version of the current PR build where
> we
> > > have sanity-test(lint)->build on linux(cpu)->unit test on linux(cpu) ?
> > >
> > >
> > > Thanks, Naveen
> > >
> > >
> > >
> > >
> > > On Tue, Sep 12, 2017 at 4:27 AM, Joern Kottmann <ko...@gmail.com>
> > > wrote:
> > >
> > > > Not sure how it works with jenkins, but other CI serves can look at
> > > > the commit message and skip the CI run based on certain commands in
> > > > it.
> > > >
> > > > Might make sense for small changes such as documentation updates,
> half
> > > > done PRs, etc.
> > > >
> > > > Jörn
> > > >
> > > > On Tue, Sep 12, 2017 at 11:17 AM, Larroy, Pedro <pl...@amazon.de>
> > > > wrote:
> > > > > Hi
> > > > >
> > > > > I would like to integrate our CI system for devices to make sure
> PRs
> > > > build on ARM / android etc. Who has admin rights on the repository so
> > we
> > > > can install the necessary hooks to trigger our builds?
> > > > >
> > > > >
> > > > > Kind regards.
> > > > > --
> > > > >
> > > > > Pedro
> > > > >
> > > > > On 12/09/17 02:50, "Meghna Baijal" <me...@gmail.com>
> > wrote:
> > > > >
> > > > > Hi All,
> > > > > We would like to initiate a change in the way the PR builds are
> > > > being triggered. At the moment, every time a Pull Request is
> created, a
> > > > build gets triggered on Jenkins. Additional builds also get triggered
> > due
> > > > to changes to the same PR.
> > > > > Too many PR builds leads to resource starvation and very long
> > > queues
> > > > and long build times. Hence we would like to add some checks where a
> > > human
> > > > reviewer manually marks it to something like “ok to build” before a
> PR
> > > > build is triggered.
> > > > >
> > > > > Do you think this approach would be helpful and we should move
> > > > forward with it?
> > > > >
> > > > > Thanks,
> > > > > Meghna Baijal
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Amazon Development Center Germany GmbH
> > > > > Berlin - Dresden - Aachen
> > > > > main office: Krausenstr. 38, 10117 Berlin
> > > > > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
> > > > > Ust-ID: DE289237879
> > > > > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
> > > >
> > >
> >
>
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is reviewed
Posted by Chris Olivier <cj...@gmail.com>.
The majority of these iterations is to trigger a build on the broken CI in
hopes it will finally pass...
On Tue, Sep 12, 2017 at 11:15 AM Madan Jampani <ma...@gmail.com>
wrote:
> +1
> I second only running sanity test (lint) until manual approval.
>
> On Tue, Sep 12, 2017 at 11:05 AM, Naveen Swamy <mn...@gmail.com> wrote:
>
> > Just to be clear, the proposal is not to remove the PR build. It's only
> to
> > delay the PR build until a reviewer has looked at it and marks it
> Approved
> > or adds a Label to build. Once it's approved and PR build succeeds a
> > reviewer/committer can see the build result and merge to the master.
> >
> > I don't mean to pick on the author of this PR(Chris), I am just using
> this
> > build to support my argument.
> > https://builds.apache.org/view/Incubator%20Projects/job/
> > incubator-mxnet/view/change-requests/job/PR-7577/
> > This has gone through 74 iterations, I understand not all of them are
> due
> > to changes and some of them are due to build instability and some dummy
> > commits to getting the PR build to pass. However, the PR has been under
> > review and changes being made for the last several days. I don't think it
> > warrants a build trigger for every new change. Each build on average
> takes
> > about 2 hours running on all platforms.
> >
> > Probably we can run a lean down version of the current PR build where we
> > have sanity-test(lint)->build on linux(cpu)->unit test on linux(cpu) ?
> >
> >
> > Thanks, Naveen
> >
> >
> >
> >
> > On Tue, Sep 12, 2017 at 4:27 AM, Joern Kottmann <ko...@gmail.com>
> > wrote:
> >
> > > Not sure how it works with jenkins, but other CI serves can look at
> > > the commit message and skip the CI run based on certain commands in
> > > it.
> > >
> > > Might make sense for small changes such as documentation updates, half
> > > done PRs, etc.
> > >
> > > Jörn
> > >
> > > On Tue, Sep 12, 2017 at 11:17 AM, Larroy, Pedro <pl...@amazon.de>
> > > wrote:
> > > > Hi
> > > >
> > > > I would like to integrate our CI system for devices to make sure PRs
> > > build on ARM / android etc. Who has admin rights on the repository so
> we
> > > can install the necessary hooks to trigger our builds?
> > > >
> > > >
> > > > Kind regards.
> > > > --
> > > >
> > > > Pedro
> > > >
> > > > On 12/09/17 02:50, "Meghna Baijal" <me...@gmail.com>
> wrote:
> > > >
> > > > Hi All,
> > > > We would like to initiate a change in the way the PR builds are
> > > being triggered. At the moment, every time a Pull Request is created, a
> > > build gets triggered on Jenkins. Additional builds also get triggered
> due
> > > to changes to the same PR.
> > > > Too many PR builds leads to resource starvation and very long
> > queues
> > > and long build times. Hence we would like to add some checks where a
> > human
> > > reviewer manually marks it to something like “ok to build” before a PR
> > > build is triggered.
> > > >
> > > > Do you think this approach would be helpful and we should move
> > > forward with it?
> > > >
> > > > Thanks,
> > > > Meghna Baijal
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Amazon Development Center Germany GmbH
> > > > Berlin - Dresden - Aachen
> > > > main office: Krausenstr. 38, 10117 Berlin
> > > > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
> > > > Ust-ID: DE289237879
> > > > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
> > >
> >
>
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is reviewed
Posted by Madan Jampani <ma...@gmail.com>.
+1
I second only running sanity test (lint) until manual approval.
On Tue, Sep 12, 2017 at 11:05 AM, Naveen Swamy <mn...@gmail.com> wrote:
> Just to be clear, the proposal is not to remove the PR build. It's only to
> delay the PR build until a reviewer has looked at it and marks it Approved
> or adds a Label to build. Once it's approved and PR build succeeds a
> reviewer/committer can see the build result and merge to the master.
>
> I don't mean to pick on the author of this PR(Chris), I am just using this
> build to support my argument.
> https://builds.apache.org/view/Incubator%20Projects/job/
> incubator-mxnet/view/change-requests/job/PR-7577/
> This has gone through 74 iterations, I understand not all of them are due
> to changes and some of them are due to build instability and some dummy
> commits to getting the PR build to pass. However, the PR has been under
> review and changes being made for the last several days. I don't think it
> warrants a build trigger for every new change. Each build on average takes
> about 2 hours running on all platforms.
>
> Probably we can run a lean down version of the current PR build where we
> have sanity-test(lint)->build on linux(cpu)->unit test on linux(cpu) ?
>
>
> Thanks, Naveen
>
>
>
>
> On Tue, Sep 12, 2017 at 4:27 AM, Joern Kottmann <ko...@gmail.com>
> wrote:
>
> > Not sure how it works with jenkins, but other CI serves can look at
> > the commit message and skip the CI run based on certain commands in
> > it.
> >
> > Might make sense for small changes such as documentation updates, half
> > done PRs, etc.
> >
> > Jörn
> >
> > On Tue, Sep 12, 2017 at 11:17 AM, Larroy, Pedro <pl...@amazon.de>
> > wrote:
> > > Hi
> > >
> > > I would like to integrate our CI system for devices to make sure PRs
> > build on ARM / android etc. Who has admin rights on the repository so we
> > can install the necessary hooks to trigger our builds?
> > >
> > >
> > > Kind regards.
> > > --
> > >
> > > Pedro
> > >
> > > On 12/09/17 02:50, "Meghna Baijal" <me...@gmail.com> wrote:
> > >
> > > Hi All,
> > > We would like to initiate a change in the way the PR builds are
> > being triggered. At the moment, every time a Pull Request is created, a
> > build gets triggered on Jenkins. Additional builds also get triggered due
> > to changes to the same PR.
> > > Too many PR builds leads to resource starvation and very long
> queues
> > and long build times. Hence we would like to add some checks where a
> human
> > reviewer manually marks it to something like “ok to build” before a PR
> > build is triggered.
> > >
> > > Do you think this approach would be helpful and we should move
> > forward with it?
> > >
> > > Thanks,
> > > Meghna Baijal
> > >
> > >
> > >
> > >
> > >
> > >
> > > Amazon Development Center Germany GmbH
> > > Berlin - Dresden - Aachen
> > > main office: Krausenstr. 38, 10117 Berlin
> > > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
> > > Ust-ID: DE289237879
> > > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
> >
>
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is reviewed
Posted by Naveen Swamy <mn...@gmail.com>.
Just to be clear, the proposal is not to remove the PR build. It's only to
delay the PR build until a reviewer has looked at it and marks it Approved
or adds a Label to build. Once it's approved and PR build succeeds a
reviewer/committer can see the build result and merge to the master.
I don't mean to pick on the author of this PR(Chris), I am just using this
build to support my argument.
https://builds.apache.org/view/Incubator%20Projects/job/incubator-mxnet/view/change-requests/job/PR-7577/
This has gone through 74 iterations, I understand not all of them are due
to changes and some of them are due to build instability and some dummy
commits to getting the PR build to pass. However, the PR has been under
review and changes being made for the last several days. I don't think it
warrants a build trigger for every new change. Each build on average takes
about 2 hours running on all platforms.
Probably we can run a lean down version of the current PR build where we
have sanity-test(lint)->build on linux(cpu)->unit test on linux(cpu) ?
Thanks, Naveen
On Tue, Sep 12, 2017 at 4:27 AM, Joern Kottmann <ko...@gmail.com> wrote:
> Not sure how it works with jenkins, but other CI serves can look at
> the commit message and skip the CI run based on certain commands in
> it.
>
> Might make sense for small changes such as documentation updates, half
> done PRs, etc.
>
> Jörn
>
> On Tue, Sep 12, 2017 at 11:17 AM, Larroy, Pedro <pl...@amazon.de>
> wrote:
> > Hi
> >
> > I would like to integrate our CI system for devices to make sure PRs
> build on ARM / android etc. Who has admin rights on the repository so we
> can install the necessary hooks to trigger our builds?
> >
> >
> > Kind regards.
> > --
> >
> > Pedro
> >
> > On 12/09/17 02:50, "Meghna Baijal" <me...@gmail.com> wrote:
> >
> > Hi All,
> > We would like to initiate a change in the way the PR builds are
> being triggered. At the moment, every time a Pull Request is created, a
> build gets triggered on Jenkins. Additional builds also get triggered due
> to changes to the same PR.
> > Too many PR builds leads to resource starvation and very long queues
> and long build times. Hence we would like to add some checks where a human
> reviewer manually marks it to something like “ok to build” before a PR
> build is triggered.
> >
> > Do you think this approach would be helpful and we should move
> forward with it?
> >
> > Thanks,
> > Meghna Baijal
> >
> >
> >
> >
> >
> >
> > Amazon Development Center Germany GmbH
> > Berlin - Dresden - Aachen
> > main office: Krausenstr. 38, 10117 Berlin
> > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
> > Ust-ID: DE289237879
> > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
>
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is reviewed
Posted by Joern Kottmann <ko...@gmail.com>.
Not sure how it works with jenkins, but other CI serves can look at
the commit message and skip the CI run based on certain commands in
it.
Might make sense for small changes such as documentation updates, half
done PRs, etc.
Jörn
On Tue, Sep 12, 2017 at 11:17 AM, Larroy, Pedro <pl...@amazon.de> wrote:
> Hi
>
> I would like to integrate our CI system for devices to make sure PRs build on ARM / android etc. Who has admin rights on the repository so we can install the necessary hooks to trigger our builds?
>
>
> Kind regards.
> --
>
> Pedro
>
> On 12/09/17 02:50, "Meghna Baijal" <me...@gmail.com> wrote:
>
> Hi All,
> We would like to initiate a change in the way the PR builds are being triggered. At the moment, every time a Pull Request is created, a build gets triggered on Jenkins. Additional builds also get triggered due to changes to the same PR.
> Too many PR builds leads to resource starvation and very long queues and long build times. Hence we would like to add some checks where a human reviewer manually marks it to something like “ok to build” before a PR build is triggered.
>
> Do you think this approach would be helpful and we should move forward with it?
>
> Thanks,
> Meghna Baijal
>
>
>
>
>
>
> Amazon Development Center Germany GmbH
> Berlin - Dresden - Aachen
> main office: Krausenstr. 38, 10117 Berlin
> Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
> Ust-ID: DE289237879
> Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
Re: MXNet: Run PR builds on Apache Jenkins only after the commit is
reviewed
Posted by "Larroy, Pedro" <pl...@amazon.de>.
Hi
I would like to integrate our CI system for devices to make sure PRs build on ARM / android etc. Who has admin rights on the repository so we can install the necessary hooks to trigger our builds?
Kind regards.
--
Pedro
On 12/09/17 02:50, "Meghna Baijal" <me...@gmail.com> wrote:
Hi All,
We would like to initiate a change in the way the PR builds are being triggered. At the moment, every time a Pull Request is created, a build gets triggered on Jenkins. Additional builds also get triggered due to changes to the same PR.
Too many PR builds leads to resource starvation and very long queues and long build times. Hence we would like to add some checks where a human reviewer manually marks it to something like “ok to build” before a PR build is triggered.
Do you think this approach would be helpful and we should move forward with it?
Thanks,
Meghna Baijal
Amazon Development Center Germany GmbH
Berlin - Dresden - Aachen
main office: Krausenstr. 38, 10117 Berlin
Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
Ust-ID: DE289237879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B