You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mxnet.apache.org by Chaitanya Bapat <ch...@gmail.com> on 2020/03/12 21:00:22 UTC
MXNet Bot Demo
Hello MXNet community,
I have built an MXNet Bot <https://github.com/mxnet-bot> that allows PR
Authors, Committers and Jenkins Admins to trigger CI manually.
It handles 2 problems
1. Manual CI trigger instead of existing automated CI trigger
2. Gives permissions to PR Authors (in addition to MXNet Committers and
Jenkins Admins)
Design Doc :
https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
I urge you all to attend the demonstration meeting and lend your views on
the same.
Thank you,
Chai
*Meeting Details*:
==============Conference Bridge Information==============
You have been invited to an online meeting, powered by Amazon Chime.
*Chime meeting ID*: *9272158344*
Join via Chime clients (manually): Select 'Meetings > Join a Meeting', and
enter 9272158344
Join via Chime clients (auto-call): If you invite auto-call as attendee,
Chime will call you when the meeting starts, select 'Answer'
*Join via browser screen share*: https://chime.aws/9272158344
*Join via phone* (US): +1-929-432-4463,,,9272158344#
*Join via phone (US toll-free)*: +1-855-552-4463,,,9272158344#
International dial-in: https://chime.aws/dialinnumbers/
In-room video system: Ext: 62000, Meeting PIN: 9272158344#
--
*Chaitanya Prakash Bapat*
*+1 (973) 953-6299*
[image: https://www.linkedin.com//in/chaibapat25]
<https://github.com/ChaiBapchya>[image: https://www.facebook.com/chaibapat]
<https://www.facebook.com/chaibapchya>[image:
https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
https://www.linkedin.com//in/chaibapat25]
<https://www.linkedin.com//in/chaibapchya/>
Re: MXNet Bot Demo
Posted by Kshitij Kalambarkar <ks...@gmail.com>.
Great Job. Would be really helpful.
Regards,
Kshiteej
On Wed, 25 Mar, 2020, 7:29 AM Chaitanya Bapat, <ch...@gmail.com> wrote:
> Yes Denisa, you can use it on existing PRs as well. Sorry for miswording.
>
> Bot will comment instructions on how-to-use for every new PR.
> You can invoke bot for all the PRs.
>
>
> On Tue, 24 Mar 2020 at 18:00, Denisa Roberts <de...@gmail.com>
> wrote:
>
> > Hi- Only for new PRs? Can I use it for an existing PR to retrigger only
> > specific tests?
> >
> > On Tue, Mar 24, 2020 at 8:27 PM Chaitanya Bapat <ch...@gmail.com>
> > wrote:
> >
> >> Hello MXNet community,
> >>
> >> Update: Bot has been deployed 🚀 on apache/incubator-mxnet.
> >>
> >> For every new PR, bot will comment with a help message (instructing what
> >> command to comment)
> >> It can trigger all jobs or specific jobs for users.
> >>
> >> Do use and if you find issues/suggestions do comment on this mail thread
> >> or
> >> the GitHub issue :
> https://github.com/apache/incubator-mxnet/issues/17801
> >>
> >> Thanks to Denis, Sandeep, Joe, Pedro, Marco and the design doc reviewers
> >> for valuable feedback and guidance.
> >>
> >> Thank you,
> >> Chai
> >>
> >> On Mon, 23 Mar 2020 at 13:08, sandeep krishnamurthy <
> >> sandeep.krishna98@gmail.com> wrote:
> >>
> >> > Thank you Chaitanya and Marco for helping the MXNet community.
> >> >
> >> > On Mon, Mar 23, 2020 at 12:56 PM Marco de Abreu <
> >> marco.g.abreu@gmail.com>
> >> > wrote:
> >> >
> >> > > Sure, already done.
> >> > >
> >> > > -Marco
> >> > >
> >> > > On Mon, Mar 23, 2020 at 8:53 PM Chaitanya Bapat <
> chai.bapat@gmail.com
> >> >
> >> > > wrote:
> >> > >
> >> > > > Hello,
> >> > > > Update: Apache Infra Ticket for MXNet Bot
> >> > > > Thanks once again, Marco for opening the ticket. But turns out,
> >> Apache
> >> > > > Infra folks closed it stating: "Security concerns around allowing
> >> > unknown
> >> > > > person to submit PR and run our hardware". Furthermore, it goes
> onto
> >> > > state
> >> > > > that bot circumvents the dependence on Jenkins Admins which is
> like
> >> > > solving
> >> > > > a problem that doesn't exist.
> >> > > >
> >> > > > I sense there is some confusion in the communication (maybe on my
> >> > part).
> >> > > It
> >> > > > turns out the security concerns aren't actually correct.
> >> > > >
> >> > > > 1. Unknown person can submit a PR (before & after bot proposal),
> and
> >> > run
> >> > > > our hardware (pre as well as post bot).
> >> > > > 2. Code should be reviewed by somebody with an ICLA on file. This
> >> > doesn't
> >> > > > change either. Prior to merging a PR, code has to be approved by a
> >> > > > committer just like before.
> >> > > > Overall it looks like the job of the bot isn't clear to folks in
> >> Apache
> >> > > > Infra. Bot simply is a means for triggering CI (which could be
> done
> >> > > > manually by Log In to Jenkins -> PR -> Job -> Build) and doesn't
> >> quite
> >> > > > tweak with merging procedure. Yes, only addition is now unknown
> >> person
> >> > > (PR
> >> > > > Author) can trigger CI with a message (but that was possible
> anyway
> >> by
> >> > > > pushing a commit. Bot just prevents users from pushing empty
> commits
> >> > and
> >> > > > building entire suite).
> >> > > >
> >> > > > As can be seen from last 10 open PRs as of Monday 23rd March, 12pm
> >> PT
> >> > > most
> >> > > > PRs fail on 1/2 jobs. In such a scenario, the proposed MXNet bot
> >> would
> >> > > come
> >> > > > in handy for just invoking CI on that specific build (instead of a
> >> > > > non-committer PR Author to push empty commit : hurting on the
> >> resource,
> >> > > > time & cost considerations apart from undesirable dev experience)
> >> > > >
> >> > > > @Marco Since I am a non-committer, I guess these 2 clarifications
> >> need
> >> > to
> >> > > > be conveyed to the Apache Infra by someone with Committer access.
> >> > > >
> >> > > > What do you think?
> >> > > >
> >> > > > Thanks,
> >> > > > Chai
> >> > > >
> >> > > > On Sat, 21 Mar 2020 at 16:08, Marco de Abreu <
> >> marco.g.abreu@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > > Hello,
> >> > > > >
> >> > > > > the ticket has been created:
> >> > > > > https://issues.apache.org/jira/browse/INFRA-20005
> >> > > > >
> >> > > > > Best regards,
> >> > > > > Marco
> >> > > > >
> >> > > > > On Thu, Mar 19, 2020 at 11:49 PM Marco de Abreu <
> >> > > marco.g.abreu@gmail.com
> >> > > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Sounds like a good plan!
> >> > > > > >
> >> > > > > > Please send me the URL (please make sure it's backed by DNS
> and
> >> not
> >> > > > just
> >> > > > > > the gateway URL) of the webhook handler, GitHub events you're
> >> > > > interested
> >> > > > > in
> >> > > > > > and the shared secret in a private email to my personal email
> >> > > address.
> >> > > > I
> >> > > > > > will then create the ticket with Apache infra.
> >> > > > > >
> >> > > > > > -Marco
> >> > > > > >
> >> > > > > > Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 19.
> März
> >> > > 2020,
> >> > > > > > 23:07:
> >> > > > > >
> >> > > > > >> @Marco Alright, it makes total sense to test out the Bot
> >> feature
> >> > > > > alongside
> >> > > > > >> auto-trigger as a transition.
> >> > > > > >>
> >> > > > > >> Path Forward:
> >> > > > > >> 1. Setup MXNet Bot on apache/incubator-mxnet repo (GitHub
> >> WebHook
> >> > > and
> >> > > > > >> Infra)
> >> > > > > >> 2. We don't turn off automatic trigger of PR builds for now.
> >> > > > > >> 3. Hopefully, bot is used by developers to trigger specific
> >> jobs
> >> > > > > >> 4. Later on (say around April 20), let's discuss the
> >> possibility
> >> > of
> >> > > > > >> switching off auto-trigger (with appropriate data) if it
> makes
> >> > > sense.
> >> > > > > >> Thanks Marco for volunteering to help enable the web hook on
> >> > > > > >> apache/incubator-mxnet. Let me know if we can sync up on
> Slack
> >> > > channel
> >> > > > > to
> >> > > > > >> get the ball rolling.
> >> > > > > >>
> >> > > > > >> Thanks once again for the entire community to step in and
> help
> >> try
> >> > > out
> >> > > > > >> this
> >> > > > > >> Bot.
> >> > > > > >> Chai
> >> > > > > >>
> >> > > > > >> On Wed, 18 Mar 2020 at 17:07, Marco de Abreu <
> >> > > marco.g.abreu@gmail.com
> >> > > > >
> >> > > > > >> wrote:
> >> > > > > >>
> >> > > > > >> > Hi, that's correct. But as stated previously, it's not an
> >> option
> >> > > to
> >> > > > > >> remove
> >> > > > > >> > the hook. For now, I'd like to see how the system behaves
> >> while
> >> > > it's
> >> > > > > >> > optional. Later on, we can talk about revisiting this
> >> decision.
> >> > > But
> >> > > > to
> >> > > > > >> me
> >> > > > > >> > it's not an option to deploy an entirely new system and
> >> approach
> >> > > > > without
> >> > > > > >> > having a transition or even a timeframe in which we are
> able
> >> to
> >> > > fall
> >> > > > > >> back.
> >> > > > > >> >
> >> > > > > >> > I'm happy to support the deployment of the bot and add an
> >> > > additional
> >> > > > > >> > webhook to enable it's functionality to support selective
> >> > > triggering
> >> > > > > by
> >> > > > > >> PR
> >> > > > > >> > authors and committers, but I will not support the
> disabling
> >> of
> >> > > > > >> automatic
> >> > > > > >> > triggering of branches or PRs.
> >> > > > > >> >
> >> > > > > >> > -Marco
> >> > > > > >> >
> >> > > > > >> > Chaitanya Bapat <ch...@gmail.com> schrieb am Mi., 18.
> >> März
> >> > > > 2020,
> >> > > > > >> > 21:00:
> >> > > > > >> >
> >> > > > > >> > > Hey Marco,
> >> > > > > >> > >
> >> > > > > >> > > I thought currently every commit on PR and master
> triggers
> >> CI
> >> > > > > >> > > because
> >> > > > > >> > > a. github webhook points to Jenkins Server
> >> > > > > >> > > b. GH Webhook events trigger builds on Jenkins for all
> >> commits
> >> > > to
> >> > > > > any
> >> > > > > >> > > branch in apache/incubator-mxnet
> >> > > > > >> > > may it be master/PR/non-PR
> >> > > > > >> > > Reason:
> >> > > > > >> > > Because all the 3 types of branches are discovered by
> >> Jenkins
> >> > > > > (non-PR
> >> > > > > >> > > (including master) and PR)
> >> > > > > >> > >
> >> > > > > >> > > Proposal: Remove GitHub WebHook to Jenkins and replace
> >> with GH
> >> > > > > >> Webhook to
> >> > > > > >> > > Lambda
> >> > > > > >> > > But after I remove the github webhook that points to
> >> Jenkins :
> >> > > > *N**o
> >> > > > > >> > commit
> >> > > > > >> > > will trigger Jenkins build by default* (as Jenkins wont
> >> > receive
> >> > > GH
> >> > > > > >> > events)
> >> > > > > >> > > Only those that Bot deems fit will be triggered (using
> >> Jenkins
> >> > > API
> >> > > > > >> > invoked
> >> > > > > >> > > by Lambda).
> >> > > > > >> > > Hence its needed to handle that case of master merge.
> >> > > > > >> > > Am I understanding this correctly?
> >> > > > > >> > >
> >> > > > > >> > > On Wed, 18 Mar 2020 at 04:23, Marco de Abreu <
> >> > > > > marco.g.abreu@gmail.com
> >> > > > > >> >
> >> > > > > >> > > wrote:
> >> > > > > >> > >
> >> > > > > >> > > > Thanks Chai, sounds good to me. Could you elaborate a
> >> bit on
> >> > > the
> >> > > > > >> point
> >> > > > > >> > > > about triggering a CI run after the PR has been merged?
> >> We
> >> > > > already
> >> > > > > >> to
> >> > > > > >> > > that
> >> > > > > >> > > > automatically for the master, so what's the benefit to
> >> do it
> >> > > > > twice?
> >> > > > > >> > > >
> >> > > > > >> > > > -Marco
> >> > > > > >> > > >
> >> > > > > >> > > > Chaitanya Bapat <ch...@gmail.com> schrieb am Mi.,
> >> 18.
> >> > > März
> >> > > > > >> 2020,
> >> > > > > >> > > > 09:30:
> >> > > > > >> > > >
> >> > > > > >> > > > > Update:
> >> > > > > >> > > > >
> >> > > > > >> > > > > > we can ensure that all CI runs ran on the commit
> >> that
> >> > > will
> >> > > > be
> >> > > > > >> > merged
> >> > > > > >> > > > > @Sam Skalicky <sa...@gmail.com> Branch
> >> Protection
> >> > is
> >> > > > > added
> >> > > > > >> to
> >> > > > > >> > > > public
> >> > > > > >> > > > > MXNet repo. It ensures that for every PR to be
> merged,
> >> the
> >> > > CI
> >> > > > > >> passes.
> >> > > > > >> > > All
> >> > > > > >> > > > > the jobs selected "required" jobs will have to be
> green
> >> > for
> >> > > > the
> >> > > > > >> PR to
> >> > > > > >> > > be
> >> > > > > >> > > > > merged. Ofcourse, users with "Adminstrator" access
> can
> >> > merge
> >> > > > > >> without
> >> > > > > >> > it
> >> > > > > >> > > > but
> >> > > > > >> > > > > that's just a backdoor. It is the case now and will
> >> > continue
> >> > > > to
> >> > > > > be
> >> > > > > >> > the
> >> > > > > >> > > > case
> >> > > > > >> > > > > with the inclusion of Bot.
> >> > > > > >> > > > >
> >> > > > > >> > > > > > easily verify that the CI has executed all runs on
> >> the
> >> > > > commit
> >> > > > > >> that
> >> > > > > >> > > will
> >> > > > > >> > > > > be merged
> >> > > > > >> > > > > GitHub UI shows all the jobs and the status
> >> corresponding
> >> > to
> >> > > > it
> >> > > > > on
> >> > > > > >> > > every
> >> > > > > >> > > > > commit. That should suffice. For the merged commits,
> >> Repo
> >> > ->
> >> > > > > >> Commits
> >> > > > > >> > ->
> >> > > > > >> > > > > Commit ID (Status) can be tracked currently (only way
> >> > that I
> >> > > > > know
> >> > > > > >> > of).
> >> > > > > >> > > > > Moreover, it is beyond the scope of this project (and
> >> > > possibly
> >> > > > > >> out of
> >> > > > > >> > > our
> >> > > > > >> > > > > control since this is purely GitHub UI specific
> >> use-case).
> >> > > > > >> > > > >
> >> > > > > >> > > > > Thanks @przemyslaw for supporting the opt-in.
> >> > > > > >> > > > >
> >> > > > > >> > > > > Thanks everyone in the community for sharing
> concerns,
> >> > > voicing
> >> > > > > >> your
> >> > > > > >> > > > opinion
> >> > > > > >> > > > > and participating in the discussion.
> >> > > > > >> > > > > Thanks to those who attended the demo last Friday.
> >> > > > > >> > > > >
> >> > > > > >> > > > > Action items from that discussion
> >> > > > > >> > > > > 1. Handle master merge builds [Done]
> >> > > > > >> > > > > Bot runs entire CI suite after the PR is merged and
> >> > comments
> >> > > > on
> >> > > > > >> the
> >> > > > > >> > PR
> >> > > > > >> > > > > about the same.
> >> > > > > >> > > > > Design decision :
> >> > > > > >> > > > > MXNet Bot comment about master merge build on the
> >> *merge
> >> > > > commit
> >> > > > > vs
> >> > > > > >> > PR*.
> >> > > > > >> > > > > After the PR is merged, Bot runs entire CI and
> comments
> >> > the
> >> > > > > >> result of
> >> > > > > >> > > CI
> >> > > > > >> > > > > trigger on the PR (because it is easy to track on a
> PR
> >> > > rather
> >> > > > > than
> >> > > > > >> > > > > commenting inside the merge commit)
> >> > > > > >> > > > >
> >> > > > > >> > > > > 2. Idempotent condition
> >> > > > > >> > > > > In case of already running build, if an attempt is
> >> made to
> >> > > > > >> retrigger
> >> > > > > >> > > the
> >> > > > > >> > > > > job then what should be the response
> >> > > > > >> > > > > a. Not to re-trigger, let the ongoing build continue
> >> till
> >> > > > > >> completion
> >> > > > > >> > > > > b. End the ongoing build and re-trigger
> >> > > > > >> > > > > c. Let the ongoing build continue, re-trigger new
> build
> >> > > > > >> > > > >
> >> > > > > >> > > > > From resource saving point of view, *c* looks costly
> >> and a
> >> > > can
> >> > > > > be
> >> > > > > >> > > > > avoided/optimized by B.
> >> > > > > >> > > > > In case when a re-trigger was started "erroneously"
> >> then
> >> > > > killing
> >> > > > > >> > > ongoing
> >> > > > > >> > > > > build and re-trigger is a waste.
> >> > > > > >> > > > > In case when ongoing build failed in one sub-part,
> then
> >> > > > > >> re-triggering
> >> > > > > >> > > is
> >> > > > > >> > > > > justified.
> >> > > > > >> > > > > Erroneous re-triggers would be less often while
> >> conscious
> >> > > > > >> re-triggers
> >> > > > > >> > > to
> >> > > > > >> > > > > suppress failure is more common use-case. It looks
> >> like a
> >> > > safe
> >> > > > > >> > > assumption
> >> > > > > >> > > > > to make given the trade-off.
> >> > > > > >> > > > > [Open to debate]
> >> > > > > >> > > > >
> >> > > > > >> > > > > 3. Add security consideration [Use of secret manager,
> >> but
> >> > > > > without
> >> > > > > >> > > > > auto-rotation due to Jenkins manual config
> requirement]
> >> > > [Done]
> >> > > > > >> > > > > 4. New PR Instruction message by the Bot [Done]
> >> > > > > >> > > > > Thanks to the suggestion of Leonard, supported by
> >> others.
> >> > > I've
> >> > > > > now
> >> > > > > >> > > added
> >> > > > > >> > > > > the feature where the Bot comments a help message.
> [For
> >> > > > > reference
> >> > > > > >> -
> >> > > > > >> > > > >
> https://github.com/ChaiBapchya/incubator-mxnet/pull/52
> >> ]
> >> > > > > >> > > > >
> >> > > > > >> > > > > Barring the opt-in vs opt-out debate & idempotency,
> >> > > consensus
> >> > > > > was
> >> > > > > >> > > quickly
> >> > > > > >> > > > > reached for the rest.
> >> > > > > >> > > > >
> >> > > > > >> > > > > In the coming days, I hope to roll-out this feature
> >> into
> >> > > Prod
> >> > > > > >> (public
> >> > > > > >> > > > > MXNet) for all devs to use.
> >> > > > > >> > > > >
> >> > > > > >> > > > > Thanks,
> >> > > > > >> > > > > Chai
> >> > > > > >> > > > >
> >> > > > > >> > > > > On Mon, 16 Mar 2020 at 11:57, Marco de Abreu <
> >> > > > > >> > marco.g.abreu@gmail.com>
> >> > > > > >> > > > > wrote:
> >> > > > > >> > > > >
> >> > > > > >> > > > > > Well that's generally a problem with a deferred CI
> >> > > approach
> >> > > > > (CI
> >> > > > > >> is
> >> > > > > >> > > run
> >> > > > > >> > > > at
> >> > > > > >> > > > > > commit and not at merge time). This can either be
> >> solved
> >> > > > > through
> >> > > > > >> > the
> >> > > > > >> > > > > other
> >> > > > > >> > > > > > proposal that's currently on dev@, by having a bot
> >> > which
> >> > > > does
> >> > > > > >> > merges
> >> > > > > >> > > > by
> >> > > > > >> > > > > > having a global lock and a merge queue or by
> >> accepting
> >> > the
> >> > > > > >> issue.
> >> > > > > >> > > > Reality
> >> > > > > >> > > > > > right now is that we're running that model where
> two
> >> PRs
> >> > > > which
> >> > > > > >> are
> >> > > > > >> > > > merged
> >> > > > > >> > > > > > in parallel might break one another. One thing to
> >> > consider
> >> > > > > >> though
> >> > > > > >> > is
> >> > > > > >> > > > that
> >> > > > > >> > > > > > this breakage would have to be introduced in two
> >> > separate
> >> > > > > parts
> >> > > > > >> > since
> >> > > > > >> > > > > > otherwise there'd be merge conflicts. I think we
> had
> >> > that
> >> > > > > >> situation
> >> > > > > >> > > > twice
> >> > > > > >> > > > > > so far and the result was a quick revert, so I'd
> say
> >> > that
> >> > > > > it's a
> >> > > > > >> > > > problem
> >> > > > > >> > > > > > that can happily be accepted. All other solutions
> >> > > basically
> >> > > > > >> require
> >> > > > > >> > > > some
> >> > > > > >> > > > > > form of single-threaded and globally locked
> solution
> >> > which
> >> > > > > >> limits
> >> > > > > >> > us
> >> > > > > >> > > in
> >> > > > > >> > > > > > scalability. I'd recommend to just accept that risk
> >> and
> >> > > > revert
> >> > > > > >> a PR
> >> > > > > >> > > in
> >> > > > > >> > > > > case
> >> > > > > >> > > > > > it actually had a conflict.
> >> > > > > >> > > > > >
> >> > > > > >> > > > > > -Marco
> >> > > > > >> > > > > >
> >> > > > > >> > > > > > On Mon, Mar 16, 2020 at 6:29 PM Skalicky, Sam
> >> > > > > >> > > > <sskalic@amazon.com.invalid
> >> > > > > >> > > > > >
> >> > > > > >> > > > > > wrote:
> >> > > > > >> > > > > >
> >> > > > > >> > > > > > > We probably need some way to track which CI runs
> >> ran
> >> > for
> >> > > > > which
> >> > > > > >> > > commit
> >> > > > > >> > > > > > too,
> >> > > > > >> > > > > > > that way we can ensure that all CI runs ran on
> the
> >> > > commit
> >> > > > > that
> >> > > > > >> > will
> >> > > > > >> > > > be
> >> > > > > >> > > > > > > merged. Maybe the bot can comment with the
> commit
> >> > hash
> >> > > > when
> >> > > > > >> > users
> >> > > > > >> > > > > > command
> >> > > > > >> > > > > > > it to do something. Although since users can
> >> trigger
> >> > > > > >> individual
> >> > > > > >> > CI
> >> > > > > >> > > > runs
> >> > > > > >> > > > > > its
> >> > > > > >> > > > > > > possible to have some commits run some CI runs
> but
> >> not
> >> > > > > >> others. We
> >> > > > > >> > > > need
> >> > > > > >> > > > > > some
> >> > > > > >> > > > > > > way to easily verify that the CI has executed all
> >> runs
> >> > > on
> >> > > > > the
> >> > > > > >> > > commit
> >> > > > > >> > > > > that
> >> > > > > >> > > > > > > will be merged.
> >> > > > > >> > > > > > >
> >> > > > > >> > > > > > > Sam
> >> > > > > >> > > > > > >
> >> > > > > >> > > > > > > > On Mar 13, 2020, at 8:28 PM, Przemysław Trędak
> <
> >> > > > > >> > > ptrendx@apache.org
> >> > > > > >> > > > >
> >> > > > > >> > > > > > > wrote:
> >> > > > > >> > > > > > > >
> >> > > > > >> > > > > > > > CAUTION: This email originated from outside of
> >> the
> >> > > > > >> > organization.
> >> > > > > >> > > Do
> >> > > > > >> > > > > not
> >> > > > > >> > > > > > > click links or open attachments unless you can
> >> confirm
> >> > > the
> >> > > > > >> sender
> >> > > > > >> > > and
> >> > > > > >> > > > > > know
> >> > > > > >> > > > > > > the content is safe.
> >> > > > > >> > > > > > > >
> >> > > > > >> > > > > > > >
> >> > > > > >> > > > > > > >
> >> > > > > >> > > > > > > > I personally like the idea of opt-in more than
> >> > > opt-out:
> >> > > > > >> > > > > > > > - ultimately PR author wants the PR to be
> merged
> >> so
> >> > > they
> >> > > > > (or
> >> > > > > >> > > > > committer
> >> > > > > >> > > > > > > reviewing the PR) will trigger the CI
> >> > > > > >> > > > > > > > - if it is easy to trigger the PR via the bot
> >> > command
> >> > > > then
> >> > > > > >> the
> >> > > > > >> > > > amount
> >> > > > > >> > > > > > of
> >> > > > > >> > > > > > > work per PR should be less than with opt-out
> (since
> >> > most
> >> > > > of
> >> > > > > >> the
> >> > > > > >> > > > commits
> >> > > > > >> > > > > > > should then be marked as [skip ci] or something
> >> > similar
> >> > > > > >> > > > > > > >
> >> > > > > >> > > > > > > > +1 to the bot making a comment on each new PR
> >> with
> >> > its
> >> > > > > >> commands
> >> > > > > >> > > > (and
> >> > > > > >> > > > > > > also explaining, or at least giving links to the
> >> > general
> >> > > > PR
> >> > > > > >> > process
> >> > > > > >> > > > so
> >> > > > > >> > > > > > new
> >> > > > > >> > > > > > > PR authors are not lost). Maybe we could make the
> >> bot
> >> > > > > >> recognize
> >> > > > > >> > if
> >> > > > > >> > > > the
> >> > > > > >> > > > > PR
> >> > > > > >> > > > > > > author is new or existing contributor and offer
> >> advice
> >> > > > based
> >> > > > > >> on
> >> > > > > >> > > that?
> >> > > > > >> > > > > > > >
> >> > > > > >> > > > > > > > Thanks
> >> > > > > >> > > > > > > > Przemek
> >> > > > > >> > > > > > > >
> >> > > > > >> > > > > > > > On 2020/03/13 22:06:58, Marco de Abreu <
> >> > > > > >> > marco.g.abreu@gmail.com>
> >> > > > > >> > > > > > wrote:
> >> > > > > >> > > > > > > >> Hi,
> >> > > > > >> > > > > > > >>
> >> > > > > >> > > > > > > >> since it's no longer necessary to push a new
> >> commit
> >> > > to
> >> > > > > >> trigger
> >> > > > > >> > > CI,
> >> > > > > >> > > > > it
> >> > > > > >> > > > > > > will
> >> > > > > >> > > > > > > >> already reduce the costs. But to me, requiring
> >> an
> >> > > > action
> >> > > > > to
> >> > > > > >> > > enable
> >> > > > > >> > > > > CI
> >> > > > > >> > > > > > > after
> >> > > > > >> > > > > > > >> a PR has been created initially, is a no go.
> >> User
> >> > can
> >> > > > opt
> >> > > > > >> out
> >> > > > > >> > of
> >> > > > > >> > > > CI,
> >> > > > > >> > > > > > but
> >> > > > > >> > > > > > > >> the default has to be CI being triggered
> >> > > automatically
> >> > > > > for
> >> > > > > >> > every
> >> > > > > >> > > > > > commit
> >> > > > > >> > > > > > > >> unless specifically disabled by a participant.
> >> I'm
> >> > > also
> >> > > > > >> fine
> >> > > > > >> > > with
> >> > > > > >> > > > > > > >> triggering certain additional jobs (think
> about
> >> > > > running a
> >> > > > > >> > > nightly
> >> > > > > >> > > > > job
> >> > > > > >> > > > > > > upon
> >> > > > > >> > > > > > > >> request for a PR) to require a manual step,
> but
> >> the
> >> > > PR
> >> > > > > >> > > validation
> >> > > > > >> > > > > > > pipelines
> >> > > > > >> > > > > > > >> have to run automatically. Every check that is
> >> > marked
> >> > > > as
> >> > > > > >> > > > "Required"
> >> > > > > >> > > > > in
> >> > > > > >> > > > > > > >> GitHub has to be automatically kicked off.
> >> > > > > >> > > > > > > >>
> >> > > > > >> > > > > > > >> -Marco
> >> > > > > >> > > > > > > >>
> >> > > > > >> > > > > > > >> On Fri, Mar 13, 2020 at 9:50 PM Chaitanya
> Bapat
> >> <
> >> > > > > >> > > > > chai.bapat@gmail.com
> >> > > > > >> > > > > > >
> >> > > > > >> > > > > > > >> wrote:
> >> > > > > >> > > > > > > >>
> >> > > > > >> > > > > > > >>> Firstly,
> >> > > > > >> > > > > > > >>> Sorry I missed out on attaching the mail
> thread
> >> > that
> >> > > > was
> >> > > > > >> sent
> >> > > > > >> > > on
> >> > > > > >> > > > > 12th
> >> > > > > >> > > > > > > >>> February for notifying the community of the
> >> > upcoming
> >> > > > > >> changes
> >> > > > > >> > to
> >> > > > > >> > > > the
> >> > > > > >> > > > > > > MXNet
> >> > > > > >> > > > > > > >>> CI
> >> > > > > >> > > > > > > >>> For reference :
> >> > > > > >> > > > > > > >>>
> >> > > > > >> > > > > > > >>>
> >> > > > > >> > > > > > >
> >> > > > > >> > > > > >
> >> > > > > >> > > > >
> >> > > > > >> > > >
> >> > > > > >> > >
> >> > > > > >> >
> >> > > > > >>
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://lists.apache.org/thread.html/r09a6ab2803a996fc80e00fe39ed312fa4865e8805e08df847f1addad%40%3Cdev.mxnet.apache.org%3E
> >> > > > > >> > > > > > > >>>
> >> > > > > >> > > > > > > >>> Now to the questions,
> >> > > > > >> > > > > > > >>>> Is it possible for re-triggering a single
> job
> >> to
> >> > be
> >> > > > > >> abused?
> >> > > > > >> > > > > > > >>> @Tao In the case when a user re-triggers a
> >> single
> >> > > job
> >> > > > > >> > multiple
> >> > > > > >> > > > > times,
> >> > > > > >> > > > > > > that
> >> > > > > >> > > > > > > >>> will be visible in the PR conversation
> thread.
> >> A
> >> > > > > >> committer,
> >> > > > > >> > > even
> >> > > > > >> > > > > > after
> >> > > > > >> > > > > > > he
> >> > > > > >> > > > > > > >>> has approved the PR before, generally takes a
> >> look
> >> > > at
> >> > > > > the
> >> > > > > >> > final
> >> > > > > >> > > > > state
> >> > > > > >> > > > > > > of
> >> > > > > >> > > > > > > >>> the PR before merging. Would it be fair to
> >> assume
> >> > > the
> >> > > > > >> > committer
> >> > > > > >> > > > > could
> >> > > > > >> > > > > > > take
> >> > > > > >> > > > > > > >>> the multiple re-trigger of a single job into
> >> > account
> >> > > > > >> before
> >> > > > > >> > > > > merging?
> >> > > > > >> > > > > > > The
> >> > > > > >> > > > > > > >>> committer then has the option to invoke
> >> > `@mxnet-bot
> >> > > > run
> >> > > > > ci
> >> > > > > >> > > [all]
> >> > > > > >> > > > `
> >> > > > > >> > > > > to
> >> > > > > >> > > > > > > >>> trigger the entire build pipeline one last to
> >> > > counter
> >> > > > > the
> >> > > > > >> > > abuse.
> >> > > > > >> > > > > This
> >> > > > > >> > > > > > > is
> >> > > > > >> > > > > > > >>> aligned with what @Leonard said.
> >> > > > > >> > > > > > > >>>
> >> > > > > >> > > > > > > >>> @Sandeep Thanks a lot for collecting and
> >> sharing
> >> > > > > valuable
> >> > > > > >> > data.
> >> > > > > >> > > > I'd
> >> > > > > >> > > > > > > concur
> >> > > > > >> > > > > > > >>> with the opinion that given the existing
> things
> >> > > > > >> committers &
> >> > > > > >> > PR
> >> > > > > >> > > > > > Authors
> >> > > > > >> > > > > > > >>> take care of, invoking CI shouldn't be that
> >> big of
> >> > > an
> >> > > > > >> > > additional
> >> > > > > >> > > > > > > burden.
> >> > > > > >> > > > > > > >>>
> >> > > > > >> > > > > > > >>> @Marco With the opt-out, the onus remains on
> >> the
> >> > PR
> >> > > > > >> Author.
> >> > > > > >> > It
> >> > > > > >> > > > > > doesn't
> >> > > > > >> > > > > > > help
> >> > > > > >> > > > > > > >>> reduce the resource usage. Hence, it was
> >> suggested
> >> > > to
> >> > > > > >> switch
> >> > > > > >> > to
> >> > > > > >> > > > > > > >>> opt-in. @Leo's suggestion for proactive
> >> commenting
> >> > > on
> >> > > > > the
> >> > > > > >> > part
> >> > > > > >> > > of
> >> > > > > >> > > > > bot
> >> > > > > >> > > > > > > makes
> >> > > > > >> > > > > > > >>> sense and is doable.
> >> > > > > >> > > > > > > >>>
> >> > > > > >> > > > > > > >>> Default : opt-out and User initiated opt-in
> >> (with
> >> > > > > >> addressing
> >> > > > > >> > > > Leo's
> >> > > > > >> > > > > > fix
> >> > > > > >> > > > > > > for
> >> > > > > >> > > > > > > >>> the usability issue you correctly pointed
> out )
> >> > > > > >> > > > > > > >>> @Marco How does this sound to you?
> >> > > > > >> > > > > > > >>>
> >> > > > > >> > > > > > > >>> Again, thank you all for chiming in and
> voicing
> >> > your
> >> > > > > >> opinion.
> >> > > > > >> > > > > > > Appreciate
> >> > > > > >> > > > > > > >>> it.
> >> > > > > >> > > > > > > >>> We can take ahead these discussions in
> today's
> >> > demo
> >> > > > > >> meeting.
> >> > > > > >> > > > > [Design
> >> > > > > >> > > > > > > Doc
> >> > > > > >> > > > > > > >>> <
> >> > > > > >> > >
> >> > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> >> > > > > >> > > > >]
> >> > > > > >> > > > > > > [Demo
> >> > > > > >> > > > > > > >>> Video <
> >> > https://www.youtube.com/watch?v=gfOGwZId8aU
> >> > > >]
> >> > > > > >> > > > > > > >>>
> >> > > > > >> > > > > > > >>> Thanks,
> >> > > > > >> > > > > > > >>> Chai
> >> > > > > >> > > > > > > >>>
> >> > > > > >> > > > > > > >>> On Fri, 13 Mar 2020 at 12:34, Marco de Abreu
> <
> >> > > > > >> > > > > > marco.g.abreu@gmail.com>
> >> > > > > >> > > > > > > >>> wrote:
> >> > > > > >> > > > > > > >>>
> >> > > > > >> > > > > > > >>>> I'd recommend that the bot makes an initial
> >> > comment
> >> > > > > when
> >> > > > > >> a
> >> > > > > >> > PR
> >> > > > > >> > > > gets
> >> > > > > >> > > > > > > opened
> >> > > > > >> > > > > > > >>>> and informs the users of its commands. It
> then
> >> > > tells
> >> > > > > the
> >> > > > > >> > user
> >> > > > > >> > > > the
> >> > > > > >> > > > > > > commend
> >> > > > > >> > > > > > > >>>> to opt out of CI.
> >> > > > > >> > > > > > > >>>>
> >> > > > > >> > > > > > > >>>> -Marco
> >> > > > > >> > > > > > > >>>>
> >> > > > > >> > > > > > > >>>> Lausen, Leonard <la...@amazon.com.invalid>
> >> > > schrieb
> >> > > > am
> >> > > > > >> Fr.,
> >> > > > > >> > > 13.
> >> > > > > >> > > > > > März
> >> > > > > >> > > > > > > >>> 2020,
> >> > > > > >> > > > > > > >>>> 20:27:
> >> > > > > >> > > > > > > >>>>
> >> > > > > >> > > > > > > >>>>> On opt-out: People may be unaware of
> opt-out
> >> > would
> >> > > > not
> >> > > > > >> use
> >> > > > > >> > > it.
> >> > > > > >> > > > > > There
> >> > > > > >> > > > > > > is
> >> > > > > >> > > > > > > >>>> no
> >> > > > > >> > > > > > > >>>>> incentive to use opt-out, as the PR author
> >> > doesn't
> >> > > > pay
> >> > > > > >> any
> >> > > > > >> > > > money
> >> > > > > >> > > > > > for
> >> > > > > >> > > > > > > CI
> >> > > > > >> > > > > > > >>>>> run.
> >> > > > > >> > > > > > > >>>>>
> >> > > > > >> > > > > > > >>>>> I agree with Marco though that opt-in alone
> >> may
> >> > > > cause
> >> > > > > >> > > usability
> >> > > > > >> > > > > > > issues,
> >> > > > > >> > > > > > > >>>> as
> >> > > > > >> > > > > > > >>>>> contributors may not be aware of how to
> >> trigger
> >> > > the
> >> > > > > CI.
> >> > > > > >> > > > > > > >>>>> One solution is that the bot proactively
> >> > comments
> >> > > on
> >> > > > > >> the PR
> >> > > > > >> > > and
> >> > > > > >> > > > > > > reminds
> >> > > > > >> > > > > > > >>>> the
> >> > > > > >> > > > > > > >>>>> author to trigger running CI once the
> author
> >> > deems
> >> > > > the
> >> > > > > >> PR
> >> > > > > >> > > > ready.
> >> > > > > >> > > > > > > >>>>>
> >> > > > > >> > > > > > > >>>>> But even if we choose opt-out, the bot will
> >> > still
> >> > > > add
> >> > > > > a
> >> > > > > >> lot
> >> > > > > >> > > of
> >> > > > > >> > > > > > value,
> >> > > > > >> > > > > > > >>> as
> >> > > > > >> > > > > > > >>>> PR
> >> > > > > >> > > > > > > >>>>> authors can retrigger single jobs that have
> >> > failed
> >> > > > due
> >> > > > > >> to
> >> > > > > >> > > > > > flakiness.
> >> > > > > >> > > > > > > >>>>>
> >> > > > > >> > > > > > > >>>>>> Is it possible for re-triggering a single
> >> job
> >> > to
> >> > > be
> >> > > > > >> > abused?
> >> > > > > >> > > > For
> >> > > > > >> > > > > > > >>>> example,
> >> > > > > >> > > > > > > >>>>>> the author spends two days re-triggering a
> >> > flaky
> >> > > > job
> >> > > > > to
> >> > > > > >> > make
> >> > > > > >> > > > it
> >> > > > > >> > > > > > > pass.
> >> > > > > >> > > > > > > >>>>>
> >> > > > > >> > > > > > > >>>>> Yes, this is possible. I suggest the
> >> committer
> >> > who
> >> > > > > >> likes to
> >> > > > > >> > > > > merge a
> >> > > > > >> > > > > > > PR
> >> > > > > >> > > > > > > >>>>> needs to
> >> > > > > >> > > > > > > >>>>> make a good judgement here if a PR is
> abusing
> >> > the
> >> > > > > >> feature,
> >> > > > > >> > > and
> >> > > > > >> > > > if
> >> > > > > >> > > > > > so,
> >> > > > > >> > > > > > > >>>>> retrigger
> >> > > > > >> > > > > > > >>>>> all CI runs.
> >> > > > > >> > > > > > > >>>>>
> >> > > > > >> > > > > > > >>>>> Best regards
> >> > > > > >> > > > > > > >>>>> Leonard
> >> > > > > >> > > > > > > >>>>>
> >> > > > > >> > > > > > > >>>>> On Fri, 2020-03-13 at 08:07 +0100, Marco de
> >> > Abreu
> >> > > > > wrote:
> >> > > > > >> > > > > > > >>>>>> Thanks for the data Sandeep. In these
> cases
> >> it
> >> > > > sounds
> >> > > > > >> like
> >> > > > > >> > > it
> >> > > > > >> > > > > > would
> >> > > > > >> > > > > > > >>>> have
> >> > > > > >> > > > > > > >>>>>> rather been better when people explicitly
> >> > turned
> >> > > > off
> >> > > > > >> CI in
> >> > > > > >> > > > that
> >> > > > > >> > > > > > > case.
> >> > > > > >> > > > > > > >>>>>> What's the argument against an opt-out
> >> instead
> >> > of
> >> > > > an
> >> > > > > >> > opt-in?
> >> > > > > >> > > > > > > >>>>>>
> >> > > > > >> > > > > > > >>>>>> My intention is that I consider it quite
> >> > > cumbersome
> >> > > > > to
> >> > > > > >> > make
> >> > > > > >> > > > it a
> >> > > > > >> > > > > > > >>>>> *required*
> >> > > > > >> > > > > > > >>>>>> step to always trigger CI manually, even
> if
> >> > just
> >> > > > > >> > submitting
> >> > > > > >> > > a
> >> > > > > >> > > > > > small
> >> > > > > >> > > > > > > >>> PR.
> >> > > > > >> > > > > > > >>>>> I'd
> >> > > > > >> > > > > > > >>>>>> rather see people explicitly turning off
> CI
> >> if
> >> > > they
> >> > > > > >> > wouldn't
> >> > > > > >> > > > > like
> >> > > > > >> > > > > > to
> >> > > > > >> > > > > > > >>>> use
> >> > > > > >> > > > > > > >>>>> it
> >> > > > > >> > > > > > > >>>>>> - and there's also the "draft" stage for a
> >> PR
> >> > > which
> >> > > > > >> some
> >> > > > > >> > > > > > > contributors
> >> > > > > >> > > > > > > >>>> are
> >> > > > > >> > > > > > > >>>>>> using.
> >> > > > > >> > > > > > > >>>>>>
> >> > > > > >> > > > > > > >>>>>> With regards to WIP and do not review: I
> >> think
> >> > > > these
> >> > > > > >> are
> >> > > > > >> > use
> >> > > > > >> > > > > cases
> >> > > > > >> > > > > > > >>>> where
> >> > > > > >> > > > > > > >>>>>> you want CI feedback, as otherwise you
> >> wouldn't
> >> > > > have
> >> > > > > >> > opened
> >> > > > > >> > > > the
> >> > > > > >> > > > > > PR.
> >> > > > > >> > > > > > > >>> If
> >> > > > > >> > > > > > > >>>>> you
> >> > > > > >> > > > > > > >>>>>> don't want human feedback and neither
> >> machine
> >> > > > > feedback,
> >> > > > > >> > why
> >> > > > > >> > > > open
> >> > > > > >> > > > > > the
> >> > > > > >> > > > > > > >>> PR
> >> > > > > >> > > > > > > >>>>> at
> >> > > > > >> > > > > > > >>>>>> all?
> >> > > > > >> > > > > > > >>>>>>
> >> > > > > >> > > > > > > >>>>>> -Marco
> >> > > > > >> > > > > > > >>>>>>
> >> > > > > >> > > > > > > >>>>>>
> >> > > > > >> > > > > > > >>>>>> sandeep krishnamurthy <
> >> > > sandeep.krishna98@gmail.com
> >> > > > >
> >> > > > > >> > schrieb
> >> > > > > >> > > > am
> >> > > > > >> > > > > > Fr.,
> >> > > > > >> > > > > > > >>>> 13.
> >> > > > > >> > > > > > > >>>>>> März 2020, 05:24:
> >> > > > > >> > > > > > > >>>>>>
> >> > > > > >> > > > > > > >>>>>>> I tried to gather some data for us to
> >> discuss
> >> > > this
> >> > > > > >> topic
> >> > > > > >> > in
> >> > > > > >> > > > > this
> >> > > > > >> > > > > > > >>>>> thread. I
> >> > > > > >> > > > > > > >>>>>>> tried to count number of un-necessary
> >> builds
> >> > by
> >> > > > > >> looking
> >> > > > > >> > at
> >> > > > > >> > > > most
> >> > > > > >> > > > > > > >>>> recent
> >> > > > > >> > > > > > > >>>>> (as
> >> > > > > >> > > > > > > >>>>>>> of 12, March 9 PM PST) 50 PRs merged to
> >> master
> >> > > and
> >> > > > > 50
> >> > > > > >> > PRs.
> >> > > > > >> > > > > > > >>>>>>> Identifying un-necessary builds is bit
> >> > > > subjective. I
> >> > > > > >> > tried
> >> > > > > >> > > to
> >> > > > > >> > > > > be
> >> > > > > >> > > > > > > >>> more
> >> > > > > >> > > > > > > >>>>>>> conservative where I didn't count a build
> >> as
> >> > > > > >> un-necessary
> >> > > > > >> > > if
> >> > > > > >> > > > I
> >> > > > > >> > > > > > was
> >> > > > > >> > > > > > > >>> in
> >> > > > > >> > > > > > > >>>>>>> doubt. Hence, I was not able to automate,
> >> but
> >> > I
> >> > > > made
> >> > > > > >> an
> >> > > > > >> > > > effort
> >> > > > > >> > > > > to
> >> > > > > >> > > > > > > >>> go
> >> > > > > >> > > > > > > >>>>>>> through PRs manually and use below
> >> criteria to
> >> > > > > >> identify
> >> > > > > >> > > > > > > >>> un-necessary
> >> > > > > >> > > > > > > >>>>>>> commits triggering the builds.
> >> > > > > >> > > > > > > >>>>>>>
> >> > > > > >> > > > > > > >>>>>>> 1. Explicitly marked as WIP / do not
> >> review
> >> > > PR
> >> > > > > >> > > > > > > >>>>>>> 2. Incremental WIP commit and finally
> >> > > > commenting a
> >> > > > > >> > commit
> >> > > > > >> > > > > > > >>> “trigger
> >> > > > > >> > > > > > > >>>>> CI”
> >> > > > > >> > > > > > > >>>>>>> 3. Multiple commits to address all
> >> comments
> >> > > from
> >> > > > > >> single
> >> > > > > >> > > > > review.
> >> > > > > >> > > > > > > >>>>> This is
> >> > > > > >> > > > > > > >>>>>>> assuming we see a comment, address
> them,
> >> > > commit,
> >> > > > > >> next
> >> > > > > >> > the
> >> > > > > >> > > > > > > >>>> following
> >> > > > > >> > > > > > > >>>>>>> comment
> >> > > > > >> > > > > > > >>>>>>> 4. Sequence of documentation only
> changes
> >> > > > > >> > > > > > > >>>>>>>
> >> > > > > >> > > > > > > >>>>>>> I found there were around 42 avoidable
> >> builds
> >> > > from
> >> > > > > >> most
> >> > > > > >> > > > recent
> >> > > > > >> > > > > 50
> >> > > > > >> > > > > > > >>>>> merged
> >> > > > > >> > > > > > > >>>>>>> PRs and around 86 builds from recent 50
> >> open
> >> > > PRs.
> >> > > > > >> > > > > > > >>>>>>>
> >> > > > > >> > > > > > > >>>>>>> I synced up with other contributors (Joe
> >> > Evans,
> >> > > > > Chai)
> >> > > > > >> > from
> >> > > > > >> > > > > Amazon
> >> > > > > >> > > > > > > >>> who
> >> > > > > >> > > > > > > >>>>> is
> >> > > > > >> > > > > > > >>>>>>> contributing to MXNet CI system. I was
> told
> >> > that
> >> > > > on
> >> > > > > an
> >> > > > > >> > > > average
> >> > > > > >> > > > > it
> >> > > > > >> > > > > > > >>>> costs
> >> > > > > >> > > > > > > >>>>>>> around $84 per build and on an average 6
> >> > commits
> >> > > > per
> >> > > > > >> > merged
> >> > > > > >> > > > PR
> >> > > > > >> > > > > > (for
> >> > > > > >> > > > > > > >>>>> year
> >> > > > > >> > > > > > > >>>>>>> 2019). Going by that, it is approximately
> >> 1/6
> >> > > > builds
> >> > > > > >> are
> >> > > > > >> > > > > > avoidable.
> >> > > > > >> > > > > > > >>>>> [100 /
> >> > > > > >> > > > > > > >>>>>>> 300 + 300 ]
> >> > > > > >> > > > > > > >>>>>>>
> >> > > > > >> > > > > > > >>>>>>>
> >> > > > > >> > > > > > > >>>>>>> Usability should be top most priority.
> But,
> >> > > since
> >> > > > > >> either
> >> > > > > >> > a
> >> > > > > >> > > > > > reviewer
> >> > > > > >> > > > > > > >>>> or
> >> > > > > >> > > > > > > >>>>> pr
> >> > > > > >> > > > > > > >>>>>>> author can trigger the bot, is it really
> a
> >> > > hurdle
> >> > > > > for
> >> > > > > >> pr
> >> > > > > >> > > > author
> >> > > > > >> > > > > > or
> >> > > > > >> > > > > > > >>>>> reviewer
> >> > > > > >> > > > > > > >>>>>>> to call a bot to trigger CI? Given that
> PR
> >> > > author
> >> > > > > and
> >> > > > > >> > > > reviewer
> >> > > > > >> > > > > is
> >> > > > > >> > > > > > > >>>>> already
> >> > > > > >> > > > > > > >>>>>>> actively commenting various details such
> >> as -
> >> > PR
> >> > > > > >> > > description,
> >> > > > > >> > > > > > > >>> review
> >> > > > > >> > > > > > > >>>>>>> comments and responses, adding labels
> etc.
> >> > > > > >> > > > > > > >>>>>>>
> >> > > > > >> > > > > > > >>>>>>>
> >> > > > > >> > > > > > > >>>>>>> Me too curious to know the behavior for
> >> Tao's
> >> > > > above
> >> > > > > >> use
> >> > > > > >> > > case.
> >> > > > > >> > > > > > > >>>>>>>
> >> > > > > >> > > > > > > >>>>>>>
> >> > > > > >> > > > > > > >>>>>>> Best,
> >> > > > > >> > > > > > > >>>>>>>
> >> > > > > >> > > > > > > >>>>>>> Sandeep
> >> > > > > >> > > > > > > >>>>>>>
> >> > > > > >> > > > > > > >>>>>>> On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <
> >> > > > > >> > mutouorz@gmail.com
> >> > > > > >> > > >
> >> > > > > >> > > > > > wrote:
> >> > > > > >> > > > > > > >>>>>>>
> >> > > > > >> > > > > > > >>>>>>>> Is it possible for re-triggering a
> single
> >> job
> >> > > to
> >> > > > be
> >> > > > > >> > > abused?
> >> > > > > >> > > > > For
> >> > > > > >> > > > > > > >>>>> example,
> >> > > > > >> > > > > > > >>>>>>>> the author spends two days
> re-triggering a
> >> > > flaky
> >> > > > > job
> >> > > > > >> to
> >> > > > > >> > > make
> >> > > > > >> > > > > it
> >> > > > > >> > > > > > > >>>>> pass. But
> >> > > > > >> > > > > > > >>>>>>>> other jobs which have passed the
> >> validation
> >> > may
> >> > > > be
> >> > > > > >> > broken
> >> > > > > >> > > by
> >> > > > > >> > > > > > > >>> other
> >> > > > > >> > > > > > > >>>>>>> commits
> >> > > > > >> > > > > > > >>>>>>>> during the two day without being
> noticed.
> >> And
> >> > > > > finally
> >> > > > > >> > the
> >> > > > > >> > > PR
> >> > > > > >> > > > > is
> >> > > > > >> > > > > > > >>>>> merged
> >> > > > > >> > > > > > > >>>>>>> with
> >> > > > > >> > > > > > > >>>>>>>> underlying problems.
> >> > > > > >> > > > > > > >>>>>>>>
> >> > > > > >> > > > > > > >>>>>>>> On Fri, Mar 13, 2020 at 6:19 AM Marco de
> >> > Abreu
> >> > > <
> >> > > > > >> > > > > > > >>>>> marco.g.abreu@gmail.com>
> >> > > > > >> > > > > > > >>>>>>>> wrote:
> >> > > > > >> > > > > > > >>>>>>>>
> >> > > > > >> > > > > > > >>>>>>>>> In the end it only comes down to money,
> >> > > > > considering
> >> > > > > >> > that
> >> > > > > >> > > > the
> >> > > > > >> > > > > > > >>>>> system is
> >> > > > > >> > > > > > > >>>>>>>> auto
> >> > > > > >> > > > > > > >>>>>>>>> scaling, making the execution time
> >> constant.
> >> > > > > >> > > > > > > >>>>>>>>>
> >> > > > > >> > > > > > > >>>>>>>>> If we're trading money for usability, I
> >> > > > certainly
> >> > > > > >> would
> >> > > > > >> > > > > prefer
> >> > > > > >> > > > > > > >>>>>>> usability.
> >> > > > > >> > > > > > > >>>>>>>>> I'd rather recommend to spend time on
> >> > > > > parallelizing
> >> > > > > >> > test
> >> > > > > >> > > > > > > >>>> execution
> >> > > > > >> > > > > > > >>>>> or
> >> > > > > >> > > > > > > >>>>>>>>> getting rid of integration tests in the
> >> PR
> >> > > stage
> >> > > > > >> > instead
> >> > > > > >> > > > > > > >>> reducing
> >> > > > > >> > > > > > > >>>>> the
> >> > > > > >> > > > > > > >>>>>>>> costs
> >> > > > > >> > > > > > > >>>>>>>>> by making people not use it. But
> taking a
> >> > step
> >> > > > > back
> >> > > > > >> to
> >> > > > > >> > > > > > > >>> requiring
> >> > > > > >> > > > > > > >>>>> people
> >> > > > > >> > > > > > > >>>>>>>> to
> >> > > > > >> > > > > > > >>>>>>>>> manually trigger CI again doesn't feel
> >> > right.
> >> > > > > >> > > > > > > >>>>>>>>>
> >> > > > > >> > > > > > > >>>>>>>>> I'm happy to see that bot deployed, but
> >> I do
> >> > > not
> >> > > > > >> agree
> >> > > > > >> > > with
> >> > > > > >> > > > > > > >>>>> removing
> >> > > > > >> > > > > > > >>>>>>> the
> >> > > > > >> > > > > > > >>>>>>>>> auto trigger functionality for new
> >> commits.
> >> > > > > >> > > > > > > >>>>>>>>>
> >> > > > > >> > > > > > > >>>>>>>>> -Marco
> >> > > > > >> > > > > > > >>>>>>>>>
> >> > > > > >> > > > > > > >>>>>>>>> Chaitanya Bapat <ch...@gmail.com>
> >> > > schrieb
> >> > > > am
> >> > > > > >> Do.,
> >> > > > > >> > > 12.
> >> > > > > >> > > > > > > >>> März
> >> > > > > >> > > > > > > >>>>> 2020,
> >> > > > > >> > > > > > > >>>>>>>>> 22:47:
> >> > > > > >> > > > > > > >>>>>>>>>
> >> > > > > >> > > > > > > >>>>>>>>>> @Marco Thanks for pointing that out.
> >> > > > > >> > > > > > > >>>>>>>>>> Tomorrow i.e. Friday, March 13, 2020
> at
> >> > 3:00
> >> > > > PM -
> >> > > > > >> 3:30
> >> > > > > >> > > PM
> >> > > > > >> > > > in
> >> > > > > >> > > > > > > >>>>>>>> (UTC-08:00)
> >> > > > > >> > > > > > > >>>>>>>>>> Pacific Time (US & Canada).
> >> > > > > >> > > > > > > >>>>>>>>>>
> >> > > > > >> > > > > > > >>>>>>>>>>> When do we expect this bot to be
> >> deployed?
> >> > > > > >> > > > > > > >>>>>>>>>> @Lin If all goes well in the next
> week I
> >> > can
> >> > > > > >> deploy it
> >> > > > > >> > > to
> >> > > > > >> > > > > > > >>>> public
> >> > > > > >> > > > > > > >>>>>>> Apache
> >> > > > > >> > > > > > > >>>>>>>>>> (provided I get permissions from
> Apache
> >> > > Infra)
> >> > > > > >> > > > > > > >>>>>>>>>>
> >> > > > > >> > > > > > > >>>>>>>>>> @Marco Thanks for your feedback.
> >> > > > > >> > > > > > > >>>>>>>>>>> CI system has to support the
> community
> >> > > without
> >> > > > > >> > > requiring
> >> > > > > >> > > > > > > >>>>> people to
> >> > > > > >> > > > > > > >>>>>>>>>> constantly shepherd every single run
> >> > > > > >> > > > > > > >>>>>>>>>> We have data for the number of times
> CI
> >> was
> >> > > > > >> triggered
> >> > > > > >> > > > > > > >>>>> unnecessarily
> >> > > > > >> > > > > > > >>>>>>>> which
> >> > > > > >> > > > > > > >>>>>>>>>> includes
> >> > > > > >> > > > > > > >>>>>>>>>> - Entire build triggered instead of
> >> > specific
> >> > > > > build
> >> > > > > >> > > > > > > >>>>>>>>>> - CI triggered when PR is still work
> in
> >> > > > progress
> >> > > > > or
> >> > > > > >> > not
> >> > > > > >> > > > yet
> >> > > > > >> > > > > > > >>>> ready
> >> > > > > >> > > > > > > >>>>>>> (say
> >> > > > > >> > > > > > > >>>>>>>> -
> >> > > > > >> > > > > > > >>>>>>>>>> intermediate commits)
> >> > > > > >> > > > > > > >>>>>>>>>> At the end its a trade-off
> >> > > > > >> > > > > > > >>>>>>>>>> Money, Resources, Time to build for
> each
> >> > and
> >> > > > > every
> >> > > > > >> > > commit
> >> > > > > >> > > > vs
> >> > > > > >> > > > > > > >>>>> Pain of
> >> > > > > >> > > > > > > >>>>>>>>>> triggering builds
> >> > > > > >> > > > > > > >>>>>>>>>>
> >> > > > > >> > > > > > > >>>>>>>>>>
> >> > > > > >> > > > > > > >>>>>>>>>>> Scan trigger plugin would poll SCM.
> >> Can we
> >> > > use
> >> > > > > >> plugin
> >> > > > > >> > > at
> >> > > > > >> > > > > > > >>>>> scale?
> >> > > > > >> > > > > > > >>>>>>>>>>
> >> > > > > >> > > > > > > >>>>>>>>>> 1. I haven't tested it on scale. But I
> >> > think
> >> > > > with
> >> > > > > >> the
> >> > > > > >> > > > > current
> >> > > > > >> > > > > > > >>>>> scale
> >> > > > > >> > > > > > > >>>>>>> of
> >> > > > > >> > > > > > > >>>>>>>>>> MXNet repo (191 open PRs i.e. checking
> >> for
> >> > > > > changes
> >> > > > > >> to
> >> > > > > >> > > 191
> >> > > > > >> > > > > > > >>>>> branches -
> >> > > > > >> > > > > > > >>>>>>> It
> >> > > > > >> > > > > > > >>>>>>>>>> should be manageable)
> >> > > > > >> > > > > > > >>>>>>>>>> 2. What's the purpose of the plugin?
> >> tldr;
> >> > > > Branch
> >> > > > > >> > > > discovery
> >> > > > > >> > > > > > > >>> or
> >> > > > > >> > > > > > > >>>>> branch
> >> > > > > >> > > > > > > >>>>>>>>>> indexing.
> >> > > > > >> > > > > > > >>>>>>>>>> Scan trigger plugin comes into the
> >> picture
> >> > > only
> >> > > > > >> once
> >> > > > > >> > per
> >> > > > > >> > > > PR
> >> > > > > >> > > > > > > >>> per
> >> > > > > >> > > > > > > >>>>> job
> >> > > > > >> > > > > > > >>>>>>>>> (i.e. 8
> >> > > > > >> > > > > > > >>>>>>>>>> times per PR for 8 jobs). It is
> >> basically
> >> > > done
> >> > > > > >> when a
> >> > > > > >> > > new
> >> > > > > >> > > > PR
> >> > > > > >> > > > > > > >>> is
> >> > > > > >> > > > > > > >>>>> made
> >> > > > > >> > > > > > > >>>>>>>> and
> >> > > > > >> > > > > > > >>>>>>>>>> the job (say unix-cpu hasn't
> discovered
> >> the
> >> > > new
> >> > > > > PR
> >> > > > > >> > > branch
> >> > > > > >> > > > > > > >>> yet).
> >> > > > > >> > > > > > > >>>>>>> That's
> >> > > > > >> > > > > > > >>>>>>>>> it.
> >> > > > > >> > > > > > > >>>>>>>>>> So it shouldn't be a problem for
> public
> >> > MXNet
> >> > > > > repo.
> >> > > > > >> > > > > > > >>>>>>>>>>
> >> > > > > >> > > > > > > >>>>>>>>>> Thanks,
> >> > > > > >> > > > > > > >>>>>>>>>> Chai
> >> > > > > >> > > > > > > >>>>>>>>>>
> >> > > > > >> > > > > > > >>>>>>>>>>
> >> > > > > >> > > > > > > >>>>>>>>>> On Thu, 12 Mar 2020 at 14:22, Marco de
> >> > Abreu
> >> > > <
> >> > > > > >> > > > > > > >>>>>>> marco.g.abreu@gmail.com>
> >> > > > > >> > > > > > > >>>>>>>>>> wrote:
> >> > > > > >> > > > > > > >>>>>>>>>>
> >> > > > > >> > > > > > > >>>>>>>>>>> Btw you forgot to set a date and time
> >> for
> >> > > the
> >> > > > > >> metting
> >> > > > > >> > > > > > > >>>>>>>>>>>
> >> > > > > >> > > > > > > >>>>>>>>>>>
> >> > > > > >> > > > > > > >>>>>>>>>>> On Thu, Mar 12, 2020 at 10:18 PM
> Marco
> >> de
> >> > > > Abreu
> >> > > > > <
> >> > > > > >> > > > > > > >>>>>>>>> marco.g.abreu@gmail.com
> >> > > > > >> > > > > > > >>>>>>>>>>> wrote:
> >> > > > > >> > > > > > > >>>>>>>>>>>
> >> > > > > >> > > > > > > >>>>>>>>>>>> Thanks Chai, I generally like the
> >> idea of
> >> > > the
> >> > > > > >> bot.
> >> > > > > >> > But
> >> > > > > >> > > > > > > >>> I'm
> >> > > > > >> > > > > > > >>>>> not a
> >> > > > > >> > > > > > > >>>>>>>>>>> supporter
> >> > > > > >> > > > > > > >>>>>>>>>>>> of the idea to disable any automatic
> >> > > > triggering
> >> > > > > >> > > > > > > >>> (disabling
> >> > > > > >> > > > > > > >>>>> the
> >> > > > > >> > > > > > > >>>>>>>>> webhook
> >> > > > > >> > > > > > > >>>>>>>>>> is
> >> > > > > >> > > > > > > >>>>>>>>>>>> also not an option, considering that
> >> this
> >> > > > will
> >> > > > > >> > disable
> >> > > > > >> > > > > > > >>>> master
> >> > > > > >> > > > > > > >>>>>>>>>> triggers).
> >> > > > > >> > > > > > > >>>>>>>>>>>> The CI system has to support the
> >> > community
> >> > > > > >> without
> >> > > > > >> > > > > > > >>>> requiring
> >> > > > > >> > > > > > > >>>>>>> people
> >> > > > > >> > > > > > > >>>>>>>>> to
> >> > > > > >> > > > > > > >>>>>>>>>>>> constantly shepherd every single
> run.
> >> > > > Disabling
> >> > > > > >> > > > automatic
> >> > > > > >> > > > > > > >>>>>>
Re: MXNet Bot Demo
Posted by Chaitanya Bapat <ch...@gmail.com>.
Yes Denisa, you can use it on existing PRs as well. Sorry for miswording.
Bot will comment instructions on how-to-use for every new PR.
You can invoke bot for all the PRs.
On Tue, 24 Mar 2020 at 18:00, Denisa Roberts <de...@gmail.com>
wrote:
> Hi- Only for new PRs? Can I use it for an existing PR to retrigger only
> specific tests?
>
> On Tue, Mar 24, 2020 at 8:27 PM Chaitanya Bapat <ch...@gmail.com>
> wrote:
>
>> Hello MXNet community,
>>
>> Update: Bot has been deployed 🚀 on apache/incubator-mxnet.
>>
>> For every new PR, bot will comment with a help message (instructing what
>> command to comment)
>> It can trigger all jobs or specific jobs for users.
>>
>> Do use and if you find issues/suggestions do comment on this mail thread
>> or
>> the GitHub issue : https://github.com/apache/incubator-mxnet/issues/17801
>>
>> Thanks to Denis, Sandeep, Joe, Pedro, Marco and the design doc reviewers
>> for valuable feedback and guidance.
>>
>> Thank you,
>> Chai
>>
>> On Mon, 23 Mar 2020 at 13:08, sandeep krishnamurthy <
>> sandeep.krishna98@gmail.com> wrote:
>>
>> > Thank you Chaitanya and Marco for helping the MXNet community.
>> >
>> > On Mon, Mar 23, 2020 at 12:56 PM Marco de Abreu <
>> marco.g.abreu@gmail.com>
>> > wrote:
>> >
>> > > Sure, already done.
>> > >
>> > > -Marco
>> > >
>> > > On Mon, Mar 23, 2020 at 8:53 PM Chaitanya Bapat <chai.bapat@gmail.com
>> >
>> > > wrote:
>> > >
>> > > > Hello,
>> > > > Update: Apache Infra Ticket for MXNet Bot
>> > > > Thanks once again, Marco for opening the ticket. But turns out,
>> Apache
>> > > > Infra folks closed it stating: "Security concerns around allowing
>> > unknown
>> > > > person to submit PR and run our hardware". Furthermore, it goes onto
>> > > state
>> > > > that bot circumvents the dependence on Jenkins Admins which is like
>> > > solving
>> > > > a problem that doesn't exist.
>> > > >
>> > > > I sense there is some confusion in the communication (maybe on my
>> > part).
>> > > It
>> > > > turns out the security concerns aren't actually correct.
>> > > >
>> > > > 1. Unknown person can submit a PR (before & after bot proposal), and
>> > run
>> > > > our hardware (pre as well as post bot).
>> > > > 2. Code should be reviewed by somebody with an ICLA on file. This
>> > doesn't
>> > > > change either. Prior to merging a PR, code has to be approved by a
>> > > > committer just like before.
>> > > > Overall it looks like the job of the bot isn't clear to folks in
>> Apache
>> > > > Infra. Bot simply is a means for triggering CI (which could be done
>> > > > manually by Log In to Jenkins -> PR -> Job -> Build) and doesn't
>> quite
>> > > > tweak with merging procedure. Yes, only addition is now unknown
>> person
>> > > (PR
>> > > > Author) can trigger CI with a message (but that was possible anyway
>> by
>> > > > pushing a commit. Bot just prevents users from pushing empty commits
>> > and
>> > > > building entire suite).
>> > > >
>> > > > As can be seen from last 10 open PRs as of Monday 23rd March, 12pm
>> PT
>> > > most
>> > > > PRs fail on 1/2 jobs. In such a scenario, the proposed MXNet bot
>> would
>> > > come
>> > > > in handy for just invoking CI on that specific build (instead of a
>> > > > non-committer PR Author to push empty commit : hurting on the
>> resource,
>> > > > time & cost considerations apart from undesirable dev experience)
>> > > >
>> > > > @Marco Since I am a non-committer, I guess these 2 clarifications
>> need
>> > to
>> > > > be conveyed to the Apache Infra by someone with Committer access.
>> > > >
>> > > > What do you think?
>> > > >
>> > > > Thanks,
>> > > > Chai
>> > > >
>> > > > On Sat, 21 Mar 2020 at 16:08, Marco de Abreu <
>> marco.g.abreu@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hello,
>> > > > >
>> > > > > the ticket has been created:
>> > > > > https://issues.apache.org/jira/browse/INFRA-20005
>> > > > >
>> > > > > Best regards,
>> > > > > Marco
>> > > > >
>> > > > > On Thu, Mar 19, 2020 at 11:49 PM Marco de Abreu <
>> > > marco.g.abreu@gmail.com
>> > > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Sounds like a good plan!
>> > > > > >
>> > > > > > Please send me the URL (please make sure it's backed by DNS and
>> not
>> > > > just
>> > > > > > the gateway URL) of the webhook handler, GitHub events you're
>> > > > interested
>> > > > > in
>> > > > > > and the shared secret in a private email to my personal email
>> > > address.
>> > > > I
>> > > > > > will then create the ticket with Apache infra.
>> > > > > >
>> > > > > > -Marco
>> > > > > >
>> > > > > > Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 19. März
>> > > 2020,
>> > > > > > 23:07:
>> > > > > >
>> > > > > >> @Marco Alright, it makes total sense to test out the Bot
>> feature
>> > > > > alongside
>> > > > > >> auto-trigger as a transition.
>> > > > > >>
>> > > > > >> Path Forward:
>> > > > > >> 1. Setup MXNet Bot on apache/incubator-mxnet repo (GitHub
>> WebHook
>> > > and
>> > > > > >> Infra)
>> > > > > >> 2. We don't turn off automatic trigger of PR builds for now.
>> > > > > >> 3. Hopefully, bot is used by developers to trigger specific
>> jobs
>> > > > > >> 4. Later on (say around April 20), let's discuss the
>> possibility
>> > of
>> > > > > >> switching off auto-trigger (with appropriate data) if it makes
>> > > sense.
>> > > > > >> Thanks Marco for volunteering to help enable the web hook on
>> > > > > >> apache/incubator-mxnet. Let me know if we can sync up on Slack
>> > > channel
>> > > > > to
>> > > > > >> get the ball rolling.
>> > > > > >>
>> > > > > >> Thanks once again for the entire community to step in and help
>> try
>> > > out
>> > > > > >> this
>> > > > > >> Bot.
>> > > > > >> Chai
>> > > > > >>
>> > > > > >> On Wed, 18 Mar 2020 at 17:07, Marco de Abreu <
>> > > marco.g.abreu@gmail.com
>> > > > >
>> > > > > >> wrote:
>> > > > > >>
>> > > > > >> > Hi, that's correct. But as stated previously, it's not an
>> option
>> > > to
>> > > > > >> remove
>> > > > > >> > the hook. For now, I'd like to see how the system behaves
>> while
>> > > it's
>> > > > > >> > optional. Later on, we can talk about revisiting this
>> decision.
>> > > But
>> > > > to
>> > > > > >> me
>> > > > > >> > it's not an option to deploy an entirely new system and
>> approach
>> > > > > without
>> > > > > >> > having a transition or even a timeframe in which we are able
>> to
>> > > fall
>> > > > > >> back.
>> > > > > >> >
>> > > > > >> > I'm happy to support the deployment of the bot and add an
>> > > additional
>> > > > > >> > webhook to enable it's functionality to support selective
>> > > triggering
>> > > > > by
>> > > > > >> PR
>> > > > > >> > authors and committers, but I will not support the disabling
>> of
>> > > > > >> automatic
>> > > > > >> > triggering of branches or PRs.
>> > > > > >> >
>> > > > > >> > -Marco
>> > > > > >> >
>> > > > > >> > Chaitanya Bapat <ch...@gmail.com> schrieb am Mi., 18.
>> März
>> > > > 2020,
>> > > > > >> > 21:00:
>> > > > > >> >
>> > > > > >> > > Hey Marco,
>> > > > > >> > >
>> > > > > >> > > I thought currently every commit on PR and master triggers
>> CI
>> > > > > >> > > because
>> > > > > >> > > a. github webhook points to Jenkins Server
>> > > > > >> > > b. GH Webhook events trigger builds on Jenkins for all
>> commits
>> > > to
>> > > > > any
>> > > > > >> > > branch in apache/incubator-mxnet
>> > > > > >> > > may it be master/PR/non-PR
>> > > > > >> > > Reason:
>> > > > > >> > > Because all the 3 types of branches are discovered by
>> Jenkins
>> > > > > (non-PR
>> > > > > >> > > (including master) and PR)
>> > > > > >> > >
>> > > > > >> > > Proposal: Remove GitHub WebHook to Jenkins and replace
>> with GH
>> > > > > >> Webhook to
>> > > > > >> > > Lambda
>> > > > > >> > > But after I remove the github webhook that points to
>> Jenkins :
>> > > > *N**o
>> > > > > >> > commit
>> > > > > >> > > will trigger Jenkins build by default* (as Jenkins wont
>> > receive
>> > > GH
>> > > > > >> > events)
>> > > > > >> > > Only those that Bot deems fit will be triggered (using
>> Jenkins
>> > > API
>> > > > > >> > invoked
>> > > > > >> > > by Lambda).
>> > > > > >> > > Hence its needed to handle that case of master merge.
>> > > > > >> > > Am I understanding this correctly?
>> > > > > >> > >
>> > > > > >> > > On Wed, 18 Mar 2020 at 04:23, Marco de Abreu <
>> > > > > marco.g.abreu@gmail.com
>> > > > > >> >
>> > > > > >> > > wrote:
>> > > > > >> > >
>> > > > > >> > > > Thanks Chai, sounds good to me. Could you elaborate a
>> bit on
>> > > the
>> > > > > >> point
>> > > > > >> > > > about triggering a CI run after the PR has been merged?
>> We
>> > > > already
>> > > > > >> to
>> > > > > >> > > that
>> > > > > >> > > > automatically for the master, so what's the benefit to
>> do it
>> > > > > twice?
>> > > > > >> > > >
>> > > > > >> > > > -Marco
>> > > > > >> > > >
>> > > > > >> > > > Chaitanya Bapat <ch...@gmail.com> schrieb am Mi.,
>> 18.
>> > > März
>> > > > > >> 2020,
>> > > > > >> > > > 09:30:
>> > > > > >> > > >
>> > > > > >> > > > > Update:
>> > > > > >> > > > >
>> > > > > >> > > > > > we can ensure that all CI runs ran on the commit
>> that
>> > > will
>> > > > be
>> > > > > >> > merged
>> > > > > >> > > > > @Sam Skalicky <sa...@gmail.com> Branch
>> Protection
>> > is
>> > > > > added
>> > > > > >> to
>> > > > > >> > > > public
>> > > > > >> > > > > MXNet repo. It ensures that for every PR to be merged,
>> the
>> > > CI
>> > > > > >> passes.
>> > > > > >> > > All
>> > > > > >> > > > > the jobs selected "required" jobs will have to be green
>> > for
>> > > > the
>> > > > > >> PR to
>> > > > > >> > > be
>> > > > > >> > > > > merged. Ofcourse, users with "Adminstrator" access can
>> > merge
>> > > > > >> without
>> > > > > >> > it
>> > > > > >> > > > but
>> > > > > >> > > > > that's just a backdoor. It is the case now and will
>> > continue
>> > > > to
>> > > > > be
>> > > > > >> > the
>> > > > > >> > > > case
>> > > > > >> > > > > with the inclusion of Bot.
>> > > > > >> > > > >
>> > > > > >> > > > > > easily verify that the CI has executed all runs on
>> the
>> > > > commit
>> > > > > >> that
>> > > > > >> > > will
>> > > > > >> > > > > be merged
>> > > > > >> > > > > GitHub UI shows all the jobs and the status
>> corresponding
>> > to
>> > > > it
>> > > > > on
>> > > > > >> > > every
>> > > > > >> > > > > commit. That should suffice. For the merged commits,
>> Repo
>> > ->
>> > > > > >> Commits
>> > > > > >> > ->
>> > > > > >> > > > > Commit ID (Status) can be tracked currently (only way
>> > that I
>> > > > > know
>> > > > > >> > of).
>> > > > > >> > > > > Moreover, it is beyond the scope of this project (and
>> > > possibly
>> > > > > >> out of
>> > > > > >> > > our
>> > > > > >> > > > > control since this is purely GitHub UI specific
>> use-case).
>> > > > > >> > > > >
>> > > > > >> > > > > Thanks @przemyslaw for supporting the opt-in.
>> > > > > >> > > > >
>> > > > > >> > > > > Thanks everyone in the community for sharing concerns,
>> > > voicing
>> > > > > >> your
>> > > > > >> > > > opinion
>> > > > > >> > > > > and participating in the discussion.
>> > > > > >> > > > > Thanks to those who attended the demo last Friday.
>> > > > > >> > > > >
>> > > > > >> > > > > Action items from that discussion
>> > > > > >> > > > > 1. Handle master merge builds [Done]
>> > > > > >> > > > > Bot runs entire CI suite after the PR is merged and
>> > comments
>> > > > on
>> > > > > >> the
>> > > > > >> > PR
>> > > > > >> > > > > about the same.
>> > > > > >> > > > > Design decision :
>> > > > > >> > > > > MXNet Bot comment about master merge build on the
>> *merge
>> > > > commit
>> > > > > vs
>> > > > > >> > PR*.
>> > > > > >> > > > > After the PR is merged, Bot runs entire CI and comments
>> > the
>> > > > > >> result of
>> > > > > >> > > CI
>> > > > > >> > > > > trigger on the PR (because it is easy to track on a PR
>> > > rather
>> > > > > than
>> > > > > >> > > > > commenting inside the merge commit)
>> > > > > >> > > > >
>> > > > > >> > > > > 2. Idempotent condition
>> > > > > >> > > > > In case of already running build, if an attempt is
>> made to
>> > > > > >> retrigger
>> > > > > >> > > the
>> > > > > >> > > > > job then what should be the response
>> > > > > >> > > > > a. Not to re-trigger, let the ongoing build continue
>> till
>> > > > > >> completion
>> > > > > >> > > > > b. End the ongoing build and re-trigger
>> > > > > >> > > > > c. Let the ongoing build continue, re-trigger new build
>> > > > > >> > > > >
>> > > > > >> > > > > From resource saving point of view, *c* looks costly
>> and a
>> > > can
>> > > > > be
>> > > > > >> > > > > avoided/optimized by B.
>> > > > > >> > > > > In case when a re-trigger was started "erroneously"
>> then
>> > > > killing
>> > > > > >> > > ongoing
>> > > > > >> > > > > build and re-trigger is a waste.
>> > > > > >> > > > > In case when ongoing build failed in one sub-part, then
>> > > > > >> re-triggering
>> > > > > >> > > is
>> > > > > >> > > > > justified.
>> > > > > >> > > > > Erroneous re-triggers would be less often while
>> conscious
>> > > > > >> re-triggers
>> > > > > >> > > to
>> > > > > >> > > > > suppress failure is more common use-case. It looks
>> like a
>> > > safe
>> > > > > >> > > assumption
>> > > > > >> > > > > to make given the trade-off.
>> > > > > >> > > > > [Open to debate]
>> > > > > >> > > > >
>> > > > > >> > > > > 3. Add security consideration [Use of secret manager,
>> but
>> > > > > without
>> > > > > >> > > > > auto-rotation due to Jenkins manual config requirement]
>> > > [Done]
>> > > > > >> > > > > 4. New PR Instruction message by the Bot [Done]
>> > > > > >> > > > > Thanks to the suggestion of Leonard, supported by
>> others.
>> > > I've
>> > > > > now
>> > > > > >> > > added
>> > > > > >> > > > > the feature where the Bot comments a help message. [For
>> > > > > reference
>> > > > > >> -
>> > > > > >> > > > > https://github.com/ChaiBapchya/incubator-mxnet/pull/52
>> ]
>> > > > > >> > > > >
>> > > > > >> > > > > Barring the opt-in vs opt-out debate & idempotency,
>> > > consensus
>> > > > > was
>> > > > > >> > > quickly
>> > > > > >> > > > > reached for the rest.
>> > > > > >> > > > >
>> > > > > >> > > > > In the coming days, I hope to roll-out this feature
>> into
>> > > Prod
>> > > > > >> (public
>> > > > > >> > > > > MXNet) for all devs to use.
>> > > > > >> > > > >
>> > > > > >> > > > > Thanks,
>> > > > > >> > > > > Chai
>> > > > > >> > > > >
>> > > > > >> > > > > On Mon, 16 Mar 2020 at 11:57, Marco de Abreu <
>> > > > > >> > marco.g.abreu@gmail.com>
>> > > > > >> > > > > wrote:
>> > > > > >> > > > >
>> > > > > >> > > > > > Well that's generally a problem with a deferred CI
>> > > approach
>> > > > > (CI
>> > > > > >> is
>> > > > > >> > > run
>> > > > > >> > > > at
>> > > > > >> > > > > > commit and not at merge time). This can either be
>> solved
>> > > > > through
>> > > > > >> > the
>> > > > > >> > > > > other
>> > > > > >> > > > > > proposal that's currently on dev@, by having a bot
>> > which
>> > > > does
>> > > > > >> > merges
>> > > > > >> > > > by
>> > > > > >> > > > > > having a global lock and a merge queue or by
>> accepting
>> > the
>> > > > > >> issue.
>> > > > > >> > > > Reality
>> > > > > >> > > > > > right now is that we're running that model where two
>> PRs
>> > > > which
>> > > > > >> are
>> > > > > >> > > > merged
>> > > > > >> > > > > > in parallel might break one another. One thing to
>> > consider
>> > > > > >> though
>> > > > > >> > is
>> > > > > >> > > > that
>> > > > > >> > > > > > this breakage would have to be introduced in two
>> > separate
>> > > > > parts
>> > > > > >> > since
>> > > > > >> > > > > > otherwise there'd be merge conflicts. I think we had
>> > that
>> > > > > >> situation
>> > > > > >> > > > twice
>> > > > > >> > > > > > so far and the result was a quick revert, so I'd say
>> > that
>> > > > > it's a
>> > > > > >> > > > problem
>> > > > > >> > > > > > that can happily be accepted. All other solutions
>> > > basically
>> > > > > >> require
>> > > > > >> > > > some
>> > > > > >> > > > > > form of single-threaded and globally locked solution
>> > which
>> > > > > >> limits
>> > > > > >> > us
>> > > > > >> > > in
>> > > > > >> > > > > > scalability. I'd recommend to just accept that risk
>> and
>> > > > revert
>> > > > > >> a PR
>> > > > > >> > > in
>> > > > > >> > > > > case
>> > > > > >> > > > > > it actually had a conflict.
>> > > > > >> > > > > >
>> > > > > >> > > > > > -Marco
>> > > > > >> > > > > >
>> > > > > >> > > > > > On Mon, Mar 16, 2020 at 6:29 PM Skalicky, Sam
>> > > > > >> > > > <sskalic@amazon.com.invalid
>> > > > > >> > > > > >
>> > > > > >> > > > > > wrote:
>> > > > > >> > > > > >
>> > > > > >> > > > > > > We probably need some way to track which CI runs
>> ran
>> > for
>> > > > > which
>> > > > > >> > > commit
>> > > > > >> > > > > > too,
>> > > > > >> > > > > > > that way we can ensure that all CI runs ran on the
>> > > commit
>> > > > > that
>> > > > > >> > will
>> > > > > >> > > > be
>> > > > > >> > > > > > > merged. Maybe the bot can comment with the commit
>> > hash
>> > > > when
>> > > > > >> > users
>> > > > > >> > > > > > command
>> > > > > >> > > > > > > it to do something. Although since users can
>> trigger
>> > > > > >> individual
>> > > > > >> > CI
>> > > > > >> > > > runs
>> > > > > >> > > > > > its
>> > > > > >> > > > > > > possible to have some commits run some CI runs but
>> not
>> > > > > >> others. We
>> > > > > >> > > > need
>> > > > > >> > > > > > some
>> > > > > >> > > > > > > way to easily verify that the CI has executed all
>> runs
>> > > on
>> > > > > the
>> > > > > >> > > commit
>> > > > > >> > > > > that
>> > > > > >> > > > > > > will be merged.
>> > > > > >> > > > > > >
>> > > > > >> > > > > > > Sam
>> > > > > >> > > > > > >
>> > > > > >> > > > > > > > On Mar 13, 2020, at 8:28 PM, Przemysław Trędak <
>> > > > > >> > > ptrendx@apache.org
>> > > > > >> > > > >
>> > > > > >> > > > > > > wrote:
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > > > CAUTION: This email originated from outside of
>> the
>> > > > > >> > organization.
>> > > > > >> > > Do
>> > > > > >> > > > > not
>> > > > > >> > > > > > > click links or open attachments unless you can
>> confirm
>> > > the
>> > > > > >> sender
>> > > > > >> > > and
>> > > > > >> > > > > > know
>> > > > > >> > > > > > > the content is safe.
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > > > I personally like the idea of opt-in more than
>> > > opt-out:
>> > > > > >> > > > > > > > - ultimately PR author wants the PR to be merged
>> so
>> > > they
>> > > > > (or
>> > > > > >> > > > > committer
>> > > > > >> > > > > > > reviewing the PR) will trigger the CI
>> > > > > >> > > > > > > > - if it is easy to trigger the PR via the bot
>> > command
>> > > > then
>> > > > > >> the
>> > > > > >> > > > amount
>> > > > > >> > > > > > of
>> > > > > >> > > > > > > work per PR should be less than with opt-out (since
>> > most
>> > > > of
>> > > > > >> the
>> > > > > >> > > > commits
>> > > > > >> > > > > > > should then be marked as [skip ci] or something
>> > similar
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > > > +1 to the bot making a comment on each new PR
>> with
>> > its
>> > > > > >> commands
>> > > > > >> > > > (and
>> > > > > >> > > > > > > also explaining, or at least giving links to the
>> > general
>> > > > PR
>> > > > > >> > process
>> > > > > >> > > > so
>> > > > > >> > > > > > new
>> > > > > >> > > > > > > PR authors are not lost). Maybe we could make the
>> bot
>> > > > > >> recognize
>> > > > > >> > if
>> > > > > >> > > > the
>> > > > > >> > > > > PR
>> > > > > >> > > > > > > author is new or existing contributor and offer
>> advice
>> > > > based
>> > > > > >> on
>> > > > > >> > > that?
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > > > Thanks
>> > > > > >> > > > > > > > Przemek
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > > > On 2020/03/13 22:06:58, Marco de Abreu <
>> > > > > >> > marco.g.abreu@gmail.com>
>> > > > > >> > > > > > wrote:
>> > > > > >> > > > > > > >> Hi,
>> > > > > >> > > > > > > >>
>> > > > > >> > > > > > > >> since it's no longer necessary to push a new
>> commit
>> > > to
>> > > > > >> trigger
>> > > > > >> > > CI,
>> > > > > >> > > > > it
>> > > > > >> > > > > > > will
>> > > > > >> > > > > > > >> already reduce the costs. But to me, requiring
>> an
>> > > > action
>> > > > > to
>> > > > > >> > > enable
>> > > > > >> > > > > CI
>> > > > > >> > > > > > > after
>> > > > > >> > > > > > > >> a PR has been created initially, is a no go.
>> User
>> > can
>> > > > opt
>> > > > > >> out
>> > > > > >> > of
>> > > > > >> > > > CI,
>> > > > > >> > > > > > but
>> > > > > >> > > > > > > >> the default has to be CI being triggered
>> > > automatically
>> > > > > for
>> > > > > >> > every
>> > > > > >> > > > > > commit
>> > > > > >> > > > > > > >> unless specifically disabled by a participant.
>> I'm
>> > > also
>> > > > > >> fine
>> > > > > >> > > with
>> > > > > >> > > > > > > >> triggering certain additional jobs (think about
>> > > > running a
>> > > > > >> > > nightly
>> > > > > >> > > > > job
>> > > > > >> > > > > > > upon
>> > > > > >> > > > > > > >> request for a PR) to require a manual step, but
>> the
>> > > PR
>> > > > > >> > > validation
>> > > > > >> > > > > > > pipelines
>> > > > > >> > > > > > > >> have to run automatically. Every check that is
>> > marked
>> > > > as
>> > > > > >> > > > "Required"
>> > > > > >> > > > > in
>> > > > > >> > > > > > > >> GitHub has to be automatically kicked off.
>> > > > > >> > > > > > > >>
>> > > > > >> > > > > > > >> -Marco
>> > > > > >> > > > > > > >>
>> > > > > >> > > > > > > >> On Fri, Mar 13, 2020 at 9:50 PM Chaitanya Bapat
>> <
>> > > > > >> > > > > chai.bapat@gmail.com
>> > > > > >> > > > > > >
>> > > > > >> > > > > > > >> wrote:
>> > > > > >> > > > > > > >>
>> > > > > >> > > > > > > >>> Firstly,
>> > > > > >> > > > > > > >>> Sorry I missed out on attaching the mail thread
>> > that
>> > > > was
>> > > > > >> sent
>> > > > > >> > > on
>> > > > > >> > > > > 12th
>> > > > > >> > > > > > > >>> February for notifying the community of the
>> > upcoming
>> > > > > >> changes
>> > > > > >> > to
>> > > > > >> > > > the
>> > > > > >> > > > > > > MXNet
>> > > > > >> > > > > > > >>> CI
>> > > > > >> > > > > > > >>> For reference :
>> > > > > >> > > > > > > >>>
>> > > > > >> > > > > > > >>>
>> > > > > >> > > > > > >
>> > > > > >> > > > > >
>> > > > > >> > > > >
>> > > > > >> > > >
>> > > > > >> > >
>> > > > > >> >
>> > > > > >>
>> > > > >
>> > > >
>> > >
>> >
>> https://lists.apache.org/thread.html/r09a6ab2803a996fc80e00fe39ed312fa4865e8805e08df847f1addad%40%3Cdev.mxnet.apache.org%3E
>> > > > > >> > > > > > > >>>
>> > > > > >> > > > > > > >>> Now to the questions,
>> > > > > >> > > > > > > >>>> Is it possible for re-triggering a single job
>> to
>> > be
>> > > > > >> abused?
>> > > > > >> > > > > > > >>> @Tao In the case when a user re-triggers a
>> single
>> > > job
>> > > > > >> > multiple
>> > > > > >> > > > > times,
>> > > > > >> > > > > > > that
>> > > > > >> > > > > > > >>> will be visible in the PR conversation thread.
>> A
>> > > > > >> committer,
>> > > > > >> > > even
>> > > > > >> > > > > > after
>> > > > > >> > > > > > > he
>> > > > > >> > > > > > > >>> has approved the PR before, generally takes a
>> look
>> > > at
>> > > > > the
>> > > > > >> > final
>> > > > > >> > > > > state
>> > > > > >> > > > > > > of
>> > > > > >> > > > > > > >>> the PR before merging. Would it be fair to
>> assume
>> > > the
>> > > > > >> > committer
>> > > > > >> > > > > could
>> > > > > >> > > > > > > take
>> > > > > >> > > > > > > >>> the multiple re-trigger of a single job into
>> > account
>> > > > > >> before
>> > > > > >> > > > > merging?
>> > > > > >> > > > > > > The
>> > > > > >> > > > > > > >>> committer then has the option to invoke
>> > `@mxnet-bot
>> > > > run
>> > > > > ci
>> > > > > >> > > [all]
>> > > > > >> > > > `
>> > > > > >> > > > > to
>> > > > > >> > > > > > > >>> trigger the entire build pipeline one last to
>> > > counter
>> > > > > the
>> > > > > >> > > abuse.
>> > > > > >> > > > > This
>> > > > > >> > > > > > > is
>> > > > > >> > > > > > > >>> aligned with what @Leonard said.
>> > > > > >> > > > > > > >>>
>> > > > > >> > > > > > > >>> @Sandeep Thanks a lot for collecting and
>> sharing
>> > > > > valuable
>> > > > > >> > data.
>> > > > > >> > > > I'd
>> > > > > >> > > > > > > concur
>> > > > > >> > > > > > > >>> with the opinion that given the existing things
>> > > > > >> committers &
>> > > > > >> > PR
>> > > > > >> > > > > > Authors
>> > > > > >> > > > > > > >>> take care of, invoking CI shouldn't be that
>> big of
>> > > an
>> > > > > >> > > additional
>> > > > > >> > > > > > > burden.
>> > > > > >> > > > > > > >>>
>> > > > > >> > > > > > > >>> @Marco With the opt-out, the onus remains on
>> the
>> > PR
>> > > > > >> Author.
>> > > > > >> > It
>> > > > > >> > > > > > doesn't
>> > > > > >> > > > > > > help
>> > > > > >> > > > > > > >>> reduce the resource usage. Hence, it was
>> suggested
>> > > to
>> > > > > >> switch
>> > > > > >> > to
>> > > > > >> > > > > > > >>> opt-in. @Leo's suggestion for proactive
>> commenting
>> > > on
>> > > > > the
>> > > > > >> > part
>> > > > > >> > > of
>> > > > > >> > > > > bot
>> > > > > >> > > > > > > makes
>> > > > > >> > > > > > > >>> sense and is doable.
>> > > > > >> > > > > > > >>>
>> > > > > >> > > > > > > >>> Default : opt-out and User initiated opt-in
>> (with
>> > > > > >> addressing
>> > > > > >> > > > Leo's
>> > > > > >> > > > > > fix
>> > > > > >> > > > > > > for
>> > > > > >> > > > > > > >>> the usability issue you correctly pointed out )
>> > > > > >> > > > > > > >>> @Marco How does this sound to you?
>> > > > > >> > > > > > > >>>
>> > > > > >> > > > > > > >>> Again, thank you all for chiming in and voicing
>> > your
>> > > > > >> opinion.
>> > > > > >> > > > > > > Appreciate
>> > > > > >> > > > > > > >>> it.
>> > > > > >> > > > > > > >>> We can take ahead these discussions in today's
>> > demo
>> > > > > >> meeting.
>> > > > > >> > > > > [Design
>> > > > > >> > > > > > > Doc
>> > > > > >> > > > > > > >>> <
>> > > > > >> > >
>> > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
>> > > > > >> > > > >]
>> > > > > >> > > > > > > [Demo
>> > > > > >> > > > > > > >>> Video <
>> > https://www.youtube.com/watch?v=gfOGwZId8aU
>> > > >]
>> > > > > >> > > > > > > >>>
>> > > > > >> > > > > > > >>> Thanks,
>> > > > > >> > > > > > > >>> Chai
>> > > > > >> > > > > > > >>>
>> > > > > >> > > > > > > >>> On Fri, 13 Mar 2020 at 12:34, Marco de Abreu <
>> > > > > >> > > > > > marco.g.abreu@gmail.com>
>> > > > > >> > > > > > > >>> wrote:
>> > > > > >> > > > > > > >>>
>> > > > > >> > > > > > > >>>> I'd recommend that the bot makes an initial
>> > comment
>> > > > > when
>> > > > > >> a
>> > > > > >> > PR
>> > > > > >> > > > gets
>> > > > > >> > > > > > > opened
>> > > > > >> > > > > > > >>>> and informs the users of its commands. It then
>> > > tells
>> > > > > the
>> > > > > >> > user
>> > > > > >> > > > the
>> > > > > >> > > > > > > commend
>> > > > > >> > > > > > > >>>> to opt out of CI.
>> > > > > >> > > > > > > >>>>
>> > > > > >> > > > > > > >>>> -Marco
>> > > > > >> > > > > > > >>>>
>> > > > > >> > > > > > > >>>> Lausen, Leonard <la...@amazon.com.invalid>
>> > > schrieb
>> > > > am
>> > > > > >> Fr.,
>> > > > > >> > > 13.
>> > > > > >> > > > > > März
>> > > > > >> > > > > > > >>> 2020,
>> > > > > >> > > > > > > >>>> 20:27:
>> > > > > >> > > > > > > >>>>
>> > > > > >> > > > > > > >>>>> On opt-out: People may be unaware of opt-out
>> > would
>> > > > not
>> > > > > >> use
>> > > > > >> > > it.
>> > > > > >> > > > > > There
>> > > > > >> > > > > > > is
>> > > > > >> > > > > > > >>>> no
>> > > > > >> > > > > > > >>>>> incentive to use opt-out, as the PR author
>> > doesn't
>> > > > pay
>> > > > > >> any
>> > > > > >> > > > money
>> > > > > >> > > > > > for
>> > > > > >> > > > > > > CI
>> > > > > >> > > > > > > >>>>> run.
>> > > > > >> > > > > > > >>>>>
>> > > > > >> > > > > > > >>>>> I agree with Marco though that opt-in alone
>> may
>> > > > cause
>> > > > > >> > > usability
>> > > > > >> > > > > > > issues,
>> > > > > >> > > > > > > >>>> as
>> > > > > >> > > > > > > >>>>> contributors may not be aware of how to
>> trigger
>> > > the
>> > > > > CI.
>> > > > > >> > > > > > > >>>>> One solution is that the bot proactively
>> > comments
>> > > on
>> > > > > >> the PR
>> > > > > >> > > and
>> > > > > >> > > > > > > reminds
>> > > > > >> > > > > > > >>>> the
>> > > > > >> > > > > > > >>>>> author to trigger running CI once the author
>> > deems
>> > > > the
>> > > > > >> PR
>> > > > > >> > > > ready.
>> > > > > >> > > > > > > >>>>>
>> > > > > >> > > > > > > >>>>> But even if we choose opt-out, the bot will
>> > still
>> > > > add
>> > > > > a
>> > > > > >> lot
>> > > > > >> > > of
>> > > > > >> > > > > > value,
>> > > > > >> > > > > > > >>> as
>> > > > > >> > > > > > > >>>> PR
>> > > > > >> > > > > > > >>>>> authors can retrigger single jobs that have
>> > failed
>> > > > due
>> > > > > >> to
>> > > > > >> > > > > > flakiness.
>> > > > > >> > > > > > > >>>>>
>> > > > > >> > > > > > > >>>>>> Is it possible for re-triggering a single
>> job
>> > to
>> > > be
>> > > > > >> > abused?
>> > > > > >> > > > For
>> > > > > >> > > > > > > >>>> example,
>> > > > > >> > > > > > > >>>>>> the author spends two days re-triggering a
>> > flaky
>> > > > job
>> > > > > to
>> > > > > >> > make
>> > > > > >> > > > it
>> > > > > >> > > > > > > pass.
>> > > > > >> > > > > > > >>>>>
>> > > > > >> > > > > > > >>>>> Yes, this is possible. I suggest the
>> committer
>> > who
>> > > > > >> likes to
>> > > > > >> > > > > merge a
>> > > > > >> > > > > > > PR
>> > > > > >> > > > > > > >>>>> needs to
>> > > > > >> > > > > > > >>>>> make a good judgement here if a PR is abusing
>> > the
>> > > > > >> feature,
>> > > > > >> > > and
>> > > > > >> > > > if
>> > > > > >> > > > > > so,
>> > > > > >> > > > > > > >>>>> retrigger
>> > > > > >> > > > > > > >>>>> all CI runs.
>> > > > > >> > > > > > > >>>>>
>> > > > > >> > > > > > > >>>>> Best regards
>> > > > > >> > > > > > > >>>>> Leonard
>> > > > > >> > > > > > > >>>>>
>> > > > > >> > > > > > > >>>>> On Fri, 2020-03-13 at 08:07 +0100, Marco de
>> > Abreu
>> > > > > wrote:
>> > > > > >> > > > > > > >>>>>> Thanks for the data Sandeep. In these cases
>> it
>> > > > sounds
>> > > > > >> like
>> > > > > >> > > it
>> > > > > >> > > > > > would
>> > > > > >> > > > > > > >>>> have
>> > > > > >> > > > > > > >>>>>> rather been better when people explicitly
>> > turned
>> > > > off
>> > > > > >> CI in
>> > > > > >> > > > that
>> > > > > >> > > > > > > case.
>> > > > > >> > > > > > > >>>>>> What's the argument against an opt-out
>> instead
>> > of
>> > > > an
>> > > > > >> > opt-in?
>> > > > > >> > > > > > > >>>>>>
>> > > > > >> > > > > > > >>>>>> My intention is that I consider it quite
>> > > cumbersome
>> > > > > to
>> > > > > >> > make
>> > > > > >> > > > it a
>> > > > > >> > > > > > > >>>>> *required*
>> > > > > >> > > > > > > >>>>>> step to always trigger CI manually, even if
>> > just
>> > > > > >> > submitting
>> > > > > >> > > a
>> > > > > >> > > > > > small
>> > > > > >> > > > > > > >>> PR.
>> > > > > >> > > > > > > >>>>> I'd
>> > > > > >> > > > > > > >>>>>> rather see people explicitly turning off CI
>> if
>> > > they
>> > > > > >> > wouldn't
>> > > > > >> > > > > like
>> > > > > >> > > > > > to
>> > > > > >> > > > > > > >>>> use
>> > > > > >> > > > > > > >>>>> it
>> > > > > >> > > > > > > >>>>>> - and there's also the "draft" stage for a
>> PR
>> > > which
>> > > > > >> some
>> > > > > >> > > > > > > contributors
>> > > > > >> > > > > > > >>>> are
>> > > > > >> > > > > > > >>>>>> using.
>> > > > > >> > > > > > > >>>>>>
>> > > > > >> > > > > > > >>>>>> With regards to WIP and do not review: I
>> think
>> > > > these
>> > > > > >> are
>> > > > > >> > use
>> > > > > >> > > > > cases
>> > > > > >> > > > > > > >>>> where
>> > > > > >> > > > > > > >>>>>> you want CI feedback, as otherwise you
>> wouldn't
>> > > > have
>> > > > > >> > opened
>> > > > > >> > > > the
>> > > > > >> > > > > > PR.
>> > > > > >> > > > > > > >>> If
>> > > > > >> > > > > > > >>>>> you
>> > > > > >> > > > > > > >>>>>> don't want human feedback and neither
>> machine
>> > > > > feedback,
>> > > > > >> > why
>> > > > > >> > > > open
>> > > > > >> > > > > > the
>> > > > > >> > > > > > > >>> PR
>> > > > > >> > > > > > > >>>>> at
>> > > > > >> > > > > > > >>>>>> all?
>> > > > > >> > > > > > > >>>>>>
>> > > > > >> > > > > > > >>>>>> -Marco
>> > > > > >> > > > > > > >>>>>>
>> > > > > >> > > > > > > >>>>>>
>> > > > > >> > > > > > > >>>>>> sandeep krishnamurthy <
>> > > sandeep.krishna98@gmail.com
>> > > > >
>> > > > > >> > schrieb
>> > > > > >> > > > am
>> > > > > >> > > > > > Fr.,
>> > > > > >> > > > > > > >>>> 13.
>> > > > > >> > > > > > > >>>>>> März 2020, 05:24:
>> > > > > >> > > > > > > >>>>>>
>> > > > > >> > > > > > > >>>>>>> I tried to gather some data for us to
>> discuss
>> > > this
>> > > > > >> topic
>> > > > > >> > in
>> > > > > >> > > > > this
>> > > > > >> > > > > > > >>>>> thread. I
>> > > > > >> > > > > > > >>>>>>> tried to count number of un-necessary
>> builds
>> > by
>> > > > > >> looking
>> > > > > >> > at
>> > > > > >> > > > most
>> > > > > >> > > > > > > >>>> recent
>> > > > > >> > > > > > > >>>>> (as
>> > > > > >> > > > > > > >>>>>>> of 12, March 9 PM PST) 50 PRs merged to
>> master
>> > > and
>> > > > > 50
>> > > > > >> > PRs.
>> > > > > >> > > > > > > >>>>>>> Identifying un-necessary builds is bit
>> > > > subjective. I
>> > > > > >> > tried
>> > > > > >> > > to
>> > > > > >> > > > > be
>> > > > > >> > > > > > > >>> more
>> > > > > >> > > > > > > >>>>>>> conservative where I didn't count a build
>> as
>> > > > > >> un-necessary
>> > > > > >> > > if
>> > > > > >> > > > I
>> > > > > >> > > > > > was
>> > > > > >> > > > > > > >>> in
>> > > > > >> > > > > > > >>>>>>> doubt. Hence, I was not able to automate,
>> but
>> > I
>> > > > made
>> > > > > >> an
>> > > > > >> > > > effort
>> > > > > >> > > > > to
>> > > > > >> > > > > > > >>> go
>> > > > > >> > > > > > > >>>>>>> through PRs manually and use below
>> criteria to
>> > > > > >> identify
>> > > > > >> > > > > > > >>> un-necessary
>> > > > > >> > > > > > > >>>>>>> commits triggering the builds.
>> > > > > >> > > > > > > >>>>>>>
>> > > > > >> > > > > > > >>>>>>> 1. Explicitly marked as WIP / do not
>> review
>> > > PR
>> > > > > >> > > > > > > >>>>>>> 2. Incremental WIP commit and finally
>> > > > commenting a
>> > > > > >> > commit
>> > > > > >> > > > > > > >>> “trigger
>> > > > > >> > > > > > > >>>>> CI”
>> > > > > >> > > > > > > >>>>>>> 3. Multiple commits to address all
>> comments
>> > > from
>> > > > > >> single
>> > > > > >> > > > > review.
>> > > > > >> > > > > > > >>>>> This is
>> > > > > >> > > > > > > >>>>>>> assuming we see a comment, address them,
>> > > commit,
>> > > > > >> next
>> > > > > >> > the
>> > > > > >> > > > > > > >>>> following
>> > > > > >> > > > > > > >>>>>>> comment
>> > > > > >> > > > > > > >>>>>>> 4. Sequence of documentation only changes
>> > > > > >> > > > > > > >>>>>>>
>> > > > > >> > > > > > > >>>>>>> I found there were around 42 avoidable
>> builds
>> > > from
>> > > > > >> most
>> > > > > >> > > > recent
>> > > > > >> > > > > 50
>> > > > > >> > > > > > > >>>>> merged
>> > > > > >> > > > > > > >>>>>>> PRs and around 86 builds from recent 50
>> open
>> > > PRs.
>> > > > > >> > > > > > > >>>>>>>
>> > > > > >> > > > > > > >>>>>>> I synced up with other contributors (Joe
>> > Evans,
>> > > > > Chai)
>> > > > > >> > from
>> > > > > >> > > > > Amazon
>> > > > > >> > > > > > > >>> who
>> > > > > >> > > > > > > >>>>> is
>> > > > > >> > > > > > > >>>>>>> contributing to MXNet CI system. I was told
>> > that
>> > > > on
>> > > > > an
>> > > > > >> > > > average
>> > > > > >> > > > > it
>> > > > > >> > > > > > > >>>> costs
>> > > > > >> > > > > > > >>>>>>> around $84 per build and on an average 6
>> > commits
>> > > > per
>> > > > > >> > merged
>> > > > > >> > > > PR
>> > > > > >> > > > > > (for
>> > > > > >> > > > > > > >>>>> year
>> > > > > >> > > > > > > >>>>>>> 2019). Going by that, it is approximately
>> 1/6
>> > > > builds
>> > > > > >> are
>> > > > > >> > > > > > avoidable.
>> > > > > >> > > > > > > >>>>> [100 /
>> > > > > >> > > > > > > >>>>>>> 300 + 300 ]
>> > > > > >> > > > > > > >>>>>>>
>> > > > > >> > > > > > > >>>>>>>
>> > > > > >> > > > > > > >>>>>>> Usability should be top most priority. But,
>> > > since
>> > > > > >> either
>> > > > > >> > a
>> > > > > >> > > > > > reviewer
>> > > > > >> > > > > > > >>>> or
>> > > > > >> > > > > > > >>>>> pr
>> > > > > >> > > > > > > >>>>>>> author can trigger the bot, is it really a
>> > > hurdle
>> > > > > for
>> > > > > >> pr
>> > > > > >> > > > author
>> > > > > >> > > > > > or
>> > > > > >> > > > > > > >>>>> reviewer
>> > > > > >> > > > > > > >>>>>>> to call a bot to trigger CI? Given that PR
>> > > author
>> > > > > and
>> > > > > >> > > > reviewer
>> > > > > >> > > > > is
>> > > > > >> > > > > > > >>>>> already
>> > > > > >> > > > > > > >>>>>>> actively commenting various details such
>> as -
>> > PR
>> > > > > >> > > description,
>> > > > > >> > > > > > > >>> review
>> > > > > >> > > > > > > >>>>>>> comments and responses, adding labels etc.
>> > > > > >> > > > > > > >>>>>>>
>> > > > > >> > > > > > > >>>>>>>
>> > > > > >> > > > > > > >>>>>>> Me too curious to know the behavior for
>> Tao's
>> > > > above
>> > > > > >> use
>> > > > > >> > > case.
>> > > > > >> > > > > > > >>>>>>>
>> > > > > >> > > > > > > >>>>>>>
>> > > > > >> > > > > > > >>>>>>> Best,
>> > > > > >> > > > > > > >>>>>>>
>> > > > > >> > > > > > > >>>>>>> Sandeep
>> > > > > >> > > > > > > >>>>>>>
>> > > > > >> > > > > > > >>>>>>> On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <
>> > > > > >> > mutouorz@gmail.com
>> > > > > >> > > >
>> > > > > >> > > > > > wrote:
>> > > > > >> > > > > > > >>>>>>>
>> > > > > >> > > > > > > >>>>>>>> Is it possible for re-triggering a single
>> job
>> > > to
>> > > > be
>> > > > > >> > > abused?
>> > > > > >> > > > > For
>> > > > > >> > > > > > > >>>>> example,
>> > > > > >> > > > > > > >>>>>>>> the author spends two days re-triggering a
>> > > flaky
>> > > > > job
>> > > > > >> to
>> > > > > >> > > make
>> > > > > >> > > > > it
>> > > > > >> > > > > > > >>>>> pass. But
>> > > > > >> > > > > > > >>>>>>>> other jobs which have passed the
>> validation
>> > may
>> > > > be
>> > > > > >> > broken
>> > > > > >> > > by
>> > > > > >> > > > > > > >>> other
>> > > > > >> > > > > > > >>>>>>> commits
>> > > > > >> > > > > > > >>>>>>>> during the two day without being noticed.
>> And
>> > > > > finally
>> > > > > >> > the
>> > > > > >> > > PR
>> > > > > >> > > > > is
>> > > > > >> > > > > > > >>>>> merged
>> > > > > >> > > > > > > >>>>>>> with
>> > > > > >> > > > > > > >>>>>>>> underlying problems.
>> > > > > >> > > > > > > >>>>>>>>
>> > > > > >> > > > > > > >>>>>>>> On Fri, Mar 13, 2020 at 6:19 AM Marco de
>> > Abreu
>> > > <
>> > > > > >> > > > > > > >>>>> marco.g.abreu@gmail.com>
>> > > > > >> > > > > > > >>>>>>>> wrote:
>> > > > > >> > > > > > > >>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>> In the end it only comes down to money,
>> > > > > considering
>> > > > > >> > that
>> > > > > >> > > > the
>> > > > > >> > > > > > > >>>>> system is
>> > > > > >> > > > > > > >>>>>>>> auto
>> > > > > >> > > > > > > >>>>>>>>> scaling, making the execution time
>> constant.
>> > > > > >> > > > > > > >>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>> If we're trading money for usability, I
>> > > > certainly
>> > > > > >> would
>> > > > > >> > > > > prefer
>> > > > > >> > > > > > > >>>>>>> usability.
>> > > > > >> > > > > > > >>>>>>>>> I'd rather recommend to spend time on
>> > > > > parallelizing
>> > > > > >> > test
>> > > > > >> > > > > > > >>>> execution
>> > > > > >> > > > > > > >>>>> or
>> > > > > >> > > > > > > >>>>>>>>> getting rid of integration tests in the
>> PR
>> > > stage
>> > > > > >> > instead
>> > > > > >> > > > > > > >>> reducing
>> > > > > >> > > > > > > >>>>> the
>> > > > > >> > > > > > > >>>>>>>> costs
>> > > > > >> > > > > > > >>>>>>>>> by making people not use it. But taking a
>> > step
>> > > > > back
>> > > > > >> to
>> > > > > >> > > > > > > >>> requiring
>> > > > > >> > > > > > > >>>>> people
>> > > > > >> > > > > > > >>>>>>>> to
>> > > > > >> > > > > > > >>>>>>>>> manually trigger CI again doesn't feel
>> > right.
>> > > > > >> > > > > > > >>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>> I'm happy to see that bot deployed, but
>> I do
>> > > not
>> > > > > >> agree
>> > > > > >> > > with
>> > > > > >> > > > > > > >>>>> removing
>> > > > > >> > > > > > > >>>>>>> the
>> > > > > >> > > > > > > >>>>>>>>> auto trigger functionality for new
>> commits.
>> > > > > >> > > > > > > >>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>> -Marco
>> > > > > >> > > > > > > >>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>> Chaitanya Bapat <ch...@gmail.com>
>> > > schrieb
>> > > > am
>> > > > > >> Do.,
>> > > > > >> > > 12.
>> > > > > >> > > > > > > >>> März
>> > > > > >> > > > > > > >>>>> 2020,
>> > > > > >> > > > > > > >>>>>>>>> 22:47:
>> > > > > >> > > > > > > >>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>> @Marco Thanks for pointing that out.
>> > > > > >> > > > > > > >>>>>>>>>> Tomorrow i.e. Friday, March 13, 2020 at
>> > 3:00
>> > > > PM -
>> > > > > >> 3:30
>> > > > > >> > > PM
>> > > > > >> > > > in
>> > > > > >> > > > > > > >>>>>>>> (UTC-08:00)
>> > > > > >> > > > > > > >>>>>>>>>> Pacific Time (US & Canada).
>> > > > > >> > > > > > > >>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>> When do we expect this bot to be
>> deployed?
>> > > > > >> > > > > > > >>>>>>>>>> @Lin If all goes well in the next week I
>> > can
>> > > > > >> deploy it
>> > > > > >> > > to
>> > > > > >> > > > > > > >>>> public
>> > > > > >> > > > > > > >>>>>>> Apache
>> > > > > >> > > > > > > >>>>>>>>>> (provided I get permissions from Apache
>> > > Infra)
>> > > > > >> > > > > > > >>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>> @Marco Thanks for your feedback.
>> > > > > >> > > > > > > >>>>>>>>>>> CI system has to support the community
>> > > without
>> > > > > >> > > requiring
>> > > > > >> > > > > > > >>>>> people to
>> > > > > >> > > > > > > >>>>>>>>>> constantly shepherd every single run
>> > > > > >> > > > > > > >>>>>>>>>> We have data for the number of times CI
>> was
>> > > > > >> triggered
>> > > > > >> > > > > > > >>>>> unnecessarily
>> > > > > >> > > > > > > >>>>>>>> which
>> > > > > >> > > > > > > >>>>>>>>>> includes
>> > > > > >> > > > > > > >>>>>>>>>> - Entire build triggered instead of
>> > specific
>> > > > > build
>> > > > > >> > > > > > > >>>>>>>>>> - CI triggered when PR is still work in
>> > > > progress
>> > > > > or
>> > > > > >> > not
>> > > > > >> > > > yet
>> > > > > >> > > > > > > >>>> ready
>> > > > > >> > > > > > > >>>>>>> (say
>> > > > > >> > > > > > > >>>>>>>> -
>> > > > > >> > > > > > > >>>>>>>>>> intermediate commits)
>> > > > > >> > > > > > > >>>>>>>>>> At the end its a trade-off
>> > > > > >> > > > > > > >>>>>>>>>> Money, Resources, Time to build for each
>> > and
>> > > > > every
>> > > > > >> > > commit
>> > > > > >> > > > vs
>> > > > > >> > > > > > > >>>>> Pain of
>> > > > > >> > > > > > > >>>>>>>>>> triggering builds
>> > > > > >> > > > > > > >>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>> Scan trigger plugin would poll SCM.
>> Can we
>> > > use
>> > > > > >> plugin
>> > > > > >> > > at
>> > > > > >> > > > > > > >>>>> scale?
>> > > > > >> > > > > > > >>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>> 1. I haven't tested it on scale. But I
>> > think
>> > > > with
>> > > > > >> the
>> > > > > >> > > > > current
>> > > > > >> > > > > > > >>>>> scale
>> > > > > >> > > > > > > >>>>>>> of
>> > > > > >> > > > > > > >>>>>>>>>> MXNet repo (191 open PRs i.e. checking
>> for
>> > > > > changes
>> > > > > >> to
>> > > > > >> > > 191
>> > > > > >> > > > > > > >>>>> branches -
>> > > > > >> > > > > > > >>>>>>> It
>> > > > > >> > > > > > > >>>>>>>>>> should be manageable)
>> > > > > >> > > > > > > >>>>>>>>>> 2. What's the purpose of the plugin?
>> tldr;
>> > > > Branch
>> > > > > >> > > > discovery
>> > > > > >> > > > > > > >>> or
>> > > > > >> > > > > > > >>>>> branch
>> > > > > >> > > > > > > >>>>>>>>>> indexing.
>> > > > > >> > > > > > > >>>>>>>>>> Scan trigger plugin comes into the
>> picture
>> > > only
>> > > > > >> once
>> > > > > >> > per
>> > > > > >> > > > PR
>> > > > > >> > > > > > > >>> per
>> > > > > >> > > > > > > >>>>> job
>> > > > > >> > > > > > > >>>>>>>>> (i.e. 8
>> > > > > >> > > > > > > >>>>>>>>>> times per PR for 8 jobs). It is
>> basically
>> > > done
>> > > > > >> when a
>> > > > > >> > > new
>> > > > > >> > > > PR
>> > > > > >> > > > > > > >>> is
>> > > > > >> > > > > > > >>>>> made
>> > > > > >> > > > > > > >>>>>>>> and
>> > > > > >> > > > > > > >>>>>>>>>> the job (say unix-cpu hasn't discovered
>> the
>> > > new
>> > > > > PR
>> > > > > >> > > branch
>> > > > > >> > > > > > > >>> yet).
>> > > > > >> > > > > > > >>>>>>> That's
>> > > > > >> > > > > > > >>>>>>>>> it.
>> > > > > >> > > > > > > >>>>>>>>>> So it shouldn't be a problem for public
>> > MXNet
>> > > > > repo.
>> > > > > >> > > > > > > >>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>> Thanks,
>> > > > > >> > > > > > > >>>>>>>>>> Chai
>> > > > > >> > > > > > > >>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>> On Thu, 12 Mar 2020 at 14:22, Marco de
>> > Abreu
>> > > <
>> > > > > >> > > > > > > >>>>>>> marco.g.abreu@gmail.com>
>> > > > > >> > > > > > > >>>>>>>>>> wrote:
>> > > > > >> > > > > > > >>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>> Btw you forgot to set a date and time
>> for
>> > > the
>> > > > > >> metting
>> > > > > >> > > > > > > >>>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>> On Thu, Mar 12, 2020 at 10:18 PM Marco
>> de
>> > > > Abreu
>> > > > > <
>> > > > > >> > > > > > > >>>>>>>>> marco.g.abreu@gmail.com
>> > > > > >> > > > > > > >>>>>>>>>>> wrote:
>> > > > > >> > > > > > > >>>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>>> Thanks Chai, I generally like the
>> idea of
>> > > the
>> > > > > >> bot.
>> > > > > >> > But
>> > > > > >> > > > > > > >>> I'm
>> > > > > >> > > > > > > >>>>> not a
>> > > > > >> > > > > > > >>>>>>>>>>> supporter
>> > > > > >> > > > > > > >>>>>>>>>>>> of the idea to disable any automatic
>> > > > triggering
>> > > > > >> > > > > > > >>> (disabling
>> > > > > >> > > > > > > >>>>> the
>> > > > > >> > > > > > > >>>>>>>>> webhook
>> > > > > >> > > > > > > >>>>>>>>>> is
>> > > > > >> > > > > > > >>>>>>>>>>>> also not an option, considering that
>> this
>> > > > will
>> > > > > >> > disable
>> > > > > >> > > > > > > >>>> master
>> > > > > >> > > > > > > >>>>>>>>>> triggers).
>> > > > > >> > > > > > > >>>>>>>>>>>> The CI system has to support the
>> > community
>> > > > > >> without
>> > > > > >> > > > > > > >>>> requiring
>> > > > > >> > > > > > > >>>>>>> people
>> > > > > >> > > > > > > >>>>>>>>> to
>> > > > > >> > > > > > > >>>>>>>>>>>> constantly shepherd every single run.
>> > > > Disabling
>> > > > > >> > > > automatic
>> > > > > >> > > > > > > >>>>>>>> triggering
>> > > > > >> > > > > > > >>>>>>>>>>> seems
>> > > > > >> > > > > > > >>>>>>>>>>>> like a step back to me.
>> > > > > >> > > > > > > >>>>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>>> Instead, I'd recommend that CI gets
>> > > triggered
>> > > > > >> upon
>> > > > > >> > > every
>> > > > > >> > > > > > > >>>>> commit
>> > > > > >> > > > > > > >>>>>>> as
>> > > > > >> > > > > > > >>>>>>>>>> usual,
>> > > > > >> > > > > > > >>>>>>>>>>>> but people have the possibility to
>> call a
>> > > > > >> "command"
>> > > > > >> > > > (i.e.
>> > > > > >> > > > > > > >>>>> make a
>> > > > > >> > > > > > > >>>>>>>>>> message
>> > > > > >> > > > > > > >>>>>>>>>>>> which results in the bot setting a
>> label)
>> > > to
>> > > > > >> disable
>> > > > > >> > > CI
>> > > > > >> > > > > > > >>>> until
>> > > > > >> > > > > > > >>>>>>> they
>> > > > > >> > > > > > > >>>>>>>>>> revoke
>> > > > > >> > > > > > > >>>>>>>>>>>> it. But the standard should still be
>> > that a
>> > > > new
>> > > > > >> > commit
>> > > > > >> > > > > > > >>>>> triggers a
>> > > > > >> > > > > > > >>>>>>>> new
>> > > > > >> > > > > > > >>>>>>>>>> CI
>> > > > > >> > > > > > > >>>>>>>>>>>> run.
>> > > > > >> > > > > > > >>>>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>>>
>> > > > > >> > > > > > > >>>>
>> > > > > >> > https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
>> > > > > >> > > > > > > >>>>>>>> seems
>> > > > > >> > > > > > > >>>>>>>>>> like
>> > > > > >> > > > > > > >>>>>>>>>>>> this would poll SCM. This will incur
>> high
>> > > > quota
>> > > > > >> > > > > > > >>>>> restrictions. Are
>> > > > > >> > > > > > > >>>>>>>> you
>> > > > > >> > > > > > > >>>>>>>>>>> sure
>> > > > > >> > > > > > > >>>>>>>>>>>> that you can use that plugin at scale?
>> > > > > >> > > > > > > >>>>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>>> -Marco
>> > > > > >> > > > > > > >>>>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>>> On Thu, Mar 12, 2020 at 10:04 PM Lin
>> > Yuan <
>> > > > > >> > > > > > > >>>>> apeforest@gmail.com>
>> > > > > >> > > > > > > >>>>>>>>> wrote:
>> > > > > >> > > > > > > >>>>>>>>>>>>> Chai,
>> > > > > >> > > > > > > >>>>>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>>>> Awesome work. When do we expect this
>> bot
>> > > to
>> > > > be
>> > > > > >> > > > > > > >>> deployed?
>> > > > > >> > > > > > > >>>>>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>>>> Best,
>> > > > > >> > > > > > > >>>>>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>>>> Lin
>> > > > > >> > > > > > > >>>>>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>>>> On Thu, Mar 12, 2020 at 2:00 PM
>> > Chaitanya
>> > > > > Bapat
>> > > > > >> <
>> > > > > >> > > > > > > >>>>>>>>> chai.bapat@gmail.com
>> > > > > >> > > > > > > >>>>>>>>>>>>> wrote:
>> > > > > >> > > > > > > >>>>>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>>>>> Hello MXNet community,
>> > > > > >> > > > > > > >>>>>>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>>>>> I have built an MXNet Bot <
>> > > > > >> > > > > > > >>>> https://github.com/mxnet-bot>
>> > > > > >> > > > > > > >>>>> that
>> > > > > >> > > > > > > >>>>>>>>>> allows
>> > > > > >> > > > > > > >>>>>>>>>>> PR
>> > > > > >> > > > > > > >>>>>>>>>>>>>> Authors, Committers and Jenkins
>> Admins
>> > to
>> > > > > >> trigger
>> > > > > >> > CI
>> > > > > >> > > > > > > >>>>> manually.
>> > > > > >> > > > > > > >>>>>>>>>>>>>> It handles 2 problems
>> > > > > >> > > > > > > >>>>>>>>>>>>>> 1. Manual CI trigger instead of
>> > existing
>> > > > > >> automated
>> > > > > >> > > CI
>> > > > > >> > > > > > > >>>>> trigger
>> > > > > >> > > > > > > >>>>>>>>>>>>>> 2. Gives permissions to PR Authors
>> (in
>> > > > > >> addition to
>> > > > > >> > > > > > > >>>> MXNet
>> > > > > >> > > > > > > >>>>>>>>> Committers
>> > > > > >> > > > > > > >>>>>>>>>>> and
>> > > > > >> > > > > > > >>>>>>>>>>>>>> Jenkins Admins)
>> > > > > >> > > > > > > >>>>>>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>>>>> Design Doc :
>> > > > > >> > > > > > > >>>>>>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>
>> > > > > >> > > >
>> > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
>> > > > > >> > > > > > > >>>>>>>>>>>>>> I urge you all to attend the
>> > > demonstration
>> > > > > >> meeting
>> > > > > >> > > > > > > >>> and
>> > > > > >> > > > > > > >>>>> lend
>> > > > > >> > > > > > > >>>>>>> your
>> > > > > >> > > > > > > >>>>>>>>>> views
>> > > > > >> > > > > > > >>>>>>>>>>>>> on
>> > > > > >> > > > > > > >>>>>>>>>>>>>> the same.
>> > > > > >> > > > > > > >>>>>>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>>>>> Thank you,
>> > > > > >> > > > > > > >>>>>>>>>>>>>> Chai
>> > > > > >> > > > > > > >>>>>>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>>>>> *Meeting Details*:
>> > > > > >> > > > > > > >>>>>>>>>>>>>> ==============Conference Bridge
>> > > > > >> > > > > > > >>>> Information==============
>> > > > > >> > > > > > > >>>>>>>>>>>>>> You have been invited to an online
>> > > meeting,
>> > > > > >> > powered
>> > > > > >> > > > > > > >>> by
>> > > > > >> > > > > > > >>>>> Amazon
>> > > > > >> > > > > > > >>>>>>>>> Chime.
>> > > > > >> > > > > > > >>>>>>>>>>>>>> *Chime meeting ID*: *9272158344*
>> > > > > >> > > > > > > >>>>>>>>>>>>>> Join via Chime clients (manually):
>> > Select
>> > > > > >> > 'Meetings
>> > > > > >> > > >
>> > > > > >> > > > > > > >>>>> Join a
>> > > > > >> > > > > > > >>>>>>>>>> Meeting',
>> > > > > >> > > > > > > >>>>>>>>>>>>> and
>> > > > > >> > > > > > > >>>>>>>>>>>>>> enter 9272158344
>> > > > > >> > > > > > > >>>>>>>>>>>>>> Join via Chime clients (auto-call):
>> If
>> > > you
>> > > > > >> invite
>> > > > > >> > > > > > > >>>>> auto-call as
>> > > > > >> > > > > > > >>>>>>>>>>> attendee,
>> > > > > >> > > > > > > >>>>>>>>>>>>>> Chime will call you when the meeting
>> > > > starts,
>> > > > > >> > select
>> > > > > >> > > > > > > >>>>> 'Answer'
>> > > > > >> > > > > > > >>>>>>>>>>>>>> *Join via browser screen share*:
>> > > > > >> > > > > > > >>>>> https://chime.aws/9272158344
>> > > > > >> > > > > > > >>>>>>>>>>>>>> *Join via phone* (US):
>> > > > > >> > +1-929-432-4463,,,9272158344#
>> > > > > >> > > > > > > >>>>>>>>>>>>>> *Join via phone (US toll-free)*:
>> > > > > >> > > > > > > >>>>> +1-855-552-4463,,,9272158344#
>> > > > > >> > > > > > > >>>>>>>>>>>>>> International dial-in:
>> > > > > >> > > > > > > >>>> https://chime.aws/dialinnumbers/
>> > > > > >> > > > > > > >>>>>>>>>>>>>> In-room video system: Ext: 62000,
>> > Meeting
>> > > > > PIN:
>> > > > > >> > > > > > > >>>>> 9272158344#
>> > > > > >> > > > > > > >>>>>>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>>>>> --
>> > > > > >> > > > > > > >>>>>>>>>>>>>> *Chaitanya Prakash Bapat*
>> > > > > >> > > > > > > >>>>>>>>>>>>>> *+1 (973) 953-6299*
>> > > > > >> > > > > > > >>>>>>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>>>>> [image:
>> > > > > >> https://www.linkedin.com//in/chaibapat25]
>> > > > > >> > > > > > > >>>>>>>>>>>>>> <https://github.com/ChaiBapchya
>> > >[image:
>> > > > > >> > > > > > > >>>>>>>>>>>>> https://www.facebook.com/chaibapat
>> > > > > >> > > > > > > >>>>>>>>>>>>>> ]
>> > > > > >> > > > > > > >>>>>>>>>>>>>> <
>> https://www.facebook.com/chaibapchya
>> > > > > >[image:
>> > > > > >> > > > > > > >>>>>>>>>>>>>> https://twitter.com/ChaiBapchya] <
>> > > > > >> > > > > > > >>>>>>>> https://twitter.com/ChaiBapchya
>> > > > > >> > > > > > > >>>>>>>>>>>>>> [image:
>> > > > > >> > > > > > > >>>>>>>>>>>>>>
>> > https://www.linkedin.com//in/chaibapat25
>> > > ]
>> > > > > >> > > > > > > >>>>>>>>>>>>>> <
>> > > https://www.linkedin.com//in/chaibapchya/
>> > > > >
>> > > > > >> > > > > > > >>>>>>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>> --
>> > > > > >> > > > > > > >>>>>>>>>> *Chaitanya Prakash Bapat*
>> > > > > >> > > > > > > >>>>>>>>>> *+1 (973) 953-6299*
>> > > > > >> > > > > > > >>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>>>> [image:
>> > > > https://www.linkedin.com//in/chaibapat25
>> > > > > ]
>> > > > > >> > > > > > > >>>>>>>>>> <https://github.com/ChaiBapchya>[image:
>> > > > > >> > > > > > > >>>>>>>>> https://www.facebook.com/chaibapat
>> > > > > >> > > > > > > >>>>>>>>>> ]
>> > > > > >> > > > > > > >>>>>>>>>> <https://www.facebook.com/chaibapchya
>> > > >[image:
>> > > > > >> > > > > > > >>>>>>>>>> https://twitter.com/ChaiBapchya] <
>> > > > > >> > > > > > > >>>>> https://twitter.com/ChaiBapchya
>> > > > > >> > > > > > > >>>>>>>>>> [image:
>> > > > > >> > > > > > > >>>>>>>>>>
>> https://www.linkedin.com//in/chaibapat25]
>> > > > > >> > > > > > > >>>>>>>>>> <
>> https://www.linkedin.com//in/chaibapchya/
>> > >
>> > > > > >> > > > > > > >>>>>>>>>>
>> > > > > >> > > > > > > >>>>>>>
>> > > > > >> > > > > > > >>>>>>> --
>> > > > > >> > > > > > > >>>>>>> Sandeep Krishnamurthy
>> > > > > >> > > > > > > >>>>>>>
>> > > > > >> > > > > > > >>>>>
>> > > > > >> > > > > > > >>>>
>> > > > > >> > > > > > > >>>
>> > > > > >> > > > > > > >>>
>> > > > > >> > > > > > > >>> --
>> > > > > >> > > > > > > >>> *Chaitanya Prakash Bapat*
>> > > > > >> > > > > > > >>> *+1 (973) 953-6299*
>> > > > > >> > > > > > > >>>
>> > > > > >> > > > > > > >>> [image:
>> https://www.linkedin.com//in/chaibapat25]
>> > > > > >> > > > > > > >>> <https://github.com/ChaiBapchya>[image:
>> > > > > >> > > > > > > https://www.facebook.com/chaibapat
>> > > > > >> > > > > > > >>> ]
>> > > > > >> > > > > > > >>> <https://www.facebook.com/chaibapchya>[image:
>> > > > > >> > > > > > > >>> https://twitter.com/ChaiBapchya] <
>> > > > > >> > > > https://twitter.com/ChaiBapchya
>> > > > > >> > > > > > > >[image:
>> > > > > >> > > > > > > >>> https://www.linkedin.com//in/chaibapat25]
>> > > > > >> > > > > > > >>> <https://www.linkedin.com//in/chaibapchya/>
>> > > > > >> > > > > > > >>>
>> > > > > >> > > > > > > >>
>> > > > > >> > > > > > >
>> > > > > >> > > > > > >
>> > > > > >> > > > > >
>> > > > > >> > > > >
>> > > > > >> > > > >
>> > > > > >> > > > > --
>> > > > > >> > > > > *Chaitanya Prakash Bapat*
>> > > > > >> > > > > *+1 (973) 953-6299*
>> > > > > >> > > > >
>> > > > > >> > > > > [image: https://www.linkedin.com//in/chaibapat25]
>> > > > > >> > > > > <https://github.com/ChaiBapchya>[image:
>> > > > > >> > > > https://www.facebook.com/chaibapat
>> > > > > >> > > > > ]
>> > > > > >> > > > > <https://www.facebook.com/chaibapchya>[image:
>> > > > > >> > > > > https://twitter.com/ChaiBapchya] <
>> > > > > https://twitter.com/ChaiBapchya
>> > > > > >> > > > >[image:
>> > > > > >> > > > > https://www.linkedin.com//in/chaibapat25]
>> > > > > >> > > > > <https://www.linkedin.com//in/chaibapchya/>
>> > > > > >> > > > >
>> > > > > >> > > >
>> > > > > >> > >
>> > > > > >> > >
>> > > > > >> > > --
>> > > > > >> > > *Chaitanya Prakash Bapat*
>> > > > > >> > > *+1 (973) 953-6299*
>> > > > > >> > >
>> > > > > >> > > [image: https://www.linkedin.com//in/chaibapat25]
>> > > > > >> > > <https://github.com/ChaiBapchya>[image:
>> > > > > >> > https://www.facebook.com/chaibapat
>> > > > > >> > > ]
>> > > > > >> > > <https://www.facebook.com/chaibapchya>[image:
>> > > > > >> > > https://twitter.com/ChaiBapchya] <
>> > > https://twitter.com/ChaiBapchya
>> > > > > >> > >[image:
>> > > > > >> > > https://www.linkedin.com//in/chaibapat25]
>> > > > > >> > > <https://www.linkedin.com//in/chaibapchya/>
>> > > > > >> > >
>> > > > > >> >
>> > > > > >>
>> > > > > >>
>> > > > > >> --
>> > > > > >> *Chaitanya Prakash Bapat*
>> > > > > >> *+1 (973) 953-6299*
>> > > > > >>
>> > > > > >> [image: https://www.linkedin.com//in/chaibapat25]
>> > > > > >> <https://github.com/ChaiBapchya>[image:
>> > > > > >> https://www.facebook.com/chaibapat]
>> > > > > >> <https://www.facebook.com/chaibapchya>[image:
>> > > > > >> https://twitter.com/ChaiBapchya] <
>> https://twitter.com/ChaiBapchya
>> > > > > >[image:
>> > > > > >> https://www.linkedin.com//in/chaibapat25]
>> > > > > >> <https://www.linkedin.com//in/chaibapchya/>
>> > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > *Chaitanya Prakash Bapat*
>> > > > *+1 (973) 953-6299*
>> > > >
>> > > > [image: https://www.linkedin.com//in/chaibapat25]
>> > > > <https://github.com/ChaiBapchya>[image:
>> > > https://www.facebook.com/chaibapat
>> > > > ]
>> > > > <https://www.facebook.com/chaibapchya>[image:
>> > > > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
>> > > >[image:
>> > > > https://www.linkedin.com//in/chaibapat25]
>> > > > <https://www.linkedin.com//in/chaibapchya/>
>> > > >
>> > >
>> >
>> >
>> > --
>> > Sandeep Krishnamurthy
>> >
>>
>>
>> --
>> *Chaitanya Prakash Bapat*
>> *+1 (973) 953-6299*
>>
>> [image: https://www.linkedin.com//in/chaibapat25]
>> <https://github.com/ChaiBapchya>[image:
>> https://www.facebook.com/chaibapat]
>> <https://www.facebook.com/chaibapchya>[image:
>> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
>> https://www.linkedin.com//in/chaibapat25]
>> <https://www.linkedin.com//in/chaibapchya/>
>>
>
--
*Chaitanya Prakash Bapat*
*+1 (973) 953-6299*
[image: https://www.linkedin.com//in/chaibapat25]
<https://github.com/ChaiBapchya>[image: https://www.facebook.com/chaibapat]
<https://www.facebook.com/chaibapchya>[image:
https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
https://www.linkedin.com//in/chaibapat25]
<https://www.linkedin.com//in/chaibapchya/>
Re: MXNet Bot Demo
Posted by Chaitanya Bapat <ch...@gmail.com>.
Hello MXNet community,
Update: Bot has been deployed 🚀 on apache/incubator-mxnet.
For every new PR, bot will comment with a help message (instructing what
command to comment)
It can trigger all jobs or specific jobs for users.
Do use and if you find issues/suggestions do comment on this mail thread or
the GitHub issue : https://github.com/apache/incubator-mxnet/issues/17801
Thanks to Denis, Sandeep, Joe, Pedro, Marco and the design doc reviewers
for valuable feedback and guidance.
Thank you,
Chai
On Mon, 23 Mar 2020 at 13:08, sandeep krishnamurthy <
sandeep.krishna98@gmail.com> wrote:
> Thank you Chaitanya and Marco for helping the MXNet community.
>
> On Mon, Mar 23, 2020 at 12:56 PM Marco de Abreu <ma...@gmail.com>
> wrote:
>
> > Sure, already done.
> >
> > -Marco
> >
> > On Mon, Mar 23, 2020 at 8:53 PM Chaitanya Bapat <ch...@gmail.com>
> > wrote:
> >
> > > Hello,
> > > Update: Apache Infra Ticket for MXNet Bot
> > > Thanks once again, Marco for opening the ticket. But turns out, Apache
> > > Infra folks closed it stating: "Security concerns around allowing
> unknown
> > > person to submit PR and run our hardware". Furthermore, it goes onto
> > state
> > > that bot circumvents the dependence on Jenkins Admins which is like
> > solving
> > > a problem that doesn't exist.
> > >
> > > I sense there is some confusion in the communication (maybe on my
> part).
> > It
> > > turns out the security concerns aren't actually correct.
> > >
> > > 1. Unknown person can submit a PR (before & after bot proposal), and
> run
> > > our hardware (pre as well as post bot).
> > > 2. Code should be reviewed by somebody with an ICLA on file. This
> doesn't
> > > change either. Prior to merging a PR, code has to be approved by a
> > > committer just like before.
> > > Overall it looks like the job of the bot isn't clear to folks in Apache
> > > Infra. Bot simply is a means for triggering CI (which could be done
> > > manually by Log In to Jenkins -> PR -> Job -> Build) and doesn't quite
> > > tweak with merging procedure. Yes, only addition is now unknown person
> > (PR
> > > Author) can trigger CI with a message (but that was possible anyway by
> > > pushing a commit. Bot just prevents users from pushing empty commits
> and
> > > building entire suite).
> > >
> > > As can be seen from last 10 open PRs as of Monday 23rd March, 12pm PT
> > most
> > > PRs fail on 1/2 jobs. In such a scenario, the proposed MXNet bot would
> > come
> > > in handy for just invoking CI on that specific build (instead of a
> > > non-committer PR Author to push empty commit : hurting on the resource,
> > > time & cost considerations apart from undesirable dev experience)
> > >
> > > @Marco Since I am a non-committer, I guess these 2 clarifications need
> to
> > > be conveyed to the Apache Infra by someone with Committer access.
> > >
> > > What do you think?
> > >
> > > Thanks,
> > > Chai
> > >
> > > On Sat, 21 Mar 2020 at 16:08, Marco de Abreu <ma...@gmail.com>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > the ticket has been created:
> > > > https://issues.apache.org/jira/browse/INFRA-20005
> > > >
> > > > Best regards,
> > > > Marco
> > > >
> > > > On Thu, Mar 19, 2020 at 11:49 PM Marco de Abreu <
> > marco.g.abreu@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Sounds like a good plan!
> > > > >
> > > > > Please send me the URL (please make sure it's backed by DNS and not
> > > just
> > > > > the gateway URL) of the webhook handler, GitHub events you're
> > > interested
> > > > in
> > > > > and the shared secret in a private email to my personal email
> > address.
> > > I
> > > > > will then create the ticket with Apache infra.
> > > > >
> > > > > -Marco
> > > > >
> > > > > Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 19. März
> > 2020,
> > > > > 23:07:
> > > > >
> > > > >> @Marco Alright, it makes total sense to test out the Bot feature
> > > > alongside
> > > > >> auto-trigger as a transition.
> > > > >>
> > > > >> Path Forward:
> > > > >> 1. Setup MXNet Bot on apache/incubator-mxnet repo (GitHub WebHook
> > and
> > > > >> Infra)
> > > > >> 2. We don't turn off automatic trigger of PR builds for now.
> > > > >> 3. Hopefully, bot is used by developers to trigger specific jobs
> > > > >> 4. Later on (say around April 20), let's discuss the possibility
> of
> > > > >> switching off auto-trigger (with appropriate data) if it makes
> > sense.
> > > > >> Thanks Marco for volunteering to help enable the web hook on
> > > > >> apache/incubator-mxnet. Let me know if we can sync up on Slack
> > channel
> > > > to
> > > > >> get the ball rolling.
> > > > >>
> > > > >> Thanks once again for the entire community to step in and help try
> > out
> > > > >> this
> > > > >> Bot.
> > > > >> Chai
> > > > >>
> > > > >> On Wed, 18 Mar 2020 at 17:07, Marco de Abreu <
> > marco.g.abreu@gmail.com
> > > >
> > > > >> wrote:
> > > > >>
> > > > >> > Hi, that's correct. But as stated previously, it's not an option
> > to
> > > > >> remove
> > > > >> > the hook. For now, I'd like to see how the system behaves while
> > it's
> > > > >> > optional. Later on, we can talk about revisiting this decision.
> > But
> > > to
> > > > >> me
> > > > >> > it's not an option to deploy an entirely new system and approach
> > > > without
> > > > >> > having a transition or even a timeframe in which we are able to
> > fall
> > > > >> back.
> > > > >> >
> > > > >> > I'm happy to support the deployment of the bot and add an
> > additional
> > > > >> > webhook to enable it's functionality to support selective
> > triggering
> > > > by
> > > > >> PR
> > > > >> > authors and committers, but I will not support the disabling of
> > > > >> automatic
> > > > >> > triggering of branches or PRs.
> > > > >> >
> > > > >> > -Marco
> > > > >> >
> > > > >> > Chaitanya Bapat <ch...@gmail.com> schrieb am Mi., 18. März
> > > 2020,
> > > > >> > 21:00:
> > > > >> >
> > > > >> > > Hey Marco,
> > > > >> > >
> > > > >> > > I thought currently every commit on PR and master triggers CI
> > > > >> > > because
> > > > >> > > a. github webhook points to Jenkins Server
> > > > >> > > b. GH Webhook events trigger builds on Jenkins for all commits
> > to
> > > > any
> > > > >> > > branch in apache/incubator-mxnet
> > > > >> > > may it be master/PR/non-PR
> > > > >> > > Reason:
> > > > >> > > Because all the 3 types of branches are discovered by Jenkins
> > > > (non-PR
> > > > >> > > (including master) and PR)
> > > > >> > >
> > > > >> > > Proposal: Remove GitHub WebHook to Jenkins and replace with GH
> > > > >> Webhook to
> > > > >> > > Lambda
> > > > >> > > But after I remove the github webhook that points to Jenkins :
> > > *N**o
> > > > >> > commit
> > > > >> > > will trigger Jenkins build by default* (as Jenkins wont
> receive
> > GH
> > > > >> > events)
> > > > >> > > Only those that Bot deems fit will be triggered (using Jenkins
> > API
> > > > >> > invoked
> > > > >> > > by Lambda).
> > > > >> > > Hence its needed to handle that case of master merge.
> > > > >> > > Am I understanding this correctly?
> > > > >> > >
> > > > >> > > On Wed, 18 Mar 2020 at 04:23, Marco de Abreu <
> > > > marco.g.abreu@gmail.com
> > > > >> >
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > Thanks Chai, sounds good to me. Could you elaborate a bit on
> > the
> > > > >> point
> > > > >> > > > about triggering a CI run after the PR has been merged? We
> > > already
> > > > >> to
> > > > >> > > that
> > > > >> > > > automatically for the master, so what's the benefit to do it
> > > > twice?
> > > > >> > > >
> > > > >> > > > -Marco
> > > > >> > > >
> > > > >> > > > Chaitanya Bapat <ch...@gmail.com> schrieb am Mi., 18.
> > März
> > > > >> 2020,
> > > > >> > > > 09:30:
> > > > >> > > >
> > > > >> > > > > Update:
> > > > >> > > > >
> > > > >> > > > > > we can ensure that all CI runs ran on the commit that
> > will
> > > be
> > > > >> > merged
> > > > >> > > > > @Sam Skalicky <sa...@gmail.com> Branch Protection
> is
> > > > added
> > > > >> to
> > > > >> > > > public
> > > > >> > > > > MXNet repo. It ensures that for every PR to be merged, the
> > CI
> > > > >> passes.
> > > > >> > > All
> > > > >> > > > > the jobs selected "required" jobs will have to be green
> for
> > > the
> > > > >> PR to
> > > > >> > > be
> > > > >> > > > > merged. Ofcourse, users with "Adminstrator" access can
> merge
> > > > >> without
> > > > >> > it
> > > > >> > > > but
> > > > >> > > > > that's just a backdoor. It is the case now and will
> continue
> > > to
> > > > be
> > > > >> > the
> > > > >> > > > case
> > > > >> > > > > with the inclusion of Bot.
> > > > >> > > > >
> > > > >> > > > > > easily verify that the CI has executed all runs on the
> > > commit
> > > > >> that
> > > > >> > > will
> > > > >> > > > > be merged
> > > > >> > > > > GitHub UI shows all the jobs and the status corresponding
> to
> > > it
> > > > on
> > > > >> > > every
> > > > >> > > > > commit. That should suffice. For the merged commits, Repo
> ->
> > > > >> Commits
> > > > >> > ->
> > > > >> > > > > Commit ID (Status) can be tracked currently (only way
> that I
> > > > know
> > > > >> > of).
> > > > >> > > > > Moreover, it is beyond the scope of this project (and
> > possibly
> > > > >> out of
> > > > >> > > our
> > > > >> > > > > control since this is purely GitHub UI specific use-case).
> > > > >> > > > >
> > > > >> > > > > Thanks @przemyslaw for supporting the opt-in.
> > > > >> > > > >
> > > > >> > > > > Thanks everyone in the community for sharing concerns,
> > voicing
> > > > >> your
> > > > >> > > > opinion
> > > > >> > > > > and participating in the discussion.
> > > > >> > > > > Thanks to those who attended the demo last Friday.
> > > > >> > > > >
> > > > >> > > > > Action items from that discussion
> > > > >> > > > > 1. Handle master merge builds [Done]
> > > > >> > > > > Bot runs entire CI suite after the PR is merged and
> comments
> > > on
> > > > >> the
> > > > >> > PR
> > > > >> > > > > about the same.
> > > > >> > > > > Design decision :
> > > > >> > > > > MXNet Bot comment about master merge build on the *merge
> > > commit
> > > > vs
> > > > >> > PR*.
> > > > >> > > > > After the PR is merged, Bot runs entire CI and comments
> the
> > > > >> result of
> > > > >> > > CI
> > > > >> > > > > trigger on the PR (because it is easy to track on a PR
> > rather
> > > > than
> > > > >> > > > > commenting inside the merge commit)
> > > > >> > > > >
> > > > >> > > > > 2. Idempotent condition
> > > > >> > > > > In case of already running build, if an attempt is made to
> > > > >> retrigger
> > > > >> > > the
> > > > >> > > > > job then what should be the response
> > > > >> > > > > a. Not to re-trigger, let the ongoing build continue till
> > > > >> completion
> > > > >> > > > > b. End the ongoing build and re-trigger
> > > > >> > > > > c. Let the ongoing build continue, re-trigger new build
> > > > >> > > > >
> > > > >> > > > > From resource saving point of view, *c* looks costly and a
> > can
> > > > be
> > > > >> > > > > avoided/optimized by B.
> > > > >> > > > > In case when a re-trigger was started "erroneously" then
> > > killing
> > > > >> > > ongoing
> > > > >> > > > > build and re-trigger is a waste.
> > > > >> > > > > In case when ongoing build failed in one sub-part, then
> > > > >> re-triggering
> > > > >> > > is
> > > > >> > > > > justified.
> > > > >> > > > > Erroneous re-triggers would be less often while conscious
> > > > >> re-triggers
> > > > >> > > to
> > > > >> > > > > suppress failure is more common use-case. It looks like a
> > safe
> > > > >> > > assumption
> > > > >> > > > > to make given the trade-off.
> > > > >> > > > > [Open to debate]
> > > > >> > > > >
> > > > >> > > > > 3. Add security consideration [Use of secret manager, but
> > > > without
> > > > >> > > > > auto-rotation due to Jenkins manual config requirement]
> > [Done]
> > > > >> > > > > 4. New PR Instruction message by the Bot [Done]
> > > > >> > > > > Thanks to the suggestion of Leonard, supported by others.
> > I've
> > > > now
> > > > >> > > added
> > > > >> > > > > the feature where the Bot comments a help message. [For
> > > > reference
> > > > >> -
> > > > >> > > > > https://github.com/ChaiBapchya/incubator-mxnet/pull/52]
> > > > >> > > > >
> > > > >> > > > > Barring the opt-in vs opt-out debate & idempotency,
> > consensus
> > > > was
> > > > >> > > quickly
> > > > >> > > > > reached for the rest.
> > > > >> > > > >
> > > > >> > > > > In the coming days, I hope to roll-out this feature into
> > Prod
> > > > >> (public
> > > > >> > > > > MXNet) for all devs to use.
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > > > >> > > > > Chai
> > > > >> > > > >
> > > > >> > > > > On Mon, 16 Mar 2020 at 11:57, Marco de Abreu <
> > > > >> > marco.g.abreu@gmail.com>
> > > > >> > > > > wrote:
> > > > >> > > > >
> > > > >> > > > > > Well that's generally a problem with a deferred CI
> > approach
> > > > (CI
> > > > >> is
> > > > >> > > run
> > > > >> > > > at
> > > > >> > > > > > commit and not at merge time). This can either be solved
> > > > through
> > > > >> > the
> > > > >> > > > > other
> > > > >> > > > > > proposal that's currently on dev@, by having a bot
> which
> > > does
> > > > >> > merges
> > > > >> > > > by
> > > > >> > > > > > having a global lock and a merge queue or by accepting
> the
> > > > >> issue.
> > > > >> > > > Reality
> > > > >> > > > > > right now is that we're running that model where two PRs
> > > which
> > > > >> are
> > > > >> > > > merged
> > > > >> > > > > > in parallel might break one another. One thing to
> consider
> > > > >> though
> > > > >> > is
> > > > >> > > > that
> > > > >> > > > > > this breakage would have to be introduced in two
> separate
> > > > parts
> > > > >> > since
> > > > >> > > > > > otherwise there'd be merge conflicts. I think we had
> that
> > > > >> situation
> > > > >> > > > twice
> > > > >> > > > > > so far and the result was a quick revert, so I'd say
> that
> > > > it's a
> > > > >> > > > problem
> > > > >> > > > > > that can happily be accepted. All other solutions
> > basically
> > > > >> require
> > > > >> > > > some
> > > > >> > > > > > form of single-threaded and globally locked solution
> which
> > > > >> limits
> > > > >> > us
> > > > >> > > in
> > > > >> > > > > > scalability. I'd recommend to just accept that risk and
> > > revert
> > > > >> a PR
> > > > >> > > in
> > > > >> > > > > case
> > > > >> > > > > > it actually had a conflict.
> > > > >> > > > > >
> > > > >> > > > > > -Marco
> > > > >> > > > > >
> > > > >> > > > > > On Mon, Mar 16, 2020 at 6:29 PM Skalicky, Sam
> > > > >> > > > <sskalic@amazon.com.invalid
> > > > >> > > > > >
> > > > >> > > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > We probably need some way to track which CI runs ran
> for
> > > > which
> > > > >> > > commit
> > > > >> > > > > > too,
> > > > >> > > > > > > that way we can ensure that all CI runs ran on the
> > commit
> > > > that
> > > > >> > will
> > > > >> > > > be
> > > > >> > > > > > > merged. Maybe the bot can comment with the commit
> hash
> > > when
> > > > >> > users
> > > > >> > > > > > command
> > > > >> > > > > > > it to do something. Although since users can trigger
> > > > >> individual
> > > > >> > CI
> > > > >> > > > runs
> > > > >> > > > > > its
> > > > >> > > > > > > possible to have some commits run some CI runs but not
> > > > >> others. We
> > > > >> > > > need
> > > > >> > > > > > some
> > > > >> > > > > > > way to easily verify that the CI has executed all runs
> > on
> > > > the
> > > > >> > > commit
> > > > >> > > > > that
> > > > >> > > > > > > will be merged.
> > > > >> > > > > > >
> > > > >> > > > > > > Sam
> > > > >> > > > > > >
> > > > >> > > > > > > > On Mar 13, 2020, at 8:28 PM, Przemysław Trędak <
> > > > >> > > ptrendx@apache.org
> > > > >> > > > >
> > > > >> > > > > > > wrote:
> > > > >> > > > > > > >
> > > > >> > > > > > > > CAUTION: This email originated from outside of the
> > > > >> > organization.
> > > > >> > > Do
> > > > >> > > > > not
> > > > >> > > > > > > click links or open attachments unless you can confirm
> > the
> > > > >> sender
> > > > >> > > and
> > > > >> > > > > > know
> > > > >> > > > > > > the content is safe.
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > I personally like the idea of opt-in more than
> > opt-out:
> > > > >> > > > > > > > - ultimately PR author wants the PR to be merged so
> > they
> > > > (or
> > > > >> > > > > committer
> > > > >> > > > > > > reviewing the PR) will trigger the CI
> > > > >> > > > > > > > - if it is easy to trigger the PR via the bot
> command
> > > then
> > > > >> the
> > > > >> > > > amount
> > > > >> > > > > > of
> > > > >> > > > > > > work per PR should be less than with opt-out (since
> most
> > > of
> > > > >> the
> > > > >> > > > commits
> > > > >> > > > > > > should then be marked as [skip ci] or something
> similar
> > > > >> > > > > > > >
> > > > >> > > > > > > > +1 to the bot making a comment on each new PR with
> its
> > > > >> commands
> > > > >> > > > (and
> > > > >> > > > > > > also explaining, or at least giving links to the
> general
> > > PR
> > > > >> > process
> > > > >> > > > so
> > > > >> > > > > > new
> > > > >> > > > > > > PR authors are not lost). Maybe we could make the bot
> > > > >> recognize
> > > > >> > if
> > > > >> > > > the
> > > > >> > > > > PR
> > > > >> > > > > > > author is new or existing contributor and offer advice
> > > based
> > > > >> on
> > > > >> > > that?
> > > > >> > > > > > > >
> > > > >> > > > > > > > Thanks
> > > > >> > > > > > > > Przemek
> > > > >> > > > > > > >
> > > > >> > > > > > > > On 2020/03/13 22:06:58, Marco de Abreu <
> > > > >> > marco.g.abreu@gmail.com>
> > > > >> > > > > > wrote:
> > > > >> > > > > > > >> Hi,
> > > > >> > > > > > > >>
> > > > >> > > > > > > >> since it's no longer necessary to push a new commit
> > to
> > > > >> trigger
> > > > >> > > CI,
> > > > >> > > > > it
> > > > >> > > > > > > will
> > > > >> > > > > > > >> already reduce the costs. But to me, requiring an
> > > action
> > > > to
> > > > >> > > enable
> > > > >> > > > > CI
> > > > >> > > > > > > after
> > > > >> > > > > > > >> a PR has been created initially, is a no go. User
> can
> > > opt
> > > > >> out
> > > > >> > of
> > > > >> > > > CI,
> > > > >> > > > > > but
> > > > >> > > > > > > >> the default has to be CI being triggered
> > automatically
> > > > for
> > > > >> > every
> > > > >> > > > > > commit
> > > > >> > > > > > > >> unless specifically disabled by a participant. I'm
> > also
> > > > >> fine
> > > > >> > > with
> > > > >> > > > > > > >> triggering certain additional jobs (think about
> > > running a
> > > > >> > > nightly
> > > > >> > > > > job
> > > > >> > > > > > > upon
> > > > >> > > > > > > >> request for a PR) to require a manual step, but the
> > PR
> > > > >> > > validation
> > > > >> > > > > > > pipelines
> > > > >> > > > > > > >> have to run automatically. Every check that is
> marked
> > > as
> > > > >> > > > "Required"
> > > > >> > > > > in
> > > > >> > > > > > > >> GitHub has to be automatically kicked off.
> > > > >> > > > > > > >>
> > > > >> > > > > > > >> -Marco
> > > > >> > > > > > > >>
> > > > >> > > > > > > >> On Fri, Mar 13, 2020 at 9:50 PM Chaitanya Bapat <
> > > > >> > > > > chai.bapat@gmail.com
> > > > >> > > > > > >
> > > > >> > > > > > > >> wrote:
> > > > >> > > > > > > >>
> > > > >> > > > > > > >>> Firstly,
> > > > >> > > > > > > >>> Sorry I missed out on attaching the mail thread
> that
> > > was
> > > > >> sent
> > > > >> > > on
> > > > >> > > > > 12th
> > > > >> > > > > > > >>> February for notifying the community of the
> upcoming
> > > > >> changes
> > > > >> > to
> > > > >> > > > the
> > > > >> > > > > > > MXNet
> > > > >> > > > > > > >>> CI
> > > > >> > > > > > > >>> For reference :
> > > > >> > > > > > > >>>
> > > > >> > > > > > > >>>
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://lists.apache.org/thread.html/r09a6ab2803a996fc80e00fe39ed312fa4865e8805e08df847f1addad%40%3Cdev.mxnet.apache.org%3E
> > > > >> > > > > > > >>>
> > > > >> > > > > > > >>> Now to the questions,
> > > > >> > > > > > > >>>> Is it possible for re-triggering a single job to
> be
> > > > >> abused?
> > > > >> > > > > > > >>> @Tao In the case when a user re-triggers a single
> > job
> > > > >> > multiple
> > > > >> > > > > times,
> > > > >> > > > > > > that
> > > > >> > > > > > > >>> will be visible in the PR conversation thread. A
> > > > >> committer,
> > > > >> > > even
> > > > >> > > > > > after
> > > > >> > > > > > > he
> > > > >> > > > > > > >>> has approved the PR before, generally takes a look
> > at
> > > > the
> > > > >> > final
> > > > >> > > > > state
> > > > >> > > > > > > of
> > > > >> > > > > > > >>> the PR before merging. Would it be fair to assume
> > the
> > > > >> > committer
> > > > >> > > > > could
> > > > >> > > > > > > take
> > > > >> > > > > > > >>> the multiple re-trigger of a single job into
> account
> > > > >> before
> > > > >> > > > > merging?
> > > > >> > > > > > > The
> > > > >> > > > > > > >>> committer then has the option to invoke
> `@mxnet-bot
> > > run
> > > > ci
> > > > >> > > [all]
> > > > >> > > > `
> > > > >> > > > > to
> > > > >> > > > > > > >>> trigger the entire build pipeline one last to
> > counter
> > > > the
> > > > >> > > abuse.
> > > > >> > > > > This
> > > > >> > > > > > > is
> > > > >> > > > > > > >>> aligned with what @Leonard said.
> > > > >> > > > > > > >>>
> > > > >> > > > > > > >>> @Sandeep Thanks a lot for collecting and sharing
> > > > valuable
> > > > >> > data.
> > > > >> > > > I'd
> > > > >> > > > > > > concur
> > > > >> > > > > > > >>> with the opinion that given the existing things
> > > > >> committers &
> > > > >> > PR
> > > > >> > > > > > Authors
> > > > >> > > > > > > >>> take care of, invoking CI shouldn't be that big of
> > an
> > > > >> > > additional
> > > > >> > > > > > > burden.
> > > > >> > > > > > > >>>
> > > > >> > > > > > > >>> @Marco With the opt-out, the onus remains on the
> PR
> > > > >> Author.
> > > > >> > It
> > > > >> > > > > > doesn't
> > > > >> > > > > > > help
> > > > >> > > > > > > >>> reduce the resource usage. Hence, it was suggested
> > to
> > > > >> switch
> > > > >> > to
> > > > >> > > > > > > >>> opt-in. @Leo's suggestion for proactive commenting
> > on
> > > > the
> > > > >> > part
> > > > >> > > of
> > > > >> > > > > bot
> > > > >> > > > > > > makes
> > > > >> > > > > > > >>> sense and is doable.
> > > > >> > > > > > > >>>
> > > > >> > > > > > > >>> Default : opt-out and User initiated opt-in (with
> > > > >> addressing
> > > > >> > > > Leo's
> > > > >> > > > > > fix
> > > > >> > > > > > > for
> > > > >> > > > > > > >>> the usability issue you correctly pointed out )
> > > > >> > > > > > > >>> @Marco How does this sound to you?
> > > > >> > > > > > > >>>
> > > > >> > > > > > > >>> Again, thank you all for chiming in and voicing
> your
> > > > >> opinion.
> > > > >> > > > > > > Appreciate
> > > > >> > > > > > > >>> it.
> > > > >> > > > > > > >>> We can take ahead these discussions in today's
> demo
> > > > >> meeting.
> > > > >> > > > > [Design
> > > > >> > > > > > > Doc
> > > > >> > > > > > > >>> <
> > > > >> > >
> https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > > > >> > > > >]
> > > > >> > > > > > > [Demo
> > > > >> > > > > > > >>> Video <
> https://www.youtube.com/watch?v=gfOGwZId8aU
> > >]
> > > > >> > > > > > > >>>
> > > > >> > > > > > > >>> Thanks,
> > > > >> > > > > > > >>> Chai
> > > > >> > > > > > > >>>
> > > > >> > > > > > > >>> On Fri, 13 Mar 2020 at 12:34, Marco de Abreu <
> > > > >> > > > > > marco.g.abreu@gmail.com>
> > > > >> > > > > > > >>> wrote:
> > > > >> > > > > > > >>>
> > > > >> > > > > > > >>>> I'd recommend that the bot makes an initial
> comment
> > > > when
> > > > >> a
> > > > >> > PR
> > > > >> > > > gets
> > > > >> > > > > > > opened
> > > > >> > > > > > > >>>> and informs the users of its commands. It then
> > tells
> > > > the
> > > > >> > user
> > > > >> > > > the
> > > > >> > > > > > > commend
> > > > >> > > > > > > >>>> to opt out of CI.
> > > > >> > > > > > > >>>>
> > > > >> > > > > > > >>>> -Marco
> > > > >> > > > > > > >>>>
> > > > >> > > > > > > >>>> Lausen, Leonard <la...@amazon.com.invalid>
> > schrieb
> > > am
> > > > >> Fr.,
> > > > >> > > 13.
> > > > >> > > > > > März
> > > > >> > > > > > > >>> 2020,
> > > > >> > > > > > > >>>> 20:27:
> > > > >> > > > > > > >>>>
> > > > >> > > > > > > >>>>> On opt-out: People may be unaware of opt-out
> would
> > > not
> > > > >> use
> > > > >> > > it.
> > > > >> > > > > > There
> > > > >> > > > > > > is
> > > > >> > > > > > > >>>> no
> > > > >> > > > > > > >>>>> incentive to use opt-out, as the PR author
> doesn't
> > > pay
> > > > >> any
> > > > >> > > > money
> > > > >> > > > > > for
> > > > >> > > > > > > CI
> > > > >> > > > > > > >>>>> run.
> > > > >> > > > > > > >>>>>
> > > > >> > > > > > > >>>>> I agree with Marco though that opt-in alone may
> > > cause
> > > > >> > > usability
> > > > >> > > > > > > issues,
> > > > >> > > > > > > >>>> as
> > > > >> > > > > > > >>>>> contributors may not be aware of how to trigger
> > the
> > > > CI.
> > > > >> > > > > > > >>>>> One solution is that the bot proactively
> comments
> > on
> > > > >> the PR
> > > > >> > > and
> > > > >> > > > > > > reminds
> > > > >> > > > > > > >>>> the
> > > > >> > > > > > > >>>>> author to trigger running CI once the author
> deems
> > > the
> > > > >> PR
> > > > >> > > > ready.
> > > > >> > > > > > > >>>>>
> > > > >> > > > > > > >>>>> But even if we choose opt-out, the bot will
> still
> > > add
> > > > a
> > > > >> lot
> > > > >> > > of
> > > > >> > > > > > value,
> > > > >> > > > > > > >>> as
> > > > >> > > > > > > >>>> PR
> > > > >> > > > > > > >>>>> authors can retrigger single jobs that have
> failed
> > > due
> > > > >> to
> > > > >> > > > > > flakiness.
> > > > >> > > > > > > >>>>>
> > > > >> > > > > > > >>>>>> Is it possible for re-triggering a single job
> to
> > be
> > > > >> > abused?
> > > > >> > > > For
> > > > >> > > > > > > >>>> example,
> > > > >> > > > > > > >>>>>> the author spends two days re-triggering a
> flaky
> > > job
> > > > to
> > > > >> > make
> > > > >> > > > it
> > > > >> > > > > > > pass.
> > > > >> > > > > > > >>>>>
> > > > >> > > > > > > >>>>> Yes, this is possible. I suggest the committer
> who
> > > > >> likes to
> > > > >> > > > > merge a
> > > > >> > > > > > > PR
> > > > >> > > > > > > >>>>> needs to
> > > > >> > > > > > > >>>>> make a good judgement here if a PR is abusing
> the
> > > > >> feature,
> > > > >> > > and
> > > > >> > > > if
> > > > >> > > > > > so,
> > > > >> > > > > > > >>>>> retrigger
> > > > >> > > > > > > >>>>> all CI runs.
> > > > >> > > > > > > >>>>>
> > > > >> > > > > > > >>>>> Best regards
> > > > >> > > > > > > >>>>> Leonard
> > > > >> > > > > > > >>>>>
> > > > >> > > > > > > >>>>> On Fri, 2020-03-13 at 08:07 +0100, Marco de
> Abreu
> > > > wrote:
> > > > >> > > > > > > >>>>>> Thanks for the data Sandeep. In these cases it
> > > sounds
> > > > >> like
> > > > >> > > it
> > > > >> > > > > > would
> > > > >> > > > > > > >>>> have
> > > > >> > > > > > > >>>>>> rather been better when people explicitly
> turned
> > > off
> > > > >> CI in
> > > > >> > > > that
> > > > >> > > > > > > case.
> > > > >> > > > > > > >>>>>> What's the argument against an opt-out instead
> of
> > > an
> > > > >> > opt-in?
> > > > >> > > > > > > >>>>>>
> > > > >> > > > > > > >>>>>> My intention is that I consider it quite
> > cumbersome
> > > > to
> > > > >> > make
> > > > >> > > > it a
> > > > >> > > > > > > >>>>> *required*
> > > > >> > > > > > > >>>>>> step to always trigger CI manually, even if
> just
> > > > >> > submitting
> > > > >> > > a
> > > > >> > > > > > small
> > > > >> > > > > > > >>> PR.
> > > > >> > > > > > > >>>>> I'd
> > > > >> > > > > > > >>>>>> rather see people explicitly turning off CI if
> > they
> > > > >> > wouldn't
> > > > >> > > > > like
> > > > >> > > > > > to
> > > > >> > > > > > > >>>> use
> > > > >> > > > > > > >>>>> it
> > > > >> > > > > > > >>>>>> - and there's also the "draft" stage for a PR
> > which
> > > > >> some
> > > > >> > > > > > > contributors
> > > > >> > > > > > > >>>> are
> > > > >> > > > > > > >>>>>> using.
> > > > >> > > > > > > >>>>>>
> > > > >> > > > > > > >>>>>> With regards to WIP and do not review: I think
> > > these
> > > > >> are
> > > > >> > use
> > > > >> > > > > cases
> > > > >> > > > > > > >>>> where
> > > > >> > > > > > > >>>>>> you want CI feedback, as otherwise you wouldn't
> > > have
> > > > >> > opened
> > > > >> > > > the
> > > > >> > > > > > PR.
> > > > >> > > > > > > >>> If
> > > > >> > > > > > > >>>>> you
> > > > >> > > > > > > >>>>>> don't want human feedback and neither machine
> > > > feedback,
> > > > >> > why
> > > > >> > > > open
> > > > >> > > > > > the
> > > > >> > > > > > > >>> PR
> > > > >> > > > > > > >>>>> at
> > > > >> > > > > > > >>>>>> all?
> > > > >> > > > > > > >>>>>>
> > > > >> > > > > > > >>>>>> -Marco
> > > > >> > > > > > > >>>>>>
> > > > >> > > > > > > >>>>>>
> > > > >> > > > > > > >>>>>> sandeep krishnamurthy <
> > sandeep.krishna98@gmail.com
> > > >
> > > > >> > schrieb
> > > > >> > > > am
> > > > >> > > > > > Fr.,
> > > > >> > > > > > > >>>> 13.
> > > > >> > > > > > > >>>>>> März 2020, 05:24:
> > > > >> > > > > > > >>>>>>
> > > > >> > > > > > > >>>>>>> I tried to gather some data for us to discuss
> > this
> > > > >> topic
> > > > >> > in
> > > > >> > > > > this
> > > > >> > > > > > > >>>>> thread. I
> > > > >> > > > > > > >>>>>>> tried to count number of un-necessary builds
> by
> > > > >> looking
> > > > >> > at
> > > > >> > > > most
> > > > >> > > > > > > >>>> recent
> > > > >> > > > > > > >>>>> (as
> > > > >> > > > > > > >>>>>>> of 12, March 9 PM PST) 50 PRs merged to master
> > and
> > > > 50
> > > > >> > PRs.
> > > > >> > > > > > > >>>>>>> Identifying un-necessary builds is bit
> > > subjective. I
> > > > >> > tried
> > > > >> > > to
> > > > >> > > > > be
> > > > >> > > > > > > >>> more
> > > > >> > > > > > > >>>>>>> conservative where I didn't count a build as
> > > > >> un-necessary
> > > > >> > > if
> > > > >> > > > I
> > > > >> > > > > > was
> > > > >> > > > > > > >>> in
> > > > >> > > > > > > >>>>>>> doubt. Hence, I was not able to automate, but
> I
> > > made
> > > > >> an
> > > > >> > > > effort
> > > > >> > > > > to
> > > > >> > > > > > > >>> go
> > > > >> > > > > > > >>>>>>> through PRs manually and use below criteria to
> > > > >> identify
> > > > >> > > > > > > >>> un-necessary
> > > > >> > > > > > > >>>>>>> commits triggering the builds.
> > > > >> > > > > > > >>>>>>>
> > > > >> > > > > > > >>>>>>> 1. Explicitly marked as WIP / do not review
> > PR
> > > > >> > > > > > > >>>>>>> 2. Incremental WIP commit and finally
> > > commenting a
> > > > >> > commit
> > > > >> > > > > > > >>> “trigger
> > > > >> > > > > > > >>>>> CI”
> > > > >> > > > > > > >>>>>>> 3. Multiple commits to address all comments
> > from
> > > > >> single
> > > > >> > > > > review.
> > > > >> > > > > > > >>>>> This is
> > > > >> > > > > > > >>>>>>> assuming we see a comment, address them,
> > commit,
> > > > >> next
> > > > >> > the
> > > > >> > > > > > > >>>> following
> > > > >> > > > > > > >>>>>>> comment
> > > > >> > > > > > > >>>>>>> 4. Sequence of documentation only changes
> > > > >> > > > > > > >>>>>>>
> > > > >> > > > > > > >>>>>>> I found there were around 42 avoidable builds
> > from
> > > > >> most
> > > > >> > > > recent
> > > > >> > > > > 50
> > > > >> > > > > > > >>>>> merged
> > > > >> > > > > > > >>>>>>> PRs and around 86 builds from recent 50 open
> > PRs.
> > > > >> > > > > > > >>>>>>>
> > > > >> > > > > > > >>>>>>> I synced up with other contributors (Joe
> Evans,
> > > > Chai)
> > > > >> > from
> > > > >> > > > > Amazon
> > > > >> > > > > > > >>> who
> > > > >> > > > > > > >>>>> is
> > > > >> > > > > > > >>>>>>> contributing to MXNet CI system. I was told
> that
> > > on
> > > > an
> > > > >> > > > average
> > > > >> > > > > it
> > > > >> > > > > > > >>>> costs
> > > > >> > > > > > > >>>>>>> around $84 per build and on an average 6
> commits
> > > per
> > > > >> > merged
> > > > >> > > > PR
> > > > >> > > > > > (for
> > > > >> > > > > > > >>>>> year
> > > > >> > > > > > > >>>>>>> 2019). Going by that, it is approximately 1/6
> > > builds
> > > > >> are
> > > > >> > > > > > avoidable.
> > > > >> > > > > > > >>>>> [100 /
> > > > >> > > > > > > >>>>>>> 300 + 300 ]
> > > > >> > > > > > > >>>>>>>
> > > > >> > > > > > > >>>>>>>
> > > > >> > > > > > > >>>>>>> Usability should be top most priority. But,
> > since
> > > > >> either
> > > > >> > a
> > > > >> > > > > > reviewer
> > > > >> > > > > > > >>>> or
> > > > >> > > > > > > >>>>> pr
> > > > >> > > > > > > >>>>>>> author can trigger the bot, is it really a
> > hurdle
> > > > for
> > > > >> pr
> > > > >> > > > author
> > > > >> > > > > > or
> > > > >> > > > > > > >>>>> reviewer
> > > > >> > > > > > > >>>>>>> to call a bot to trigger CI? Given that PR
> > author
> > > > and
> > > > >> > > > reviewer
> > > > >> > > > > is
> > > > >> > > > > > > >>>>> already
> > > > >> > > > > > > >>>>>>> actively commenting various details such as -
> PR
> > > > >> > > description,
> > > > >> > > > > > > >>> review
> > > > >> > > > > > > >>>>>>> comments and responses, adding labels etc.
> > > > >> > > > > > > >>>>>>>
> > > > >> > > > > > > >>>>>>>
> > > > >> > > > > > > >>>>>>> Me too curious to know the behavior for Tao's
> > > above
> > > > >> use
> > > > >> > > case.
> > > > >> > > > > > > >>>>>>>
> > > > >> > > > > > > >>>>>>>
> > > > >> > > > > > > >>>>>>> Best,
> > > > >> > > > > > > >>>>>>>
> > > > >> > > > > > > >>>>>>> Sandeep
> > > > >> > > > > > > >>>>>>>
> > > > >> > > > > > > >>>>>>> On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <
> > > > >> > mutouorz@gmail.com
> > > > >> > > >
> > > > >> > > > > > wrote:
> > > > >> > > > > > > >>>>>>>
> > > > >> > > > > > > >>>>>>>> Is it possible for re-triggering a single job
> > to
> > > be
> > > > >> > > abused?
> > > > >> > > > > For
> > > > >> > > > > > > >>>>> example,
> > > > >> > > > > > > >>>>>>>> the author spends two days re-triggering a
> > flaky
> > > > job
> > > > >> to
> > > > >> > > make
> > > > >> > > > > it
> > > > >> > > > > > > >>>>> pass. But
> > > > >> > > > > > > >>>>>>>> other jobs which have passed the validation
> may
> > > be
> > > > >> > broken
> > > > >> > > by
> > > > >> > > > > > > >>> other
> > > > >> > > > > > > >>>>>>> commits
> > > > >> > > > > > > >>>>>>>> during the two day without being noticed. And
> > > > finally
> > > > >> > the
> > > > >> > > PR
> > > > >> > > > > is
> > > > >> > > > > > > >>>>> merged
> > > > >> > > > > > > >>>>>>> with
> > > > >> > > > > > > >>>>>>>> underlying problems.
> > > > >> > > > > > > >>>>>>>>
> > > > >> > > > > > > >>>>>>>> On Fri, Mar 13, 2020 at 6:19 AM Marco de
> Abreu
> > <
> > > > >> > > > > > > >>>>> marco.g.abreu@gmail.com>
> > > > >> > > > > > > >>>>>>>> wrote:
> > > > >> > > > > > > >>>>>>>>
> > > > >> > > > > > > >>>>>>>>> In the end it only comes down to money,
> > > > considering
> > > > >> > that
> > > > >> > > > the
> > > > >> > > > > > > >>>>> system is
> > > > >> > > > > > > >>>>>>>> auto
> > > > >> > > > > > > >>>>>>>>> scaling, making the execution time constant.
> > > > >> > > > > > > >>>>>>>>>
> > > > >> > > > > > > >>>>>>>>> If we're trading money for usability, I
> > > certainly
> > > > >> would
> > > > >> > > > > prefer
> > > > >> > > > > > > >>>>>>> usability.
> > > > >> > > > > > > >>>>>>>>> I'd rather recommend to spend time on
> > > > parallelizing
> > > > >> > test
> > > > >> > > > > > > >>>> execution
> > > > >> > > > > > > >>>>> or
> > > > >> > > > > > > >>>>>>>>> getting rid of integration tests in the PR
> > stage
> > > > >> > instead
> > > > >> > > > > > > >>> reducing
> > > > >> > > > > > > >>>>> the
> > > > >> > > > > > > >>>>>>>> costs
> > > > >> > > > > > > >>>>>>>>> by making people not use it. But taking a
> step
> > > > back
> > > > >> to
> > > > >> > > > > > > >>> requiring
> > > > >> > > > > > > >>>>> people
> > > > >> > > > > > > >>>>>>>> to
> > > > >> > > > > > > >>>>>>>>> manually trigger CI again doesn't feel
> right.
> > > > >> > > > > > > >>>>>>>>>
> > > > >> > > > > > > >>>>>>>>> I'm happy to see that bot deployed, but I do
> > not
> > > > >> agree
> > > > >> > > with
> > > > >> > > > > > > >>>>> removing
> > > > >> > > > > > > >>>>>>> the
> > > > >> > > > > > > >>>>>>>>> auto trigger functionality for new commits.
> > > > >> > > > > > > >>>>>>>>>
> > > > >> > > > > > > >>>>>>>>> -Marco
> > > > >> > > > > > > >>>>>>>>>
> > > > >> > > > > > > >>>>>>>>> Chaitanya Bapat <ch...@gmail.com>
> > schrieb
> > > am
> > > > >> Do.,
> > > > >> > > 12.
> > > > >> > > > > > > >>> März
> > > > >> > > > > > > >>>>> 2020,
> > > > >> > > > > > > >>>>>>>>> 22:47:
> > > > >> > > > > > > >>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>> @Marco Thanks for pointing that out.
> > > > >> > > > > > > >>>>>>>>>> Tomorrow i.e. Friday, March 13, 2020 at
> 3:00
> > > PM -
> > > > >> 3:30
> > > > >> > > PM
> > > > >> > > > in
> > > > >> > > > > > > >>>>>>>> (UTC-08:00)
> > > > >> > > > > > > >>>>>>>>>> Pacific Time (US & Canada).
> > > > >> > > > > > > >>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>> When do we expect this bot to be deployed?
> > > > >> > > > > > > >>>>>>>>>> @Lin If all goes well in the next week I
> can
> > > > >> deploy it
> > > > >> > > to
> > > > >> > > > > > > >>>> public
> > > > >> > > > > > > >>>>>>> Apache
> > > > >> > > > > > > >>>>>>>>>> (provided I get permissions from Apache
> > Infra)
> > > > >> > > > > > > >>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>> @Marco Thanks for your feedback.
> > > > >> > > > > > > >>>>>>>>>>> CI system has to support the community
> > without
> > > > >> > > requiring
> > > > >> > > > > > > >>>>> people to
> > > > >> > > > > > > >>>>>>>>>> constantly shepherd every single run
> > > > >> > > > > > > >>>>>>>>>> We have data for the number of times CI was
> > > > >> triggered
> > > > >> > > > > > > >>>>> unnecessarily
> > > > >> > > > > > > >>>>>>>> which
> > > > >> > > > > > > >>>>>>>>>> includes
> > > > >> > > > > > > >>>>>>>>>> - Entire build triggered instead of
> specific
> > > > build
> > > > >> > > > > > > >>>>>>>>>> - CI triggered when PR is still work in
> > > progress
> > > > or
> > > > >> > not
> > > > >> > > > yet
> > > > >> > > > > > > >>>> ready
> > > > >> > > > > > > >>>>>>> (say
> > > > >> > > > > > > >>>>>>>> -
> > > > >> > > > > > > >>>>>>>>>> intermediate commits)
> > > > >> > > > > > > >>>>>>>>>> At the end its a trade-off
> > > > >> > > > > > > >>>>>>>>>> Money, Resources, Time to build for each
> and
> > > > every
> > > > >> > > commit
> > > > >> > > > vs
> > > > >> > > > > > > >>>>> Pain of
> > > > >> > > > > > > >>>>>>>>>> triggering builds
> > > > >> > > > > > > >>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>> Scan trigger plugin would poll SCM. Can we
> > use
> > > > >> plugin
> > > > >> > > at
> > > > >> > > > > > > >>>>> scale?
> > > > >> > > > > > > >>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>> 1. I haven't tested it on scale. But I
> think
> > > with
> > > > >> the
> > > > >> > > > > current
> > > > >> > > > > > > >>>>> scale
> > > > >> > > > > > > >>>>>>> of
> > > > >> > > > > > > >>>>>>>>>> MXNet repo (191 open PRs i.e. checking for
> > > > changes
> > > > >> to
> > > > >> > > 191
> > > > >> > > > > > > >>>>> branches -
> > > > >> > > > > > > >>>>>>> It
> > > > >> > > > > > > >>>>>>>>>> should be manageable)
> > > > >> > > > > > > >>>>>>>>>> 2. What's the purpose of the plugin? tldr;
> > > Branch
> > > > >> > > > discovery
> > > > >> > > > > > > >>> or
> > > > >> > > > > > > >>>>> branch
> > > > >> > > > > > > >>>>>>>>>> indexing.
> > > > >> > > > > > > >>>>>>>>>> Scan trigger plugin comes into the picture
> > only
> > > > >> once
> > > > >> > per
> > > > >> > > > PR
> > > > >> > > > > > > >>> per
> > > > >> > > > > > > >>>>> job
> > > > >> > > > > > > >>>>>>>>> (i.e. 8
> > > > >> > > > > > > >>>>>>>>>> times per PR for 8 jobs). It is basically
> > done
> > > > >> when a
> > > > >> > > new
> > > > >> > > > PR
> > > > >> > > > > > > >>> is
> > > > >> > > > > > > >>>>> made
> > > > >> > > > > > > >>>>>>>> and
> > > > >> > > > > > > >>>>>>>>>> the job (say unix-cpu hasn't discovered the
> > new
> > > > PR
> > > > >> > > branch
> > > > >> > > > > > > >>> yet).
> > > > >> > > > > > > >>>>>>> That's
> > > > >> > > > > > > >>>>>>>>> it.
> > > > >> > > > > > > >>>>>>>>>> So it shouldn't be a problem for public
> MXNet
> > > > repo.
> > > > >> > > > > > > >>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>> Thanks,
> > > > >> > > > > > > >>>>>>>>>> Chai
> > > > >> > > > > > > >>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>> On Thu, 12 Mar 2020 at 14:22, Marco de
> Abreu
> > <
> > > > >> > > > > > > >>>>>>> marco.g.abreu@gmail.com>
> > > > >> > > > > > > >>>>>>>>>> wrote:
> > > > >> > > > > > > >>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>> Btw you forgot to set a date and time for
> > the
> > > > >> metting
> > > > >> > > > > > > >>>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>> On Thu, Mar 12, 2020 at 10:18 PM Marco de
> > > Abreu
> > > > <
> > > > >> > > > > > > >>>>>>>>> marco.g.abreu@gmail.com
> > > > >> > > > > > > >>>>>>>>>>> wrote:
> > > > >> > > > > > > >>>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>>> Thanks Chai, I generally like the idea of
> > the
> > > > >> bot.
> > > > >> > But
> > > > >> > > > > > > >>> I'm
> > > > >> > > > > > > >>>>> not a
> > > > >> > > > > > > >>>>>>>>>>> supporter
> > > > >> > > > > > > >>>>>>>>>>>> of the idea to disable any automatic
> > > triggering
> > > > >> > > > > > > >>> (disabling
> > > > >> > > > > > > >>>>> the
> > > > >> > > > > > > >>>>>>>>> webhook
> > > > >> > > > > > > >>>>>>>>>> is
> > > > >> > > > > > > >>>>>>>>>>>> also not an option, considering that this
> > > will
> > > > >> > disable
> > > > >> > > > > > > >>>> master
> > > > >> > > > > > > >>>>>>>>>> triggers).
> > > > >> > > > > > > >>>>>>>>>>>> The CI system has to support the
> community
> > > > >> without
> > > > >> > > > > > > >>>> requiring
> > > > >> > > > > > > >>>>>>> people
> > > > >> > > > > > > >>>>>>>>> to
> > > > >> > > > > > > >>>>>>>>>>>> constantly shepherd every single run.
> > > Disabling
> > > > >> > > > automatic
> > > > >> > > > > > > >>>>>>>> triggering
> > > > >> > > > > > > >>>>>>>>>>> seems
> > > > >> > > > > > > >>>>>>>>>>>> like a step back to me.
> > > > >> > > > > > > >>>>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>>> Instead, I'd recommend that CI gets
> > triggered
> > > > >> upon
> > > > >> > > every
> > > > >> > > > > > > >>>>> commit
> > > > >> > > > > > > >>>>>>> as
> > > > >> > > > > > > >>>>>>>>>> usual,
> > > > >> > > > > > > >>>>>>>>>>>> but people have the possibility to call a
> > > > >> "command"
> > > > >> > > > (i.e.
> > > > >> > > > > > > >>>>> make a
> > > > >> > > > > > > >>>>>>>>>> message
> > > > >> > > > > > > >>>>>>>>>>>> which results in the bot setting a label)
> > to
> > > > >> disable
> > > > >> > > CI
> > > > >> > > > > > > >>>> until
> > > > >> > > > > > > >>>>>>> they
> > > > >> > > > > > > >>>>>>>>>> revoke
> > > > >> > > > > > > >>>>>>>>>>>> it. But the standard should still be
> that a
> > > new
> > > > >> > commit
> > > > >> > > > > > > >>>>> triggers a
> > > > >> > > > > > > >>>>>>>> new
> > > > >> > > > > > > >>>>>>>>>> CI
> > > > >> > > > > > > >>>>>>>>>>>> run.
> > > > >> > > > > > > >>>>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>>>
> > > > >> > > > > > > >>>>
> > > > >> > https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
> > > > >> > > > > > > >>>>>>>> seems
> > > > >> > > > > > > >>>>>>>>>> like
> > > > >> > > > > > > >>>>>>>>>>>> this would poll SCM. This will incur high
> > > quota
> > > > >> > > > > > > >>>>> restrictions. Are
> > > > >> > > > > > > >>>>>>>> you
> > > > >> > > > > > > >>>>>>>>>>> sure
> > > > >> > > > > > > >>>>>>>>>>>> that you can use that plugin at scale?
> > > > >> > > > > > > >>>>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>>> -Marco
> > > > >> > > > > > > >>>>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>>> On Thu, Mar 12, 2020 at 10:04 PM Lin
> Yuan <
> > > > >> > > > > > > >>>>> apeforest@gmail.com>
> > > > >> > > > > > > >>>>>>>>> wrote:
> > > > >> > > > > > > >>>>>>>>>>>>> Chai,
> > > > >> > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>>>> Awesome work. When do we expect this bot
> > to
> > > be
> > > > >> > > > > > > >>> deployed?
> > > > >> > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>>>> Best,
> > > > >> > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>>>> Lin
> > > > >> > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>>>> On Thu, Mar 12, 2020 at 2:00 PM
> Chaitanya
> > > > Bapat
> > > > >> <
> > > > >> > > > > > > >>>>>>>>> chai.bapat@gmail.com
> > > > >> > > > > > > >>>>>>>>>>>>> wrote:
> > > > >> > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>>>>> Hello MXNet community,
> > > > >> > > > > > > >>>>>>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>>>>> I have built an MXNet Bot <
> > > > >> > > > > > > >>>> https://github.com/mxnet-bot>
> > > > >> > > > > > > >>>>> that
> > > > >> > > > > > > >>>>>>>>>> allows
> > > > >> > > > > > > >>>>>>>>>>> PR
> > > > >> > > > > > > >>>>>>>>>>>>>> Authors, Committers and Jenkins Admins
> to
> > > > >> trigger
> > > > >> > CI
> > > > >> > > > > > > >>>>> manually.
> > > > >> > > > > > > >>>>>>>>>>>>>> It handles 2 problems
> > > > >> > > > > > > >>>>>>>>>>>>>> 1. Manual CI trigger instead of
> existing
> > > > >> automated
> > > > >> > > CI
> > > > >> > > > > > > >>>>> trigger
> > > > >> > > > > > > >>>>>>>>>>>>>> 2. Gives permissions to PR Authors (in
> > > > >> addition to
> > > > >> > > > > > > >>>> MXNet
> > > > >> > > > > > > >>>>>>>>> Committers
> > > > >> > > > > > > >>>>>>>>>>> and
> > > > >> > > > > > > >>>>>>>>>>>>>> Jenkins Admins)
> > > > >> > > > > > > >>>>>>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>>>>> Design Doc :
> > > > >> > > > > > > >>>>>>>>>>>>>>
> > > > >> > > > > > > >>>>>>>
> > > > >> > > >
> > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > > > >> > > > > > > >>>>>>>>>>>>>> I urge you all to attend the
> > demonstration
> > > > >> meeting
> > > > >> > > > > > > >>> and
> > > > >> > > > > > > >>>>> lend
> > > > >> > > > > > > >>>>>>> your
> > > > >> > > > > > > >>>>>>>>>> views
> > > > >> > > > > > > >>>>>>>>>>>>> on
> > > > >> > > > > > > >>>>>>>>>>>>>> the same.
> > > > >> > > > > > > >>>>>>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>>>>> Thank you,
> > > > >> > > > > > > >>>>>>>>>>>>>> Chai
> > > > >> > > > > > > >>>>>>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>>>>> *Meeting Details*:
> > > > >> > > > > > > >>>>>>>>>>>>>> ==============Conference Bridge
> > > > >> > > > > > > >>>> Information==============
> > > > >> > > > > > > >>>>>>>>>>>>>> You have been invited to an online
> > meeting,
> > > > >> > powered
> > > > >> > > > > > > >>> by
> > > > >> > > > > > > >>>>> Amazon
> > > > >> > > > > > > >>>>>>>>> Chime.
> > > > >> > > > > > > >>>>>>>>>>>>>> *Chime meeting ID*: *9272158344*
> > > > >> > > > > > > >>>>>>>>>>>>>> Join via Chime clients (manually):
> Select
> > > > >> > 'Meetings
> > > > >> > > >
> > > > >> > > > > > > >>>>> Join a
> > > > >> > > > > > > >>>>>>>>>> Meeting',
> > > > >> > > > > > > >>>>>>>>>>>>> and
> > > > >> > > > > > > >>>>>>>>>>>>>> enter 9272158344
> > > > >> > > > > > > >>>>>>>>>>>>>> Join via Chime clients (auto-call): If
> > you
> > > > >> invite
> > > > >> > > > > > > >>>>> auto-call as
> > > > >> > > > > > > >>>>>>>>>>> attendee,
> > > > >> > > > > > > >>>>>>>>>>>>>> Chime will call you when the meeting
> > > starts,
> > > > >> > select
> > > > >> > > > > > > >>>>> 'Answer'
> > > > >> > > > > > > >>>>>>>>>>>>>> *Join via browser screen share*:
> > > > >> > > > > > > >>>>> https://chime.aws/9272158344
> > > > >> > > > > > > >>>>>>>>>>>>>> *Join via phone* (US):
> > > > >> > +1-929-432-4463,,,9272158344#
> > > > >> > > > > > > >>>>>>>>>>>>>> *Join via phone (US toll-free)*:
> > > > >> > > > > > > >>>>> +1-855-552-4463,,,9272158344#
> > > > >> > > > > > > >>>>>>>>>>>>>> International dial-in:
> > > > >> > > > > > > >>>> https://chime.aws/dialinnumbers/
> > > > >> > > > > > > >>>>>>>>>>>>>> In-room video system: Ext: 62000,
> Meeting
> > > > PIN:
> > > > >> > > > > > > >>>>> 9272158344#
> > > > >> > > > > > > >>>>>>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>>>>> --
> > > > >> > > > > > > >>>>>>>>>>>>>> *Chaitanya Prakash Bapat*
> > > > >> > > > > > > >>>>>>>>>>>>>> *+1 (973) 953-6299*
> > > > >> > > > > > > >>>>>>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>>>>> [image:
> > > > >> https://www.linkedin.com//in/chaibapat25]
> > > > >> > > > > > > >>>>>>>>>>>>>> <https://github.com/ChaiBapchya
> >[image:
> > > > >> > > > > > > >>>>>>>>>>>>> https://www.facebook.com/chaibapat
> > > > >> > > > > > > >>>>>>>>>>>>>> ]
> > > > >> > > > > > > >>>>>>>>>>>>>> <https://www.facebook.com/chaibapchya
> > > > >[image:
> > > > >> > > > > > > >>>>>>>>>>>>>> https://twitter.com/ChaiBapchya] <
> > > > >> > > > > > > >>>>>>>> https://twitter.com/ChaiBapchya
> > > > >> > > > > > > >>>>>>>>>>>>>> [image:
> > > > >> > > > > > > >>>>>>>>>>>>>>
> https://www.linkedin.com//in/chaibapat25
> > ]
> > > > >> > > > > > > >>>>>>>>>>>>>> <
> > https://www.linkedin.com//in/chaibapchya/
> > > >
> > > > >> > > > > > > >>>>>>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>> --
> > > > >> > > > > > > >>>>>>>>>> *Chaitanya Prakash Bapat*
> > > > >> > > > > > > >>>>>>>>>> *+1 (973) 953-6299*
> > > > >> > > > > > > >>>>>>>>>>
> > > > >> > > > > > > >>>>>>>>>> [image:
> > > https://www.linkedin.com//in/chaibapat25
> > > > ]
> > > > >> > > > > > > >>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> > > > >> > > > > > > >>>>>>>>> https://www.facebook.com/chaibapat
> > > > >> > > > > > > >>>>>>>>>> ]
> > > > >> > > > > > > >>>>>>>>>> <https://www.facebook.com/chaibapchya
> > >[image:
> > > > >> > > > > > > >>>>>>>>>> https://twitter.com/ChaiBapchya] <
> > > > >> > > > > > > >>>>> https://twitter.com/ChaiBapchya
> > > > >> > > > > > > >>>>>>>>>> [image:
> > > > >> > > > > > > >>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> > > > >> > > > > > > >>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/
> >
> > > > >> > > > > > > >>>>>>>>>>
> > > > >> > > > > > > >>>>>>>
> > > > >> > > > > > > >>>>>>> --
> > > > >> > > > > > > >>>>>>> Sandeep Krishnamurthy
> > > > >> > > > > > > >>>>>>>
> > > > >> > > > > > > >>>>>
> > > > >> > > > > > > >>>>
> > > > >> > > > > > > >>>
> > > > >> > > > > > > >>>
> > > > >> > > > > > > >>> --
> > > > >> > > > > > > >>> *Chaitanya Prakash Bapat*
> > > > >> > > > > > > >>> *+1 (973) 953-6299*
> > > > >> > > > > > > >>>
> > > > >> > > > > > > >>> [image: https://www.linkedin.com//in/chaibapat25]
> > > > >> > > > > > > >>> <https://github.com/ChaiBapchya>[image:
> > > > >> > > > > > > https://www.facebook.com/chaibapat
> > > > >> > > > > > > >>> ]
> > > > >> > > > > > > >>> <https://www.facebook.com/chaibapchya>[image:
> > > > >> > > > > > > >>> https://twitter.com/ChaiBapchya] <
> > > > >> > > > https://twitter.com/ChaiBapchya
> > > > >> > > > > > > >[image:
> > > > >> > > > > > > >>> https://www.linkedin.com//in/chaibapat25]
> > > > >> > > > > > > >>> <https://www.linkedin.com//in/chaibapchya/>
> > > > >> > > > > > > >>>
> > > > >> > > > > > > >>
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > --
> > > > >> > > > > *Chaitanya Prakash Bapat*
> > > > >> > > > > *+1 (973) 953-6299*
> > > > >> > > > >
> > > > >> > > > > [image: https://www.linkedin.com//in/chaibapat25]
> > > > >> > > > > <https://github.com/ChaiBapchya>[image:
> > > > >> > > > https://www.facebook.com/chaibapat
> > > > >> > > > > ]
> > > > >> > > > > <https://www.facebook.com/chaibapchya>[image:
> > > > >> > > > > https://twitter.com/ChaiBapchya] <
> > > > https://twitter.com/ChaiBapchya
> > > > >> > > > >[image:
> > > > >> > > > > https://www.linkedin.com//in/chaibapat25]
> > > > >> > > > > <https://www.linkedin.com//in/chaibapchya/>
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> > >
> > > > >> > > --
> > > > >> > > *Chaitanya Prakash Bapat*
> > > > >> > > *+1 (973) 953-6299*
> > > > >> > >
> > > > >> > > [image: https://www.linkedin.com//in/chaibapat25]
> > > > >> > > <https://github.com/ChaiBapchya>[image:
> > > > >> > https://www.facebook.com/chaibapat
> > > > >> > > ]
> > > > >> > > <https://www.facebook.com/chaibapchya>[image:
> > > > >> > > https://twitter.com/ChaiBapchya] <
> > https://twitter.com/ChaiBapchya
> > > > >> > >[image:
> > > > >> > > https://www.linkedin.com//in/chaibapat25]
> > > > >> > > <https://www.linkedin.com//in/chaibapchya/>
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >> --
> > > > >> *Chaitanya Prakash Bapat*
> > > > >> *+1 (973) 953-6299*
> > > > >>
> > > > >> [image: https://www.linkedin.com//in/chaibapat25]
> > > > >> <https://github.com/ChaiBapchya>[image:
> > > > >> https://www.facebook.com/chaibapat]
> > > > >> <https://www.facebook.com/chaibapchya>[image:
> > > > >> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> > > > >[image:
> > > > >> https://www.linkedin.com//in/chaibapat25]
> > > > >> <https://www.linkedin.com//in/chaibapchya/>
> > > > >>
> > > > >
> > > >
> > >
> > >
> > > --
> > > *Chaitanya Prakash Bapat*
> > > *+1 (973) 953-6299*
> > >
> > > [image: https://www.linkedin.com//in/chaibapat25]
> > > <https://github.com/ChaiBapchya>[image:
> > https://www.facebook.com/chaibapat
> > > ]
> > > <https://www.facebook.com/chaibapchya>[image:
> > > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> > >[image:
> > > https://www.linkedin.com//in/chaibapat25]
> > > <https://www.linkedin.com//in/chaibapchya/>
> > >
> >
>
>
> --
> Sandeep Krishnamurthy
>
--
*Chaitanya Prakash Bapat*
*+1 (973) 953-6299*
[image: https://www.linkedin.com//in/chaibapat25]
<https://github.com/ChaiBapchya>[image: https://www.facebook.com/chaibapat]
<https://www.facebook.com/chaibapchya>[image:
https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
https://www.linkedin.com//in/chaibapat25]
<https://www.linkedin.com//in/chaibapchya/>
Re: MXNet Bot Demo
Posted by sandeep krishnamurthy <sa...@gmail.com>.
Thank you Chaitanya and Marco for helping the MXNet community.
On Mon, Mar 23, 2020 at 12:56 PM Marco de Abreu <ma...@gmail.com>
wrote:
> Sure, already done.
>
> -Marco
>
> On Mon, Mar 23, 2020 at 8:53 PM Chaitanya Bapat <ch...@gmail.com>
> wrote:
>
> > Hello,
> > Update: Apache Infra Ticket for MXNet Bot
> > Thanks once again, Marco for opening the ticket. But turns out, Apache
> > Infra folks closed it stating: "Security concerns around allowing unknown
> > person to submit PR and run our hardware". Furthermore, it goes onto
> state
> > that bot circumvents the dependence on Jenkins Admins which is like
> solving
> > a problem that doesn't exist.
> >
> > I sense there is some confusion in the communication (maybe on my part).
> It
> > turns out the security concerns aren't actually correct.
> >
> > 1. Unknown person can submit a PR (before & after bot proposal), and run
> > our hardware (pre as well as post bot).
> > 2. Code should be reviewed by somebody with an ICLA on file. This doesn't
> > change either. Prior to merging a PR, code has to be approved by a
> > committer just like before.
> > Overall it looks like the job of the bot isn't clear to folks in Apache
> > Infra. Bot simply is a means for triggering CI (which could be done
> > manually by Log In to Jenkins -> PR -> Job -> Build) and doesn't quite
> > tweak with merging procedure. Yes, only addition is now unknown person
> (PR
> > Author) can trigger CI with a message (but that was possible anyway by
> > pushing a commit. Bot just prevents users from pushing empty commits and
> > building entire suite).
> >
> > As can be seen from last 10 open PRs as of Monday 23rd March, 12pm PT
> most
> > PRs fail on 1/2 jobs. In such a scenario, the proposed MXNet bot would
> come
> > in handy for just invoking CI on that specific build (instead of a
> > non-committer PR Author to push empty commit : hurting on the resource,
> > time & cost considerations apart from undesirable dev experience)
> >
> > @Marco Since I am a non-committer, I guess these 2 clarifications need to
> > be conveyed to the Apache Infra by someone with Committer access.
> >
> > What do you think?
> >
> > Thanks,
> > Chai
> >
> > On Sat, 21 Mar 2020 at 16:08, Marco de Abreu <ma...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > the ticket has been created:
> > > https://issues.apache.org/jira/browse/INFRA-20005
> > >
> > > Best regards,
> > > Marco
> > >
> > > On Thu, Mar 19, 2020 at 11:49 PM Marco de Abreu <
> marco.g.abreu@gmail.com
> > >
> > > wrote:
> > >
> > > > Sounds like a good plan!
> > > >
> > > > Please send me the URL (please make sure it's backed by DNS and not
> > just
> > > > the gateway URL) of the webhook handler, GitHub events you're
> > interested
> > > in
> > > > and the shared secret in a private email to my personal email
> address.
> > I
> > > > will then create the ticket with Apache infra.
> > > >
> > > > -Marco
> > > >
> > > > Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 19. März
> 2020,
> > > > 23:07:
> > > >
> > > >> @Marco Alright, it makes total sense to test out the Bot feature
> > > alongside
> > > >> auto-trigger as a transition.
> > > >>
> > > >> Path Forward:
> > > >> 1. Setup MXNet Bot on apache/incubator-mxnet repo (GitHub WebHook
> and
> > > >> Infra)
> > > >> 2. We don't turn off automatic trigger of PR builds for now.
> > > >> 3. Hopefully, bot is used by developers to trigger specific jobs
> > > >> 4. Later on (say around April 20), let's discuss the possibility of
> > > >> switching off auto-trigger (with appropriate data) if it makes
> sense.
> > > >> Thanks Marco for volunteering to help enable the web hook on
> > > >> apache/incubator-mxnet. Let me know if we can sync up on Slack
> channel
> > > to
> > > >> get the ball rolling.
> > > >>
> > > >> Thanks once again for the entire community to step in and help try
> out
> > > >> this
> > > >> Bot.
> > > >> Chai
> > > >>
> > > >> On Wed, 18 Mar 2020 at 17:07, Marco de Abreu <
> marco.g.abreu@gmail.com
> > >
> > > >> wrote:
> > > >>
> > > >> > Hi, that's correct. But as stated previously, it's not an option
> to
> > > >> remove
> > > >> > the hook. For now, I'd like to see how the system behaves while
> it's
> > > >> > optional. Later on, we can talk about revisiting this decision.
> But
> > to
> > > >> me
> > > >> > it's not an option to deploy an entirely new system and approach
> > > without
> > > >> > having a transition or even a timeframe in which we are able to
> fall
> > > >> back.
> > > >> >
> > > >> > I'm happy to support the deployment of the bot and add an
> additional
> > > >> > webhook to enable it's functionality to support selective
> triggering
> > > by
> > > >> PR
> > > >> > authors and committers, but I will not support the disabling of
> > > >> automatic
> > > >> > triggering of branches or PRs.
> > > >> >
> > > >> > -Marco
> > > >> >
> > > >> > Chaitanya Bapat <ch...@gmail.com> schrieb am Mi., 18. März
> > 2020,
> > > >> > 21:00:
> > > >> >
> > > >> > > Hey Marco,
> > > >> > >
> > > >> > > I thought currently every commit on PR and master triggers CI
> > > >> > > because
> > > >> > > a. github webhook points to Jenkins Server
> > > >> > > b. GH Webhook events trigger builds on Jenkins for all commits
> to
> > > any
> > > >> > > branch in apache/incubator-mxnet
> > > >> > > may it be master/PR/non-PR
> > > >> > > Reason:
> > > >> > > Because all the 3 types of branches are discovered by Jenkins
> > > (non-PR
> > > >> > > (including master) and PR)
> > > >> > >
> > > >> > > Proposal: Remove GitHub WebHook to Jenkins and replace with GH
> > > >> Webhook to
> > > >> > > Lambda
> > > >> > > But after I remove the github webhook that points to Jenkins :
> > *N**o
> > > >> > commit
> > > >> > > will trigger Jenkins build by default* (as Jenkins wont receive
> GH
> > > >> > events)
> > > >> > > Only those that Bot deems fit will be triggered (using Jenkins
> API
> > > >> > invoked
> > > >> > > by Lambda).
> > > >> > > Hence its needed to handle that case of master merge.
> > > >> > > Am I understanding this correctly?
> > > >> > >
> > > >> > > On Wed, 18 Mar 2020 at 04:23, Marco de Abreu <
> > > marco.g.abreu@gmail.com
> > > >> >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Thanks Chai, sounds good to me. Could you elaborate a bit on
> the
> > > >> point
> > > >> > > > about triggering a CI run after the PR has been merged? We
> > already
> > > >> to
> > > >> > > that
> > > >> > > > automatically for the master, so what's the benefit to do it
> > > twice?
> > > >> > > >
> > > >> > > > -Marco
> > > >> > > >
> > > >> > > > Chaitanya Bapat <ch...@gmail.com> schrieb am Mi., 18.
> März
> > > >> 2020,
> > > >> > > > 09:30:
> > > >> > > >
> > > >> > > > > Update:
> > > >> > > > >
> > > >> > > > > > we can ensure that all CI runs ran on the commit that
> will
> > be
> > > >> > merged
> > > >> > > > > @Sam Skalicky <sa...@gmail.com> Branch Protection is
> > > added
> > > >> to
> > > >> > > > public
> > > >> > > > > MXNet repo. It ensures that for every PR to be merged, the
> CI
> > > >> passes.
> > > >> > > All
> > > >> > > > > the jobs selected "required" jobs will have to be green for
> > the
> > > >> PR to
> > > >> > > be
> > > >> > > > > merged. Ofcourse, users with "Adminstrator" access can merge
> > > >> without
> > > >> > it
> > > >> > > > but
> > > >> > > > > that's just a backdoor. It is the case now and will continue
> > to
> > > be
> > > >> > the
> > > >> > > > case
> > > >> > > > > with the inclusion of Bot.
> > > >> > > > >
> > > >> > > > > > easily verify that the CI has executed all runs on the
> > commit
> > > >> that
> > > >> > > will
> > > >> > > > > be merged
> > > >> > > > > GitHub UI shows all the jobs and the status corresponding to
> > it
> > > on
> > > >> > > every
> > > >> > > > > commit. That should suffice. For the merged commits, Repo ->
> > > >> Commits
> > > >> > ->
> > > >> > > > > Commit ID (Status) can be tracked currently (only way that I
> > > know
> > > >> > of).
> > > >> > > > > Moreover, it is beyond the scope of this project (and
> possibly
> > > >> out of
> > > >> > > our
> > > >> > > > > control since this is purely GitHub UI specific use-case).
> > > >> > > > >
> > > >> > > > > Thanks @przemyslaw for supporting the opt-in.
> > > >> > > > >
> > > >> > > > > Thanks everyone in the community for sharing concerns,
> voicing
> > > >> your
> > > >> > > > opinion
> > > >> > > > > and participating in the discussion.
> > > >> > > > > Thanks to those who attended the demo last Friday.
> > > >> > > > >
> > > >> > > > > Action items from that discussion
> > > >> > > > > 1. Handle master merge builds [Done]
> > > >> > > > > Bot runs entire CI suite after the PR is merged and comments
> > on
> > > >> the
> > > >> > PR
> > > >> > > > > about the same.
> > > >> > > > > Design decision :
> > > >> > > > > MXNet Bot comment about master merge build on the *merge
> > commit
> > > vs
> > > >> > PR*.
> > > >> > > > > After the PR is merged, Bot runs entire CI and comments the
> > > >> result of
> > > >> > > CI
> > > >> > > > > trigger on the PR (because it is easy to track on a PR
> rather
> > > than
> > > >> > > > > commenting inside the merge commit)
> > > >> > > > >
> > > >> > > > > 2. Idempotent condition
> > > >> > > > > In case of already running build, if an attempt is made to
> > > >> retrigger
> > > >> > > the
> > > >> > > > > job then what should be the response
> > > >> > > > > a. Not to re-trigger, let the ongoing build continue till
> > > >> completion
> > > >> > > > > b. End the ongoing build and re-trigger
> > > >> > > > > c. Let the ongoing build continue, re-trigger new build
> > > >> > > > >
> > > >> > > > > From resource saving point of view, *c* looks costly and a
> can
> > > be
> > > >> > > > > avoided/optimized by B.
> > > >> > > > > In case when a re-trigger was started "erroneously" then
> > killing
> > > >> > > ongoing
> > > >> > > > > build and re-trigger is a waste.
> > > >> > > > > In case when ongoing build failed in one sub-part, then
> > > >> re-triggering
> > > >> > > is
> > > >> > > > > justified.
> > > >> > > > > Erroneous re-triggers would be less often while conscious
> > > >> re-triggers
> > > >> > > to
> > > >> > > > > suppress failure is more common use-case. It looks like a
> safe
> > > >> > > assumption
> > > >> > > > > to make given the trade-off.
> > > >> > > > > [Open to debate]
> > > >> > > > >
> > > >> > > > > 3. Add security consideration [Use of secret manager, but
> > > without
> > > >> > > > > auto-rotation due to Jenkins manual config requirement]
> [Done]
> > > >> > > > > 4. New PR Instruction message by the Bot [Done]
> > > >> > > > > Thanks to the suggestion of Leonard, supported by others.
> I've
> > > now
> > > >> > > added
> > > >> > > > > the feature where the Bot comments a help message. [For
> > > reference
> > > >> -
> > > >> > > > > https://github.com/ChaiBapchya/incubator-mxnet/pull/52]
> > > >> > > > >
> > > >> > > > > Barring the opt-in vs opt-out debate & idempotency,
> consensus
> > > was
> > > >> > > quickly
> > > >> > > > > reached for the rest.
> > > >> > > > >
> > > >> > > > > In the coming days, I hope to roll-out this feature into
> Prod
> > > >> (public
> > > >> > > > > MXNet) for all devs to use.
> > > >> > > > >
> > > >> > > > > Thanks,
> > > >> > > > > Chai
> > > >> > > > >
> > > >> > > > > On Mon, 16 Mar 2020 at 11:57, Marco de Abreu <
> > > >> > marco.g.abreu@gmail.com>
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > Well that's generally a problem with a deferred CI
> approach
> > > (CI
> > > >> is
> > > >> > > run
> > > >> > > > at
> > > >> > > > > > commit and not at merge time). This can either be solved
> > > through
> > > >> > the
> > > >> > > > > other
> > > >> > > > > > proposal that's currently on dev@, by having a bot which
> > does
> > > >> > merges
> > > >> > > > by
> > > >> > > > > > having a global lock and a merge queue or by accepting the
> > > >> issue.
> > > >> > > > Reality
> > > >> > > > > > right now is that we're running that model where two PRs
> > which
> > > >> are
> > > >> > > > merged
> > > >> > > > > > in parallel might break one another. One thing to consider
> > > >> though
> > > >> > is
> > > >> > > > that
> > > >> > > > > > this breakage would have to be introduced in two separate
> > > parts
> > > >> > since
> > > >> > > > > > otherwise there'd be merge conflicts. I think we had that
> > > >> situation
> > > >> > > > twice
> > > >> > > > > > so far and the result was a quick revert, so I'd say that
> > > it's a
> > > >> > > > problem
> > > >> > > > > > that can happily be accepted. All other solutions
> basically
> > > >> require
> > > >> > > > some
> > > >> > > > > > form of single-threaded and globally locked solution which
> > > >> limits
> > > >> > us
> > > >> > > in
> > > >> > > > > > scalability. I'd recommend to just accept that risk and
> > revert
> > > >> a PR
> > > >> > > in
> > > >> > > > > case
> > > >> > > > > > it actually had a conflict.
> > > >> > > > > >
> > > >> > > > > > -Marco
> > > >> > > > > >
> > > >> > > > > > On Mon, Mar 16, 2020 at 6:29 PM Skalicky, Sam
> > > >> > > > <sskalic@amazon.com.invalid
> > > >> > > > > >
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > We probably need some way to track which CI runs ran for
> > > which
> > > >> > > commit
> > > >> > > > > > too,
> > > >> > > > > > > that way we can ensure that all CI runs ran on the
> commit
> > > that
> > > >> > will
> > > >> > > > be
> > > >> > > > > > > merged. Maybe the bot can comment with the commit hash
> > when
> > > >> > users
> > > >> > > > > > command
> > > >> > > > > > > it to do something. Although since users can trigger
> > > >> individual
> > > >> > CI
> > > >> > > > runs
> > > >> > > > > > its
> > > >> > > > > > > possible to have some commits run some CI runs but not
> > > >> others. We
> > > >> > > > need
> > > >> > > > > > some
> > > >> > > > > > > way to easily verify that the CI has executed all runs
> on
> > > the
> > > >> > > commit
> > > >> > > > > that
> > > >> > > > > > > will be merged.
> > > >> > > > > > >
> > > >> > > > > > > Sam
> > > >> > > > > > >
> > > >> > > > > > > > On Mar 13, 2020, at 8:28 PM, Przemysław Trędak <
> > > >> > > ptrendx@apache.org
> > > >> > > > >
> > > >> > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > CAUTION: This email originated from outside of the
> > > >> > organization.
> > > >> > > Do
> > > >> > > > > not
> > > >> > > > > > > click links or open attachments unless you can confirm
> the
> > > >> sender
> > > >> > > and
> > > >> > > > > > know
> > > >> > > > > > > the content is safe.
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > I personally like the idea of opt-in more than
> opt-out:
> > > >> > > > > > > > - ultimately PR author wants the PR to be merged so
> they
> > > (or
> > > >> > > > > committer
> > > >> > > > > > > reviewing the PR) will trigger the CI
> > > >> > > > > > > > - if it is easy to trigger the PR via the bot command
> > then
> > > >> the
> > > >> > > > amount
> > > >> > > > > > of
> > > >> > > > > > > work per PR should be less than with opt-out (since most
> > of
> > > >> the
> > > >> > > > commits
> > > >> > > > > > > should then be marked as [skip ci] or something similar
> > > >> > > > > > > >
> > > >> > > > > > > > +1 to the bot making a comment on each new PR with its
> > > >> commands
> > > >> > > > (and
> > > >> > > > > > > also explaining, or at least giving links to the general
> > PR
> > > >> > process
> > > >> > > > so
> > > >> > > > > > new
> > > >> > > > > > > PR authors are not lost). Maybe we could make the bot
> > > >> recognize
> > > >> > if
> > > >> > > > the
> > > >> > > > > PR
> > > >> > > > > > > author is new or existing contributor and offer advice
> > based
> > > >> on
> > > >> > > that?
> > > >> > > > > > > >
> > > >> > > > > > > > Thanks
> > > >> > > > > > > > Przemek
> > > >> > > > > > > >
> > > >> > > > > > > > On 2020/03/13 22:06:58, Marco de Abreu <
> > > >> > marco.g.abreu@gmail.com>
> > > >> > > > > > wrote:
> > > >> > > > > > > >> Hi,
> > > >> > > > > > > >>
> > > >> > > > > > > >> since it's no longer necessary to push a new commit
> to
> > > >> trigger
> > > >> > > CI,
> > > >> > > > > it
> > > >> > > > > > > will
> > > >> > > > > > > >> already reduce the costs. But to me, requiring an
> > action
> > > to
> > > >> > > enable
> > > >> > > > > CI
> > > >> > > > > > > after
> > > >> > > > > > > >> a PR has been created initially, is a no go. User can
> > opt
> > > >> out
> > > >> > of
> > > >> > > > CI,
> > > >> > > > > > but
> > > >> > > > > > > >> the default has to be CI being triggered
> automatically
> > > for
> > > >> > every
> > > >> > > > > > commit
> > > >> > > > > > > >> unless specifically disabled by a participant. I'm
> also
> > > >> fine
> > > >> > > with
> > > >> > > > > > > >> triggering certain additional jobs (think about
> > running a
> > > >> > > nightly
> > > >> > > > > job
> > > >> > > > > > > upon
> > > >> > > > > > > >> request for a PR) to require a manual step, but the
> PR
> > > >> > > validation
> > > >> > > > > > > pipelines
> > > >> > > > > > > >> have to run automatically. Every check that is marked
> > as
> > > >> > > > "Required"
> > > >> > > > > in
> > > >> > > > > > > >> GitHub has to be automatically kicked off.
> > > >> > > > > > > >>
> > > >> > > > > > > >> -Marco
> > > >> > > > > > > >>
> > > >> > > > > > > >> On Fri, Mar 13, 2020 at 9:50 PM Chaitanya Bapat <
> > > >> > > > > chai.bapat@gmail.com
> > > >> > > > > > >
> > > >> > > > > > > >> wrote:
> > > >> > > > > > > >>
> > > >> > > > > > > >>> Firstly,
> > > >> > > > > > > >>> Sorry I missed out on attaching the mail thread that
> > was
> > > >> sent
> > > >> > > on
> > > >> > > > > 12th
> > > >> > > > > > > >>> February for notifying the community of the upcoming
> > > >> changes
> > > >> > to
> > > >> > > > the
> > > >> > > > > > > MXNet
> > > >> > > > > > > >>> CI
> > > >> > > > > > > >>> For reference :
> > > >> > > > > > > >>>
> > > >> > > > > > > >>>
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://lists.apache.org/thread.html/r09a6ab2803a996fc80e00fe39ed312fa4865e8805e08df847f1addad%40%3Cdev.mxnet.apache.org%3E
> > > >> > > > > > > >>>
> > > >> > > > > > > >>> Now to the questions,
> > > >> > > > > > > >>>> Is it possible for re-triggering a single job to be
> > > >> abused?
> > > >> > > > > > > >>> @Tao In the case when a user re-triggers a single
> job
> > > >> > multiple
> > > >> > > > > times,
> > > >> > > > > > > that
> > > >> > > > > > > >>> will be visible in the PR conversation thread. A
> > > >> committer,
> > > >> > > even
> > > >> > > > > > after
> > > >> > > > > > > he
> > > >> > > > > > > >>> has approved the PR before, generally takes a look
> at
> > > the
> > > >> > final
> > > >> > > > > state
> > > >> > > > > > > of
> > > >> > > > > > > >>> the PR before merging. Would it be fair to assume
> the
> > > >> > committer
> > > >> > > > > could
> > > >> > > > > > > take
> > > >> > > > > > > >>> the multiple re-trigger of a single job into account
> > > >> before
> > > >> > > > > merging?
> > > >> > > > > > > The
> > > >> > > > > > > >>> committer then has the option to invoke `@mxnet-bot
> > run
> > > ci
> > > >> > > [all]
> > > >> > > > `
> > > >> > > > > to
> > > >> > > > > > > >>> trigger the entire build pipeline one last to
> counter
> > > the
> > > >> > > abuse.
> > > >> > > > > This
> > > >> > > > > > > is
> > > >> > > > > > > >>> aligned with what @Leonard said.
> > > >> > > > > > > >>>
> > > >> > > > > > > >>> @Sandeep Thanks a lot for collecting and sharing
> > > valuable
> > > >> > data.
> > > >> > > > I'd
> > > >> > > > > > > concur
> > > >> > > > > > > >>> with the opinion that given the existing things
> > > >> committers &
> > > >> > PR
> > > >> > > > > > Authors
> > > >> > > > > > > >>> take care of, invoking CI shouldn't be that big of
> an
> > > >> > > additional
> > > >> > > > > > > burden.
> > > >> > > > > > > >>>
> > > >> > > > > > > >>> @Marco With the opt-out, the onus remains on the PR
> > > >> Author.
> > > >> > It
> > > >> > > > > > doesn't
> > > >> > > > > > > help
> > > >> > > > > > > >>> reduce the resource usage. Hence, it was suggested
> to
> > > >> switch
> > > >> > to
> > > >> > > > > > > >>> opt-in. @Leo's suggestion for proactive commenting
> on
> > > the
> > > >> > part
> > > >> > > of
> > > >> > > > > bot
> > > >> > > > > > > makes
> > > >> > > > > > > >>> sense and is doable.
> > > >> > > > > > > >>>
> > > >> > > > > > > >>> Default : opt-out and User initiated opt-in (with
> > > >> addressing
> > > >> > > > Leo's
> > > >> > > > > > fix
> > > >> > > > > > > for
> > > >> > > > > > > >>> the usability issue you correctly pointed out )
> > > >> > > > > > > >>> @Marco How does this sound to you?
> > > >> > > > > > > >>>
> > > >> > > > > > > >>> Again, thank you all for chiming in and voicing your
> > > >> opinion.
> > > >> > > > > > > Appreciate
> > > >> > > > > > > >>> it.
> > > >> > > > > > > >>> We can take ahead these discussions in today's demo
> > > >> meeting.
> > > >> > > > > [Design
> > > >> > > > > > > Doc
> > > >> > > > > > > >>> <
> > > >> > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > > >> > > > >]
> > > >> > > > > > > [Demo
> > > >> > > > > > > >>> Video <https://www.youtube.com/watch?v=gfOGwZId8aU
> >]
> > > >> > > > > > > >>>
> > > >> > > > > > > >>> Thanks,
> > > >> > > > > > > >>> Chai
> > > >> > > > > > > >>>
> > > >> > > > > > > >>> On Fri, 13 Mar 2020 at 12:34, Marco de Abreu <
> > > >> > > > > > marco.g.abreu@gmail.com>
> > > >> > > > > > > >>> wrote:
> > > >> > > > > > > >>>
> > > >> > > > > > > >>>> I'd recommend that the bot makes an initial comment
> > > when
> > > >> a
> > > >> > PR
> > > >> > > > gets
> > > >> > > > > > > opened
> > > >> > > > > > > >>>> and informs the users of its commands. It then
> tells
> > > the
> > > >> > user
> > > >> > > > the
> > > >> > > > > > > commend
> > > >> > > > > > > >>>> to opt out of CI.
> > > >> > > > > > > >>>>
> > > >> > > > > > > >>>> -Marco
> > > >> > > > > > > >>>>
> > > >> > > > > > > >>>> Lausen, Leonard <la...@amazon.com.invalid>
> schrieb
> > am
> > > >> Fr.,
> > > >> > > 13.
> > > >> > > > > > März
> > > >> > > > > > > >>> 2020,
> > > >> > > > > > > >>>> 20:27:
> > > >> > > > > > > >>>>
> > > >> > > > > > > >>>>> On opt-out: People may be unaware of opt-out would
> > not
> > > >> use
> > > >> > > it.
> > > >> > > > > > There
> > > >> > > > > > > is
> > > >> > > > > > > >>>> no
> > > >> > > > > > > >>>>> incentive to use opt-out, as the PR author doesn't
> > pay
> > > >> any
> > > >> > > > money
> > > >> > > > > > for
> > > >> > > > > > > CI
> > > >> > > > > > > >>>>> run.
> > > >> > > > > > > >>>>>
> > > >> > > > > > > >>>>> I agree with Marco though that opt-in alone may
> > cause
> > > >> > > usability
> > > >> > > > > > > issues,
> > > >> > > > > > > >>>> as
> > > >> > > > > > > >>>>> contributors may not be aware of how to trigger
> the
> > > CI.
> > > >> > > > > > > >>>>> One solution is that the bot proactively comments
> on
> > > >> the PR
> > > >> > > and
> > > >> > > > > > > reminds
> > > >> > > > > > > >>>> the
> > > >> > > > > > > >>>>> author to trigger running CI once the author deems
> > the
> > > >> PR
> > > >> > > > ready.
> > > >> > > > > > > >>>>>
> > > >> > > > > > > >>>>> But even if we choose opt-out, the bot will still
> > add
> > > a
> > > >> lot
> > > >> > > of
> > > >> > > > > > value,
> > > >> > > > > > > >>> as
> > > >> > > > > > > >>>> PR
> > > >> > > > > > > >>>>> authors can retrigger single jobs that have failed
> > due
> > > >> to
> > > >> > > > > > flakiness.
> > > >> > > > > > > >>>>>
> > > >> > > > > > > >>>>>> Is it possible for re-triggering a single job to
> be
> > > >> > abused?
> > > >> > > > For
> > > >> > > > > > > >>>> example,
> > > >> > > > > > > >>>>>> the author spends two days re-triggering a flaky
> > job
> > > to
> > > >> > make
> > > >> > > > it
> > > >> > > > > > > pass.
> > > >> > > > > > > >>>>>
> > > >> > > > > > > >>>>> Yes, this is possible. I suggest the committer who
> > > >> likes to
> > > >> > > > > merge a
> > > >> > > > > > > PR
> > > >> > > > > > > >>>>> needs to
> > > >> > > > > > > >>>>> make a good judgement here if a PR is abusing the
> > > >> feature,
> > > >> > > and
> > > >> > > > if
> > > >> > > > > > so,
> > > >> > > > > > > >>>>> retrigger
> > > >> > > > > > > >>>>> all CI runs.
> > > >> > > > > > > >>>>>
> > > >> > > > > > > >>>>> Best regards
> > > >> > > > > > > >>>>> Leonard
> > > >> > > > > > > >>>>>
> > > >> > > > > > > >>>>> On Fri, 2020-03-13 at 08:07 +0100, Marco de Abreu
> > > wrote:
> > > >> > > > > > > >>>>>> Thanks for the data Sandeep. In these cases it
> > sounds
> > > >> like
> > > >> > > it
> > > >> > > > > > would
> > > >> > > > > > > >>>> have
> > > >> > > > > > > >>>>>> rather been better when people explicitly turned
> > off
> > > >> CI in
> > > >> > > > that
> > > >> > > > > > > case.
> > > >> > > > > > > >>>>>> What's the argument against an opt-out instead of
> > an
> > > >> > opt-in?
> > > >> > > > > > > >>>>>>
> > > >> > > > > > > >>>>>> My intention is that I consider it quite
> cumbersome
> > > to
> > > >> > make
> > > >> > > > it a
> > > >> > > > > > > >>>>> *required*
> > > >> > > > > > > >>>>>> step to always trigger CI manually, even if just
> > > >> > submitting
> > > >> > > a
> > > >> > > > > > small
> > > >> > > > > > > >>> PR.
> > > >> > > > > > > >>>>> I'd
> > > >> > > > > > > >>>>>> rather see people explicitly turning off CI if
> they
> > > >> > wouldn't
> > > >> > > > > like
> > > >> > > > > > to
> > > >> > > > > > > >>>> use
> > > >> > > > > > > >>>>> it
> > > >> > > > > > > >>>>>> - and there's also the "draft" stage for a PR
> which
> > > >> some
> > > >> > > > > > > contributors
> > > >> > > > > > > >>>> are
> > > >> > > > > > > >>>>>> using.
> > > >> > > > > > > >>>>>>
> > > >> > > > > > > >>>>>> With regards to WIP and do not review: I think
> > these
> > > >> are
> > > >> > use
> > > >> > > > > cases
> > > >> > > > > > > >>>> where
> > > >> > > > > > > >>>>>> you want CI feedback, as otherwise you wouldn't
> > have
> > > >> > opened
> > > >> > > > the
> > > >> > > > > > PR.
> > > >> > > > > > > >>> If
> > > >> > > > > > > >>>>> you
> > > >> > > > > > > >>>>>> don't want human feedback and neither machine
> > > feedback,
> > > >> > why
> > > >> > > > open
> > > >> > > > > > the
> > > >> > > > > > > >>> PR
> > > >> > > > > > > >>>>> at
> > > >> > > > > > > >>>>>> all?
> > > >> > > > > > > >>>>>>
> > > >> > > > > > > >>>>>> -Marco
> > > >> > > > > > > >>>>>>
> > > >> > > > > > > >>>>>>
> > > >> > > > > > > >>>>>> sandeep krishnamurthy <
> sandeep.krishna98@gmail.com
> > >
> > > >> > schrieb
> > > >> > > > am
> > > >> > > > > > Fr.,
> > > >> > > > > > > >>>> 13.
> > > >> > > > > > > >>>>>> März 2020, 05:24:
> > > >> > > > > > > >>>>>>
> > > >> > > > > > > >>>>>>> I tried to gather some data for us to discuss
> this
> > > >> topic
> > > >> > in
> > > >> > > > > this
> > > >> > > > > > > >>>>> thread. I
> > > >> > > > > > > >>>>>>> tried to count number of un-necessary builds by
> > > >> looking
> > > >> > at
> > > >> > > > most
> > > >> > > > > > > >>>> recent
> > > >> > > > > > > >>>>> (as
> > > >> > > > > > > >>>>>>> of 12, March 9 PM PST) 50 PRs merged to master
> and
> > > 50
> > > >> > PRs.
> > > >> > > > > > > >>>>>>> Identifying un-necessary builds is bit
> > subjective. I
> > > >> > tried
> > > >> > > to
> > > >> > > > > be
> > > >> > > > > > > >>> more
> > > >> > > > > > > >>>>>>> conservative where I didn't count a build as
> > > >> un-necessary
> > > >> > > if
> > > >> > > > I
> > > >> > > > > > was
> > > >> > > > > > > >>> in
> > > >> > > > > > > >>>>>>> doubt. Hence, I was not able to automate, but I
> > made
> > > >> an
> > > >> > > > effort
> > > >> > > > > to
> > > >> > > > > > > >>> go
> > > >> > > > > > > >>>>>>> through PRs manually and use below criteria to
> > > >> identify
> > > >> > > > > > > >>> un-necessary
> > > >> > > > > > > >>>>>>> commits triggering the builds.
> > > >> > > > > > > >>>>>>>
> > > >> > > > > > > >>>>>>> 1. Explicitly marked as WIP / do not review
> PR
> > > >> > > > > > > >>>>>>> 2. Incremental WIP commit and finally
> > commenting a
> > > >> > commit
> > > >> > > > > > > >>> “trigger
> > > >> > > > > > > >>>>> CI”
> > > >> > > > > > > >>>>>>> 3. Multiple commits to address all comments
> from
> > > >> single
> > > >> > > > > review.
> > > >> > > > > > > >>>>> This is
> > > >> > > > > > > >>>>>>> assuming we see a comment, address them,
> commit,
> > > >> next
> > > >> > the
> > > >> > > > > > > >>>> following
> > > >> > > > > > > >>>>>>> comment
> > > >> > > > > > > >>>>>>> 4. Sequence of documentation only changes
> > > >> > > > > > > >>>>>>>
> > > >> > > > > > > >>>>>>> I found there were around 42 avoidable builds
> from
> > > >> most
> > > >> > > > recent
> > > >> > > > > 50
> > > >> > > > > > > >>>>> merged
> > > >> > > > > > > >>>>>>> PRs and around 86 builds from recent 50 open
> PRs.
> > > >> > > > > > > >>>>>>>
> > > >> > > > > > > >>>>>>> I synced up with other contributors (Joe Evans,
> > > Chai)
> > > >> > from
> > > >> > > > > Amazon
> > > >> > > > > > > >>> who
> > > >> > > > > > > >>>>> is
> > > >> > > > > > > >>>>>>> contributing to MXNet CI system. I was told that
> > on
> > > an
> > > >> > > > average
> > > >> > > > > it
> > > >> > > > > > > >>>> costs
> > > >> > > > > > > >>>>>>> around $84 per build and on an average 6 commits
> > per
> > > >> > merged
> > > >> > > > PR
> > > >> > > > > > (for
> > > >> > > > > > > >>>>> year
> > > >> > > > > > > >>>>>>> 2019). Going by that, it is approximately 1/6
> > builds
> > > >> are
> > > >> > > > > > avoidable.
> > > >> > > > > > > >>>>> [100 /
> > > >> > > > > > > >>>>>>> 300 + 300 ]
> > > >> > > > > > > >>>>>>>
> > > >> > > > > > > >>>>>>>
> > > >> > > > > > > >>>>>>> Usability should be top most priority. But,
> since
> > > >> either
> > > >> > a
> > > >> > > > > > reviewer
> > > >> > > > > > > >>>> or
> > > >> > > > > > > >>>>> pr
> > > >> > > > > > > >>>>>>> author can trigger the bot, is it really a
> hurdle
> > > for
> > > >> pr
> > > >> > > > author
> > > >> > > > > > or
> > > >> > > > > > > >>>>> reviewer
> > > >> > > > > > > >>>>>>> to call a bot to trigger CI? Given that PR
> author
> > > and
> > > >> > > > reviewer
> > > >> > > > > is
> > > >> > > > > > > >>>>> already
> > > >> > > > > > > >>>>>>> actively commenting various details such as - PR
> > > >> > > description,
> > > >> > > > > > > >>> review
> > > >> > > > > > > >>>>>>> comments and responses, adding labels etc.
> > > >> > > > > > > >>>>>>>
> > > >> > > > > > > >>>>>>>
> > > >> > > > > > > >>>>>>> Me too curious to know the behavior for Tao's
> > above
> > > >> use
> > > >> > > case.
> > > >> > > > > > > >>>>>>>
> > > >> > > > > > > >>>>>>>
> > > >> > > > > > > >>>>>>> Best,
> > > >> > > > > > > >>>>>>>
> > > >> > > > > > > >>>>>>> Sandeep
> > > >> > > > > > > >>>>>>>
> > > >> > > > > > > >>>>>>> On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <
> > > >> > mutouorz@gmail.com
> > > >> > > >
> > > >> > > > > > wrote:
> > > >> > > > > > > >>>>>>>
> > > >> > > > > > > >>>>>>>> Is it possible for re-triggering a single job
> to
> > be
> > > >> > > abused?
> > > >> > > > > For
> > > >> > > > > > > >>>>> example,
> > > >> > > > > > > >>>>>>>> the author spends two days re-triggering a
> flaky
> > > job
> > > >> to
> > > >> > > make
> > > >> > > > > it
> > > >> > > > > > > >>>>> pass. But
> > > >> > > > > > > >>>>>>>> other jobs which have passed the validation may
> > be
> > > >> > broken
> > > >> > > by
> > > >> > > > > > > >>> other
> > > >> > > > > > > >>>>>>> commits
> > > >> > > > > > > >>>>>>>> during the two day without being noticed. And
> > > finally
> > > >> > the
> > > >> > > PR
> > > >> > > > > is
> > > >> > > > > > > >>>>> merged
> > > >> > > > > > > >>>>>>> with
> > > >> > > > > > > >>>>>>>> underlying problems.
> > > >> > > > > > > >>>>>>>>
> > > >> > > > > > > >>>>>>>> On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu
> <
> > > >> > > > > > > >>>>> marco.g.abreu@gmail.com>
> > > >> > > > > > > >>>>>>>> wrote:
> > > >> > > > > > > >>>>>>>>
> > > >> > > > > > > >>>>>>>>> In the end it only comes down to money,
> > > considering
> > > >> > that
> > > >> > > > the
> > > >> > > > > > > >>>>> system is
> > > >> > > > > > > >>>>>>>> auto
> > > >> > > > > > > >>>>>>>>> scaling, making the execution time constant.
> > > >> > > > > > > >>>>>>>>>
> > > >> > > > > > > >>>>>>>>> If we're trading money for usability, I
> > certainly
> > > >> would
> > > >> > > > > prefer
> > > >> > > > > > > >>>>>>> usability.
> > > >> > > > > > > >>>>>>>>> I'd rather recommend to spend time on
> > > parallelizing
> > > >> > test
> > > >> > > > > > > >>>> execution
> > > >> > > > > > > >>>>> or
> > > >> > > > > > > >>>>>>>>> getting rid of integration tests in the PR
> stage
> > > >> > instead
> > > >> > > > > > > >>> reducing
> > > >> > > > > > > >>>>> the
> > > >> > > > > > > >>>>>>>> costs
> > > >> > > > > > > >>>>>>>>> by making people not use it. But taking a step
> > > back
> > > >> to
> > > >> > > > > > > >>> requiring
> > > >> > > > > > > >>>>> people
> > > >> > > > > > > >>>>>>>> to
> > > >> > > > > > > >>>>>>>>> manually trigger CI again doesn't feel right.
> > > >> > > > > > > >>>>>>>>>
> > > >> > > > > > > >>>>>>>>> I'm happy to see that bot deployed, but I do
> not
> > > >> agree
> > > >> > > with
> > > >> > > > > > > >>>>> removing
> > > >> > > > > > > >>>>>>> the
> > > >> > > > > > > >>>>>>>>> auto trigger functionality for new commits.
> > > >> > > > > > > >>>>>>>>>
> > > >> > > > > > > >>>>>>>>> -Marco
> > > >> > > > > > > >>>>>>>>>
> > > >> > > > > > > >>>>>>>>> Chaitanya Bapat <ch...@gmail.com>
> schrieb
> > am
> > > >> Do.,
> > > >> > > 12.
> > > >> > > > > > > >>> März
> > > >> > > > > > > >>>>> 2020,
> > > >> > > > > > > >>>>>>>>> 22:47:
> > > >> > > > > > > >>>>>>>>>
> > > >> > > > > > > >>>>>>>>>> @Marco Thanks for pointing that out.
> > > >> > > > > > > >>>>>>>>>> Tomorrow i.e. Friday, March 13, 2020 at 3:00
> > PM -
> > > >> 3:30
> > > >> > > PM
> > > >> > > > in
> > > >> > > > > > > >>>>>>>> (UTC-08:00)
> > > >> > > > > > > >>>>>>>>>> Pacific Time (US & Canada).
> > > >> > > > > > > >>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>> When do we expect this bot to be deployed?
> > > >> > > > > > > >>>>>>>>>> @Lin If all goes well in the next week I can
> > > >> deploy it
> > > >> > > to
> > > >> > > > > > > >>>> public
> > > >> > > > > > > >>>>>>> Apache
> > > >> > > > > > > >>>>>>>>>> (provided I get permissions from Apache
> Infra)
> > > >> > > > > > > >>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>> @Marco Thanks for your feedback.
> > > >> > > > > > > >>>>>>>>>>> CI system has to support the community
> without
> > > >> > > requiring
> > > >> > > > > > > >>>>> people to
> > > >> > > > > > > >>>>>>>>>> constantly shepherd every single run
> > > >> > > > > > > >>>>>>>>>> We have data for the number of times CI was
> > > >> triggered
> > > >> > > > > > > >>>>> unnecessarily
> > > >> > > > > > > >>>>>>>> which
> > > >> > > > > > > >>>>>>>>>> includes
> > > >> > > > > > > >>>>>>>>>> - Entire build triggered instead of specific
> > > build
> > > >> > > > > > > >>>>>>>>>> - CI triggered when PR is still work in
> > progress
> > > or
> > > >> > not
> > > >> > > > yet
> > > >> > > > > > > >>>> ready
> > > >> > > > > > > >>>>>>> (say
> > > >> > > > > > > >>>>>>>> -
> > > >> > > > > > > >>>>>>>>>> intermediate commits)
> > > >> > > > > > > >>>>>>>>>> At the end its a trade-off
> > > >> > > > > > > >>>>>>>>>> Money, Resources, Time to build for each and
> > > every
> > > >> > > commit
> > > >> > > > vs
> > > >> > > > > > > >>>>> Pain of
> > > >> > > > > > > >>>>>>>>>> triggering builds
> > > >> > > > > > > >>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>> Scan trigger plugin would poll SCM. Can we
> use
> > > >> plugin
> > > >> > > at
> > > >> > > > > > > >>>>> scale?
> > > >> > > > > > > >>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>> 1. I haven't tested it on scale. But I think
> > with
> > > >> the
> > > >> > > > > current
> > > >> > > > > > > >>>>> scale
> > > >> > > > > > > >>>>>>> of
> > > >> > > > > > > >>>>>>>>>> MXNet repo (191 open PRs i.e. checking for
> > > changes
> > > >> to
> > > >> > > 191
> > > >> > > > > > > >>>>> branches -
> > > >> > > > > > > >>>>>>> It
> > > >> > > > > > > >>>>>>>>>> should be manageable)
> > > >> > > > > > > >>>>>>>>>> 2. What's the purpose of the plugin? tldr;
> > Branch
> > > >> > > > discovery
> > > >> > > > > > > >>> or
> > > >> > > > > > > >>>>> branch
> > > >> > > > > > > >>>>>>>>>> indexing.
> > > >> > > > > > > >>>>>>>>>> Scan trigger plugin comes into the picture
> only
> > > >> once
> > > >> > per
> > > >> > > > PR
> > > >> > > > > > > >>> per
> > > >> > > > > > > >>>>> job
> > > >> > > > > > > >>>>>>>>> (i.e. 8
> > > >> > > > > > > >>>>>>>>>> times per PR for 8 jobs). It is basically
> done
> > > >> when a
> > > >> > > new
> > > >> > > > PR
> > > >> > > > > > > >>> is
> > > >> > > > > > > >>>>> made
> > > >> > > > > > > >>>>>>>> and
> > > >> > > > > > > >>>>>>>>>> the job (say unix-cpu hasn't discovered the
> new
> > > PR
> > > >> > > branch
> > > >> > > > > > > >>> yet).
> > > >> > > > > > > >>>>>>> That's
> > > >> > > > > > > >>>>>>>>> it.
> > > >> > > > > > > >>>>>>>>>> So it shouldn't be a problem for public MXNet
> > > repo.
> > > >> > > > > > > >>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>> Thanks,
> > > >> > > > > > > >>>>>>>>>> Chai
> > > >> > > > > > > >>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>> On Thu, 12 Mar 2020 at 14:22, Marco de Abreu
> <
> > > >> > > > > > > >>>>>>> marco.g.abreu@gmail.com>
> > > >> > > > > > > >>>>>>>>>> wrote:
> > > >> > > > > > > >>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>> Btw you forgot to set a date and time for
> the
> > > >> metting
> > > >> > > > > > > >>>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>> On Thu, Mar 12, 2020 at 10:18 PM Marco de
> > Abreu
> > > <
> > > >> > > > > > > >>>>>>>>> marco.g.abreu@gmail.com
> > > >> > > > > > > >>>>>>>>>>> wrote:
> > > >> > > > > > > >>>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>>> Thanks Chai, I generally like the idea of
> the
> > > >> bot.
> > > >> > But
> > > >> > > > > > > >>> I'm
> > > >> > > > > > > >>>>> not a
> > > >> > > > > > > >>>>>>>>>>> supporter
> > > >> > > > > > > >>>>>>>>>>>> of the idea to disable any automatic
> > triggering
> > > >> > > > > > > >>> (disabling
> > > >> > > > > > > >>>>> the
> > > >> > > > > > > >>>>>>>>> webhook
> > > >> > > > > > > >>>>>>>>>> is
> > > >> > > > > > > >>>>>>>>>>>> also not an option, considering that this
> > will
> > > >> > disable
> > > >> > > > > > > >>>> master
> > > >> > > > > > > >>>>>>>>>> triggers).
> > > >> > > > > > > >>>>>>>>>>>> The CI system has to support the community
> > > >> without
> > > >> > > > > > > >>>> requiring
> > > >> > > > > > > >>>>>>> people
> > > >> > > > > > > >>>>>>>>> to
> > > >> > > > > > > >>>>>>>>>>>> constantly shepherd every single run.
> > Disabling
> > > >> > > > automatic
> > > >> > > > > > > >>>>>>>> triggering
> > > >> > > > > > > >>>>>>>>>>> seems
> > > >> > > > > > > >>>>>>>>>>>> like a step back to me.
> > > >> > > > > > > >>>>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>>> Instead, I'd recommend that CI gets
> triggered
> > > >> upon
> > > >> > > every
> > > >> > > > > > > >>>>> commit
> > > >> > > > > > > >>>>>>> as
> > > >> > > > > > > >>>>>>>>>> usual,
> > > >> > > > > > > >>>>>>>>>>>> but people have the possibility to call a
> > > >> "command"
> > > >> > > > (i.e.
> > > >> > > > > > > >>>>> make a
> > > >> > > > > > > >>>>>>>>>> message
> > > >> > > > > > > >>>>>>>>>>>> which results in the bot setting a label)
> to
> > > >> disable
> > > >> > > CI
> > > >> > > > > > > >>>> until
> > > >> > > > > > > >>>>>>> they
> > > >> > > > > > > >>>>>>>>>> revoke
> > > >> > > > > > > >>>>>>>>>>>> it. But the standard should still be that a
> > new
> > > >> > commit
> > > >> > > > > > > >>>>> triggers a
> > > >> > > > > > > >>>>>>>> new
> > > >> > > > > > > >>>>>>>>>> CI
> > > >> > > > > > > >>>>>>>>>>>> run.
> > > >> > > > > > > >>>>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>>>
> > > >> > > > > > > >>>>
> > > >> > https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
> > > >> > > > > > > >>>>>>>> seems
> > > >> > > > > > > >>>>>>>>>> like
> > > >> > > > > > > >>>>>>>>>>>> this would poll SCM. This will incur high
> > quota
> > > >> > > > > > > >>>>> restrictions. Are
> > > >> > > > > > > >>>>>>>> you
> > > >> > > > > > > >>>>>>>>>>> sure
> > > >> > > > > > > >>>>>>>>>>>> that you can use that plugin at scale?
> > > >> > > > > > > >>>>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>>> -Marco
> > > >> > > > > > > >>>>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>>> On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <
> > > >> > > > > > > >>>>> apeforest@gmail.com>
> > > >> > > > > > > >>>>>>>>> wrote:
> > > >> > > > > > > >>>>>>>>>>>>> Chai,
> > > >> > > > > > > >>>>>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>>>> Awesome work. When do we expect this bot
> to
> > be
> > > >> > > > > > > >>> deployed?
> > > >> > > > > > > >>>>>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>>>> Best,
> > > >> > > > > > > >>>>>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>>>> Lin
> > > >> > > > > > > >>>>>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>>>> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya
> > > Bapat
> > > >> <
> > > >> > > > > > > >>>>>>>>> chai.bapat@gmail.com
> > > >> > > > > > > >>>>>>>>>>>>> wrote:
> > > >> > > > > > > >>>>>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>>>>> Hello MXNet community,
> > > >> > > > > > > >>>>>>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>>>>> I have built an MXNet Bot <
> > > >> > > > > > > >>>> https://github.com/mxnet-bot>
> > > >> > > > > > > >>>>> that
> > > >> > > > > > > >>>>>>>>>> allows
> > > >> > > > > > > >>>>>>>>>>> PR
> > > >> > > > > > > >>>>>>>>>>>>>> Authors, Committers and Jenkins Admins to
> > > >> trigger
> > > >> > CI
> > > >> > > > > > > >>>>> manually.
> > > >> > > > > > > >>>>>>>>>>>>>> It handles 2 problems
> > > >> > > > > > > >>>>>>>>>>>>>> 1. Manual CI trigger instead of existing
> > > >> automated
> > > >> > > CI
> > > >> > > > > > > >>>>> trigger
> > > >> > > > > > > >>>>>>>>>>>>>> 2. Gives permissions to PR Authors (in
> > > >> addition to
> > > >> > > > > > > >>>> MXNet
> > > >> > > > > > > >>>>>>>>> Committers
> > > >> > > > > > > >>>>>>>>>>> and
> > > >> > > > > > > >>>>>>>>>>>>>> Jenkins Admins)
> > > >> > > > > > > >>>>>>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>>>>> Design Doc :
> > > >> > > > > > > >>>>>>>>>>>>>>
> > > >> > > > > > > >>>>>>>
> > > >> > > >
> https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > > >> > > > > > > >>>>>>>>>>>>>> I urge you all to attend the
> demonstration
> > > >> meeting
> > > >> > > > > > > >>> and
> > > >> > > > > > > >>>>> lend
> > > >> > > > > > > >>>>>>> your
> > > >> > > > > > > >>>>>>>>>> views
> > > >> > > > > > > >>>>>>>>>>>>> on
> > > >> > > > > > > >>>>>>>>>>>>>> the same.
> > > >> > > > > > > >>>>>>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>>>>> Thank you,
> > > >> > > > > > > >>>>>>>>>>>>>> Chai
> > > >> > > > > > > >>>>>>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>>>>> *Meeting Details*:
> > > >> > > > > > > >>>>>>>>>>>>>> ==============Conference Bridge
> > > >> > > > > > > >>>> Information==============
> > > >> > > > > > > >>>>>>>>>>>>>> You have been invited to an online
> meeting,
> > > >> > powered
> > > >> > > > > > > >>> by
> > > >> > > > > > > >>>>> Amazon
> > > >> > > > > > > >>>>>>>>> Chime.
> > > >> > > > > > > >>>>>>>>>>>>>> *Chime meeting ID*: *9272158344*
> > > >> > > > > > > >>>>>>>>>>>>>> Join via Chime clients (manually): Select
> > > >> > 'Meetings
> > > >> > > >
> > > >> > > > > > > >>>>> Join a
> > > >> > > > > > > >>>>>>>>>> Meeting',
> > > >> > > > > > > >>>>>>>>>>>>> and
> > > >> > > > > > > >>>>>>>>>>>>>> enter 9272158344
> > > >> > > > > > > >>>>>>>>>>>>>> Join via Chime clients (auto-call): If
> you
> > > >> invite
> > > >> > > > > > > >>>>> auto-call as
> > > >> > > > > > > >>>>>>>>>>> attendee,
> > > >> > > > > > > >>>>>>>>>>>>>> Chime will call you when the meeting
> > starts,
> > > >> > select
> > > >> > > > > > > >>>>> 'Answer'
> > > >> > > > > > > >>>>>>>>>>>>>> *Join via browser screen share*:
> > > >> > > > > > > >>>>> https://chime.aws/9272158344
> > > >> > > > > > > >>>>>>>>>>>>>> *Join via phone* (US):
> > > >> > +1-929-432-4463,,,9272158344#
> > > >> > > > > > > >>>>>>>>>>>>>> *Join via phone (US toll-free)*:
> > > >> > > > > > > >>>>> +1-855-552-4463,,,9272158344#
> > > >> > > > > > > >>>>>>>>>>>>>> International dial-in:
> > > >> > > > > > > >>>> https://chime.aws/dialinnumbers/
> > > >> > > > > > > >>>>>>>>>>>>>> In-room video system: Ext: 62000, Meeting
> > > PIN:
> > > >> > > > > > > >>>>> 9272158344#
> > > >> > > > > > > >>>>>>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>>>>> --
> > > >> > > > > > > >>>>>>>>>>>>>> *Chaitanya Prakash Bapat*
> > > >> > > > > > > >>>>>>>>>>>>>> *+1 (973) 953-6299*
> > > >> > > > > > > >>>>>>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>>>>> [image:
> > > >> https://www.linkedin.com//in/chaibapat25]
> > > >> > > > > > > >>>>>>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> > > >> > > > > > > >>>>>>>>>>>>> https://www.facebook.com/chaibapat
> > > >> > > > > > > >>>>>>>>>>>>>> ]
> > > >> > > > > > > >>>>>>>>>>>>>> <https://www.facebook.com/chaibapchya
> > > >[image:
> > > >> > > > > > > >>>>>>>>>>>>>> https://twitter.com/ChaiBapchya] <
> > > >> > > > > > > >>>>>>>> https://twitter.com/ChaiBapchya
> > > >> > > > > > > >>>>>>>>>>>>>> [image:
> > > >> > > > > > > >>>>>>>>>>>>>> https://www.linkedin.com//in/chaibapat25
> ]
> > > >> > > > > > > >>>>>>>>>>>>>> <
> https://www.linkedin.com//in/chaibapchya/
> > >
> > > >> > > > > > > >>>>>>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>> --
> > > >> > > > > > > >>>>>>>>>> *Chaitanya Prakash Bapat*
> > > >> > > > > > > >>>>>>>>>> *+1 (973) 953-6299*
> > > >> > > > > > > >>>>>>>>>>
> > > >> > > > > > > >>>>>>>>>> [image:
> > https://www.linkedin.com//in/chaibapat25
> > > ]
> > > >> > > > > > > >>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> > > >> > > > > > > >>>>>>>>> https://www.facebook.com/chaibapat
> > > >> > > > > > > >>>>>>>>>> ]
> > > >> > > > > > > >>>>>>>>>> <https://www.facebook.com/chaibapchya
> >[image:
> > > >> > > > > > > >>>>>>>>>> https://twitter.com/ChaiBapchya] <
> > > >> > > > > > > >>>>> https://twitter.com/ChaiBapchya
> > > >> > > > > > > >>>>>>>>>> [image:
> > > >> > > > > > > >>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> > > >> > > > > > > >>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
> > > >> > > > > > > >>>>>>>>>>
> > > >> > > > > > > >>>>>>>
> > > >> > > > > > > >>>>>>> --
> > > >> > > > > > > >>>>>>> Sandeep Krishnamurthy
> > > >> > > > > > > >>>>>>>
> > > >> > > > > > > >>>>>
> > > >> > > > > > > >>>>
> > > >> > > > > > > >>>
> > > >> > > > > > > >>>
> > > >> > > > > > > >>> --
> > > >> > > > > > > >>> *Chaitanya Prakash Bapat*
> > > >> > > > > > > >>> *+1 (973) 953-6299*
> > > >> > > > > > > >>>
> > > >> > > > > > > >>> [image: https://www.linkedin.com//in/chaibapat25]
> > > >> > > > > > > >>> <https://github.com/ChaiBapchya>[image:
> > > >> > > > > > > https://www.facebook.com/chaibapat
> > > >> > > > > > > >>> ]
> > > >> > > > > > > >>> <https://www.facebook.com/chaibapchya>[image:
> > > >> > > > > > > >>> https://twitter.com/ChaiBapchya] <
> > > >> > > > https://twitter.com/ChaiBapchya
> > > >> > > > > > > >[image:
> > > >> > > > > > > >>> https://www.linkedin.com//in/chaibapat25]
> > > >> > > > > > > >>> <https://www.linkedin.com//in/chaibapchya/>
> > > >> > > > > > > >>>
> > > >> > > > > > > >>
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > --
> > > >> > > > > *Chaitanya Prakash Bapat*
> > > >> > > > > *+1 (973) 953-6299*
> > > >> > > > >
> > > >> > > > > [image: https://www.linkedin.com//in/chaibapat25]
> > > >> > > > > <https://github.com/ChaiBapchya>[image:
> > > >> > > > https://www.facebook.com/chaibapat
> > > >> > > > > ]
> > > >> > > > > <https://www.facebook.com/chaibapchya>[image:
> > > >> > > > > https://twitter.com/ChaiBapchya] <
> > > https://twitter.com/ChaiBapchya
> > > >> > > > >[image:
> > > >> > > > > https://www.linkedin.com//in/chaibapat25]
> > > >> > > > > <https://www.linkedin.com//in/chaibapchya/>
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> > > --
> > > >> > > *Chaitanya Prakash Bapat*
> > > >> > > *+1 (973) 953-6299*
> > > >> > >
> > > >> > > [image: https://www.linkedin.com//in/chaibapat25]
> > > >> > > <https://github.com/ChaiBapchya>[image:
> > > >> > https://www.facebook.com/chaibapat
> > > >> > > ]
> > > >> > > <https://www.facebook.com/chaibapchya>[image:
> > > >> > > https://twitter.com/ChaiBapchya] <
> https://twitter.com/ChaiBapchya
> > > >> > >[image:
> > > >> > > https://www.linkedin.com//in/chaibapat25]
> > > >> > > <https://www.linkedin.com//in/chaibapchya/>
> > > >> > >
> > > >> >
> > > >>
> > > >>
> > > >> --
> > > >> *Chaitanya Prakash Bapat*
> > > >> *+1 (973) 953-6299*
> > > >>
> > > >> [image: https://www.linkedin.com//in/chaibapat25]
> > > >> <https://github.com/ChaiBapchya>[image:
> > > >> https://www.facebook.com/chaibapat]
> > > >> <https://www.facebook.com/chaibapchya>[image:
> > > >> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> > > >[image:
> > > >> https://www.linkedin.com//in/chaibapat25]
> > > >> <https://www.linkedin.com//in/chaibapchya/>
> > > >>
> > > >
> > >
> >
> >
> > --
> > *Chaitanya Prakash Bapat*
> > *+1 (973) 953-6299*
> >
> > [image: https://www.linkedin.com//in/chaibapat25]
> > <https://github.com/ChaiBapchya>[image:
> https://www.facebook.com/chaibapat
> > ]
> > <https://www.facebook.com/chaibapchya>[image:
> > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> >[image:
> > https://www.linkedin.com//in/chaibapat25]
> > <https://www.linkedin.com//in/chaibapchya/>
> >
>
--
Sandeep Krishnamurthy
Re: MXNet Bot Demo
Posted by Marco de Abreu <ma...@gmail.com>.
Sure, already done.
-Marco
On Mon, Mar 23, 2020 at 8:53 PM Chaitanya Bapat <ch...@gmail.com>
wrote:
> Hello,
> Update: Apache Infra Ticket for MXNet Bot
> Thanks once again, Marco for opening the ticket. But turns out, Apache
> Infra folks closed it stating: "Security concerns around allowing unknown
> person to submit PR and run our hardware". Furthermore, it goes onto state
> that bot circumvents the dependence on Jenkins Admins which is like solving
> a problem that doesn't exist.
>
> I sense there is some confusion in the communication (maybe on my part). It
> turns out the security concerns aren't actually correct.
>
> 1. Unknown person can submit a PR (before & after bot proposal), and run
> our hardware (pre as well as post bot).
> 2. Code should be reviewed by somebody with an ICLA on file. This doesn't
> change either. Prior to merging a PR, code has to be approved by a
> committer just like before.
> Overall it looks like the job of the bot isn't clear to folks in Apache
> Infra. Bot simply is a means for triggering CI (which could be done
> manually by Log In to Jenkins -> PR -> Job -> Build) and doesn't quite
> tweak with merging procedure. Yes, only addition is now unknown person (PR
> Author) can trigger CI with a message (but that was possible anyway by
> pushing a commit. Bot just prevents users from pushing empty commits and
> building entire suite).
>
> As can be seen from last 10 open PRs as of Monday 23rd March, 12pm PT most
> PRs fail on 1/2 jobs. In such a scenario, the proposed MXNet bot would come
> in handy for just invoking CI on that specific build (instead of a
> non-committer PR Author to push empty commit : hurting on the resource,
> time & cost considerations apart from undesirable dev experience)
>
> @Marco Since I am a non-committer, I guess these 2 clarifications need to
> be conveyed to the Apache Infra by someone with Committer access.
>
> What do you think?
>
> Thanks,
> Chai
>
> On Sat, 21 Mar 2020 at 16:08, Marco de Abreu <ma...@gmail.com>
> wrote:
>
> > Hello,
> >
> > the ticket has been created:
> > https://issues.apache.org/jira/browse/INFRA-20005
> >
> > Best regards,
> > Marco
> >
> > On Thu, Mar 19, 2020 at 11:49 PM Marco de Abreu <marco.g.abreu@gmail.com
> >
> > wrote:
> >
> > > Sounds like a good plan!
> > >
> > > Please send me the URL (please make sure it's backed by DNS and not
> just
> > > the gateway URL) of the webhook handler, GitHub events you're
> interested
> > in
> > > and the shared secret in a private email to my personal email address.
> I
> > > will then create the ticket with Apache infra.
> > >
> > > -Marco
> > >
> > > Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 19. März 2020,
> > > 23:07:
> > >
> > >> @Marco Alright, it makes total sense to test out the Bot feature
> > alongside
> > >> auto-trigger as a transition.
> > >>
> > >> Path Forward:
> > >> 1. Setup MXNet Bot on apache/incubator-mxnet repo (GitHub WebHook and
> > >> Infra)
> > >> 2. We don't turn off automatic trigger of PR builds for now.
> > >> 3. Hopefully, bot is used by developers to trigger specific jobs
> > >> 4. Later on (say around April 20), let's discuss the possibility of
> > >> switching off auto-trigger (with appropriate data) if it makes sense.
> > >> Thanks Marco for volunteering to help enable the web hook on
> > >> apache/incubator-mxnet. Let me know if we can sync up on Slack channel
> > to
> > >> get the ball rolling.
> > >>
> > >> Thanks once again for the entire community to step in and help try out
> > >> this
> > >> Bot.
> > >> Chai
> > >>
> > >> On Wed, 18 Mar 2020 at 17:07, Marco de Abreu <marco.g.abreu@gmail.com
> >
> > >> wrote:
> > >>
> > >> > Hi, that's correct. But as stated previously, it's not an option to
> > >> remove
> > >> > the hook. For now, I'd like to see how the system behaves while it's
> > >> > optional. Later on, we can talk about revisiting this decision. But
> to
> > >> me
> > >> > it's not an option to deploy an entirely new system and approach
> > without
> > >> > having a transition or even a timeframe in which we are able to fall
> > >> back.
> > >> >
> > >> > I'm happy to support the deployment of the bot and add an additional
> > >> > webhook to enable it's functionality to support selective triggering
> > by
> > >> PR
> > >> > authors and committers, but I will not support the disabling of
> > >> automatic
> > >> > triggering of branches or PRs.
> > >> >
> > >> > -Marco
> > >> >
> > >> > Chaitanya Bapat <ch...@gmail.com> schrieb am Mi., 18. März
> 2020,
> > >> > 21:00:
> > >> >
> > >> > > Hey Marco,
> > >> > >
> > >> > > I thought currently every commit on PR and master triggers CI
> > >> > > because
> > >> > > a. github webhook points to Jenkins Server
> > >> > > b. GH Webhook events trigger builds on Jenkins for all commits to
> > any
> > >> > > branch in apache/incubator-mxnet
> > >> > > may it be master/PR/non-PR
> > >> > > Reason:
> > >> > > Because all the 3 types of branches are discovered by Jenkins
> > (non-PR
> > >> > > (including master) and PR)
> > >> > >
> > >> > > Proposal: Remove GitHub WebHook to Jenkins and replace with GH
> > >> Webhook to
> > >> > > Lambda
> > >> > > But after I remove the github webhook that points to Jenkins :
> *N**o
> > >> > commit
> > >> > > will trigger Jenkins build by default* (as Jenkins wont receive GH
> > >> > events)
> > >> > > Only those that Bot deems fit will be triggered (using Jenkins API
> > >> > invoked
> > >> > > by Lambda).
> > >> > > Hence its needed to handle that case of master merge.
> > >> > > Am I understanding this correctly?
> > >> > >
> > >> > > On Wed, 18 Mar 2020 at 04:23, Marco de Abreu <
> > marco.g.abreu@gmail.com
> > >> >
> > >> > > wrote:
> > >> > >
> > >> > > > Thanks Chai, sounds good to me. Could you elaborate a bit on the
> > >> point
> > >> > > > about triggering a CI run after the PR has been merged? We
> already
> > >> to
> > >> > > that
> > >> > > > automatically for the master, so what's the benefit to do it
> > twice?
> > >> > > >
> > >> > > > -Marco
> > >> > > >
> > >> > > > Chaitanya Bapat <ch...@gmail.com> schrieb am Mi., 18. März
> > >> 2020,
> > >> > > > 09:30:
> > >> > > >
> > >> > > > > Update:
> > >> > > > >
> > >> > > > > > we can ensure that all CI runs ran on the commit that will
> be
> > >> > merged
> > >> > > > > @Sam Skalicky <sa...@gmail.com> Branch Protection is
> > added
> > >> to
> > >> > > > public
> > >> > > > > MXNet repo. It ensures that for every PR to be merged, the CI
> > >> passes.
> > >> > > All
> > >> > > > > the jobs selected "required" jobs will have to be green for
> the
> > >> PR to
> > >> > > be
> > >> > > > > merged. Ofcourse, users with "Adminstrator" access can merge
> > >> without
> > >> > it
> > >> > > > but
> > >> > > > > that's just a backdoor. It is the case now and will continue
> to
> > be
> > >> > the
> > >> > > > case
> > >> > > > > with the inclusion of Bot.
> > >> > > > >
> > >> > > > > > easily verify that the CI has executed all runs on the
> commit
> > >> that
> > >> > > will
> > >> > > > > be merged
> > >> > > > > GitHub UI shows all the jobs and the status corresponding to
> it
> > on
> > >> > > every
> > >> > > > > commit. That should suffice. For the merged commits, Repo ->
> > >> Commits
> > >> > ->
> > >> > > > > Commit ID (Status) can be tracked currently (only way that I
> > know
> > >> > of).
> > >> > > > > Moreover, it is beyond the scope of this project (and possibly
> > >> out of
> > >> > > our
> > >> > > > > control since this is purely GitHub UI specific use-case).
> > >> > > > >
> > >> > > > > Thanks @przemyslaw for supporting the opt-in.
> > >> > > > >
> > >> > > > > Thanks everyone in the community for sharing concerns, voicing
> > >> your
> > >> > > > opinion
> > >> > > > > and participating in the discussion.
> > >> > > > > Thanks to those who attended the demo last Friday.
> > >> > > > >
> > >> > > > > Action items from that discussion
> > >> > > > > 1. Handle master merge builds [Done]
> > >> > > > > Bot runs entire CI suite after the PR is merged and comments
> on
> > >> the
> > >> > PR
> > >> > > > > about the same.
> > >> > > > > Design decision :
> > >> > > > > MXNet Bot comment about master merge build on the *merge
> commit
> > vs
> > >> > PR*.
> > >> > > > > After the PR is merged, Bot runs entire CI and comments the
> > >> result of
> > >> > > CI
> > >> > > > > trigger on the PR (because it is easy to track on a PR rather
> > than
> > >> > > > > commenting inside the merge commit)
> > >> > > > >
> > >> > > > > 2. Idempotent condition
> > >> > > > > In case of already running build, if an attempt is made to
> > >> retrigger
> > >> > > the
> > >> > > > > job then what should be the response
> > >> > > > > a. Not to re-trigger, let the ongoing build continue till
> > >> completion
> > >> > > > > b. End the ongoing build and re-trigger
> > >> > > > > c. Let the ongoing build continue, re-trigger new build
> > >> > > > >
> > >> > > > > From resource saving point of view, *c* looks costly and a can
> > be
> > >> > > > > avoided/optimized by B.
> > >> > > > > In case when a re-trigger was started "erroneously" then
> killing
> > >> > > ongoing
> > >> > > > > build and re-trigger is a waste.
> > >> > > > > In case when ongoing build failed in one sub-part, then
> > >> re-triggering
> > >> > > is
> > >> > > > > justified.
> > >> > > > > Erroneous re-triggers would be less often while conscious
> > >> re-triggers
> > >> > > to
> > >> > > > > suppress failure is more common use-case. It looks like a safe
> > >> > > assumption
> > >> > > > > to make given the trade-off.
> > >> > > > > [Open to debate]
> > >> > > > >
> > >> > > > > 3. Add security consideration [Use of secret manager, but
> > without
> > >> > > > > auto-rotation due to Jenkins manual config requirement] [Done]
> > >> > > > > 4. New PR Instruction message by the Bot [Done]
> > >> > > > > Thanks to the suggestion of Leonard, supported by others. I've
> > now
> > >> > > added
> > >> > > > > the feature where the Bot comments a help message. [For
> > reference
> > >> -
> > >> > > > > https://github.com/ChaiBapchya/incubator-mxnet/pull/52]
> > >> > > > >
> > >> > > > > Barring the opt-in vs opt-out debate & idempotency, consensus
> > was
> > >> > > quickly
> > >> > > > > reached for the rest.
> > >> > > > >
> > >> > > > > In the coming days, I hope to roll-out this feature into Prod
> > >> (public
> > >> > > > > MXNet) for all devs to use.
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > Chai
> > >> > > > >
> > >> > > > > On Mon, 16 Mar 2020 at 11:57, Marco de Abreu <
> > >> > marco.g.abreu@gmail.com>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Well that's generally a problem with a deferred CI approach
> > (CI
> > >> is
> > >> > > run
> > >> > > > at
> > >> > > > > > commit and not at merge time). This can either be solved
> > through
> > >> > the
> > >> > > > > other
> > >> > > > > > proposal that's currently on dev@, by having a bot which
> does
> > >> > merges
> > >> > > > by
> > >> > > > > > having a global lock and a merge queue or by accepting the
> > >> issue.
> > >> > > > Reality
> > >> > > > > > right now is that we're running that model where two PRs
> which
> > >> are
> > >> > > > merged
> > >> > > > > > in parallel might break one another. One thing to consider
> > >> though
> > >> > is
> > >> > > > that
> > >> > > > > > this breakage would have to be introduced in two separate
> > parts
> > >> > since
> > >> > > > > > otherwise there'd be merge conflicts. I think we had that
> > >> situation
> > >> > > > twice
> > >> > > > > > so far and the result was a quick revert, so I'd say that
> > it's a
> > >> > > > problem
> > >> > > > > > that can happily be accepted. All other solutions basically
> > >> require
> > >> > > > some
> > >> > > > > > form of single-threaded and globally locked solution which
> > >> limits
> > >> > us
> > >> > > in
> > >> > > > > > scalability. I'd recommend to just accept that risk and
> revert
> > >> a PR
> > >> > > in
> > >> > > > > case
> > >> > > > > > it actually had a conflict.
> > >> > > > > >
> > >> > > > > > -Marco
> > >> > > > > >
> > >> > > > > > On Mon, Mar 16, 2020 at 6:29 PM Skalicky, Sam
> > >> > > > <sskalic@amazon.com.invalid
> > >> > > > > >
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > We probably need some way to track which CI runs ran for
> > which
> > >> > > commit
> > >> > > > > > too,
> > >> > > > > > > that way we can ensure that all CI runs ran on the commit
> > that
> > >> > will
> > >> > > > be
> > >> > > > > > > merged. Maybe the bot can comment with the commit hash
> when
> > >> > users
> > >> > > > > > command
> > >> > > > > > > it to do something. Although since users can trigger
> > >> individual
> > >> > CI
> > >> > > > runs
> > >> > > > > > its
> > >> > > > > > > possible to have some commits run some CI runs but not
> > >> others. We
> > >> > > > need
> > >> > > > > > some
> > >> > > > > > > way to easily verify that the CI has executed all runs on
> > the
> > >> > > commit
> > >> > > > > that
> > >> > > > > > > will be merged.
> > >> > > > > > >
> > >> > > > > > > Sam
> > >> > > > > > >
> > >> > > > > > > > On Mar 13, 2020, at 8:28 PM, Przemysław Trędak <
> > >> > > ptrendx@apache.org
> > >> > > > >
> > >> > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > CAUTION: This email originated from outside of the
> > >> > organization.
> > >> > > Do
> > >> > > > > not
> > >> > > > > > > click links or open attachments unless you can confirm the
> > >> sender
> > >> > > and
> > >> > > > > > know
> > >> > > > > > > the content is safe.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > I personally like the idea of opt-in more than opt-out:
> > >> > > > > > > > - ultimately PR author wants the PR to be merged so they
> > (or
> > >> > > > > committer
> > >> > > > > > > reviewing the PR) will trigger the CI
> > >> > > > > > > > - if it is easy to trigger the PR via the bot command
> then
> > >> the
> > >> > > > amount
> > >> > > > > > of
> > >> > > > > > > work per PR should be less than with opt-out (since most
> of
> > >> the
> > >> > > > commits
> > >> > > > > > > should then be marked as [skip ci] or something similar
> > >> > > > > > > >
> > >> > > > > > > > +1 to the bot making a comment on each new PR with its
> > >> commands
> > >> > > > (and
> > >> > > > > > > also explaining, or at least giving links to the general
> PR
> > >> > process
> > >> > > > so
> > >> > > > > > new
> > >> > > > > > > PR authors are not lost). Maybe we could make the bot
> > >> recognize
> > >> > if
> > >> > > > the
> > >> > > > > PR
> > >> > > > > > > author is new or existing contributor and offer advice
> based
> > >> on
> > >> > > that?
> > >> > > > > > > >
> > >> > > > > > > > Thanks
> > >> > > > > > > > Przemek
> > >> > > > > > > >
> > >> > > > > > > > On 2020/03/13 22:06:58, Marco de Abreu <
> > >> > marco.g.abreu@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > > > >> Hi,
> > >> > > > > > > >>
> > >> > > > > > > >> since it's no longer necessary to push a new commit to
> > >> trigger
> > >> > > CI,
> > >> > > > > it
> > >> > > > > > > will
> > >> > > > > > > >> already reduce the costs. But to me, requiring an
> action
> > to
> > >> > > enable
> > >> > > > > CI
> > >> > > > > > > after
> > >> > > > > > > >> a PR has been created initially, is a no go. User can
> opt
> > >> out
> > >> > of
> > >> > > > CI,
> > >> > > > > > but
> > >> > > > > > > >> the default has to be CI being triggered automatically
> > for
> > >> > every
> > >> > > > > > commit
> > >> > > > > > > >> unless specifically disabled by a participant. I'm also
> > >> fine
> > >> > > with
> > >> > > > > > > >> triggering certain additional jobs (think about
> running a
> > >> > > nightly
> > >> > > > > job
> > >> > > > > > > upon
> > >> > > > > > > >> request for a PR) to require a manual step, but the PR
> > >> > > validation
> > >> > > > > > > pipelines
> > >> > > > > > > >> have to run automatically. Every check that is marked
> as
> > >> > > > "Required"
> > >> > > > > in
> > >> > > > > > > >> GitHub has to be automatically kicked off.
> > >> > > > > > > >>
> > >> > > > > > > >> -Marco
> > >> > > > > > > >>
> > >> > > > > > > >> On Fri, Mar 13, 2020 at 9:50 PM Chaitanya Bapat <
> > >> > > > > chai.bapat@gmail.com
> > >> > > > > > >
> > >> > > > > > > >> wrote:
> > >> > > > > > > >>
> > >> > > > > > > >>> Firstly,
> > >> > > > > > > >>> Sorry I missed out on attaching the mail thread that
> was
> > >> sent
> > >> > > on
> > >> > > > > 12th
> > >> > > > > > > >>> February for notifying the community of the upcoming
> > >> changes
> > >> > to
> > >> > > > the
> > >> > > > > > > MXNet
> > >> > > > > > > >>> CI
> > >> > > > > > > >>> For reference :
> > >> > > > > > > >>>
> > >> > > > > > > >>>
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://lists.apache.org/thread.html/r09a6ab2803a996fc80e00fe39ed312fa4865e8805e08df847f1addad%40%3Cdev.mxnet.apache.org%3E
> > >> > > > > > > >>>
> > >> > > > > > > >>> Now to the questions,
> > >> > > > > > > >>>> Is it possible for re-triggering a single job to be
> > >> abused?
> > >> > > > > > > >>> @Tao In the case when a user re-triggers a single job
> > >> > multiple
> > >> > > > > times,
> > >> > > > > > > that
> > >> > > > > > > >>> will be visible in the PR conversation thread. A
> > >> committer,
> > >> > > even
> > >> > > > > > after
> > >> > > > > > > he
> > >> > > > > > > >>> has approved the PR before, generally takes a look at
> > the
> > >> > final
> > >> > > > > state
> > >> > > > > > > of
> > >> > > > > > > >>> the PR before merging. Would it be fair to assume the
> > >> > committer
> > >> > > > > could
> > >> > > > > > > take
> > >> > > > > > > >>> the multiple re-trigger of a single job into account
> > >> before
> > >> > > > > merging?
> > >> > > > > > > The
> > >> > > > > > > >>> committer then has the option to invoke `@mxnet-bot
> run
> > ci
> > >> > > [all]
> > >> > > > `
> > >> > > > > to
> > >> > > > > > > >>> trigger the entire build pipeline one last to counter
> > the
> > >> > > abuse.
> > >> > > > > This
> > >> > > > > > > is
> > >> > > > > > > >>> aligned with what @Leonard said.
> > >> > > > > > > >>>
> > >> > > > > > > >>> @Sandeep Thanks a lot for collecting and sharing
> > valuable
> > >> > data.
> > >> > > > I'd
> > >> > > > > > > concur
> > >> > > > > > > >>> with the opinion that given the existing things
> > >> committers &
> > >> > PR
> > >> > > > > > Authors
> > >> > > > > > > >>> take care of, invoking CI shouldn't be that big of an
> > >> > > additional
> > >> > > > > > > burden.
> > >> > > > > > > >>>
> > >> > > > > > > >>> @Marco With the opt-out, the onus remains on the PR
> > >> Author.
> > >> > It
> > >> > > > > > doesn't
> > >> > > > > > > help
> > >> > > > > > > >>> reduce the resource usage. Hence, it was suggested to
> > >> switch
> > >> > to
> > >> > > > > > > >>> opt-in. @Leo's suggestion for proactive commenting on
> > the
> > >> > part
> > >> > > of
> > >> > > > > bot
> > >> > > > > > > makes
> > >> > > > > > > >>> sense and is doable.
> > >> > > > > > > >>>
> > >> > > > > > > >>> Default : opt-out and User initiated opt-in (with
> > >> addressing
> > >> > > > Leo's
> > >> > > > > > fix
> > >> > > > > > > for
> > >> > > > > > > >>> the usability issue you correctly pointed out )
> > >> > > > > > > >>> @Marco How does this sound to you?
> > >> > > > > > > >>>
> > >> > > > > > > >>> Again, thank you all for chiming in and voicing your
> > >> opinion.
> > >> > > > > > > Appreciate
> > >> > > > > > > >>> it.
> > >> > > > > > > >>> We can take ahead these discussions in today's demo
> > >> meeting.
> > >> > > > > [Design
> > >> > > > > > > Doc
> > >> > > > > > > >>> <
> > >> > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > >> > > > >]
> > >> > > > > > > [Demo
> > >> > > > > > > >>> Video <https://www.youtube.com/watch?v=gfOGwZId8aU>]
> > >> > > > > > > >>>
> > >> > > > > > > >>> Thanks,
> > >> > > > > > > >>> Chai
> > >> > > > > > > >>>
> > >> > > > > > > >>> On Fri, 13 Mar 2020 at 12:34, Marco de Abreu <
> > >> > > > > > marco.g.abreu@gmail.com>
> > >> > > > > > > >>> wrote:
> > >> > > > > > > >>>
> > >> > > > > > > >>>> I'd recommend that the bot makes an initial comment
> > when
> > >> a
> > >> > PR
> > >> > > > gets
> > >> > > > > > > opened
> > >> > > > > > > >>>> and informs the users of its commands. It then tells
> > the
> > >> > user
> > >> > > > the
> > >> > > > > > > commend
> > >> > > > > > > >>>> to opt out of CI.
> > >> > > > > > > >>>>
> > >> > > > > > > >>>> -Marco
> > >> > > > > > > >>>>
> > >> > > > > > > >>>> Lausen, Leonard <la...@amazon.com.invalid> schrieb
> am
> > >> Fr.,
> > >> > > 13.
> > >> > > > > > März
> > >> > > > > > > >>> 2020,
> > >> > > > > > > >>>> 20:27:
> > >> > > > > > > >>>>
> > >> > > > > > > >>>>> On opt-out: People may be unaware of opt-out would
> not
> > >> use
> > >> > > it.
> > >> > > > > > There
> > >> > > > > > > is
> > >> > > > > > > >>>> no
> > >> > > > > > > >>>>> incentive to use opt-out, as the PR author doesn't
> pay
> > >> any
> > >> > > > money
> > >> > > > > > for
> > >> > > > > > > CI
> > >> > > > > > > >>>>> run.
> > >> > > > > > > >>>>>
> > >> > > > > > > >>>>> I agree with Marco though that opt-in alone may
> cause
> > >> > > usability
> > >> > > > > > > issues,
> > >> > > > > > > >>>> as
> > >> > > > > > > >>>>> contributors may not be aware of how to trigger the
> > CI.
> > >> > > > > > > >>>>> One solution is that the bot proactively comments on
> > >> the PR
> > >> > > and
> > >> > > > > > > reminds
> > >> > > > > > > >>>> the
> > >> > > > > > > >>>>> author to trigger running CI once the author deems
> the
> > >> PR
> > >> > > > ready.
> > >> > > > > > > >>>>>
> > >> > > > > > > >>>>> But even if we choose opt-out, the bot will still
> add
> > a
> > >> lot
> > >> > > of
> > >> > > > > > value,
> > >> > > > > > > >>> as
> > >> > > > > > > >>>> PR
> > >> > > > > > > >>>>> authors can retrigger single jobs that have failed
> due
> > >> to
> > >> > > > > > flakiness.
> > >> > > > > > > >>>>>
> > >> > > > > > > >>>>>> Is it possible for re-triggering a single job to be
> > >> > abused?
> > >> > > > For
> > >> > > > > > > >>>> example,
> > >> > > > > > > >>>>>> the author spends two days re-triggering a flaky
> job
> > to
> > >> > make
> > >> > > > it
> > >> > > > > > > pass.
> > >> > > > > > > >>>>>
> > >> > > > > > > >>>>> Yes, this is possible. I suggest the committer who
> > >> likes to
> > >> > > > > merge a
> > >> > > > > > > PR
> > >> > > > > > > >>>>> needs to
> > >> > > > > > > >>>>> make a good judgement here if a PR is abusing the
> > >> feature,
> > >> > > and
> > >> > > > if
> > >> > > > > > so,
> > >> > > > > > > >>>>> retrigger
> > >> > > > > > > >>>>> all CI runs.
> > >> > > > > > > >>>>>
> > >> > > > > > > >>>>> Best regards
> > >> > > > > > > >>>>> Leonard
> > >> > > > > > > >>>>>
> > >> > > > > > > >>>>> On Fri, 2020-03-13 at 08:07 +0100, Marco de Abreu
> > wrote:
> > >> > > > > > > >>>>>> Thanks for the data Sandeep. In these cases it
> sounds
> > >> like
> > >> > > it
> > >> > > > > > would
> > >> > > > > > > >>>> have
> > >> > > > > > > >>>>>> rather been better when people explicitly turned
> off
> > >> CI in
> > >> > > > that
> > >> > > > > > > case.
> > >> > > > > > > >>>>>> What's the argument against an opt-out instead of
> an
> > >> > opt-in?
> > >> > > > > > > >>>>>>
> > >> > > > > > > >>>>>> My intention is that I consider it quite cumbersome
> > to
> > >> > make
> > >> > > > it a
> > >> > > > > > > >>>>> *required*
> > >> > > > > > > >>>>>> step to always trigger CI manually, even if just
> > >> > submitting
> > >> > > a
> > >> > > > > > small
> > >> > > > > > > >>> PR.
> > >> > > > > > > >>>>> I'd
> > >> > > > > > > >>>>>> rather see people explicitly turning off CI if they
> > >> > wouldn't
> > >> > > > > like
> > >> > > > > > to
> > >> > > > > > > >>>> use
> > >> > > > > > > >>>>> it
> > >> > > > > > > >>>>>> - and there's also the "draft" stage for a PR which
> > >> some
> > >> > > > > > > contributors
> > >> > > > > > > >>>> are
> > >> > > > > > > >>>>>> using.
> > >> > > > > > > >>>>>>
> > >> > > > > > > >>>>>> With regards to WIP and do not review: I think
> these
> > >> are
> > >> > use
> > >> > > > > cases
> > >> > > > > > > >>>> where
> > >> > > > > > > >>>>>> you want CI feedback, as otherwise you wouldn't
> have
> > >> > opened
> > >> > > > the
> > >> > > > > > PR.
> > >> > > > > > > >>> If
> > >> > > > > > > >>>>> you
> > >> > > > > > > >>>>>> don't want human feedback and neither machine
> > feedback,
> > >> > why
> > >> > > > open
> > >> > > > > > the
> > >> > > > > > > >>> PR
> > >> > > > > > > >>>>> at
> > >> > > > > > > >>>>>> all?
> > >> > > > > > > >>>>>>
> > >> > > > > > > >>>>>> -Marco
> > >> > > > > > > >>>>>>
> > >> > > > > > > >>>>>>
> > >> > > > > > > >>>>>> sandeep krishnamurthy <sandeep.krishna98@gmail.com
> >
> > >> > schrieb
> > >> > > > am
> > >> > > > > > Fr.,
> > >> > > > > > > >>>> 13.
> > >> > > > > > > >>>>>> März 2020, 05:24:
> > >> > > > > > > >>>>>>
> > >> > > > > > > >>>>>>> I tried to gather some data for us to discuss this
> > >> topic
> > >> > in
> > >> > > > > this
> > >> > > > > > > >>>>> thread. I
> > >> > > > > > > >>>>>>> tried to count number of un-necessary builds by
> > >> looking
> > >> > at
> > >> > > > most
> > >> > > > > > > >>>> recent
> > >> > > > > > > >>>>> (as
> > >> > > > > > > >>>>>>> of 12, March 9 PM PST) 50 PRs merged to master and
> > 50
> > >> > PRs.
> > >> > > > > > > >>>>>>> Identifying un-necessary builds is bit
> subjective. I
> > >> > tried
> > >> > > to
> > >> > > > > be
> > >> > > > > > > >>> more
> > >> > > > > > > >>>>>>> conservative where I didn't count a build as
> > >> un-necessary
> > >> > > if
> > >> > > > I
> > >> > > > > > was
> > >> > > > > > > >>> in
> > >> > > > > > > >>>>>>> doubt. Hence, I was not able to automate, but I
> made
> > >> an
> > >> > > > effort
> > >> > > > > to
> > >> > > > > > > >>> go
> > >> > > > > > > >>>>>>> through PRs manually and use below criteria to
> > >> identify
> > >> > > > > > > >>> un-necessary
> > >> > > > > > > >>>>>>> commits triggering the builds.
> > >> > > > > > > >>>>>>>
> > >> > > > > > > >>>>>>> 1. Explicitly marked as WIP / do not review PR
> > >> > > > > > > >>>>>>> 2. Incremental WIP commit and finally
> commenting a
> > >> > commit
> > >> > > > > > > >>> “trigger
> > >> > > > > > > >>>>> CI”
> > >> > > > > > > >>>>>>> 3. Multiple commits to address all comments from
> > >> single
> > >> > > > > review.
> > >> > > > > > > >>>>> This is
> > >> > > > > > > >>>>>>> assuming we see a comment, address them, commit,
> > >> next
> > >> > the
> > >> > > > > > > >>>> following
> > >> > > > > > > >>>>>>> comment
> > >> > > > > > > >>>>>>> 4. Sequence of documentation only changes
> > >> > > > > > > >>>>>>>
> > >> > > > > > > >>>>>>> I found there were around 42 avoidable builds from
> > >> most
> > >> > > > recent
> > >> > > > > 50
> > >> > > > > > > >>>>> merged
> > >> > > > > > > >>>>>>> PRs and around 86 builds from recent 50 open PRs.
> > >> > > > > > > >>>>>>>
> > >> > > > > > > >>>>>>> I synced up with other contributors (Joe Evans,
> > Chai)
> > >> > from
> > >> > > > > Amazon
> > >> > > > > > > >>> who
> > >> > > > > > > >>>>> is
> > >> > > > > > > >>>>>>> contributing to MXNet CI system. I was told that
> on
> > an
> > >> > > > average
> > >> > > > > it
> > >> > > > > > > >>>> costs
> > >> > > > > > > >>>>>>> around $84 per build and on an average 6 commits
> per
> > >> > merged
> > >> > > > PR
> > >> > > > > > (for
> > >> > > > > > > >>>>> year
> > >> > > > > > > >>>>>>> 2019). Going by that, it is approximately 1/6
> builds
> > >> are
> > >> > > > > > avoidable.
> > >> > > > > > > >>>>> [100 /
> > >> > > > > > > >>>>>>> 300 + 300 ]
> > >> > > > > > > >>>>>>>
> > >> > > > > > > >>>>>>>
> > >> > > > > > > >>>>>>> Usability should be top most priority. But, since
> > >> either
> > >> > a
> > >> > > > > > reviewer
> > >> > > > > > > >>>> or
> > >> > > > > > > >>>>> pr
> > >> > > > > > > >>>>>>> author can trigger the bot, is it really a hurdle
> > for
> > >> pr
> > >> > > > author
> > >> > > > > > or
> > >> > > > > > > >>>>> reviewer
> > >> > > > > > > >>>>>>> to call a bot to trigger CI? Given that PR author
> > and
> > >> > > > reviewer
> > >> > > > > is
> > >> > > > > > > >>>>> already
> > >> > > > > > > >>>>>>> actively commenting various details such as - PR
> > >> > > description,
> > >> > > > > > > >>> review
> > >> > > > > > > >>>>>>> comments and responses, adding labels etc.
> > >> > > > > > > >>>>>>>
> > >> > > > > > > >>>>>>>
> > >> > > > > > > >>>>>>> Me too curious to know the behavior for Tao's
> above
> > >> use
> > >> > > case.
> > >> > > > > > > >>>>>>>
> > >> > > > > > > >>>>>>>
> > >> > > > > > > >>>>>>> Best,
> > >> > > > > > > >>>>>>>
> > >> > > > > > > >>>>>>> Sandeep
> > >> > > > > > > >>>>>>>
> > >> > > > > > > >>>>>>> On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <
> > >> > mutouorz@gmail.com
> > >> > > >
> > >> > > > > > wrote:
> > >> > > > > > > >>>>>>>
> > >> > > > > > > >>>>>>>> Is it possible for re-triggering a single job to
> be
> > >> > > abused?
> > >> > > > > For
> > >> > > > > > > >>>>> example,
> > >> > > > > > > >>>>>>>> the author spends two days re-triggering a flaky
> > job
> > >> to
> > >> > > make
> > >> > > > > it
> > >> > > > > > > >>>>> pass. But
> > >> > > > > > > >>>>>>>> other jobs which have passed the validation may
> be
> > >> > broken
> > >> > > by
> > >> > > > > > > >>> other
> > >> > > > > > > >>>>>>> commits
> > >> > > > > > > >>>>>>>> during the two day without being noticed. And
> > finally
> > >> > the
> > >> > > PR
> > >> > > > > is
> > >> > > > > > > >>>>> merged
> > >> > > > > > > >>>>>>> with
> > >> > > > > > > >>>>>>>> underlying problems.
> > >> > > > > > > >>>>>>>>
> > >> > > > > > > >>>>>>>> On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <
> > >> > > > > > > >>>>> marco.g.abreu@gmail.com>
> > >> > > > > > > >>>>>>>> wrote:
> > >> > > > > > > >>>>>>>>
> > >> > > > > > > >>>>>>>>> In the end it only comes down to money,
> > considering
> > >> > that
> > >> > > > the
> > >> > > > > > > >>>>> system is
> > >> > > > > > > >>>>>>>> auto
> > >> > > > > > > >>>>>>>>> scaling, making the execution time constant.
> > >> > > > > > > >>>>>>>>>
> > >> > > > > > > >>>>>>>>> If we're trading money for usability, I
> certainly
> > >> would
> > >> > > > > prefer
> > >> > > > > > > >>>>>>> usability.
> > >> > > > > > > >>>>>>>>> I'd rather recommend to spend time on
> > parallelizing
> > >> > test
> > >> > > > > > > >>>> execution
> > >> > > > > > > >>>>> or
> > >> > > > > > > >>>>>>>>> getting rid of integration tests in the PR stage
> > >> > instead
> > >> > > > > > > >>> reducing
> > >> > > > > > > >>>>> the
> > >> > > > > > > >>>>>>>> costs
> > >> > > > > > > >>>>>>>>> by making people not use it. But taking a step
> > back
> > >> to
> > >> > > > > > > >>> requiring
> > >> > > > > > > >>>>> people
> > >> > > > > > > >>>>>>>> to
> > >> > > > > > > >>>>>>>>> manually trigger CI again doesn't feel right.
> > >> > > > > > > >>>>>>>>>
> > >> > > > > > > >>>>>>>>> I'm happy to see that bot deployed, but I do not
> > >> agree
> > >> > > with
> > >> > > > > > > >>>>> removing
> > >> > > > > > > >>>>>>> the
> > >> > > > > > > >>>>>>>>> auto trigger functionality for new commits.
> > >> > > > > > > >>>>>>>>>
> > >> > > > > > > >>>>>>>>> -Marco
> > >> > > > > > > >>>>>>>>>
> > >> > > > > > > >>>>>>>>> Chaitanya Bapat <ch...@gmail.com> schrieb
> am
> > >> Do.,
> > >> > > 12.
> > >> > > > > > > >>> März
> > >> > > > > > > >>>>> 2020,
> > >> > > > > > > >>>>>>>>> 22:47:
> > >> > > > > > > >>>>>>>>>
> > >> > > > > > > >>>>>>>>>> @Marco Thanks for pointing that out.
> > >> > > > > > > >>>>>>>>>> Tomorrow i.e. Friday, March 13, 2020 at 3:00
> PM -
> > >> 3:30
> > >> > > PM
> > >> > > > in
> > >> > > > > > > >>>>>>>> (UTC-08:00)
> > >> > > > > > > >>>>>>>>>> Pacific Time (US & Canada).
> > >> > > > > > > >>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>> When do we expect this bot to be deployed?
> > >> > > > > > > >>>>>>>>>> @Lin If all goes well in the next week I can
> > >> deploy it
> > >> > > to
> > >> > > > > > > >>>> public
> > >> > > > > > > >>>>>>> Apache
> > >> > > > > > > >>>>>>>>>> (provided I get permissions from Apache Infra)
> > >> > > > > > > >>>>>>>>>>
> > >> > > > > > > >>>>>>>>>> @Marco Thanks for your feedback.
> > >> > > > > > > >>>>>>>>>>> CI system has to support the community without
> > >> > > requiring
> > >> > > > > > > >>>>> people to
> > >> > > > > > > >>>>>>>>>> constantly shepherd every single run
> > >> > > > > > > >>>>>>>>>> We have data for the number of times CI was
> > >> triggered
> > >> > > > > > > >>>>> unnecessarily
> > >> > > > > > > >>>>>>>> which
> > >> > > > > > > >>>>>>>>>> includes
> > >> > > > > > > >>>>>>>>>> - Entire build triggered instead of specific
> > build
> > >> > > > > > > >>>>>>>>>> - CI triggered when PR is still work in
> progress
> > or
> > >> > not
> > >> > > > yet
> > >> > > > > > > >>>> ready
> > >> > > > > > > >>>>>>> (say
> > >> > > > > > > >>>>>>>> -
> > >> > > > > > > >>>>>>>>>> intermediate commits)
> > >> > > > > > > >>>>>>>>>> At the end its a trade-off
> > >> > > > > > > >>>>>>>>>> Money, Resources, Time to build for each and
> > every
> > >> > > commit
> > >> > > > vs
> > >> > > > > > > >>>>> Pain of
> > >> > > > > > > >>>>>>>>>> triggering builds
> > >> > > > > > > >>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>> Scan trigger plugin would poll SCM. Can we use
> > >> plugin
> > >> > > at
> > >> > > > > > > >>>>> scale?
> > >> > > > > > > >>>>>>>>>>
> > >> > > > > > > >>>>>>>>>> 1. I haven't tested it on scale. But I think
> with
> > >> the
> > >> > > > > current
> > >> > > > > > > >>>>> scale
> > >> > > > > > > >>>>>>> of
> > >> > > > > > > >>>>>>>>>> MXNet repo (191 open PRs i.e. checking for
> > changes
> > >> to
> > >> > > 191
> > >> > > > > > > >>>>> branches -
> > >> > > > > > > >>>>>>> It
> > >> > > > > > > >>>>>>>>>> should be manageable)
> > >> > > > > > > >>>>>>>>>> 2. What's the purpose of the plugin? tldr;
> Branch
> > >> > > > discovery
> > >> > > > > > > >>> or
> > >> > > > > > > >>>>> branch
> > >> > > > > > > >>>>>>>>>> indexing.
> > >> > > > > > > >>>>>>>>>> Scan trigger plugin comes into the picture only
> > >> once
> > >> > per
> > >> > > > PR
> > >> > > > > > > >>> per
> > >> > > > > > > >>>>> job
> > >> > > > > > > >>>>>>>>> (i.e. 8
> > >> > > > > > > >>>>>>>>>> times per PR for 8 jobs). It is basically done
> > >> when a
> > >> > > new
> > >> > > > PR
> > >> > > > > > > >>> is
> > >> > > > > > > >>>>> made
> > >> > > > > > > >>>>>>>> and
> > >> > > > > > > >>>>>>>>>> the job (say unix-cpu hasn't discovered the new
> > PR
> > >> > > branch
> > >> > > > > > > >>> yet).
> > >> > > > > > > >>>>>>> That's
> > >> > > > > > > >>>>>>>>> it.
> > >> > > > > > > >>>>>>>>>> So it shouldn't be a problem for public MXNet
> > repo.
> > >> > > > > > > >>>>>>>>>>
> > >> > > > > > > >>>>>>>>>> Thanks,
> > >> > > > > > > >>>>>>>>>> Chai
> > >> > > > > > > >>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>
> > >> > > > > > > >>>>>>>>>> On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <
> > >> > > > > > > >>>>>>> marco.g.abreu@gmail.com>
> > >> > > > > > > >>>>>>>>>> wrote:
> > >> > > > > > > >>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>> Btw you forgot to set a date and time for the
> > >> metting
> > >> > > > > > > >>>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>> On Thu, Mar 12, 2020 at 10:18 PM Marco de
> Abreu
> > <
> > >> > > > > > > >>>>>>>>> marco.g.abreu@gmail.com
> > >> > > > > > > >>>>>>>>>>> wrote:
> > >> > > > > > > >>>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>>> Thanks Chai, I generally like the idea of the
> > >> bot.
> > >> > But
> > >> > > > > > > >>> I'm
> > >> > > > > > > >>>>> not a
> > >> > > > > > > >>>>>>>>>>> supporter
> > >> > > > > > > >>>>>>>>>>>> of the idea to disable any automatic
> triggering
> > >> > > > > > > >>> (disabling
> > >> > > > > > > >>>>> the
> > >> > > > > > > >>>>>>>>> webhook
> > >> > > > > > > >>>>>>>>>> is
> > >> > > > > > > >>>>>>>>>>>> also not an option, considering that this
> will
> > >> > disable
> > >> > > > > > > >>>> master
> > >> > > > > > > >>>>>>>>>> triggers).
> > >> > > > > > > >>>>>>>>>>>> The CI system has to support the community
> > >> without
> > >> > > > > > > >>>> requiring
> > >> > > > > > > >>>>>>> people
> > >> > > > > > > >>>>>>>>> to
> > >> > > > > > > >>>>>>>>>>>> constantly shepherd every single run.
> Disabling
> > >> > > > automatic
> > >> > > > > > > >>>>>>>> triggering
> > >> > > > > > > >>>>>>>>>>> seems
> > >> > > > > > > >>>>>>>>>>>> like a step back to me.
> > >> > > > > > > >>>>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>>> Instead, I'd recommend that CI gets triggered
> > >> upon
> > >> > > every
> > >> > > > > > > >>>>> commit
> > >> > > > > > > >>>>>>> as
> > >> > > > > > > >>>>>>>>>> usual,
> > >> > > > > > > >>>>>>>>>>>> but people have the possibility to call a
> > >> "command"
> > >> > > > (i.e.
> > >> > > > > > > >>>>> make a
> > >> > > > > > > >>>>>>>>>> message
> > >> > > > > > > >>>>>>>>>>>> which results in the bot setting a label) to
> > >> disable
> > >> > > CI
> > >> > > > > > > >>>> until
> > >> > > > > > > >>>>>>> they
> > >> > > > > > > >>>>>>>>>> revoke
> > >> > > > > > > >>>>>>>>>>>> it. But the standard should still be that a
> new
> > >> > commit
> > >> > > > > > > >>>>> triggers a
> > >> > > > > > > >>>>>>>> new
> > >> > > > > > > >>>>>>>>>> CI
> > >> > > > > > > >>>>>>>>>>>> run.
> > >> > > > > > > >>>>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>>>
> > >> > > > > > > >>>>
> > >> > https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
> > >> > > > > > > >>>>>>>> seems
> > >> > > > > > > >>>>>>>>>> like
> > >> > > > > > > >>>>>>>>>>>> this would poll SCM. This will incur high
> quota
> > >> > > > > > > >>>>> restrictions. Are
> > >> > > > > > > >>>>>>>> you
> > >> > > > > > > >>>>>>>>>>> sure
> > >> > > > > > > >>>>>>>>>>>> that you can use that plugin at scale?
> > >> > > > > > > >>>>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>>> -Marco
> > >> > > > > > > >>>>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>>> On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <
> > >> > > > > > > >>>>> apeforest@gmail.com>
> > >> > > > > > > >>>>>>>>> wrote:
> > >> > > > > > > >>>>>>>>>>>>> Chai,
> > >> > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>>>> Awesome work. When do we expect this bot to
> be
> > >> > > > > > > >>> deployed?
> > >> > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>>>> Best,
> > >> > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>>>> Lin
> > >> > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>>>> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya
> > Bapat
> > >> <
> > >> > > > > > > >>>>>>>>> chai.bapat@gmail.com
> > >> > > > > > > >>>>>>>>>>>>> wrote:
> > >> > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>>>>> Hello MXNet community,
> > >> > > > > > > >>>>>>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>>>>> I have built an MXNet Bot <
> > >> > > > > > > >>>> https://github.com/mxnet-bot>
> > >> > > > > > > >>>>> that
> > >> > > > > > > >>>>>>>>>> allows
> > >> > > > > > > >>>>>>>>>>> PR
> > >> > > > > > > >>>>>>>>>>>>>> Authors, Committers and Jenkins Admins to
> > >> trigger
> > >> > CI
> > >> > > > > > > >>>>> manually.
> > >> > > > > > > >>>>>>>>>>>>>> It handles 2 problems
> > >> > > > > > > >>>>>>>>>>>>>> 1. Manual CI trigger instead of existing
> > >> automated
> > >> > > CI
> > >> > > > > > > >>>>> trigger
> > >> > > > > > > >>>>>>>>>>>>>> 2. Gives permissions to PR Authors (in
> > >> addition to
> > >> > > > > > > >>>> MXNet
> > >> > > > > > > >>>>>>>>> Committers
> > >> > > > > > > >>>>>>>>>>> and
> > >> > > > > > > >>>>>>>>>>>>>> Jenkins Admins)
> > >> > > > > > > >>>>>>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>>>>> Design Doc :
> > >> > > > > > > >>>>>>>>>>>>>>
> > >> > > > > > > >>>>>>>
> > >> > > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > >> > > > > > > >>>>>>>>>>>>>> I urge you all to attend the demonstration
> > >> meeting
> > >> > > > > > > >>> and
> > >> > > > > > > >>>>> lend
> > >> > > > > > > >>>>>>> your
> > >> > > > > > > >>>>>>>>>> views
> > >> > > > > > > >>>>>>>>>>>>> on
> > >> > > > > > > >>>>>>>>>>>>>> the same.
> > >> > > > > > > >>>>>>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>>>>> Thank you,
> > >> > > > > > > >>>>>>>>>>>>>> Chai
> > >> > > > > > > >>>>>>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>>>>> *Meeting Details*:
> > >> > > > > > > >>>>>>>>>>>>>> ==============Conference Bridge
> > >> > > > > > > >>>> Information==============
> > >> > > > > > > >>>>>>>>>>>>>> You have been invited to an online meeting,
> > >> > powered
> > >> > > > > > > >>> by
> > >> > > > > > > >>>>> Amazon
> > >> > > > > > > >>>>>>>>> Chime.
> > >> > > > > > > >>>>>>>>>>>>>> *Chime meeting ID*: *9272158344*
> > >> > > > > > > >>>>>>>>>>>>>> Join via Chime clients (manually): Select
> > >> > 'Meetings
> > >> > > >
> > >> > > > > > > >>>>> Join a
> > >> > > > > > > >>>>>>>>>> Meeting',
> > >> > > > > > > >>>>>>>>>>>>> and
> > >> > > > > > > >>>>>>>>>>>>>> enter 9272158344
> > >> > > > > > > >>>>>>>>>>>>>> Join via Chime clients (auto-call): If you
> > >> invite
> > >> > > > > > > >>>>> auto-call as
> > >> > > > > > > >>>>>>>>>>> attendee,
> > >> > > > > > > >>>>>>>>>>>>>> Chime will call you when the meeting
> starts,
> > >> > select
> > >> > > > > > > >>>>> 'Answer'
> > >> > > > > > > >>>>>>>>>>>>>> *Join via browser screen share*:
> > >> > > > > > > >>>>> https://chime.aws/9272158344
> > >> > > > > > > >>>>>>>>>>>>>> *Join via phone* (US):
> > >> > +1-929-432-4463,,,9272158344#
> > >> > > > > > > >>>>>>>>>>>>>> *Join via phone (US toll-free)*:
> > >> > > > > > > >>>>> +1-855-552-4463,,,9272158344#
> > >> > > > > > > >>>>>>>>>>>>>> International dial-in:
> > >> > > > > > > >>>> https://chime.aws/dialinnumbers/
> > >> > > > > > > >>>>>>>>>>>>>> In-room video system: Ext: 62000, Meeting
> > PIN:
> > >> > > > > > > >>>>> 9272158344#
> > >> > > > > > > >>>>>>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>>>>> --
> > >> > > > > > > >>>>>>>>>>>>>> *Chaitanya Prakash Bapat*
> > >> > > > > > > >>>>>>>>>>>>>> *+1 (973) 953-6299*
> > >> > > > > > > >>>>>>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>>>>> [image:
> > >> https://www.linkedin.com//in/chaibapat25]
> > >> > > > > > > >>>>>>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> > >> > > > > > > >>>>>>>>>>>>> https://www.facebook.com/chaibapat
> > >> > > > > > > >>>>>>>>>>>>>> ]
> > >> > > > > > > >>>>>>>>>>>>>> <https://www.facebook.com/chaibapchya
> > >[image:
> > >> > > > > > > >>>>>>>>>>>>>> https://twitter.com/ChaiBapchya] <
> > >> > > > > > > >>>>>>>> https://twitter.com/ChaiBapchya
> > >> > > > > > > >>>>>>>>>>>>>> [image:
> > >> > > > > > > >>>>>>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> > >> > > > > > > >>>>>>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/
> >
> > >> > > > > > > >>>>>>>>>>>>>>
> > >> > > > > > > >>>>>>>>>>
> > >> > > > > > > >>>>>>>>>> --
> > >> > > > > > > >>>>>>>>>> *Chaitanya Prakash Bapat*
> > >> > > > > > > >>>>>>>>>> *+1 (973) 953-6299*
> > >> > > > > > > >>>>>>>>>>
> > >> > > > > > > >>>>>>>>>> [image:
> https://www.linkedin.com//in/chaibapat25
> > ]
> > >> > > > > > > >>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> > >> > > > > > > >>>>>>>>> https://www.facebook.com/chaibapat
> > >> > > > > > > >>>>>>>>>> ]
> > >> > > > > > > >>>>>>>>>> <https://www.facebook.com/chaibapchya>[image:
> > >> > > > > > > >>>>>>>>>> https://twitter.com/ChaiBapchya] <
> > >> > > > > > > >>>>> https://twitter.com/ChaiBapchya
> > >> > > > > > > >>>>>>>>>> [image:
> > >> > > > > > > >>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> > >> > > > > > > >>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
> > >> > > > > > > >>>>>>>>>>
> > >> > > > > > > >>>>>>>
> > >> > > > > > > >>>>>>> --
> > >> > > > > > > >>>>>>> Sandeep Krishnamurthy
> > >> > > > > > > >>>>>>>
> > >> > > > > > > >>>>>
> > >> > > > > > > >>>>
> > >> > > > > > > >>>
> > >> > > > > > > >>>
> > >> > > > > > > >>> --
> > >> > > > > > > >>> *Chaitanya Prakash Bapat*
> > >> > > > > > > >>> *+1 (973) 953-6299*
> > >> > > > > > > >>>
> > >> > > > > > > >>> [image: https://www.linkedin.com//in/chaibapat25]
> > >> > > > > > > >>> <https://github.com/ChaiBapchya>[image:
> > >> > > > > > > https://www.facebook.com/chaibapat
> > >> > > > > > > >>> ]
> > >> > > > > > > >>> <https://www.facebook.com/chaibapchya>[image:
> > >> > > > > > > >>> https://twitter.com/ChaiBapchya] <
> > >> > > > https://twitter.com/ChaiBapchya
> > >> > > > > > > >[image:
> > >> > > > > > > >>> https://www.linkedin.com//in/chaibapat25]
> > >> > > > > > > >>> <https://www.linkedin.com//in/chaibapchya/>
> > >> > > > > > > >>>
> > >> > > > > > > >>
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > --
> > >> > > > > *Chaitanya Prakash Bapat*
> > >> > > > > *+1 (973) 953-6299*
> > >> > > > >
> > >> > > > > [image: https://www.linkedin.com//in/chaibapat25]
> > >> > > > > <https://github.com/ChaiBapchya>[image:
> > >> > > > https://www.facebook.com/chaibapat
> > >> > > > > ]
> > >> > > > > <https://www.facebook.com/chaibapchya>[image:
> > >> > > > > https://twitter.com/ChaiBapchya] <
> > https://twitter.com/ChaiBapchya
> > >> > > > >[image:
> > >> > > > > https://www.linkedin.com//in/chaibapat25]
> > >> > > > > <https://www.linkedin.com//in/chaibapchya/>
> > >> > > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > > *Chaitanya Prakash Bapat*
> > >> > > *+1 (973) 953-6299*
> > >> > >
> > >> > > [image: https://www.linkedin.com//in/chaibapat25]
> > >> > > <https://github.com/ChaiBapchya>[image:
> > >> > https://www.facebook.com/chaibapat
> > >> > > ]
> > >> > > <https://www.facebook.com/chaibapchya>[image:
> > >> > > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> > >> > >[image:
> > >> > > https://www.linkedin.com//in/chaibapat25]
> > >> > > <https://www.linkedin.com//in/chaibapchya/>
> > >> > >
> > >> >
> > >>
> > >>
> > >> --
> > >> *Chaitanya Prakash Bapat*
> > >> *+1 (973) 953-6299*
> > >>
> > >> [image: https://www.linkedin.com//in/chaibapat25]
> > >> <https://github.com/ChaiBapchya>[image:
> > >> https://www.facebook.com/chaibapat]
> > >> <https://www.facebook.com/chaibapchya>[image:
> > >> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> > >[image:
> > >> https://www.linkedin.com//in/chaibapat25]
> > >> <https://www.linkedin.com//in/chaibapchya/>
> > >>
> > >
> >
>
>
> --
> *Chaitanya Prakash Bapat*
> *+1 (973) 953-6299*
>
> [image: https://www.linkedin.com//in/chaibapat25]
> <https://github.com/ChaiBapchya>[image: https://www.facebook.com/chaibapat
> ]
> <https://www.facebook.com/chaibapchya>[image:
> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
> https://www.linkedin.com//in/chaibapat25]
> <https://www.linkedin.com//in/chaibapchya/>
>
Re: MXNet Bot Demo
Posted by Chaitanya Bapat <ch...@gmail.com>.
Hello,
Update: Apache Infra Ticket for MXNet Bot
Thanks once again, Marco for opening the ticket. But turns out, Apache
Infra folks closed it stating: "Security concerns around allowing unknown
person to submit PR and run our hardware". Furthermore, it goes onto state
that bot circumvents the dependence on Jenkins Admins which is like solving
a problem that doesn't exist.
I sense there is some confusion in the communication (maybe on my part). It
turns out the security concerns aren't actually correct.
1. Unknown person can submit a PR (before & after bot proposal), and run
our hardware (pre as well as post bot).
2. Code should be reviewed by somebody with an ICLA on file. This doesn't
change either. Prior to merging a PR, code has to be approved by a
committer just like before.
Overall it looks like the job of the bot isn't clear to folks in Apache
Infra. Bot simply is a means for triggering CI (which could be done
manually by Log In to Jenkins -> PR -> Job -> Build) and doesn't quite
tweak with merging procedure. Yes, only addition is now unknown person (PR
Author) can trigger CI with a message (but that was possible anyway by
pushing a commit. Bot just prevents users from pushing empty commits and
building entire suite).
As can be seen from last 10 open PRs as of Monday 23rd March, 12pm PT most
PRs fail on 1/2 jobs. In such a scenario, the proposed MXNet bot would come
in handy for just invoking CI on that specific build (instead of a
non-committer PR Author to push empty commit : hurting on the resource,
time & cost considerations apart from undesirable dev experience)
@Marco Since I am a non-committer, I guess these 2 clarifications need to
be conveyed to the Apache Infra by someone with Committer access.
What do you think?
Thanks,
Chai
On Sat, 21 Mar 2020 at 16:08, Marco de Abreu <ma...@gmail.com>
wrote:
> Hello,
>
> the ticket has been created:
> https://issues.apache.org/jira/browse/INFRA-20005
>
> Best regards,
> Marco
>
> On Thu, Mar 19, 2020 at 11:49 PM Marco de Abreu <ma...@gmail.com>
> wrote:
>
> > Sounds like a good plan!
> >
> > Please send me the URL (please make sure it's backed by DNS and not just
> > the gateway URL) of the webhook handler, GitHub events you're interested
> in
> > and the shared secret in a private email to my personal email address. I
> > will then create the ticket with Apache infra.
> >
> > -Marco
> >
> > Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 19. März 2020,
> > 23:07:
> >
> >> @Marco Alright, it makes total sense to test out the Bot feature
> alongside
> >> auto-trigger as a transition.
> >>
> >> Path Forward:
> >> 1. Setup MXNet Bot on apache/incubator-mxnet repo (GitHub WebHook and
> >> Infra)
> >> 2. We don't turn off automatic trigger of PR builds for now.
> >> 3. Hopefully, bot is used by developers to trigger specific jobs
> >> 4. Later on (say around April 20), let's discuss the possibility of
> >> switching off auto-trigger (with appropriate data) if it makes sense.
> >> Thanks Marco for volunteering to help enable the web hook on
> >> apache/incubator-mxnet. Let me know if we can sync up on Slack channel
> to
> >> get the ball rolling.
> >>
> >> Thanks once again for the entire community to step in and help try out
> >> this
> >> Bot.
> >> Chai
> >>
> >> On Wed, 18 Mar 2020 at 17:07, Marco de Abreu <ma...@gmail.com>
> >> wrote:
> >>
> >> > Hi, that's correct. But as stated previously, it's not an option to
> >> remove
> >> > the hook. For now, I'd like to see how the system behaves while it's
> >> > optional. Later on, we can talk about revisiting this decision. But to
> >> me
> >> > it's not an option to deploy an entirely new system and approach
> without
> >> > having a transition or even a timeframe in which we are able to fall
> >> back.
> >> >
> >> > I'm happy to support the deployment of the bot and add an additional
> >> > webhook to enable it's functionality to support selective triggering
> by
> >> PR
> >> > authors and committers, but I will not support the disabling of
> >> automatic
> >> > triggering of branches or PRs.
> >> >
> >> > -Marco
> >> >
> >> > Chaitanya Bapat <ch...@gmail.com> schrieb am Mi., 18. März 2020,
> >> > 21:00:
> >> >
> >> > > Hey Marco,
> >> > >
> >> > > I thought currently every commit on PR and master triggers CI
> >> > > because
> >> > > a. github webhook points to Jenkins Server
> >> > > b. GH Webhook events trigger builds on Jenkins for all commits to
> any
> >> > > branch in apache/incubator-mxnet
> >> > > may it be master/PR/non-PR
> >> > > Reason:
> >> > > Because all the 3 types of branches are discovered by Jenkins
> (non-PR
> >> > > (including master) and PR)
> >> > >
> >> > > Proposal: Remove GitHub WebHook to Jenkins and replace with GH
> >> Webhook to
> >> > > Lambda
> >> > > But after I remove the github webhook that points to Jenkins : *N**o
> >> > commit
> >> > > will trigger Jenkins build by default* (as Jenkins wont receive GH
> >> > events)
> >> > > Only those that Bot deems fit will be triggered (using Jenkins API
> >> > invoked
> >> > > by Lambda).
> >> > > Hence its needed to handle that case of master merge.
> >> > > Am I understanding this correctly?
> >> > >
> >> > > On Wed, 18 Mar 2020 at 04:23, Marco de Abreu <
> marco.g.abreu@gmail.com
> >> >
> >> > > wrote:
> >> > >
> >> > > > Thanks Chai, sounds good to me. Could you elaborate a bit on the
> >> point
> >> > > > about triggering a CI run after the PR has been merged? We already
> >> to
> >> > > that
> >> > > > automatically for the master, so what's the benefit to do it
> twice?
> >> > > >
> >> > > > -Marco
> >> > > >
> >> > > > Chaitanya Bapat <ch...@gmail.com> schrieb am Mi., 18. März
> >> 2020,
> >> > > > 09:30:
> >> > > >
> >> > > > > Update:
> >> > > > >
> >> > > > > > we can ensure that all CI runs ran on the commit that will be
> >> > merged
> >> > > > > @Sam Skalicky <sa...@gmail.com> Branch Protection is
> added
> >> to
> >> > > > public
> >> > > > > MXNet repo. It ensures that for every PR to be merged, the CI
> >> passes.
> >> > > All
> >> > > > > the jobs selected "required" jobs will have to be green for the
> >> PR to
> >> > > be
> >> > > > > merged. Ofcourse, users with "Adminstrator" access can merge
> >> without
> >> > it
> >> > > > but
> >> > > > > that's just a backdoor. It is the case now and will continue to
> be
> >> > the
> >> > > > case
> >> > > > > with the inclusion of Bot.
> >> > > > >
> >> > > > > > easily verify that the CI has executed all runs on the commit
> >> that
> >> > > will
> >> > > > > be merged
> >> > > > > GitHub UI shows all the jobs and the status corresponding to it
> on
> >> > > every
> >> > > > > commit. That should suffice. For the merged commits, Repo ->
> >> Commits
> >> > ->
> >> > > > > Commit ID (Status) can be tracked currently (only way that I
> know
> >> > of).
> >> > > > > Moreover, it is beyond the scope of this project (and possibly
> >> out of
> >> > > our
> >> > > > > control since this is purely GitHub UI specific use-case).
> >> > > > >
> >> > > > > Thanks @przemyslaw for supporting the opt-in.
> >> > > > >
> >> > > > > Thanks everyone in the community for sharing concerns, voicing
> >> your
> >> > > > opinion
> >> > > > > and participating in the discussion.
> >> > > > > Thanks to those who attended the demo last Friday.
> >> > > > >
> >> > > > > Action items from that discussion
> >> > > > > 1. Handle master merge builds [Done]
> >> > > > > Bot runs entire CI suite after the PR is merged and comments on
> >> the
> >> > PR
> >> > > > > about the same.
> >> > > > > Design decision :
> >> > > > > MXNet Bot comment about master merge build on the *merge commit
> vs
> >> > PR*.
> >> > > > > After the PR is merged, Bot runs entire CI and comments the
> >> result of
> >> > > CI
> >> > > > > trigger on the PR (because it is easy to track on a PR rather
> than
> >> > > > > commenting inside the merge commit)
> >> > > > >
> >> > > > > 2. Idempotent condition
> >> > > > > In case of already running build, if an attempt is made to
> >> retrigger
> >> > > the
> >> > > > > job then what should be the response
> >> > > > > a. Not to re-trigger, let the ongoing build continue till
> >> completion
> >> > > > > b. End the ongoing build and re-trigger
> >> > > > > c. Let the ongoing build continue, re-trigger new build
> >> > > > >
> >> > > > > From resource saving point of view, *c* looks costly and a can
> be
> >> > > > > avoided/optimized by B.
> >> > > > > In case when a re-trigger was started "erroneously" then killing
> >> > > ongoing
> >> > > > > build and re-trigger is a waste.
> >> > > > > In case when ongoing build failed in one sub-part, then
> >> re-triggering
> >> > > is
> >> > > > > justified.
> >> > > > > Erroneous re-triggers would be less often while conscious
> >> re-triggers
> >> > > to
> >> > > > > suppress failure is more common use-case. It looks like a safe
> >> > > assumption
> >> > > > > to make given the trade-off.
> >> > > > > [Open to debate]
> >> > > > >
> >> > > > > 3. Add security consideration [Use of secret manager, but
> without
> >> > > > > auto-rotation due to Jenkins manual config requirement] [Done]
> >> > > > > 4. New PR Instruction message by the Bot [Done]
> >> > > > > Thanks to the suggestion of Leonard, supported by others. I've
> now
> >> > > added
> >> > > > > the feature where the Bot comments a help message. [For
> reference
> >> -
> >> > > > > https://github.com/ChaiBapchya/incubator-mxnet/pull/52]
> >> > > > >
> >> > > > > Barring the opt-in vs opt-out debate & idempotency, consensus
> was
> >> > > quickly
> >> > > > > reached for the rest.
> >> > > > >
> >> > > > > In the coming days, I hope to roll-out this feature into Prod
> >> (public
> >> > > > > MXNet) for all devs to use.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Chai
> >> > > > >
> >> > > > > On Mon, 16 Mar 2020 at 11:57, Marco de Abreu <
> >> > marco.g.abreu@gmail.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Well that's generally a problem with a deferred CI approach
> (CI
> >> is
> >> > > run
> >> > > > at
> >> > > > > > commit and not at merge time). This can either be solved
> through
> >> > the
> >> > > > > other
> >> > > > > > proposal that's currently on dev@, by having a bot which does
> >> > merges
> >> > > > by
> >> > > > > > having a global lock and a merge queue or by accepting the
> >> issue.
> >> > > > Reality
> >> > > > > > right now is that we're running that model where two PRs which
> >> are
> >> > > > merged
> >> > > > > > in parallel might break one another. One thing to consider
> >> though
> >> > is
> >> > > > that
> >> > > > > > this breakage would have to be introduced in two separate
> parts
> >> > since
> >> > > > > > otherwise there'd be merge conflicts. I think we had that
> >> situation
> >> > > > twice
> >> > > > > > so far and the result was a quick revert, so I'd say that
> it's a
> >> > > > problem
> >> > > > > > that can happily be accepted. All other solutions basically
> >> require
> >> > > > some
> >> > > > > > form of single-threaded and globally locked solution which
> >> limits
> >> > us
> >> > > in
> >> > > > > > scalability. I'd recommend to just accept that risk and revert
> >> a PR
> >> > > in
> >> > > > > case
> >> > > > > > it actually had a conflict.
> >> > > > > >
> >> > > > > > -Marco
> >> > > > > >
> >> > > > > > On Mon, Mar 16, 2020 at 6:29 PM Skalicky, Sam
> >> > > > <sskalic@amazon.com.invalid
> >> > > > > >
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > We probably need some way to track which CI runs ran for
> which
> >> > > commit
> >> > > > > > too,
> >> > > > > > > that way we can ensure that all CI runs ran on the commit
> that
> >> > will
> >> > > > be
> >> > > > > > > merged. Maybe the bot can comment with the commit hash when
> >> > users
> >> > > > > > command
> >> > > > > > > it to do something. Although since users can trigger
> >> individual
> >> > CI
> >> > > > runs
> >> > > > > > its
> >> > > > > > > possible to have some commits run some CI runs but not
> >> others. We
> >> > > > need
> >> > > > > > some
> >> > > > > > > way to easily verify that the CI has executed all runs on
> the
> >> > > commit
> >> > > > > that
> >> > > > > > > will be merged.
> >> > > > > > >
> >> > > > > > > Sam
> >> > > > > > >
> >> > > > > > > > On Mar 13, 2020, at 8:28 PM, Przemysław Trędak <
> >> > > ptrendx@apache.org
> >> > > > >
> >> > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > CAUTION: This email originated from outside of the
> >> > organization.
> >> > > Do
> >> > > > > not
> >> > > > > > > click links or open attachments unless you can confirm the
> >> sender
> >> > > and
> >> > > > > > know
> >> > > > > > > the content is safe.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > I personally like the idea of opt-in more than opt-out:
> >> > > > > > > > - ultimately PR author wants the PR to be merged so they
> (or
> >> > > > > committer
> >> > > > > > > reviewing the PR) will trigger the CI
> >> > > > > > > > - if it is easy to trigger the PR via the bot command then
> >> the
> >> > > > amount
> >> > > > > > of
> >> > > > > > > work per PR should be less than with opt-out (since most of
> >> the
> >> > > > commits
> >> > > > > > > should then be marked as [skip ci] or something similar
> >> > > > > > > >
> >> > > > > > > > +1 to the bot making a comment on each new PR with its
> >> commands
> >> > > > (and
> >> > > > > > > also explaining, or at least giving links to the general PR
> >> > process
> >> > > > so
> >> > > > > > new
> >> > > > > > > PR authors are not lost). Maybe we could make the bot
> >> recognize
> >> > if
> >> > > > the
> >> > > > > PR
> >> > > > > > > author is new or existing contributor and offer advice based
> >> on
> >> > > that?
> >> > > > > > > >
> >> > > > > > > > Thanks
> >> > > > > > > > Przemek
> >> > > > > > > >
> >> > > > > > > > On 2020/03/13 22:06:58, Marco de Abreu <
> >> > marco.g.abreu@gmail.com>
> >> > > > > > wrote:
> >> > > > > > > >> Hi,
> >> > > > > > > >>
> >> > > > > > > >> since it's no longer necessary to push a new commit to
> >> trigger
> >> > > CI,
> >> > > > > it
> >> > > > > > > will
> >> > > > > > > >> already reduce the costs. But to me, requiring an action
> to
> >> > > enable
> >> > > > > CI
> >> > > > > > > after
> >> > > > > > > >> a PR has been created initially, is a no go. User can opt
> >> out
> >> > of
> >> > > > CI,
> >> > > > > > but
> >> > > > > > > >> the default has to be CI being triggered automatically
> for
> >> > every
> >> > > > > > commit
> >> > > > > > > >> unless specifically disabled by a participant. I'm also
> >> fine
> >> > > with
> >> > > > > > > >> triggering certain additional jobs (think about running a
> >> > > nightly
> >> > > > > job
> >> > > > > > > upon
> >> > > > > > > >> request for a PR) to require a manual step, but the PR
> >> > > validation
> >> > > > > > > pipelines
> >> > > > > > > >> have to run automatically. Every check that is marked as
> >> > > > "Required"
> >> > > > > in
> >> > > > > > > >> GitHub has to be automatically kicked off.
> >> > > > > > > >>
> >> > > > > > > >> -Marco
> >> > > > > > > >>
> >> > > > > > > >> On Fri, Mar 13, 2020 at 9:50 PM Chaitanya Bapat <
> >> > > > > chai.bapat@gmail.com
> >> > > > > > >
> >> > > > > > > >> wrote:
> >> > > > > > > >>
> >> > > > > > > >>> Firstly,
> >> > > > > > > >>> Sorry I missed out on attaching the mail thread that was
> >> sent
> >> > > on
> >> > > > > 12th
> >> > > > > > > >>> February for notifying the community of the upcoming
> >> changes
> >> > to
> >> > > > the
> >> > > > > > > MXNet
> >> > > > > > > >>> CI
> >> > > > > > > >>> For reference :
> >> > > > > > > >>>
> >> > > > > > > >>>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://lists.apache.org/thread.html/r09a6ab2803a996fc80e00fe39ed312fa4865e8805e08df847f1addad%40%3Cdev.mxnet.apache.org%3E
> >> > > > > > > >>>
> >> > > > > > > >>> Now to the questions,
> >> > > > > > > >>>> Is it possible for re-triggering a single job to be
> >> abused?
> >> > > > > > > >>> @Tao In the case when a user re-triggers a single job
> >> > multiple
> >> > > > > times,
> >> > > > > > > that
> >> > > > > > > >>> will be visible in the PR conversation thread. A
> >> committer,
> >> > > even
> >> > > > > > after
> >> > > > > > > he
> >> > > > > > > >>> has approved the PR before, generally takes a look at
> the
> >> > final
> >> > > > > state
> >> > > > > > > of
> >> > > > > > > >>> the PR before merging. Would it be fair to assume the
> >> > committer
> >> > > > > could
> >> > > > > > > take
> >> > > > > > > >>> the multiple re-trigger of a single job into account
> >> before
> >> > > > > merging?
> >> > > > > > > The
> >> > > > > > > >>> committer then has the option to invoke `@mxnet-bot run
> ci
> >> > > [all]
> >> > > > `
> >> > > > > to
> >> > > > > > > >>> trigger the entire build pipeline one last to counter
> the
> >> > > abuse.
> >> > > > > This
> >> > > > > > > is
> >> > > > > > > >>> aligned with what @Leonard said.
> >> > > > > > > >>>
> >> > > > > > > >>> @Sandeep Thanks a lot for collecting and sharing
> valuable
> >> > data.
> >> > > > I'd
> >> > > > > > > concur
> >> > > > > > > >>> with the opinion that given the existing things
> >> committers &
> >> > PR
> >> > > > > > Authors
> >> > > > > > > >>> take care of, invoking CI shouldn't be that big of an
> >> > > additional
> >> > > > > > > burden.
> >> > > > > > > >>>
> >> > > > > > > >>> @Marco With the opt-out, the onus remains on the PR
> >> Author.
> >> > It
> >> > > > > > doesn't
> >> > > > > > > help
> >> > > > > > > >>> reduce the resource usage. Hence, it was suggested to
> >> switch
> >> > to
> >> > > > > > > >>> opt-in. @Leo's suggestion for proactive commenting on
> the
> >> > part
> >> > > of
> >> > > > > bot
> >> > > > > > > makes
> >> > > > > > > >>> sense and is doable.
> >> > > > > > > >>>
> >> > > > > > > >>> Default : opt-out and User initiated opt-in (with
> >> addressing
> >> > > > Leo's
> >> > > > > > fix
> >> > > > > > > for
> >> > > > > > > >>> the usability issue you correctly pointed out )
> >> > > > > > > >>> @Marco How does this sound to you?
> >> > > > > > > >>>
> >> > > > > > > >>> Again, thank you all for chiming in and voicing your
> >> opinion.
> >> > > > > > > Appreciate
> >> > > > > > > >>> it.
> >> > > > > > > >>> We can take ahead these discussions in today's demo
> >> meeting.
> >> > > > > [Design
> >> > > > > > > Doc
> >> > > > > > > >>> <
> >> > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> >> > > > >]
> >> > > > > > > [Demo
> >> > > > > > > >>> Video <https://www.youtube.com/watch?v=gfOGwZId8aU>]
> >> > > > > > > >>>
> >> > > > > > > >>> Thanks,
> >> > > > > > > >>> Chai
> >> > > > > > > >>>
> >> > > > > > > >>> On Fri, 13 Mar 2020 at 12:34, Marco de Abreu <
> >> > > > > > marco.g.abreu@gmail.com>
> >> > > > > > > >>> wrote:
> >> > > > > > > >>>
> >> > > > > > > >>>> I'd recommend that the bot makes an initial comment
> when
> >> a
> >> > PR
> >> > > > gets
> >> > > > > > > opened
> >> > > > > > > >>>> and informs the users of its commands. It then tells
> the
> >> > user
> >> > > > the
> >> > > > > > > commend
> >> > > > > > > >>>> to opt out of CI.
> >> > > > > > > >>>>
> >> > > > > > > >>>> -Marco
> >> > > > > > > >>>>
> >> > > > > > > >>>> Lausen, Leonard <la...@amazon.com.invalid> schrieb am
> >> Fr.,
> >> > > 13.
> >> > > > > > März
> >> > > > > > > >>> 2020,
> >> > > > > > > >>>> 20:27:
> >> > > > > > > >>>>
> >> > > > > > > >>>>> On opt-out: People may be unaware of opt-out would not
> >> use
> >> > > it.
> >> > > > > > There
> >> > > > > > > is
> >> > > > > > > >>>> no
> >> > > > > > > >>>>> incentive to use opt-out, as the PR author doesn't pay
> >> any
> >> > > > money
> >> > > > > > for
> >> > > > > > > CI
> >> > > > > > > >>>>> run.
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> I agree with Marco though that opt-in alone may cause
> >> > > usability
> >> > > > > > > issues,
> >> > > > > > > >>>> as
> >> > > > > > > >>>>> contributors may not be aware of how to trigger the
> CI.
> >> > > > > > > >>>>> One solution is that the bot proactively comments on
> >> the PR
> >> > > and
> >> > > > > > > reminds
> >> > > > > > > >>>> the
> >> > > > > > > >>>>> author to trigger running CI once the author deems the
> >> PR
> >> > > > ready.
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> But even if we choose opt-out, the bot will still add
> a
> >> lot
> >> > > of
> >> > > > > > value,
> >> > > > > > > >>> as
> >> > > > > > > >>>> PR
> >> > > > > > > >>>>> authors can retrigger single jobs that have failed due
> >> to
> >> > > > > > flakiness.
> >> > > > > > > >>>>>
> >> > > > > > > >>>>>> Is it possible for re-triggering a single job to be
> >> > abused?
> >> > > > For
> >> > > > > > > >>>> example,
> >> > > > > > > >>>>>> the author spends two days re-triggering a flaky job
> to
> >> > make
> >> > > > it
> >> > > > > > > pass.
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> Yes, this is possible. I suggest the committer who
> >> likes to
> >> > > > > merge a
> >> > > > > > > PR
> >> > > > > > > >>>>> needs to
> >> > > > > > > >>>>> make a good judgement here if a PR is abusing the
> >> feature,
> >> > > and
> >> > > > if
> >> > > > > > so,
> >> > > > > > > >>>>> retrigger
> >> > > > > > > >>>>> all CI runs.
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> Best regards
> >> > > > > > > >>>>> Leonard
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> On Fri, 2020-03-13 at 08:07 +0100, Marco de Abreu
> wrote:
> >> > > > > > > >>>>>> Thanks for the data Sandeep. In these cases it sounds
> >> like
> >> > > it
> >> > > > > > would
> >> > > > > > > >>>> have
> >> > > > > > > >>>>>> rather been better when people explicitly turned off
> >> CI in
> >> > > > that
> >> > > > > > > case.
> >> > > > > > > >>>>>> What's the argument against an opt-out instead of an
> >> > opt-in?
> >> > > > > > > >>>>>>
> >> > > > > > > >>>>>> My intention is that I consider it quite cumbersome
> to
> >> > make
> >> > > > it a
> >> > > > > > > >>>>> *required*
> >> > > > > > > >>>>>> step to always trigger CI manually, even if just
> >> > submitting
> >> > > a
> >> > > > > > small
> >> > > > > > > >>> PR.
> >> > > > > > > >>>>> I'd
> >> > > > > > > >>>>>> rather see people explicitly turning off CI if they
> >> > wouldn't
> >> > > > > like
> >> > > > > > to
> >> > > > > > > >>>> use
> >> > > > > > > >>>>> it
> >> > > > > > > >>>>>> - and there's also the "draft" stage for a PR which
> >> some
> >> > > > > > > contributors
> >> > > > > > > >>>> are
> >> > > > > > > >>>>>> using.
> >> > > > > > > >>>>>>
> >> > > > > > > >>>>>> With regards to WIP and do not review: I think these
> >> are
> >> > use
> >> > > > > cases
> >> > > > > > > >>>> where
> >> > > > > > > >>>>>> you want CI feedback, as otherwise you wouldn't have
> >> > opened
> >> > > > the
> >> > > > > > PR.
> >> > > > > > > >>> If
> >> > > > > > > >>>>> you
> >> > > > > > > >>>>>> don't want human feedback and neither machine
> feedback,
> >> > why
> >> > > > open
> >> > > > > > the
> >> > > > > > > >>> PR
> >> > > > > > > >>>>> at
> >> > > > > > > >>>>>> all?
> >> > > > > > > >>>>>>
> >> > > > > > > >>>>>> -Marco
> >> > > > > > > >>>>>>
> >> > > > > > > >>>>>>
> >> > > > > > > >>>>>> sandeep krishnamurthy <sa...@gmail.com>
> >> > schrieb
> >> > > > am
> >> > > > > > Fr.,
> >> > > > > > > >>>> 13.
> >> > > > > > > >>>>>> März 2020, 05:24:
> >> > > > > > > >>>>>>
> >> > > > > > > >>>>>>> I tried to gather some data for us to discuss this
> >> topic
> >> > in
> >> > > > > this
> >> > > > > > > >>>>> thread. I
> >> > > > > > > >>>>>>> tried to count number of un-necessary builds by
> >> looking
> >> > at
> >> > > > most
> >> > > > > > > >>>> recent
> >> > > > > > > >>>>> (as
> >> > > > > > > >>>>>>> of 12, March 9 PM PST) 50 PRs merged to master and
> 50
> >> > PRs.
> >> > > > > > > >>>>>>> Identifying un-necessary builds is bit subjective. I
> >> > tried
> >> > > to
> >> > > > > be
> >> > > > > > > >>> more
> >> > > > > > > >>>>>>> conservative where I didn't count a build as
> >> un-necessary
> >> > > if
> >> > > > I
> >> > > > > > was
> >> > > > > > > >>> in
> >> > > > > > > >>>>>>> doubt. Hence, I was not able to automate, but I made
> >> an
> >> > > > effort
> >> > > > > to
> >> > > > > > > >>> go
> >> > > > > > > >>>>>>> through PRs manually and use below criteria to
> >> identify
> >> > > > > > > >>> un-necessary
> >> > > > > > > >>>>>>> commits triggering the builds.
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>> 1. Explicitly marked as WIP / do not review PR
> >> > > > > > > >>>>>>> 2. Incremental WIP commit and finally commenting a
> >> > commit
> >> > > > > > > >>> “trigger
> >> > > > > > > >>>>> CI”
> >> > > > > > > >>>>>>> 3. Multiple commits to address all comments from
> >> single
> >> > > > > review.
> >> > > > > > > >>>>> This is
> >> > > > > > > >>>>>>> assuming we see a comment, address them, commit,
> >> next
> >> > the
> >> > > > > > > >>>> following
> >> > > > > > > >>>>>>> comment
> >> > > > > > > >>>>>>> 4. Sequence of documentation only changes
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>> I found there were around 42 avoidable builds from
> >> most
> >> > > > recent
> >> > > > > 50
> >> > > > > > > >>>>> merged
> >> > > > > > > >>>>>>> PRs and around 86 builds from recent 50 open PRs.
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>> I synced up with other contributors (Joe Evans,
> Chai)
> >> > from
> >> > > > > Amazon
> >> > > > > > > >>> who
> >> > > > > > > >>>>> is
> >> > > > > > > >>>>>>> contributing to MXNet CI system. I was told that on
> an
> >> > > > average
> >> > > > > it
> >> > > > > > > >>>> costs
> >> > > > > > > >>>>>>> around $84 per build and on an average 6 commits per
> >> > merged
> >> > > > PR
> >> > > > > > (for
> >> > > > > > > >>>>> year
> >> > > > > > > >>>>>>> 2019). Going by that, it is approximately 1/6 builds
> >> are
> >> > > > > > avoidable.
> >> > > > > > > >>>>> [100 /
> >> > > > > > > >>>>>>> 300 + 300 ]
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>> Usability should be top most priority. But, since
> >> either
> >> > a
> >> > > > > > reviewer
> >> > > > > > > >>>> or
> >> > > > > > > >>>>> pr
> >> > > > > > > >>>>>>> author can trigger the bot, is it really a hurdle
> for
> >> pr
> >> > > > author
> >> > > > > > or
> >> > > > > > > >>>>> reviewer
> >> > > > > > > >>>>>>> to call a bot to trigger CI? Given that PR author
> and
> >> > > > reviewer
> >> > > > > is
> >> > > > > > > >>>>> already
> >> > > > > > > >>>>>>> actively commenting various details such as - PR
> >> > > description,
> >> > > > > > > >>> review
> >> > > > > > > >>>>>>> comments and responses, adding labels etc.
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>> Me too curious to know the behavior for Tao's above
> >> use
> >> > > case.
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>> Best,
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>> Sandeep
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>> On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <
> >> > mutouorz@gmail.com
> >> > > >
> >> > > > > > wrote:
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>>> Is it possible for re-triggering a single job to be
> >> > > abused?
> >> > > > > For
> >> > > > > > > >>>>> example,
> >> > > > > > > >>>>>>>> the author spends two days re-triggering a flaky
> job
> >> to
> >> > > make
> >> > > > > it
> >> > > > > > > >>>>> pass. But
> >> > > > > > > >>>>>>>> other jobs which have passed the validation may be
> >> > broken
> >> > > by
> >> > > > > > > >>> other
> >> > > > > > > >>>>>>> commits
> >> > > > > > > >>>>>>>> during the two day without being noticed. And
> finally
> >> > the
> >> > > PR
> >> > > > > is
> >> > > > > > > >>>>> merged
> >> > > > > > > >>>>>>> with
> >> > > > > > > >>>>>>>> underlying problems.
> >> > > > > > > >>>>>>>>
> >> > > > > > > >>>>>>>> On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <
> >> > > > > > > >>>>> marco.g.abreu@gmail.com>
> >> > > > > > > >>>>>>>> wrote:
> >> > > > > > > >>>>>>>>
> >> > > > > > > >>>>>>>>> In the end it only comes down to money,
> considering
> >> > that
> >> > > > the
> >> > > > > > > >>>>> system is
> >> > > > > > > >>>>>>>> auto
> >> > > > > > > >>>>>>>>> scaling, making the execution time constant.
> >> > > > > > > >>>>>>>>>
> >> > > > > > > >>>>>>>>> If we're trading money for usability, I certainly
> >> would
> >> > > > > prefer
> >> > > > > > > >>>>>>> usability.
> >> > > > > > > >>>>>>>>> I'd rather recommend to spend time on
> parallelizing
> >> > test
> >> > > > > > > >>>> execution
> >> > > > > > > >>>>> or
> >> > > > > > > >>>>>>>>> getting rid of integration tests in the PR stage
> >> > instead
> >> > > > > > > >>> reducing
> >> > > > > > > >>>>> the
> >> > > > > > > >>>>>>>> costs
> >> > > > > > > >>>>>>>>> by making people not use it. But taking a step
> back
> >> to
> >> > > > > > > >>> requiring
> >> > > > > > > >>>>> people
> >> > > > > > > >>>>>>>> to
> >> > > > > > > >>>>>>>>> manually trigger CI again doesn't feel right.
> >> > > > > > > >>>>>>>>>
> >> > > > > > > >>>>>>>>> I'm happy to see that bot deployed, but I do not
> >> agree
> >> > > with
> >> > > > > > > >>>>> removing
> >> > > > > > > >>>>>>> the
> >> > > > > > > >>>>>>>>> auto trigger functionality for new commits.
> >> > > > > > > >>>>>>>>>
> >> > > > > > > >>>>>>>>> -Marco
> >> > > > > > > >>>>>>>>>
> >> > > > > > > >>>>>>>>> Chaitanya Bapat <ch...@gmail.com> schrieb am
> >> Do.,
> >> > > 12.
> >> > > > > > > >>> März
> >> > > > > > > >>>>> 2020,
> >> > > > > > > >>>>>>>>> 22:47:
> >> > > > > > > >>>>>>>>>
> >> > > > > > > >>>>>>>>>> @Marco Thanks for pointing that out.
> >> > > > > > > >>>>>>>>>> Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM -
> >> 3:30
> >> > > PM
> >> > > > in
> >> > > > > > > >>>>>>>> (UTC-08:00)
> >> > > > > > > >>>>>>>>>> Pacific Time (US & Canada).
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>>> When do we expect this bot to be deployed?
> >> > > > > > > >>>>>>>>>> @Lin If all goes well in the next week I can
> >> deploy it
> >> > > to
> >> > > > > > > >>>> public
> >> > > > > > > >>>>>>> Apache
> >> > > > > > > >>>>>>>>>> (provided I get permissions from Apache Infra)
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>> @Marco Thanks for your feedback.
> >> > > > > > > >>>>>>>>>>> CI system has to support the community without
> >> > > requiring
> >> > > > > > > >>>>> people to
> >> > > > > > > >>>>>>>>>> constantly shepherd every single run
> >> > > > > > > >>>>>>>>>> We have data for the number of times CI was
> >> triggered
> >> > > > > > > >>>>> unnecessarily
> >> > > > > > > >>>>>>>> which
> >> > > > > > > >>>>>>>>>> includes
> >> > > > > > > >>>>>>>>>> - Entire build triggered instead of specific
> build
> >> > > > > > > >>>>>>>>>> - CI triggered when PR is still work in progress
> or
> >> > not
> >> > > > yet
> >> > > > > > > >>>> ready
> >> > > > > > > >>>>>>> (say
> >> > > > > > > >>>>>>>> -
> >> > > > > > > >>>>>>>>>> intermediate commits)
> >> > > > > > > >>>>>>>>>> At the end its a trade-off
> >> > > > > > > >>>>>>>>>> Money, Resources, Time to build for each and
> every
> >> > > commit
> >> > > > vs
> >> > > > > > > >>>>> Pain of
> >> > > > > > > >>>>>>>>>> triggering builds
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>>> Scan trigger plugin would poll SCM. Can we use
> >> plugin
> >> > > at
> >> > > > > > > >>>>> scale?
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>> 1. I haven't tested it on scale. But I think with
> >> the
> >> > > > > current
> >> > > > > > > >>>>> scale
> >> > > > > > > >>>>>>> of
> >> > > > > > > >>>>>>>>>> MXNet repo (191 open PRs i.e. checking for
> changes
> >> to
> >> > > 191
> >> > > > > > > >>>>> branches -
> >> > > > > > > >>>>>>> It
> >> > > > > > > >>>>>>>>>> should be manageable)
> >> > > > > > > >>>>>>>>>> 2. What's the purpose of the plugin? tldr; Branch
> >> > > > discovery
> >> > > > > > > >>> or
> >> > > > > > > >>>>> branch
> >> > > > > > > >>>>>>>>>> indexing.
> >> > > > > > > >>>>>>>>>> Scan trigger plugin comes into the picture only
> >> once
> >> > per
> >> > > > PR
> >> > > > > > > >>> per
> >> > > > > > > >>>>> job
> >> > > > > > > >>>>>>>>> (i.e. 8
> >> > > > > > > >>>>>>>>>> times per PR for 8 jobs). It is basically done
> >> when a
> >> > > new
> >> > > > PR
> >> > > > > > > >>> is
> >> > > > > > > >>>>> made
> >> > > > > > > >>>>>>>> and
> >> > > > > > > >>>>>>>>>> the job (say unix-cpu hasn't discovered the new
> PR
> >> > > branch
> >> > > > > > > >>> yet).
> >> > > > > > > >>>>>>> That's
> >> > > > > > > >>>>>>>>> it.
> >> > > > > > > >>>>>>>>>> So it shouldn't be a problem for public MXNet
> repo.
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>> Thanks,
> >> > > > > > > >>>>>>>>>> Chai
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>> On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <
> >> > > > > > > >>>>>>> marco.g.abreu@gmail.com>
> >> > > > > > > >>>>>>>>>> wrote:
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>>> Btw you forgot to set a date and time for the
> >> metting
> >> > > > > > > >>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>> On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu
> <
> >> > > > > > > >>>>>>>>> marco.g.abreu@gmail.com
> >> > > > > > > >>>>>>>>>>> wrote:
> >> > > > > > > >>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>> Thanks Chai, I generally like the idea of the
> >> bot.
> >> > But
> >> > > > > > > >>> I'm
> >> > > > > > > >>>>> not a
> >> > > > > > > >>>>>>>>>>> supporter
> >> > > > > > > >>>>>>>>>>>> of the idea to disable any automatic triggering
> >> > > > > > > >>> (disabling
> >> > > > > > > >>>>> the
> >> > > > > > > >>>>>>>>> webhook
> >> > > > > > > >>>>>>>>>> is
> >> > > > > > > >>>>>>>>>>>> also not an option, considering that this will
> >> > disable
> >> > > > > > > >>>> master
> >> > > > > > > >>>>>>>>>> triggers).
> >> > > > > > > >>>>>>>>>>>> The CI system has to support the community
> >> without
> >> > > > > > > >>>> requiring
> >> > > > > > > >>>>>>> people
> >> > > > > > > >>>>>>>>> to
> >> > > > > > > >>>>>>>>>>>> constantly shepherd every single run. Disabling
> >> > > > automatic
> >> > > > > > > >>>>>>>> triggering
> >> > > > > > > >>>>>>>>>>> seems
> >> > > > > > > >>>>>>>>>>>> like a step back to me.
> >> > > > > > > >>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>> Instead, I'd recommend that CI gets triggered
> >> upon
> >> > > every
> >> > > > > > > >>>>> commit
> >> > > > > > > >>>>>>> as
> >> > > > > > > >>>>>>>>>> usual,
> >> > > > > > > >>>>>>>>>>>> but people have the possibility to call a
> >> "command"
> >> > > > (i.e.
> >> > > > > > > >>>>> make a
> >> > > > > > > >>>>>>>>>> message
> >> > > > > > > >>>>>>>>>>>> which results in the bot setting a label) to
> >> disable
> >> > > CI
> >> > > > > > > >>>> until
> >> > > > > > > >>>>>>> they
> >> > > > > > > >>>>>>>>>> revoke
> >> > > > > > > >>>>>>>>>>>> it. But the standard should still be that a new
> >> > commit
> >> > > > > > > >>>>> triggers a
> >> > > > > > > >>>>>>>> new
> >> > > > > > > >>>>>>>>>> CI
> >> > > > > > > >>>>>>>>>>>> run.
> >> > > > > > > >>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>
> >> > > > > > > >>>>
> >> > https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
> >> > > > > > > >>>>>>>> seems
> >> > > > > > > >>>>>>>>>> like
> >> > > > > > > >>>>>>>>>>>> this would poll SCM. This will incur high quota
> >> > > > > > > >>>>> restrictions. Are
> >> > > > > > > >>>>>>>> you
> >> > > > > > > >>>>>>>>>>> sure
> >> > > > > > > >>>>>>>>>>>> that you can use that plugin at scale?
> >> > > > > > > >>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>> -Marco
> >> > > > > > > >>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>> On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <
> >> > > > > > > >>>>> apeforest@gmail.com>
> >> > > > > > > >>>>>>>>> wrote:
> >> > > > > > > >>>>>>>>>>>>> Chai,
> >> > > > > > > >>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>> Awesome work. When do we expect this bot to be
> >> > > > > > > >>> deployed?
> >> > > > > > > >>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>> Best,
> >> > > > > > > >>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>> Lin
> >> > > > > > > >>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya
> Bapat
> >> <
> >> > > > > > > >>>>>>>>> chai.bapat@gmail.com
> >> > > > > > > >>>>>>>>>>>>> wrote:
> >> > > > > > > >>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>>> Hello MXNet community,
> >> > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>>> I have built an MXNet Bot <
> >> > > > > > > >>>> https://github.com/mxnet-bot>
> >> > > > > > > >>>>> that
> >> > > > > > > >>>>>>>>>> allows
> >> > > > > > > >>>>>>>>>>> PR
> >> > > > > > > >>>>>>>>>>>>>> Authors, Committers and Jenkins Admins to
> >> trigger
> >> > CI
> >> > > > > > > >>>>> manually.
> >> > > > > > > >>>>>>>>>>>>>> It handles 2 problems
> >> > > > > > > >>>>>>>>>>>>>> 1. Manual CI trigger instead of existing
> >> automated
> >> > > CI
> >> > > > > > > >>>>> trigger
> >> > > > > > > >>>>>>>>>>>>>> 2. Gives permissions to PR Authors (in
> >> addition to
> >> > > > > > > >>>> MXNet
> >> > > > > > > >>>>>>>>> Committers
> >> > > > > > > >>>>>>>>>>> and
> >> > > > > > > >>>>>>>>>>>>>> Jenkins Admins)
> >> > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>>> Design Doc :
> >> > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > >>>>>>>
> >> > > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> >> > > > > > > >>>>>>>>>>>>>> I urge you all to attend the demonstration
> >> meeting
> >> > > > > > > >>> and
> >> > > > > > > >>>>> lend
> >> > > > > > > >>>>>>> your
> >> > > > > > > >>>>>>>>>> views
> >> > > > > > > >>>>>>>>>>>>> on
> >> > > > > > > >>>>>>>>>>>>>> the same.
> >> > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>>> Thank you,
> >> > > > > > > >>>>>>>>>>>>>> Chai
> >> > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>>> *Meeting Details*:
> >> > > > > > > >>>>>>>>>>>>>> ==============Conference Bridge
> >> > > > > > > >>>> Information==============
> >> > > > > > > >>>>>>>>>>>>>> You have been invited to an online meeting,
> >> > powered
> >> > > > > > > >>> by
> >> > > > > > > >>>>> Amazon
> >> > > > > > > >>>>>>>>> Chime.
> >> > > > > > > >>>>>>>>>>>>>> *Chime meeting ID*: *9272158344*
> >> > > > > > > >>>>>>>>>>>>>> Join via Chime clients (manually): Select
> >> > 'Meetings
> >> > > >
> >> > > > > > > >>>>> Join a
> >> > > > > > > >>>>>>>>>> Meeting',
> >> > > > > > > >>>>>>>>>>>>> and
> >> > > > > > > >>>>>>>>>>>>>> enter 9272158344
> >> > > > > > > >>>>>>>>>>>>>> Join via Chime clients (auto-call): If you
> >> invite
> >> > > > > > > >>>>> auto-call as
> >> > > > > > > >>>>>>>>>>> attendee,
> >> > > > > > > >>>>>>>>>>>>>> Chime will call you when the meeting starts,
> >> > select
> >> > > > > > > >>>>> 'Answer'
> >> > > > > > > >>>>>>>>>>>>>> *Join via browser screen share*:
> >> > > > > > > >>>>> https://chime.aws/9272158344
> >> > > > > > > >>>>>>>>>>>>>> *Join via phone* (US):
> >> > +1-929-432-4463,,,9272158344#
> >> > > > > > > >>>>>>>>>>>>>> *Join via phone (US toll-free)*:
> >> > > > > > > >>>>> +1-855-552-4463,,,9272158344#
> >> > > > > > > >>>>>>>>>>>>>> International dial-in:
> >> > > > > > > >>>> https://chime.aws/dialinnumbers/
> >> > > > > > > >>>>>>>>>>>>>> In-room video system: Ext: 62000, Meeting
> PIN:
> >> > > > > > > >>>>> 9272158344#
> >> > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>>> --
> >> > > > > > > >>>>>>>>>>>>>> *Chaitanya Prakash Bapat*
> >> > > > > > > >>>>>>>>>>>>>> *+1 (973) 953-6299*
> >> > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>>> [image:
> >> https://www.linkedin.com//in/chaibapat25]
> >> > > > > > > >>>>>>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> >> > > > > > > >>>>>>>>>>>>> https://www.facebook.com/chaibapat
> >> > > > > > > >>>>>>>>>>>>>> ]
> >> > > > > > > >>>>>>>>>>>>>> <https://www.facebook.com/chaibapchya
> >[image:
> >> > > > > > > >>>>>>>>>>>>>> https://twitter.com/ChaiBapchya] <
> >> > > > > > > >>>>>>>> https://twitter.com/ChaiBapchya
> >> > > > > > > >>>>>>>>>>>>>> [image:
> >> > > > > > > >>>>>>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> >> > > > > > > >>>>>>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
> >> > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>> --
> >> > > > > > > >>>>>>>>>> *Chaitanya Prakash Bapat*
> >> > > > > > > >>>>>>>>>> *+1 (973) 953-6299*
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>> [image: https://www.linkedin.com//in/chaibapat25
> ]
> >> > > > > > > >>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> >> > > > > > > >>>>>>>>> https://www.facebook.com/chaibapat
> >> > > > > > > >>>>>>>>>> ]
> >> > > > > > > >>>>>>>>>> <https://www.facebook.com/chaibapchya>[image:
> >> > > > > > > >>>>>>>>>> https://twitter.com/ChaiBapchya] <
> >> > > > > > > >>>>> https://twitter.com/ChaiBapchya
> >> > > > > > > >>>>>>>>>> [image:
> >> > > > > > > >>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> >> > > > > > > >>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>> --
> >> > > > > > > >>>>>>> Sandeep Krishnamurthy
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>
> >> > > > > > > >>>>
> >> > > > > > > >>>
> >> > > > > > > >>>
> >> > > > > > > >>> --
> >> > > > > > > >>> *Chaitanya Prakash Bapat*
> >> > > > > > > >>> *+1 (973) 953-6299*
> >> > > > > > > >>>
> >> > > > > > > >>> [image: https://www.linkedin.com//in/chaibapat25]
> >> > > > > > > >>> <https://github.com/ChaiBapchya>[image:
> >> > > > > > > https://www.facebook.com/chaibapat
> >> > > > > > > >>> ]
> >> > > > > > > >>> <https://www.facebook.com/chaibapchya>[image:
> >> > > > > > > >>> https://twitter.com/ChaiBapchya] <
> >> > > > https://twitter.com/ChaiBapchya
> >> > > > > > > >[image:
> >> > > > > > > >>> https://www.linkedin.com//in/chaibapat25]
> >> > > > > > > >>> <https://www.linkedin.com//in/chaibapchya/>
> >> > > > > > > >>>
> >> > > > > > > >>
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > *Chaitanya Prakash Bapat*
> >> > > > > *+1 (973) 953-6299*
> >> > > > >
> >> > > > > [image: https://www.linkedin.com//in/chaibapat25]
> >> > > > > <https://github.com/ChaiBapchya>[image:
> >> > > > https://www.facebook.com/chaibapat
> >> > > > > ]
> >> > > > > <https://www.facebook.com/chaibapchya>[image:
> >> > > > > https://twitter.com/ChaiBapchya] <
> https://twitter.com/ChaiBapchya
> >> > > > >[image:
> >> > > > > https://www.linkedin.com//in/chaibapat25]
> >> > > > > <https://www.linkedin.com//in/chaibapchya/>
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > > --
> >> > > *Chaitanya Prakash Bapat*
> >> > > *+1 (973) 953-6299*
> >> > >
> >> > > [image: https://www.linkedin.com//in/chaibapat25]
> >> > > <https://github.com/ChaiBapchya>[image:
> >> > https://www.facebook.com/chaibapat
> >> > > ]
> >> > > <https://www.facebook.com/chaibapchya>[image:
> >> > > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> >> > >[image:
> >> > > https://www.linkedin.com//in/chaibapat25]
> >> > > <https://www.linkedin.com//in/chaibapchya/>
> >> > >
> >> >
> >>
> >>
> >> --
> >> *Chaitanya Prakash Bapat*
> >> *+1 (973) 953-6299*
> >>
> >> [image: https://www.linkedin.com//in/chaibapat25]
> >> <https://github.com/ChaiBapchya>[image:
> >> https://www.facebook.com/chaibapat]
> >> <https://www.facebook.com/chaibapchya>[image:
> >> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> >[image:
> >> https://www.linkedin.com//in/chaibapat25]
> >> <https://www.linkedin.com//in/chaibapchya/>
> >>
> >
>
--
*Chaitanya Prakash Bapat*
*+1 (973) 953-6299*
[image: https://www.linkedin.com//in/chaibapat25]
<https://github.com/ChaiBapchya>[image: https://www.facebook.com/chaibapat]
<https://www.facebook.com/chaibapchya>[image:
https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
https://www.linkedin.com//in/chaibapat25]
<https://www.linkedin.com//in/chaibapchya/>
Re: MXNet Bot Demo
Posted by Marco de Abreu <ma...@gmail.com>.
Hello,
the ticket has been created:
https://issues.apache.org/jira/browse/INFRA-20005
Best regards,
Marco
On Thu, Mar 19, 2020 at 11:49 PM Marco de Abreu <ma...@gmail.com>
wrote:
> Sounds like a good plan!
>
> Please send me the URL (please make sure it's backed by DNS and not just
> the gateway URL) of the webhook handler, GitHub events you're interested in
> and the shared secret in a private email to my personal email address. I
> will then create the ticket with Apache infra.
>
> -Marco
>
> Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 19. März 2020,
> 23:07:
>
>> @Marco Alright, it makes total sense to test out the Bot feature alongside
>> auto-trigger as a transition.
>>
>> Path Forward:
>> 1. Setup MXNet Bot on apache/incubator-mxnet repo (GitHub WebHook and
>> Infra)
>> 2. We don't turn off automatic trigger of PR builds for now.
>> 3. Hopefully, bot is used by developers to trigger specific jobs
>> 4. Later on (say around April 20), let's discuss the possibility of
>> switching off auto-trigger (with appropriate data) if it makes sense.
>> Thanks Marco for volunteering to help enable the web hook on
>> apache/incubator-mxnet. Let me know if we can sync up on Slack channel to
>> get the ball rolling.
>>
>> Thanks once again for the entire community to step in and help try out
>> this
>> Bot.
>> Chai
>>
>> On Wed, 18 Mar 2020 at 17:07, Marco de Abreu <ma...@gmail.com>
>> wrote:
>>
>> > Hi, that's correct. But as stated previously, it's not an option to
>> remove
>> > the hook. For now, I'd like to see how the system behaves while it's
>> > optional. Later on, we can talk about revisiting this decision. But to
>> me
>> > it's not an option to deploy an entirely new system and approach without
>> > having a transition or even a timeframe in which we are able to fall
>> back.
>> >
>> > I'm happy to support the deployment of the bot and add an additional
>> > webhook to enable it's functionality to support selective triggering by
>> PR
>> > authors and committers, but I will not support the disabling of
>> automatic
>> > triggering of branches or PRs.
>> >
>> > -Marco
>> >
>> > Chaitanya Bapat <ch...@gmail.com> schrieb am Mi., 18. März 2020,
>> > 21:00:
>> >
>> > > Hey Marco,
>> > >
>> > > I thought currently every commit on PR and master triggers CI
>> > > because
>> > > a. github webhook points to Jenkins Server
>> > > b. GH Webhook events trigger builds on Jenkins for all commits to any
>> > > branch in apache/incubator-mxnet
>> > > may it be master/PR/non-PR
>> > > Reason:
>> > > Because all the 3 types of branches are discovered by Jenkins (non-PR
>> > > (including master) and PR)
>> > >
>> > > Proposal: Remove GitHub WebHook to Jenkins and replace with GH
>> Webhook to
>> > > Lambda
>> > > But after I remove the github webhook that points to Jenkins : *N**o
>> > commit
>> > > will trigger Jenkins build by default* (as Jenkins wont receive GH
>> > events)
>> > > Only those that Bot deems fit will be triggered (using Jenkins API
>> > invoked
>> > > by Lambda).
>> > > Hence its needed to handle that case of master merge.
>> > > Am I understanding this correctly?
>> > >
>> > > On Wed, 18 Mar 2020 at 04:23, Marco de Abreu <marco.g.abreu@gmail.com
>> >
>> > > wrote:
>> > >
>> > > > Thanks Chai, sounds good to me. Could you elaborate a bit on the
>> point
>> > > > about triggering a CI run after the PR has been merged? We already
>> to
>> > > that
>> > > > automatically for the master, so what's the benefit to do it twice?
>> > > >
>> > > > -Marco
>> > > >
>> > > > Chaitanya Bapat <ch...@gmail.com> schrieb am Mi., 18. März
>> 2020,
>> > > > 09:30:
>> > > >
>> > > > > Update:
>> > > > >
>> > > > > > we can ensure that all CI runs ran on the commit that will be
>> > merged
>> > > > > @Sam Skalicky <sa...@gmail.com> Branch Protection is added
>> to
>> > > > public
>> > > > > MXNet repo. It ensures that for every PR to be merged, the CI
>> passes.
>> > > All
>> > > > > the jobs selected "required" jobs will have to be green for the
>> PR to
>> > > be
>> > > > > merged. Ofcourse, users with "Adminstrator" access can merge
>> without
>> > it
>> > > > but
>> > > > > that's just a backdoor. It is the case now and will continue to be
>> > the
>> > > > case
>> > > > > with the inclusion of Bot.
>> > > > >
>> > > > > > easily verify that the CI has executed all runs on the commit
>> that
>> > > will
>> > > > > be merged
>> > > > > GitHub UI shows all the jobs and the status corresponding to it on
>> > > every
>> > > > > commit. That should suffice. For the merged commits, Repo ->
>> Commits
>> > ->
>> > > > > Commit ID (Status) can be tracked currently (only way that I know
>> > of).
>> > > > > Moreover, it is beyond the scope of this project (and possibly
>> out of
>> > > our
>> > > > > control since this is purely GitHub UI specific use-case).
>> > > > >
>> > > > > Thanks @przemyslaw for supporting the opt-in.
>> > > > >
>> > > > > Thanks everyone in the community for sharing concerns, voicing
>> your
>> > > > opinion
>> > > > > and participating in the discussion.
>> > > > > Thanks to those who attended the demo last Friday.
>> > > > >
>> > > > > Action items from that discussion
>> > > > > 1. Handle master merge builds [Done]
>> > > > > Bot runs entire CI suite after the PR is merged and comments on
>> the
>> > PR
>> > > > > about the same.
>> > > > > Design decision :
>> > > > > MXNet Bot comment about master merge build on the *merge commit vs
>> > PR*.
>> > > > > After the PR is merged, Bot runs entire CI and comments the
>> result of
>> > > CI
>> > > > > trigger on the PR (because it is easy to track on a PR rather than
>> > > > > commenting inside the merge commit)
>> > > > >
>> > > > > 2. Idempotent condition
>> > > > > In case of already running build, if an attempt is made to
>> retrigger
>> > > the
>> > > > > job then what should be the response
>> > > > > a. Not to re-trigger, let the ongoing build continue till
>> completion
>> > > > > b. End the ongoing build and re-trigger
>> > > > > c. Let the ongoing build continue, re-trigger new build
>> > > > >
>> > > > > From resource saving point of view, *c* looks costly and a can be
>> > > > > avoided/optimized by B.
>> > > > > In case when a re-trigger was started "erroneously" then killing
>> > > ongoing
>> > > > > build and re-trigger is a waste.
>> > > > > In case when ongoing build failed in one sub-part, then
>> re-triggering
>> > > is
>> > > > > justified.
>> > > > > Erroneous re-triggers would be less often while conscious
>> re-triggers
>> > > to
>> > > > > suppress failure is more common use-case. It looks like a safe
>> > > assumption
>> > > > > to make given the trade-off.
>> > > > > [Open to debate]
>> > > > >
>> > > > > 3. Add security consideration [Use of secret manager, but without
>> > > > > auto-rotation due to Jenkins manual config requirement] [Done]
>> > > > > 4. New PR Instruction message by the Bot [Done]
>> > > > > Thanks to the suggestion of Leonard, supported by others. I've now
>> > > added
>> > > > > the feature where the Bot comments a help message. [For reference
>> -
>> > > > > https://github.com/ChaiBapchya/incubator-mxnet/pull/52]
>> > > > >
>> > > > > Barring the opt-in vs opt-out debate & idempotency, consensus was
>> > > quickly
>> > > > > reached for the rest.
>> > > > >
>> > > > > In the coming days, I hope to roll-out this feature into Prod
>> (public
>> > > > > MXNet) for all devs to use.
>> > > > >
>> > > > > Thanks,
>> > > > > Chai
>> > > > >
>> > > > > On Mon, 16 Mar 2020 at 11:57, Marco de Abreu <
>> > marco.g.abreu@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Well that's generally a problem with a deferred CI approach (CI
>> is
>> > > run
>> > > > at
>> > > > > > commit and not at merge time). This can either be solved through
>> > the
>> > > > > other
>> > > > > > proposal that's currently on dev@, by having a bot which does
>> > merges
>> > > > by
>> > > > > > having a global lock and a merge queue or by accepting the
>> issue.
>> > > > Reality
>> > > > > > right now is that we're running that model where two PRs which
>> are
>> > > > merged
>> > > > > > in parallel might break one another. One thing to consider
>> though
>> > is
>> > > > that
>> > > > > > this breakage would have to be introduced in two separate parts
>> > since
>> > > > > > otherwise there'd be merge conflicts. I think we had that
>> situation
>> > > > twice
>> > > > > > so far and the result was a quick revert, so I'd say that it's a
>> > > > problem
>> > > > > > that can happily be accepted. All other solutions basically
>> require
>> > > > some
>> > > > > > form of single-threaded and globally locked solution which
>> limits
>> > us
>> > > in
>> > > > > > scalability. I'd recommend to just accept that risk and revert
>> a PR
>> > > in
>> > > > > case
>> > > > > > it actually had a conflict.
>> > > > > >
>> > > > > > -Marco
>> > > > > >
>> > > > > > On Mon, Mar 16, 2020 at 6:29 PM Skalicky, Sam
>> > > > <sskalic@amazon.com.invalid
>> > > > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > We probably need some way to track which CI runs ran for which
>> > > commit
>> > > > > > too,
>> > > > > > > that way we can ensure that all CI runs ran on the commit that
>> > will
>> > > > be
>> > > > > > > merged. Maybe the bot can comment with the commit hash when
>> > users
>> > > > > > command
>> > > > > > > it to do something. Although since users can trigger
>> individual
>> > CI
>> > > > runs
>> > > > > > its
>> > > > > > > possible to have some commits run some CI runs but not
>> others. We
>> > > > need
>> > > > > > some
>> > > > > > > way to easily verify that the CI has executed all runs on the
>> > > commit
>> > > > > that
>> > > > > > > will be merged.
>> > > > > > >
>> > > > > > > Sam
>> > > > > > >
>> > > > > > > > On Mar 13, 2020, at 8:28 PM, Przemysław Trędak <
>> > > ptrendx@apache.org
>> > > > >
>> > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > CAUTION: This email originated from outside of the
>> > organization.
>> > > Do
>> > > > > not
>> > > > > > > click links or open attachments unless you can confirm the
>> sender
>> > > and
>> > > > > > know
>> > > > > > > the content is safe.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > I personally like the idea of opt-in more than opt-out:
>> > > > > > > > - ultimately PR author wants the PR to be merged so they (or
>> > > > > committer
>> > > > > > > reviewing the PR) will trigger the CI
>> > > > > > > > - if it is easy to trigger the PR via the bot command then
>> the
>> > > > amount
>> > > > > > of
>> > > > > > > work per PR should be less than with opt-out (since most of
>> the
>> > > > commits
>> > > > > > > should then be marked as [skip ci] or something similar
>> > > > > > > >
>> > > > > > > > +1 to the bot making a comment on each new PR with its
>> commands
>> > > > (and
>> > > > > > > also explaining, or at least giving links to the general PR
>> > process
>> > > > so
>> > > > > > new
>> > > > > > > PR authors are not lost). Maybe we could make the bot
>> recognize
>> > if
>> > > > the
>> > > > > PR
>> > > > > > > author is new or existing contributor and offer advice based
>> on
>> > > that?
>> > > > > > > >
>> > > > > > > > Thanks
>> > > > > > > > Przemek
>> > > > > > > >
>> > > > > > > > On 2020/03/13 22:06:58, Marco de Abreu <
>> > marco.g.abreu@gmail.com>
>> > > > > > wrote:
>> > > > > > > >> Hi,
>> > > > > > > >>
>> > > > > > > >> since it's no longer necessary to push a new commit to
>> trigger
>> > > CI,
>> > > > > it
>> > > > > > > will
>> > > > > > > >> already reduce the costs. But to me, requiring an action to
>> > > enable
>> > > > > CI
>> > > > > > > after
>> > > > > > > >> a PR has been created initially, is a no go. User can opt
>> out
>> > of
>> > > > CI,
>> > > > > > but
>> > > > > > > >> the default has to be CI being triggered automatically for
>> > every
>> > > > > > commit
>> > > > > > > >> unless specifically disabled by a participant. I'm also
>> fine
>> > > with
>> > > > > > > >> triggering certain additional jobs (think about running a
>> > > nightly
>> > > > > job
>> > > > > > > upon
>> > > > > > > >> request for a PR) to require a manual step, but the PR
>> > > validation
>> > > > > > > pipelines
>> > > > > > > >> have to run automatically. Every check that is marked as
>> > > > "Required"
>> > > > > in
>> > > > > > > >> GitHub has to be automatically kicked off.
>> > > > > > > >>
>> > > > > > > >> -Marco
>> > > > > > > >>
>> > > > > > > >> On Fri, Mar 13, 2020 at 9:50 PM Chaitanya Bapat <
>> > > > > chai.bapat@gmail.com
>> > > > > > >
>> > > > > > > >> wrote:
>> > > > > > > >>
>> > > > > > > >>> Firstly,
>> > > > > > > >>> Sorry I missed out on attaching the mail thread that was
>> sent
>> > > on
>> > > > > 12th
>> > > > > > > >>> February for notifying the community of the upcoming
>> changes
>> > to
>> > > > the
>> > > > > > > MXNet
>> > > > > > > >>> CI
>> > > > > > > >>> For reference :
>> > > > > > > >>>
>> > > > > > > >>>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://lists.apache.org/thread.html/r09a6ab2803a996fc80e00fe39ed312fa4865e8805e08df847f1addad%40%3Cdev.mxnet.apache.org%3E
>> > > > > > > >>>
>> > > > > > > >>> Now to the questions,
>> > > > > > > >>>> Is it possible for re-triggering a single job to be
>> abused?
>> > > > > > > >>> @Tao In the case when a user re-triggers a single job
>> > multiple
>> > > > > times,
>> > > > > > > that
>> > > > > > > >>> will be visible in the PR conversation thread. A
>> committer,
>> > > even
>> > > > > > after
>> > > > > > > he
>> > > > > > > >>> has approved the PR before, generally takes a look at the
>> > final
>> > > > > state
>> > > > > > > of
>> > > > > > > >>> the PR before merging. Would it be fair to assume the
>> > committer
>> > > > > could
>> > > > > > > take
>> > > > > > > >>> the multiple re-trigger of a single job into account
>> before
>> > > > > merging?
>> > > > > > > The
>> > > > > > > >>> committer then has the option to invoke `@mxnet-bot run ci
>> > > [all]
>> > > > `
>> > > > > to
>> > > > > > > >>> trigger the entire build pipeline one last to counter the
>> > > abuse.
>> > > > > This
>> > > > > > > is
>> > > > > > > >>> aligned with what @Leonard said.
>> > > > > > > >>>
>> > > > > > > >>> @Sandeep Thanks a lot for collecting and sharing valuable
>> > data.
>> > > > I'd
>> > > > > > > concur
>> > > > > > > >>> with the opinion that given the existing things
>> committers &
>> > PR
>> > > > > > Authors
>> > > > > > > >>> take care of, invoking CI shouldn't be that big of an
>> > > additional
>> > > > > > > burden.
>> > > > > > > >>>
>> > > > > > > >>> @Marco With the opt-out, the onus remains on the PR
>> Author.
>> > It
>> > > > > > doesn't
>> > > > > > > help
>> > > > > > > >>> reduce the resource usage. Hence, it was suggested to
>> switch
>> > to
>> > > > > > > >>> opt-in. @Leo's suggestion for proactive commenting on the
>> > part
>> > > of
>> > > > > bot
>> > > > > > > makes
>> > > > > > > >>> sense and is doable.
>> > > > > > > >>>
>> > > > > > > >>> Default : opt-out and User initiated opt-in (with
>> addressing
>> > > > Leo's
>> > > > > > fix
>> > > > > > > for
>> > > > > > > >>> the usability issue you correctly pointed out )
>> > > > > > > >>> @Marco How does this sound to you?
>> > > > > > > >>>
>> > > > > > > >>> Again, thank you all for chiming in and voicing your
>> opinion.
>> > > > > > > Appreciate
>> > > > > > > >>> it.
>> > > > > > > >>> We can take ahead these discussions in today's demo
>> meeting.
>> > > > > [Design
>> > > > > > > Doc
>> > > > > > > >>> <
>> > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
>> > > > >]
>> > > > > > > [Demo
>> > > > > > > >>> Video <https://www.youtube.com/watch?v=gfOGwZId8aU>]
>> > > > > > > >>>
>> > > > > > > >>> Thanks,
>> > > > > > > >>> Chai
>> > > > > > > >>>
>> > > > > > > >>> On Fri, 13 Mar 2020 at 12:34, Marco de Abreu <
>> > > > > > marco.g.abreu@gmail.com>
>> > > > > > > >>> wrote:
>> > > > > > > >>>
>> > > > > > > >>>> I'd recommend that the bot makes an initial comment when
>> a
>> > PR
>> > > > gets
>> > > > > > > opened
>> > > > > > > >>>> and informs the users of its commands. It then tells the
>> > user
>> > > > the
>> > > > > > > commend
>> > > > > > > >>>> to opt out of CI.
>> > > > > > > >>>>
>> > > > > > > >>>> -Marco
>> > > > > > > >>>>
>> > > > > > > >>>> Lausen, Leonard <la...@amazon.com.invalid> schrieb am
>> Fr.,
>> > > 13.
>> > > > > > März
>> > > > > > > >>> 2020,
>> > > > > > > >>>> 20:27:
>> > > > > > > >>>>
>> > > > > > > >>>>> On opt-out: People may be unaware of opt-out would not
>> use
>> > > it.
>> > > > > > There
>> > > > > > > is
>> > > > > > > >>>> no
>> > > > > > > >>>>> incentive to use opt-out, as the PR author doesn't pay
>> any
>> > > > money
>> > > > > > for
>> > > > > > > CI
>> > > > > > > >>>>> run.
>> > > > > > > >>>>>
>> > > > > > > >>>>> I agree with Marco though that opt-in alone may cause
>> > > usability
>> > > > > > > issues,
>> > > > > > > >>>> as
>> > > > > > > >>>>> contributors may not be aware of how to trigger the CI.
>> > > > > > > >>>>> One solution is that the bot proactively comments on
>> the PR
>> > > and
>> > > > > > > reminds
>> > > > > > > >>>> the
>> > > > > > > >>>>> author to trigger running CI once the author deems the
>> PR
>> > > > ready.
>> > > > > > > >>>>>
>> > > > > > > >>>>> But even if we choose opt-out, the bot will still add a
>> lot
>> > > of
>> > > > > > value,
>> > > > > > > >>> as
>> > > > > > > >>>> PR
>> > > > > > > >>>>> authors can retrigger single jobs that have failed due
>> to
>> > > > > > flakiness.
>> > > > > > > >>>>>
>> > > > > > > >>>>>> Is it possible for re-triggering a single job to be
>> > abused?
>> > > > For
>> > > > > > > >>>> example,
>> > > > > > > >>>>>> the author spends two days re-triggering a flaky job to
>> > make
>> > > > it
>> > > > > > > pass.
>> > > > > > > >>>>>
>> > > > > > > >>>>> Yes, this is possible. I suggest the committer who
>> likes to
>> > > > > merge a
>> > > > > > > PR
>> > > > > > > >>>>> needs to
>> > > > > > > >>>>> make a good judgement here if a PR is abusing the
>> feature,
>> > > and
>> > > > if
>> > > > > > so,
>> > > > > > > >>>>> retrigger
>> > > > > > > >>>>> all CI runs.
>> > > > > > > >>>>>
>> > > > > > > >>>>> Best regards
>> > > > > > > >>>>> Leonard
>> > > > > > > >>>>>
>> > > > > > > >>>>> On Fri, 2020-03-13 at 08:07 +0100, Marco de Abreu wrote:
>> > > > > > > >>>>>> Thanks for the data Sandeep. In these cases it sounds
>> like
>> > > it
>> > > > > > would
>> > > > > > > >>>> have
>> > > > > > > >>>>>> rather been better when people explicitly turned off
>> CI in
>> > > > that
>> > > > > > > case.
>> > > > > > > >>>>>> What's the argument against an opt-out instead of an
>> > opt-in?
>> > > > > > > >>>>>>
>> > > > > > > >>>>>> My intention is that I consider it quite cumbersome to
>> > make
>> > > > it a
>> > > > > > > >>>>> *required*
>> > > > > > > >>>>>> step to always trigger CI manually, even if just
>> > submitting
>> > > a
>> > > > > > small
>> > > > > > > >>> PR.
>> > > > > > > >>>>> I'd
>> > > > > > > >>>>>> rather see people explicitly turning off CI if they
>> > wouldn't
>> > > > > like
>> > > > > > to
>> > > > > > > >>>> use
>> > > > > > > >>>>> it
>> > > > > > > >>>>>> - and there's also the "draft" stage for a PR which
>> some
>> > > > > > > contributors
>> > > > > > > >>>> are
>> > > > > > > >>>>>> using.
>> > > > > > > >>>>>>
>> > > > > > > >>>>>> With regards to WIP and do not review: I think these
>> are
>> > use
>> > > > > cases
>> > > > > > > >>>> where
>> > > > > > > >>>>>> you want CI feedback, as otherwise you wouldn't have
>> > opened
>> > > > the
>> > > > > > PR.
>> > > > > > > >>> If
>> > > > > > > >>>>> you
>> > > > > > > >>>>>> don't want human feedback and neither machine feedback,
>> > why
>> > > > open
>> > > > > > the
>> > > > > > > >>> PR
>> > > > > > > >>>>> at
>> > > > > > > >>>>>> all?
>> > > > > > > >>>>>>
>> > > > > > > >>>>>> -Marco
>> > > > > > > >>>>>>
>> > > > > > > >>>>>>
>> > > > > > > >>>>>> sandeep krishnamurthy <sa...@gmail.com>
>> > schrieb
>> > > > am
>> > > > > > Fr.,
>> > > > > > > >>>> 13.
>> > > > > > > >>>>>> März 2020, 05:24:
>> > > > > > > >>>>>>
>> > > > > > > >>>>>>> I tried to gather some data for us to discuss this
>> topic
>> > in
>> > > > > this
>> > > > > > > >>>>> thread. I
>> > > > > > > >>>>>>> tried to count number of un-necessary builds by
>> looking
>> > at
>> > > > most
>> > > > > > > >>>> recent
>> > > > > > > >>>>> (as
>> > > > > > > >>>>>>> of 12, March 9 PM PST) 50 PRs merged to master and 50
>> > PRs.
>> > > > > > > >>>>>>> Identifying un-necessary builds is bit subjective. I
>> > tried
>> > > to
>> > > > > be
>> > > > > > > >>> more
>> > > > > > > >>>>>>> conservative where I didn't count a build as
>> un-necessary
>> > > if
>> > > > I
>> > > > > > was
>> > > > > > > >>> in
>> > > > > > > >>>>>>> doubt. Hence, I was not able to automate, but I made
>> an
>> > > > effort
>> > > > > to
>> > > > > > > >>> go
>> > > > > > > >>>>>>> through PRs manually and use below criteria to
>> identify
>> > > > > > > >>> un-necessary
>> > > > > > > >>>>>>> commits triggering the builds.
>> > > > > > > >>>>>>>
>> > > > > > > >>>>>>> 1. Explicitly marked as WIP / do not review PR
>> > > > > > > >>>>>>> 2. Incremental WIP commit and finally commenting a
>> > commit
>> > > > > > > >>> “trigger
>> > > > > > > >>>>> CI”
>> > > > > > > >>>>>>> 3. Multiple commits to address all comments from
>> single
>> > > > > review.
>> > > > > > > >>>>> This is
>> > > > > > > >>>>>>> assuming we see a comment, address them, commit,
>> next
>> > the
>> > > > > > > >>>> following
>> > > > > > > >>>>>>> comment
>> > > > > > > >>>>>>> 4. Sequence of documentation only changes
>> > > > > > > >>>>>>>
>> > > > > > > >>>>>>> I found there were around 42 avoidable builds from
>> most
>> > > > recent
>> > > > > 50
>> > > > > > > >>>>> merged
>> > > > > > > >>>>>>> PRs and around 86 builds from recent 50 open PRs.
>> > > > > > > >>>>>>>
>> > > > > > > >>>>>>> I synced up with other contributors (Joe Evans, Chai)
>> > from
>> > > > > Amazon
>> > > > > > > >>> who
>> > > > > > > >>>>> is
>> > > > > > > >>>>>>> contributing to MXNet CI system. I was told that on an
>> > > > average
>> > > > > it
>> > > > > > > >>>> costs
>> > > > > > > >>>>>>> around $84 per build and on an average 6 commits per
>> > merged
>> > > > PR
>> > > > > > (for
>> > > > > > > >>>>> year
>> > > > > > > >>>>>>> 2019). Going by that, it is approximately 1/6 builds
>> are
>> > > > > > avoidable.
>> > > > > > > >>>>> [100 /
>> > > > > > > >>>>>>> 300 + 300 ]
>> > > > > > > >>>>>>>
>> > > > > > > >>>>>>>
>> > > > > > > >>>>>>> Usability should be top most priority. But, since
>> either
>> > a
>> > > > > > reviewer
>> > > > > > > >>>> or
>> > > > > > > >>>>> pr
>> > > > > > > >>>>>>> author can trigger the bot, is it really a hurdle for
>> pr
>> > > > author
>> > > > > > or
>> > > > > > > >>>>> reviewer
>> > > > > > > >>>>>>> to call a bot to trigger CI? Given that PR author and
>> > > > reviewer
>> > > > > is
>> > > > > > > >>>>> already
>> > > > > > > >>>>>>> actively commenting various details such as - PR
>> > > description,
>> > > > > > > >>> review
>> > > > > > > >>>>>>> comments and responses, adding labels etc.
>> > > > > > > >>>>>>>
>> > > > > > > >>>>>>>
>> > > > > > > >>>>>>> Me too curious to know the behavior for Tao's above
>> use
>> > > case.
>> > > > > > > >>>>>>>
>> > > > > > > >>>>>>>
>> > > > > > > >>>>>>> Best,
>> > > > > > > >>>>>>>
>> > > > > > > >>>>>>> Sandeep
>> > > > > > > >>>>>>>
>> > > > > > > >>>>>>> On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <
>> > mutouorz@gmail.com
>> > > >
>> > > > > > wrote:
>> > > > > > > >>>>>>>
>> > > > > > > >>>>>>>> Is it possible for re-triggering a single job to be
>> > > abused?
>> > > > > For
>> > > > > > > >>>>> example,
>> > > > > > > >>>>>>>> the author spends two days re-triggering a flaky job
>> to
>> > > make
>> > > > > it
>> > > > > > > >>>>> pass. But
>> > > > > > > >>>>>>>> other jobs which have passed the validation may be
>> > broken
>> > > by
>> > > > > > > >>> other
>> > > > > > > >>>>>>> commits
>> > > > > > > >>>>>>>> during the two day without being noticed. And finally
>> > the
>> > > PR
>> > > > > is
>> > > > > > > >>>>> merged
>> > > > > > > >>>>>>> with
>> > > > > > > >>>>>>>> underlying problems.
>> > > > > > > >>>>>>>>
>> > > > > > > >>>>>>>> On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <
>> > > > > > > >>>>> marco.g.abreu@gmail.com>
>> > > > > > > >>>>>>>> wrote:
>> > > > > > > >>>>>>>>
>> > > > > > > >>>>>>>>> In the end it only comes down to money, considering
>> > that
>> > > > the
>> > > > > > > >>>>> system is
>> > > > > > > >>>>>>>> auto
>> > > > > > > >>>>>>>>> scaling, making the execution time constant.
>> > > > > > > >>>>>>>>>
>> > > > > > > >>>>>>>>> If we're trading money for usability, I certainly
>> would
>> > > > > prefer
>> > > > > > > >>>>>>> usability.
>> > > > > > > >>>>>>>>> I'd rather recommend to spend time on parallelizing
>> > test
>> > > > > > > >>>> execution
>> > > > > > > >>>>> or
>> > > > > > > >>>>>>>>> getting rid of integration tests in the PR stage
>> > instead
>> > > > > > > >>> reducing
>> > > > > > > >>>>> the
>> > > > > > > >>>>>>>> costs
>> > > > > > > >>>>>>>>> by making people not use it. But taking a step back
>> to
>> > > > > > > >>> requiring
>> > > > > > > >>>>> people
>> > > > > > > >>>>>>>> to
>> > > > > > > >>>>>>>>> manually trigger CI again doesn't feel right.
>> > > > > > > >>>>>>>>>
>> > > > > > > >>>>>>>>> I'm happy to see that bot deployed, but I do not
>> agree
>> > > with
>> > > > > > > >>>>> removing
>> > > > > > > >>>>>>> the
>> > > > > > > >>>>>>>>> auto trigger functionality for new commits.
>> > > > > > > >>>>>>>>>
>> > > > > > > >>>>>>>>> -Marco
>> > > > > > > >>>>>>>>>
>> > > > > > > >>>>>>>>> Chaitanya Bapat <ch...@gmail.com> schrieb am
>> Do.,
>> > > 12.
>> > > > > > > >>> März
>> > > > > > > >>>>> 2020,
>> > > > > > > >>>>>>>>> 22:47:
>> > > > > > > >>>>>>>>>
>> > > > > > > >>>>>>>>>> @Marco Thanks for pointing that out.
>> > > > > > > >>>>>>>>>> Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM -
>> 3:30
>> > > PM
>> > > > in
>> > > > > > > >>>>>>>> (UTC-08:00)
>> > > > > > > >>>>>>>>>> Pacific Time (US & Canada).
>> > > > > > > >>>>>>>>>>
>> > > > > > > >>>>>>>>>>> When do we expect this bot to be deployed?
>> > > > > > > >>>>>>>>>> @Lin If all goes well in the next week I can
>> deploy it
>> > > to
>> > > > > > > >>>> public
>> > > > > > > >>>>>>> Apache
>> > > > > > > >>>>>>>>>> (provided I get permissions from Apache Infra)
>> > > > > > > >>>>>>>>>>
>> > > > > > > >>>>>>>>>> @Marco Thanks for your feedback.
>> > > > > > > >>>>>>>>>>> CI system has to support the community without
>> > > requiring
>> > > > > > > >>>>> people to
>> > > > > > > >>>>>>>>>> constantly shepherd every single run
>> > > > > > > >>>>>>>>>> We have data for the number of times CI was
>> triggered
>> > > > > > > >>>>> unnecessarily
>> > > > > > > >>>>>>>> which
>> > > > > > > >>>>>>>>>> includes
>> > > > > > > >>>>>>>>>> - Entire build triggered instead of specific build
>> > > > > > > >>>>>>>>>> - CI triggered when PR is still work in progress or
>> > not
>> > > > yet
>> > > > > > > >>>> ready
>> > > > > > > >>>>>>> (say
>> > > > > > > >>>>>>>> -
>> > > > > > > >>>>>>>>>> intermediate commits)
>> > > > > > > >>>>>>>>>> At the end its a trade-off
>> > > > > > > >>>>>>>>>> Money, Resources, Time to build for each and every
>> > > commit
>> > > > vs
>> > > > > > > >>>>> Pain of
>> > > > > > > >>>>>>>>>> triggering builds
>> > > > > > > >>>>>>>>>>
>> > > > > > > >>>>>>>>>>
>> > > > > > > >>>>>>>>>>> Scan trigger plugin would poll SCM. Can we use
>> plugin
>> > > at
>> > > > > > > >>>>> scale?
>> > > > > > > >>>>>>>>>>
>> > > > > > > >>>>>>>>>> 1. I haven't tested it on scale. But I think with
>> the
>> > > > > current
>> > > > > > > >>>>> scale
>> > > > > > > >>>>>>> of
>> > > > > > > >>>>>>>>>> MXNet repo (191 open PRs i.e. checking for changes
>> to
>> > > 191
>> > > > > > > >>>>> branches -
>> > > > > > > >>>>>>> It
>> > > > > > > >>>>>>>>>> should be manageable)
>> > > > > > > >>>>>>>>>> 2. What's the purpose of the plugin? tldr; Branch
>> > > > discovery
>> > > > > > > >>> or
>> > > > > > > >>>>> branch
>> > > > > > > >>>>>>>>>> indexing.
>> > > > > > > >>>>>>>>>> Scan trigger plugin comes into the picture only
>> once
>> > per
>> > > > PR
>> > > > > > > >>> per
>> > > > > > > >>>>> job
>> > > > > > > >>>>>>>>> (i.e. 8
>> > > > > > > >>>>>>>>>> times per PR for 8 jobs). It is basically done
>> when a
>> > > new
>> > > > PR
>> > > > > > > >>> is
>> > > > > > > >>>>> made
>> > > > > > > >>>>>>>> and
>> > > > > > > >>>>>>>>>> the job (say unix-cpu hasn't discovered the new PR
>> > > branch
>> > > > > > > >>> yet).
>> > > > > > > >>>>>>> That's
>> > > > > > > >>>>>>>>> it.
>> > > > > > > >>>>>>>>>> So it shouldn't be a problem for public MXNet repo.
>> > > > > > > >>>>>>>>>>
>> > > > > > > >>>>>>>>>> Thanks,
>> > > > > > > >>>>>>>>>> Chai
>> > > > > > > >>>>>>>>>>
>> > > > > > > >>>>>>>>>>
>> > > > > > > >>>>>>>>>> On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <
>> > > > > > > >>>>>>> marco.g.abreu@gmail.com>
>> > > > > > > >>>>>>>>>> wrote:
>> > > > > > > >>>>>>>>>>
>> > > > > > > >>>>>>>>>>> Btw you forgot to set a date and time for the
>> metting
>> > > > > > > >>>>>>>>>>>
>> > > > > > > >>>>>>>>>>>
>> > > > > > > >>>>>>>>>>> On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu <
>> > > > > > > >>>>>>>>> marco.g.abreu@gmail.com
>> > > > > > > >>>>>>>>>>> wrote:
>> > > > > > > >>>>>>>>>>>
>> > > > > > > >>>>>>>>>>>> Thanks Chai, I generally like the idea of the
>> bot.
>> > But
>> > > > > > > >>> I'm
>> > > > > > > >>>>> not a
>> > > > > > > >>>>>>>>>>> supporter
>> > > > > > > >>>>>>>>>>>> of the idea to disable any automatic triggering
>> > > > > > > >>> (disabling
>> > > > > > > >>>>> the
>> > > > > > > >>>>>>>>> webhook
>> > > > > > > >>>>>>>>>> is
>> > > > > > > >>>>>>>>>>>> also not an option, considering that this will
>> > disable
>> > > > > > > >>>> master
>> > > > > > > >>>>>>>>>> triggers).
>> > > > > > > >>>>>>>>>>>> The CI system has to support the community
>> without
>> > > > > > > >>>> requiring
>> > > > > > > >>>>>>> people
>> > > > > > > >>>>>>>>> to
>> > > > > > > >>>>>>>>>>>> constantly shepherd every single run. Disabling
>> > > > automatic
>> > > > > > > >>>>>>>> triggering
>> > > > > > > >>>>>>>>>>> seems
>> > > > > > > >>>>>>>>>>>> like a step back to me.
>> > > > > > > >>>>>>>>>>>>
>> > > > > > > >>>>>>>>>>>> Instead, I'd recommend that CI gets triggered
>> upon
>> > > every
>> > > > > > > >>>>> commit
>> > > > > > > >>>>>>> as
>> > > > > > > >>>>>>>>>> usual,
>> > > > > > > >>>>>>>>>>>> but people have the possibility to call a
>> "command"
>> > > > (i.e.
>> > > > > > > >>>>> make a
>> > > > > > > >>>>>>>>>> message
>> > > > > > > >>>>>>>>>>>> which results in the bot setting a label) to
>> disable
>> > > CI
>> > > > > > > >>>> until
>> > > > > > > >>>>>>> they
>> > > > > > > >>>>>>>>>> revoke
>> > > > > > > >>>>>>>>>>>> it. But the standard should still be that a new
>> > commit
>> > > > > > > >>>>> triggers a
>> > > > > > > >>>>>>>> new
>> > > > > > > >>>>>>>>>> CI
>> > > > > > > >>>>>>>>>>>> run.
>> > > > > > > >>>>>>>>>>>>
>> > > > > > > >>>>>>>>>>>>
>> > > > > > > >>>>
>> > https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
>> > > > > > > >>>>>>>> seems
>> > > > > > > >>>>>>>>>> like
>> > > > > > > >>>>>>>>>>>> this would poll SCM. This will incur high quota
>> > > > > > > >>>>> restrictions. Are
>> > > > > > > >>>>>>>> you
>> > > > > > > >>>>>>>>>>> sure
>> > > > > > > >>>>>>>>>>>> that you can use that plugin at scale?
>> > > > > > > >>>>>>>>>>>>
>> > > > > > > >>>>>>>>>>>> -Marco
>> > > > > > > >>>>>>>>>>>>
>> > > > > > > >>>>>>>>>>>>
>> > > > > > > >>>>>>>>>>>> On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <
>> > > > > > > >>>>> apeforest@gmail.com>
>> > > > > > > >>>>>>>>> wrote:
>> > > > > > > >>>>>>>>>>>>> Chai,
>> > > > > > > >>>>>>>>>>>>>
>> > > > > > > >>>>>>>>>>>>> Awesome work. When do we expect this bot to be
>> > > > > > > >>> deployed?
>> > > > > > > >>>>>>>>>>>>>
>> > > > > > > >>>>>>>>>>>>> Best,
>> > > > > > > >>>>>>>>>>>>>
>> > > > > > > >>>>>>>>>>>>> Lin
>> > > > > > > >>>>>>>>>>>>>
>> > > > > > > >>>>>>>>>>>>> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat
>> <
>> > > > > > > >>>>>>>>> chai.bapat@gmail.com
>> > > > > > > >>>>>>>>>>>>> wrote:
>> > > > > > > >>>>>>>>>>>>>
>> > > > > > > >>>>>>>>>>>>>> Hello MXNet community,
>> > > > > > > >>>>>>>>>>>>>>
>> > > > > > > >>>>>>>>>>>>>> I have built an MXNet Bot <
>> > > > > > > >>>> https://github.com/mxnet-bot>
>> > > > > > > >>>>> that
>> > > > > > > >>>>>>>>>> allows
>> > > > > > > >>>>>>>>>>> PR
>> > > > > > > >>>>>>>>>>>>>> Authors, Committers and Jenkins Admins to
>> trigger
>> > CI
>> > > > > > > >>>>> manually.
>> > > > > > > >>>>>>>>>>>>>> It handles 2 problems
>> > > > > > > >>>>>>>>>>>>>> 1. Manual CI trigger instead of existing
>> automated
>> > > CI
>> > > > > > > >>>>> trigger
>> > > > > > > >>>>>>>>>>>>>> 2. Gives permissions to PR Authors (in
>> addition to
>> > > > > > > >>>> MXNet
>> > > > > > > >>>>>>>>> Committers
>> > > > > > > >>>>>>>>>>> and
>> > > > > > > >>>>>>>>>>>>>> Jenkins Admins)
>> > > > > > > >>>>>>>>>>>>>>
>> > > > > > > >>>>>>>>>>>>>> Design Doc :
>> > > > > > > >>>>>>>>>>>>>>
>> > > > > > > >>>>>>>
>> > > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
>> > > > > > > >>>>>>>>>>>>>> I urge you all to attend the demonstration
>> meeting
>> > > > > > > >>> and
>> > > > > > > >>>>> lend
>> > > > > > > >>>>>>> your
>> > > > > > > >>>>>>>>>> views
>> > > > > > > >>>>>>>>>>>>> on
>> > > > > > > >>>>>>>>>>>>>> the same.
>> > > > > > > >>>>>>>>>>>>>>
>> > > > > > > >>>>>>>>>>>>>> Thank you,
>> > > > > > > >>>>>>>>>>>>>> Chai
>> > > > > > > >>>>>>>>>>>>>>
>> > > > > > > >>>>>>>>>>>>>> *Meeting Details*:
>> > > > > > > >>>>>>>>>>>>>> ==============Conference Bridge
>> > > > > > > >>>> Information==============
>> > > > > > > >>>>>>>>>>>>>> You have been invited to an online meeting,
>> > powered
>> > > > > > > >>> by
>> > > > > > > >>>>> Amazon
>> > > > > > > >>>>>>>>> Chime.
>> > > > > > > >>>>>>>>>>>>>> *Chime meeting ID*: *9272158344*
>> > > > > > > >>>>>>>>>>>>>> Join via Chime clients (manually): Select
>> > 'Meetings
>> > > >
>> > > > > > > >>>>> Join a
>> > > > > > > >>>>>>>>>> Meeting',
>> > > > > > > >>>>>>>>>>>>> and
>> > > > > > > >>>>>>>>>>>>>> enter 9272158344
>> > > > > > > >>>>>>>>>>>>>> Join via Chime clients (auto-call): If you
>> invite
>> > > > > > > >>>>> auto-call as
>> > > > > > > >>>>>>>>>>> attendee,
>> > > > > > > >>>>>>>>>>>>>> Chime will call you when the meeting starts,
>> > select
>> > > > > > > >>>>> 'Answer'
>> > > > > > > >>>>>>>>>>>>>> *Join via browser screen share*:
>> > > > > > > >>>>> https://chime.aws/9272158344
>> > > > > > > >>>>>>>>>>>>>> *Join via phone* (US):
>> > +1-929-432-4463,,,9272158344#
>> > > > > > > >>>>>>>>>>>>>> *Join via phone (US toll-free)*:
>> > > > > > > >>>>> +1-855-552-4463,,,9272158344#
>> > > > > > > >>>>>>>>>>>>>> International dial-in:
>> > > > > > > >>>> https://chime.aws/dialinnumbers/
>> > > > > > > >>>>>>>>>>>>>> In-room video system: Ext: 62000, Meeting PIN:
>> > > > > > > >>>>> 9272158344#
>> > > > > > > >>>>>>>>>>>>>>
>> > > > > > > >>>>>>>>>>>>>> --
>> > > > > > > >>>>>>>>>>>>>> *Chaitanya Prakash Bapat*
>> > > > > > > >>>>>>>>>>>>>> *+1 (973) 953-6299*
>> > > > > > > >>>>>>>>>>>>>>
>> > > > > > > >>>>>>>>>>>>>> [image:
>> https://www.linkedin.com//in/chaibapat25]
>> > > > > > > >>>>>>>>>>>>>> <https://github.com/ChaiBapchya>[image:
>> > > > > > > >>>>>>>>>>>>> https://www.facebook.com/chaibapat
>> > > > > > > >>>>>>>>>>>>>> ]
>> > > > > > > >>>>>>>>>>>>>> <https://www.facebook.com/chaibapchya>[image:
>> > > > > > > >>>>>>>>>>>>>> https://twitter.com/ChaiBapchya] <
>> > > > > > > >>>>>>>> https://twitter.com/ChaiBapchya
>> > > > > > > >>>>>>>>>>>>>> [image:
>> > > > > > > >>>>>>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
>> > > > > > > >>>>>>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
>> > > > > > > >>>>>>>>>>>>>>
>> > > > > > > >>>>>>>>>>
>> > > > > > > >>>>>>>>>> --
>> > > > > > > >>>>>>>>>> *Chaitanya Prakash Bapat*
>> > > > > > > >>>>>>>>>> *+1 (973) 953-6299*
>> > > > > > > >>>>>>>>>>
>> > > > > > > >>>>>>>>>> [image: https://www.linkedin.com//in/chaibapat25]
>> > > > > > > >>>>>>>>>> <https://github.com/ChaiBapchya>[image:
>> > > > > > > >>>>>>>>> https://www.facebook.com/chaibapat
>> > > > > > > >>>>>>>>>> ]
>> > > > > > > >>>>>>>>>> <https://www.facebook.com/chaibapchya>[image:
>> > > > > > > >>>>>>>>>> https://twitter.com/ChaiBapchya] <
>> > > > > > > >>>>> https://twitter.com/ChaiBapchya
>> > > > > > > >>>>>>>>>> [image:
>> > > > > > > >>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
>> > > > > > > >>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
>> > > > > > > >>>>>>>>>>
>> > > > > > > >>>>>>>
>> > > > > > > >>>>>>> --
>> > > > > > > >>>>>>> Sandeep Krishnamurthy
>> > > > > > > >>>>>>>
>> > > > > > > >>>>>
>> > > > > > > >>>>
>> > > > > > > >>>
>> > > > > > > >>>
>> > > > > > > >>> --
>> > > > > > > >>> *Chaitanya Prakash Bapat*
>> > > > > > > >>> *+1 (973) 953-6299*
>> > > > > > > >>>
>> > > > > > > >>> [image: https://www.linkedin.com//in/chaibapat25]
>> > > > > > > >>> <https://github.com/ChaiBapchya>[image:
>> > > > > > > https://www.facebook.com/chaibapat
>> > > > > > > >>> ]
>> > > > > > > >>> <https://www.facebook.com/chaibapchya>[image:
>> > > > > > > >>> https://twitter.com/ChaiBapchya] <
>> > > > https://twitter.com/ChaiBapchya
>> > > > > > > >[image:
>> > > > > > > >>> https://www.linkedin.com//in/chaibapat25]
>> > > > > > > >>> <https://www.linkedin.com//in/chaibapchya/>
>> > > > > > > >>>
>> > > > > > > >>
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > *Chaitanya Prakash Bapat*
>> > > > > *+1 (973) 953-6299*
>> > > > >
>> > > > > [image: https://www.linkedin.com//in/chaibapat25]
>> > > > > <https://github.com/ChaiBapchya>[image:
>> > > > https://www.facebook.com/chaibapat
>> > > > > ]
>> > > > > <https://www.facebook.com/chaibapchya>[image:
>> > > > > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
>> > > > >[image:
>> > > > > https://www.linkedin.com//in/chaibapat25]
>> > > > > <https://www.linkedin.com//in/chaibapchya/>
>> > > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > *Chaitanya Prakash Bapat*
>> > > *+1 (973) 953-6299*
>> > >
>> > > [image: https://www.linkedin.com//in/chaibapat25]
>> > > <https://github.com/ChaiBapchya>[image:
>> > https://www.facebook.com/chaibapat
>> > > ]
>> > > <https://www.facebook.com/chaibapchya>[image:
>> > > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
>> > >[image:
>> > > https://www.linkedin.com//in/chaibapat25]
>> > > <https://www.linkedin.com//in/chaibapchya/>
>> > >
>> >
>>
>>
>> --
>> *Chaitanya Prakash Bapat*
>> *+1 (973) 953-6299*
>>
>> [image: https://www.linkedin.com//in/chaibapat25]
>> <https://github.com/ChaiBapchya>[image:
>> https://www.facebook.com/chaibapat]
>> <https://www.facebook.com/chaibapchya>[image:
>> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
>> https://www.linkedin.com//in/chaibapat25]
>> <https://www.linkedin.com//in/chaibapchya/>
>>
>
Re: MXNet Bot Demo
Posted by Marco de Abreu <ma...@gmail.com>.
Sounds like a good plan!
Please send me the URL (please make sure it's backed by DNS and not just
the gateway URL) of the webhook handler, GitHub events you're interested in
and the shared secret in a private email to my personal email address. I
will then create the ticket with Apache infra.
-Marco
Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 19. März 2020, 23:07:
> @Marco Alright, it makes total sense to test out the Bot feature alongside
> auto-trigger as a transition.
>
> Path Forward:
> 1. Setup MXNet Bot on apache/incubator-mxnet repo (GitHub WebHook and
> Infra)
> 2. We don't turn off automatic trigger of PR builds for now.
> 3. Hopefully, bot is used by developers to trigger specific jobs
> 4. Later on (say around April 20), let's discuss the possibility of
> switching off auto-trigger (with appropriate data) if it makes sense.
> Thanks Marco for volunteering to help enable the web hook on
> apache/incubator-mxnet. Let me know if we can sync up on Slack channel to
> get the ball rolling.
>
> Thanks once again for the entire community to step in and help try out this
> Bot.
> Chai
>
> On Wed, 18 Mar 2020 at 17:07, Marco de Abreu <ma...@gmail.com>
> wrote:
>
> > Hi, that's correct. But as stated previously, it's not an option to
> remove
> > the hook. For now, I'd like to see how the system behaves while it's
> > optional. Later on, we can talk about revisiting this decision. But to me
> > it's not an option to deploy an entirely new system and approach without
> > having a transition or even a timeframe in which we are able to fall
> back.
> >
> > I'm happy to support the deployment of the bot and add an additional
> > webhook to enable it's functionality to support selective triggering by
> PR
> > authors and committers, but I will not support the disabling of automatic
> > triggering of branches or PRs.
> >
> > -Marco
> >
> > Chaitanya Bapat <ch...@gmail.com> schrieb am Mi., 18. März 2020,
> > 21:00:
> >
> > > Hey Marco,
> > >
> > > I thought currently every commit on PR and master triggers CI
> > > because
> > > a. github webhook points to Jenkins Server
> > > b. GH Webhook events trigger builds on Jenkins for all commits to any
> > > branch in apache/incubator-mxnet
> > > may it be master/PR/non-PR
> > > Reason:
> > > Because all the 3 types of branches are discovered by Jenkins (non-PR
> > > (including master) and PR)
> > >
> > > Proposal: Remove GitHub WebHook to Jenkins and replace with GH Webhook
> to
> > > Lambda
> > > But after I remove the github webhook that points to Jenkins : *N**o
> > commit
> > > will trigger Jenkins build by default* (as Jenkins wont receive GH
> > events)
> > > Only those that Bot deems fit will be triggered (using Jenkins API
> > invoked
> > > by Lambda).
> > > Hence its needed to handle that case of master merge.
> > > Am I understanding this correctly?
> > >
> > > On Wed, 18 Mar 2020 at 04:23, Marco de Abreu <ma...@gmail.com>
> > > wrote:
> > >
> > > > Thanks Chai, sounds good to me. Could you elaborate a bit on the
> point
> > > > about triggering a CI run after the PR has been merged? We already to
> > > that
> > > > automatically for the master, so what's the benefit to do it twice?
> > > >
> > > > -Marco
> > > >
> > > > Chaitanya Bapat <ch...@gmail.com> schrieb am Mi., 18. März
> 2020,
> > > > 09:30:
> > > >
> > > > > Update:
> > > > >
> > > > > > we can ensure that all CI runs ran on the commit that will be
> > merged
> > > > > @Sam Skalicky <sa...@gmail.com> Branch Protection is added
> to
> > > > public
> > > > > MXNet repo. It ensures that for every PR to be merged, the CI
> passes.
> > > All
> > > > > the jobs selected "required" jobs will have to be green for the PR
> to
> > > be
> > > > > merged. Ofcourse, users with "Adminstrator" access can merge
> without
> > it
> > > > but
> > > > > that's just a backdoor. It is the case now and will continue to be
> > the
> > > > case
> > > > > with the inclusion of Bot.
> > > > >
> > > > > > easily verify that the CI has executed all runs on the commit
> that
> > > will
> > > > > be merged
> > > > > GitHub UI shows all the jobs and the status corresponding to it on
> > > every
> > > > > commit. That should suffice. For the merged commits, Repo ->
> Commits
> > ->
> > > > > Commit ID (Status) can be tracked currently (only way that I know
> > of).
> > > > > Moreover, it is beyond the scope of this project (and possibly out
> of
> > > our
> > > > > control since this is purely GitHub UI specific use-case).
> > > > >
> > > > > Thanks @przemyslaw for supporting the opt-in.
> > > > >
> > > > > Thanks everyone in the community for sharing concerns, voicing your
> > > > opinion
> > > > > and participating in the discussion.
> > > > > Thanks to those who attended the demo last Friday.
> > > > >
> > > > > Action items from that discussion
> > > > > 1. Handle master merge builds [Done]
> > > > > Bot runs entire CI suite after the PR is merged and comments on the
> > PR
> > > > > about the same.
> > > > > Design decision :
> > > > > MXNet Bot comment about master merge build on the *merge commit vs
> > PR*.
> > > > > After the PR is merged, Bot runs entire CI and comments the result
> of
> > > CI
> > > > > trigger on the PR (because it is easy to track on a PR rather than
> > > > > commenting inside the merge commit)
> > > > >
> > > > > 2. Idempotent condition
> > > > > In case of already running build, if an attempt is made to
> retrigger
> > > the
> > > > > job then what should be the response
> > > > > a. Not to re-trigger, let the ongoing build continue till
> completion
> > > > > b. End the ongoing build and re-trigger
> > > > > c. Let the ongoing build continue, re-trigger new build
> > > > >
> > > > > From resource saving point of view, *c* looks costly and a can be
> > > > > avoided/optimized by B.
> > > > > In case when a re-trigger was started "erroneously" then killing
> > > ongoing
> > > > > build and re-trigger is a waste.
> > > > > In case when ongoing build failed in one sub-part, then
> re-triggering
> > > is
> > > > > justified.
> > > > > Erroneous re-triggers would be less often while conscious
> re-triggers
> > > to
> > > > > suppress failure is more common use-case. It looks like a safe
> > > assumption
> > > > > to make given the trade-off.
> > > > > [Open to debate]
> > > > >
> > > > > 3. Add security consideration [Use of secret manager, but without
> > > > > auto-rotation due to Jenkins manual config requirement] [Done]
> > > > > 4. New PR Instruction message by the Bot [Done]
> > > > > Thanks to the suggestion of Leonard, supported by others. I've now
> > > added
> > > > > the feature where the Bot comments a help message. [For reference -
> > > > > https://github.com/ChaiBapchya/incubator-mxnet/pull/52]
> > > > >
> > > > > Barring the opt-in vs opt-out debate & idempotency, consensus was
> > > quickly
> > > > > reached for the rest.
> > > > >
> > > > > In the coming days, I hope to roll-out this feature into Prod
> (public
> > > > > MXNet) for all devs to use.
> > > > >
> > > > > Thanks,
> > > > > Chai
> > > > >
> > > > > On Mon, 16 Mar 2020 at 11:57, Marco de Abreu <
> > marco.g.abreu@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Well that's generally a problem with a deferred CI approach (CI
> is
> > > run
> > > > at
> > > > > > commit and not at merge time). This can either be solved through
> > the
> > > > > other
> > > > > > proposal that's currently on dev@, by having a bot which does
> > merges
> > > > by
> > > > > > having a global lock and a merge queue or by accepting the issue.
> > > > Reality
> > > > > > right now is that we're running that model where two PRs which
> are
> > > > merged
> > > > > > in parallel might break one another. One thing to consider though
> > is
> > > > that
> > > > > > this breakage would have to be introduced in two separate parts
> > since
> > > > > > otherwise there'd be merge conflicts. I think we had that
> situation
> > > > twice
> > > > > > so far and the result was a quick revert, so I'd say that it's a
> > > > problem
> > > > > > that can happily be accepted. All other solutions basically
> require
> > > > some
> > > > > > form of single-threaded and globally locked solution which limits
> > us
> > > in
> > > > > > scalability. I'd recommend to just accept that risk and revert a
> PR
> > > in
> > > > > case
> > > > > > it actually had a conflict.
> > > > > >
> > > > > > -Marco
> > > > > >
> > > > > > On Mon, Mar 16, 2020 at 6:29 PM Skalicky, Sam
> > > > <sskalic@amazon.com.invalid
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > We probably need some way to track which CI runs ran for which
> > > commit
> > > > > > too,
> > > > > > > that way we can ensure that all CI runs ran on the commit that
> > will
> > > > be
> > > > > > > merged. Maybe the bot can comment with the commit hash when
> > users
> > > > > > command
> > > > > > > it to do something. Although since users can trigger individual
> > CI
> > > > runs
> > > > > > its
> > > > > > > possible to have some commits run some CI runs but not others.
> We
> > > > need
> > > > > > some
> > > > > > > way to easily verify that the CI has executed all runs on the
> > > commit
> > > > > that
> > > > > > > will be merged.
> > > > > > >
> > > > > > > Sam
> > > > > > >
> > > > > > > > On Mar 13, 2020, at 8:28 PM, Przemysław Trędak <
> > > ptrendx@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > CAUTION: This email originated from outside of the
> > organization.
> > > Do
> > > > > not
> > > > > > > click links or open attachments unless you can confirm the
> sender
> > > and
> > > > > > know
> > > > > > > the content is safe.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > I personally like the idea of opt-in more than opt-out:
> > > > > > > > - ultimately PR author wants the PR to be merged so they (or
> > > > > committer
> > > > > > > reviewing the PR) will trigger the CI
> > > > > > > > - if it is easy to trigger the PR via the bot command then
> the
> > > > amount
> > > > > > of
> > > > > > > work per PR should be less than with opt-out (since most of the
> > > > commits
> > > > > > > should then be marked as [skip ci] or something similar
> > > > > > > >
> > > > > > > > +1 to the bot making a comment on each new PR with its
> commands
> > > > (and
> > > > > > > also explaining, or at least giving links to the general PR
> > process
> > > > so
> > > > > > new
> > > > > > > PR authors are not lost). Maybe we could make the bot recognize
> > if
> > > > the
> > > > > PR
> > > > > > > author is new or existing contributor and offer advice based on
> > > that?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Przemek
> > > > > > > >
> > > > > > > > On 2020/03/13 22:06:58, Marco de Abreu <
> > marco.g.abreu@gmail.com>
> > > > > > wrote:
> > > > > > > >> Hi,
> > > > > > > >>
> > > > > > > >> since it's no longer necessary to push a new commit to
> trigger
> > > CI,
> > > > > it
> > > > > > > will
> > > > > > > >> already reduce the costs. But to me, requiring an action to
> > > enable
> > > > > CI
> > > > > > > after
> > > > > > > >> a PR has been created initially, is a no go. User can opt
> out
> > of
> > > > CI,
> > > > > > but
> > > > > > > >> the default has to be CI being triggered automatically for
> > every
> > > > > > commit
> > > > > > > >> unless specifically disabled by a participant. I'm also fine
> > > with
> > > > > > > >> triggering certain additional jobs (think about running a
> > > nightly
> > > > > job
> > > > > > > upon
> > > > > > > >> request for a PR) to require a manual step, but the PR
> > > validation
> > > > > > > pipelines
> > > > > > > >> have to run automatically. Every check that is marked as
> > > > "Required"
> > > > > in
> > > > > > > >> GitHub has to be automatically kicked off.
> > > > > > > >>
> > > > > > > >> -Marco
> > > > > > > >>
> > > > > > > >> On Fri, Mar 13, 2020 at 9:50 PM Chaitanya Bapat <
> > > > > chai.bapat@gmail.com
> > > > > > >
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >>> Firstly,
> > > > > > > >>> Sorry I missed out on attaching the mail thread that was
> sent
> > > on
> > > > > 12th
> > > > > > > >>> February for notifying the community of the upcoming
> changes
> > to
> > > > the
> > > > > > > MXNet
> > > > > > > >>> CI
> > > > > > > >>> For reference :
> > > > > > > >>>
> > > > > > > >>>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/r09a6ab2803a996fc80e00fe39ed312fa4865e8805e08df847f1addad%40%3Cdev.mxnet.apache.org%3E
> > > > > > > >>>
> > > > > > > >>> Now to the questions,
> > > > > > > >>>> Is it possible for re-triggering a single job to be
> abused?
> > > > > > > >>> @Tao In the case when a user re-triggers a single job
> > multiple
> > > > > times,
> > > > > > > that
> > > > > > > >>> will be visible in the PR conversation thread. A committer,
> > > even
> > > > > > after
> > > > > > > he
> > > > > > > >>> has approved the PR before, generally takes a look at the
> > final
> > > > > state
> > > > > > > of
> > > > > > > >>> the PR before merging. Would it be fair to assume the
> > committer
> > > > > could
> > > > > > > take
> > > > > > > >>> the multiple re-trigger of a single job into account before
> > > > > merging?
> > > > > > > The
> > > > > > > >>> committer then has the option to invoke `@mxnet-bot run ci
> > > [all]
> > > > `
> > > > > to
> > > > > > > >>> trigger the entire build pipeline one last to counter the
> > > abuse.
> > > > > This
> > > > > > > is
> > > > > > > >>> aligned with what @Leonard said.
> > > > > > > >>>
> > > > > > > >>> @Sandeep Thanks a lot for collecting and sharing valuable
> > data.
> > > > I'd
> > > > > > > concur
> > > > > > > >>> with the opinion that given the existing things committers
> &
> > PR
> > > > > > Authors
> > > > > > > >>> take care of, invoking CI shouldn't be that big of an
> > > additional
> > > > > > > burden.
> > > > > > > >>>
> > > > > > > >>> @Marco With the opt-out, the onus remains on the PR Author.
> > It
> > > > > > doesn't
> > > > > > > help
> > > > > > > >>> reduce the resource usage. Hence, it was suggested to
> switch
> > to
> > > > > > > >>> opt-in. @Leo's suggestion for proactive commenting on the
> > part
> > > of
> > > > > bot
> > > > > > > makes
> > > > > > > >>> sense and is doable.
> > > > > > > >>>
> > > > > > > >>> Default : opt-out and User initiated opt-in (with
> addressing
> > > > Leo's
> > > > > > fix
> > > > > > > for
> > > > > > > >>> the usability issue you correctly pointed out )
> > > > > > > >>> @Marco How does this sound to you?
> > > > > > > >>>
> > > > > > > >>> Again, thank you all for chiming in and voicing your
> opinion.
> > > > > > > Appreciate
> > > > > > > >>> it.
> > > > > > > >>> We can take ahead these discussions in today's demo
> meeting.
> > > > > [Design
> > > > > > > Doc
> > > > > > > >>> <
> > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > > > >]
> > > > > > > [Demo
> > > > > > > >>> Video <https://www.youtube.com/watch?v=gfOGwZId8aU>]
> > > > > > > >>>
> > > > > > > >>> Thanks,
> > > > > > > >>> Chai
> > > > > > > >>>
> > > > > > > >>> On Fri, 13 Mar 2020 at 12:34, Marco de Abreu <
> > > > > > marco.g.abreu@gmail.com>
> > > > > > > >>> wrote:
> > > > > > > >>>
> > > > > > > >>>> I'd recommend that the bot makes an initial comment when a
> > PR
> > > > gets
> > > > > > > opened
> > > > > > > >>>> and informs the users of its commands. It then tells the
> > user
> > > > the
> > > > > > > commend
> > > > > > > >>>> to opt out of CI.
> > > > > > > >>>>
> > > > > > > >>>> -Marco
> > > > > > > >>>>
> > > > > > > >>>> Lausen, Leonard <la...@amazon.com.invalid> schrieb am
> Fr.,
> > > 13.
> > > > > > März
> > > > > > > >>> 2020,
> > > > > > > >>>> 20:27:
> > > > > > > >>>>
> > > > > > > >>>>> On opt-out: People may be unaware of opt-out would not
> use
> > > it.
> > > > > > There
> > > > > > > is
> > > > > > > >>>> no
> > > > > > > >>>>> incentive to use opt-out, as the PR author doesn't pay
> any
> > > > money
> > > > > > for
> > > > > > > CI
> > > > > > > >>>>> run.
> > > > > > > >>>>>
> > > > > > > >>>>> I agree with Marco though that opt-in alone may cause
> > > usability
> > > > > > > issues,
> > > > > > > >>>> as
> > > > > > > >>>>> contributors may not be aware of how to trigger the CI.
> > > > > > > >>>>> One solution is that the bot proactively comments on the
> PR
> > > and
> > > > > > > reminds
> > > > > > > >>>> the
> > > > > > > >>>>> author to trigger running CI once the author deems the PR
> > > > ready.
> > > > > > > >>>>>
> > > > > > > >>>>> But even if we choose opt-out, the bot will still add a
> lot
> > > of
> > > > > > value,
> > > > > > > >>> as
> > > > > > > >>>> PR
> > > > > > > >>>>> authors can retrigger single jobs that have failed due to
> > > > > > flakiness.
> > > > > > > >>>>>
> > > > > > > >>>>>> Is it possible for re-triggering a single job to be
> > abused?
> > > > For
> > > > > > > >>>> example,
> > > > > > > >>>>>> the author spends two days re-triggering a flaky job to
> > make
> > > > it
> > > > > > > pass.
> > > > > > > >>>>>
> > > > > > > >>>>> Yes, this is possible. I suggest the committer who likes
> to
> > > > > merge a
> > > > > > > PR
> > > > > > > >>>>> needs to
> > > > > > > >>>>> make a good judgement here if a PR is abusing the
> feature,
> > > and
> > > > if
> > > > > > so,
> > > > > > > >>>>> retrigger
> > > > > > > >>>>> all CI runs.
> > > > > > > >>>>>
> > > > > > > >>>>> Best regards
> > > > > > > >>>>> Leonard
> > > > > > > >>>>>
> > > > > > > >>>>> On Fri, 2020-03-13 at 08:07 +0100, Marco de Abreu wrote:
> > > > > > > >>>>>> Thanks for the data Sandeep. In these cases it sounds
> like
> > > it
> > > > > > would
> > > > > > > >>>> have
> > > > > > > >>>>>> rather been better when people explicitly turned off CI
> in
> > > > that
> > > > > > > case.
> > > > > > > >>>>>> What's the argument against an opt-out instead of an
> > opt-in?
> > > > > > > >>>>>>
> > > > > > > >>>>>> My intention is that I consider it quite cumbersome to
> > make
> > > > it a
> > > > > > > >>>>> *required*
> > > > > > > >>>>>> step to always trigger CI manually, even if just
> > submitting
> > > a
> > > > > > small
> > > > > > > >>> PR.
> > > > > > > >>>>> I'd
> > > > > > > >>>>>> rather see people explicitly turning off CI if they
> > wouldn't
> > > > > like
> > > > > > to
> > > > > > > >>>> use
> > > > > > > >>>>> it
> > > > > > > >>>>>> - and there's also the "draft" stage for a PR which some
> > > > > > > contributors
> > > > > > > >>>> are
> > > > > > > >>>>>> using.
> > > > > > > >>>>>>
> > > > > > > >>>>>> With regards to WIP and do not review: I think these are
> > use
> > > > > cases
> > > > > > > >>>> where
> > > > > > > >>>>>> you want CI feedback, as otherwise you wouldn't have
> > opened
> > > > the
> > > > > > PR.
> > > > > > > >>> If
> > > > > > > >>>>> you
> > > > > > > >>>>>> don't want human feedback and neither machine feedback,
> > why
> > > > open
> > > > > > the
> > > > > > > >>> PR
> > > > > > > >>>>> at
> > > > > > > >>>>>> all?
> > > > > > > >>>>>>
> > > > > > > >>>>>> -Marco
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>> sandeep krishnamurthy <sa...@gmail.com>
> > schrieb
> > > > am
> > > > > > Fr.,
> > > > > > > >>>> 13.
> > > > > > > >>>>>> März 2020, 05:24:
> > > > > > > >>>>>>
> > > > > > > >>>>>>> I tried to gather some data for us to discuss this
> topic
> > in
> > > > > this
> > > > > > > >>>>> thread. I
> > > > > > > >>>>>>> tried to count number of un-necessary builds by looking
> > at
> > > > most
> > > > > > > >>>> recent
> > > > > > > >>>>> (as
> > > > > > > >>>>>>> of 12, March 9 PM PST) 50 PRs merged to master and 50
> > PRs.
> > > > > > > >>>>>>> Identifying un-necessary builds is bit subjective. I
> > tried
> > > to
> > > > > be
> > > > > > > >>> more
> > > > > > > >>>>>>> conservative where I didn't count a build as
> un-necessary
> > > if
> > > > I
> > > > > > was
> > > > > > > >>> in
> > > > > > > >>>>>>> doubt. Hence, I was not able to automate, but I made an
> > > > effort
> > > > > to
> > > > > > > >>> go
> > > > > > > >>>>>>> through PRs manually and use below criteria to identify
> > > > > > > >>> un-necessary
> > > > > > > >>>>>>> commits triggering the builds.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> 1. Explicitly marked as WIP / do not review PR
> > > > > > > >>>>>>> 2. Incremental WIP commit and finally commenting a
> > commit
> > > > > > > >>> “trigger
> > > > > > > >>>>> CI”
> > > > > > > >>>>>>> 3. Multiple commits to address all comments from
> single
> > > > > review.
> > > > > > > >>>>> This is
> > > > > > > >>>>>>> assuming we see a comment, address them, commit, next
> > the
> > > > > > > >>>> following
> > > > > > > >>>>>>> comment
> > > > > > > >>>>>>> 4. Sequence of documentation only changes
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> I found there were around 42 avoidable builds from most
> > > > recent
> > > > > 50
> > > > > > > >>>>> merged
> > > > > > > >>>>>>> PRs and around 86 builds from recent 50 open PRs.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> I synced up with other contributors (Joe Evans, Chai)
> > from
> > > > > Amazon
> > > > > > > >>> who
> > > > > > > >>>>> is
> > > > > > > >>>>>>> contributing to MXNet CI system. I was told that on an
> > > > average
> > > > > it
> > > > > > > >>>> costs
> > > > > > > >>>>>>> around $84 per build and on an average 6 commits per
> > merged
> > > > PR
> > > > > > (for
> > > > > > > >>>>> year
> > > > > > > >>>>>>> 2019). Going by that, it is approximately 1/6 builds
> are
> > > > > > avoidable.
> > > > > > > >>>>> [100 /
> > > > > > > >>>>>>> 300 + 300 ]
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Usability should be top most priority. But, since
> either
> > a
> > > > > > reviewer
> > > > > > > >>>> or
> > > > > > > >>>>> pr
> > > > > > > >>>>>>> author can trigger the bot, is it really a hurdle for
> pr
> > > > author
> > > > > > or
> > > > > > > >>>>> reviewer
> > > > > > > >>>>>>> to call a bot to trigger CI? Given that PR author and
> > > > reviewer
> > > > > is
> > > > > > > >>>>> already
> > > > > > > >>>>>>> actively commenting various details such as - PR
> > > description,
> > > > > > > >>> review
> > > > > > > >>>>>>> comments and responses, adding labels etc.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Me too curious to know the behavior for Tao's above use
> > > case.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Best,
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Sandeep
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <
> > mutouorz@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> Is it possible for re-triggering a single job to be
> > > abused?
> > > > > For
> > > > > > > >>>>> example,
> > > > > > > >>>>>>>> the author spends two days re-triggering a flaky job
> to
> > > make
> > > > > it
> > > > > > > >>>>> pass. But
> > > > > > > >>>>>>>> other jobs which have passed the validation may be
> > broken
> > > by
> > > > > > > >>> other
> > > > > > > >>>>>>> commits
> > > > > > > >>>>>>>> during the two day without being noticed. And finally
> > the
> > > PR
> > > > > is
> > > > > > > >>>>> merged
> > > > > > > >>>>>>> with
> > > > > > > >>>>>>>> underlying problems.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <
> > > > > > > >>>>> marco.g.abreu@gmail.com>
> > > > > > > >>>>>>>> wrote:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>>> In the end it only comes down to money, considering
> > that
> > > > the
> > > > > > > >>>>> system is
> > > > > > > >>>>>>>> auto
> > > > > > > >>>>>>>>> scaling, making the execution time constant.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> If we're trading money for usability, I certainly
> would
> > > > > prefer
> > > > > > > >>>>>>> usability.
> > > > > > > >>>>>>>>> I'd rather recommend to spend time on parallelizing
> > test
> > > > > > > >>>> execution
> > > > > > > >>>>> or
> > > > > > > >>>>>>>>> getting rid of integration tests in the PR stage
> > instead
> > > > > > > >>> reducing
> > > > > > > >>>>> the
> > > > > > > >>>>>>>> costs
> > > > > > > >>>>>>>>> by making people not use it. But taking a step back
> to
> > > > > > > >>> requiring
> > > > > > > >>>>> people
> > > > > > > >>>>>>>> to
> > > > > > > >>>>>>>>> manually trigger CI again doesn't feel right.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> I'm happy to see that bot deployed, but I do not
> agree
> > > with
> > > > > > > >>>>> removing
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>> auto trigger functionality for new commits.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> -Marco
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Chaitanya Bapat <ch...@gmail.com> schrieb am
> Do.,
> > > 12.
> > > > > > > >>> März
> > > > > > > >>>>> 2020,
> > > > > > > >>>>>>>>> 22:47:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>> @Marco Thanks for pointing that out.
> > > > > > > >>>>>>>>>> Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM -
> 3:30
> > > PM
> > > > in
> > > > > > > >>>>>>>> (UTC-08:00)
> > > > > > > >>>>>>>>>> Pacific Time (US & Canada).
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>> When do we expect this bot to be deployed?
> > > > > > > >>>>>>>>>> @Lin If all goes well in the next week I can deploy
> it
> > > to
> > > > > > > >>>> public
> > > > > > > >>>>>>> Apache
> > > > > > > >>>>>>>>>> (provided I get permissions from Apache Infra)
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> @Marco Thanks for your feedback.
> > > > > > > >>>>>>>>>>> CI system has to support the community without
> > > requiring
> > > > > > > >>>>> people to
> > > > > > > >>>>>>>>>> constantly shepherd every single run
> > > > > > > >>>>>>>>>> We have data for the number of times CI was
> triggered
> > > > > > > >>>>> unnecessarily
> > > > > > > >>>>>>>> which
> > > > > > > >>>>>>>>>> includes
> > > > > > > >>>>>>>>>> - Entire build triggered instead of specific build
> > > > > > > >>>>>>>>>> - CI triggered when PR is still work in progress or
> > not
> > > > yet
> > > > > > > >>>> ready
> > > > > > > >>>>>>> (say
> > > > > > > >>>>>>>> -
> > > > > > > >>>>>>>>>> intermediate commits)
> > > > > > > >>>>>>>>>> At the end its a trade-off
> > > > > > > >>>>>>>>>> Money, Resources, Time to build for each and every
> > > commit
> > > > vs
> > > > > > > >>>>> Pain of
> > > > > > > >>>>>>>>>> triggering builds
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>> Scan trigger plugin would poll SCM. Can we use
> plugin
> > > at
> > > > > > > >>>>> scale?
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> 1. I haven't tested it on scale. But I think with
> the
> > > > > current
> > > > > > > >>>>> scale
> > > > > > > >>>>>>> of
> > > > > > > >>>>>>>>>> MXNet repo (191 open PRs i.e. checking for changes
> to
> > > 191
> > > > > > > >>>>> branches -
> > > > > > > >>>>>>> It
> > > > > > > >>>>>>>>>> should be manageable)
> > > > > > > >>>>>>>>>> 2. What's the purpose of the plugin? tldr; Branch
> > > > discovery
> > > > > > > >>> or
> > > > > > > >>>>> branch
> > > > > > > >>>>>>>>>> indexing.
> > > > > > > >>>>>>>>>> Scan trigger plugin comes into the picture only once
> > per
> > > > PR
> > > > > > > >>> per
> > > > > > > >>>>> job
> > > > > > > >>>>>>>>> (i.e. 8
> > > > > > > >>>>>>>>>> times per PR for 8 jobs). It is basically done when
> a
> > > new
> > > > PR
> > > > > > > >>> is
> > > > > > > >>>>> made
> > > > > > > >>>>>>>> and
> > > > > > > >>>>>>>>>> the job (say unix-cpu hasn't discovered the new PR
> > > branch
> > > > > > > >>> yet).
> > > > > > > >>>>>>> That's
> > > > > > > >>>>>>>>> it.
> > > > > > > >>>>>>>>>> So it shouldn't be a problem for public MXNet repo.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>> Chai
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <
> > > > > > > >>>>>>> marco.g.abreu@gmail.com>
> > > > > > > >>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>> Btw you forgot to set a date and time for the
> metting
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu <
> > > > > > > >>>>>>>>> marco.g.abreu@gmail.com
> > > > > > > >>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Thanks Chai, I generally like the idea of the bot.
> > But
> > > > > > > >>> I'm
> > > > > > > >>>>> not a
> > > > > > > >>>>>>>>>>> supporter
> > > > > > > >>>>>>>>>>>> of the idea to disable any automatic triggering
> > > > > > > >>> (disabling
> > > > > > > >>>>> the
> > > > > > > >>>>>>>>> webhook
> > > > > > > >>>>>>>>>> is
> > > > > > > >>>>>>>>>>>> also not an option, considering that this will
> > disable
> > > > > > > >>>> master
> > > > > > > >>>>>>>>>> triggers).
> > > > > > > >>>>>>>>>>>> The CI system has to support the community without
> > > > > > > >>>> requiring
> > > > > > > >>>>>>> people
> > > > > > > >>>>>>>>> to
> > > > > > > >>>>>>>>>>>> constantly shepherd every single run. Disabling
> > > > automatic
> > > > > > > >>>>>>>> triggering
> > > > > > > >>>>>>>>>>> seems
> > > > > > > >>>>>>>>>>>> like a step back to me.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Instead, I'd recommend that CI gets triggered upon
> > > every
> > > > > > > >>>>> commit
> > > > > > > >>>>>>> as
> > > > > > > >>>>>>>>>> usual,
> > > > > > > >>>>>>>>>>>> but people have the possibility to call a
> "command"
> > > > (i.e.
> > > > > > > >>>>> make a
> > > > > > > >>>>>>>>>> message
> > > > > > > >>>>>>>>>>>> which results in the bot setting a label) to
> disable
> > > CI
> > > > > > > >>>> until
> > > > > > > >>>>>>> they
> > > > > > > >>>>>>>>>> revoke
> > > > > > > >>>>>>>>>>>> it. But the standard should still be that a new
> > commit
> > > > > > > >>>>> triggers a
> > > > > > > >>>>>>>> new
> > > > > > > >>>>>>>>>> CI
> > > > > > > >>>>>>>>>>>> run.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>
> > https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
> > > > > > > >>>>>>>> seems
> > > > > > > >>>>>>>>>> like
> > > > > > > >>>>>>>>>>>> this would poll SCM. This will incur high quota
> > > > > > > >>>>> restrictions. Are
> > > > > > > >>>>>>>> you
> > > > > > > >>>>>>>>>>> sure
> > > > > > > >>>>>>>>>>>> that you can use that plugin at scale?
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> -Marco
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <
> > > > > > > >>>>> apeforest@gmail.com>
> > > > > > > >>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>> Chai,
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Awesome work. When do we expect this bot to be
> > > > > > > >>> deployed?
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Best,
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Lin
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <
> > > > > > > >>>>>>>>> chai.bapat@gmail.com
> > > > > > > >>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Hello MXNet community,
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> I have built an MXNet Bot <
> > > > > > > >>>> https://github.com/mxnet-bot>
> > > > > > > >>>>> that
> > > > > > > >>>>>>>>>> allows
> > > > > > > >>>>>>>>>>> PR
> > > > > > > >>>>>>>>>>>>>> Authors, Committers and Jenkins Admins to
> trigger
> > CI
> > > > > > > >>>>> manually.
> > > > > > > >>>>>>>>>>>>>> It handles 2 problems
> > > > > > > >>>>>>>>>>>>>> 1. Manual CI trigger instead of existing
> automated
> > > CI
> > > > > > > >>>>> trigger
> > > > > > > >>>>>>>>>>>>>> 2. Gives permissions to PR Authors (in addition
> to
> > > > > > > >>>> MXNet
> > > > > > > >>>>>>>>> Committers
> > > > > > > >>>>>>>>>>> and
> > > > > > > >>>>>>>>>>>>>> Jenkins Admins)
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Design Doc :
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>
> > > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > > > > > > >>>>>>>>>>>>>> I urge you all to attend the demonstration
> meeting
> > > > > > > >>> and
> > > > > > > >>>>> lend
> > > > > > > >>>>>>> your
> > > > > > > >>>>>>>>>> views
> > > > > > > >>>>>>>>>>>>> on
> > > > > > > >>>>>>>>>>>>>> the same.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Thank you,
> > > > > > > >>>>>>>>>>>>>> Chai
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> *Meeting Details*:
> > > > > > > >>>>>>>>>>>>>> ==============Conference Bridge
> > > > > > > >>>> Information==============
> > > > > > > >>>>>>>>>>>>>> You have been invited to an online meeting,
> > powered
> > > > > > > >>> by
> > > > > > > >>>>> Amazon
> > > > > > > >>>>>>>>> Chime.
> > > > > > > >>>>>>>>>>>>>> *Chime meeting ID*: *9272158344*
> > > > > > > >>>>>>>>>>>>>> Join via Chime clients (manually): Select
> > 'Meetings
> > > >
> > > > > > > >>>>> Join a
> > > > > > > >>>>>>>>>> Meeting',
> > > > > > > >>>>>>>>>>>>> and
> > > > > > > >>>>>>>>>>>>>> enter 9272158344
> > > > > > > >>>>>>>>>>>>>> Join via Chime clients (auto-call): If you
> invite
> > > > > > > >>>>> auto-call as
> > > > > > > >>>>>>>>>>> attendee,
> > > > > > > >>>>>>>>>>>>>> Chime will call you when the meeting starts,
> > select
> > > > > > > >>>>> 'Answer'
> > > > > > > >>>>>>>>>>>>>> *Join via browser screen share*:
> > > > > > > >>>>> https://chime.aws/9272158344
> > > > > > > >>>>>>>>>>>>>> *Join via phone* (US):
> > +1-929-432-4463,,,9272158344#
> > > > > > > >>>>>>>>>>>>>> *Join via phone (US toll-free)*:
> > > > > > > >>>>> +1-855-552-4463,,,9272158344#
> > > > > > > >>>>>>>>>>>>>> International dial-in:
> > > > > > > >>>> https://chime.aws/dialinnumbers/
> > > > > > > >>>>>>>>>>>>>> In-room video system: Ext: 62000, Meeting PIN:
> > > > > > > >>>>> 9272158344#
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> --
> > > > > > > >>>>>>>>>>>>>> *Chaitanya Prakash Bapat*
> > > > > > > >>>>>>>>>>>>>> *+1 (973) 953-6299*
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> [image:
> https://www.linkedin.com//in/chaibapat25]
> > > > > > > >>>>>>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> > > > > > > >>>>>>>>>>>>> https://www.facebook.com/chaibapat
> > > > > > > >>>>>>>>>>>>>> ]
> > > > > > > >>>>>>>>>>>>>> <https://www.facebook.com/chaibapchya>[image:
> > > > > > > >>>>>>>>>>>>>> https://twitter.com/ChaiBapchya] <
> > > > > > > >>>>>>>> https://twitter.com/ChaiBapchya
> > > > > > > >>>>>>>>>>>>>> [image:
> > > > > > > >>>>>>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> > > > > > > >>>>>>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> --
> > > > > > > >>>>>>>>>> *Chaitanya Prakash Bapat*
> > > > > > > >>>>>>>>>> *+1 (973) 953-6299*
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> [image: https://www.linkedin.com//in/chaibapat25]
> > > > > > > >>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> > > > > > > >>>>>>>>> https://www.facebook.com/chaibapat
> > > > > > > >>>>>>>>>> ]
> > > > > > > >>>>>>>>>> <https://www.facebook.com/chaibapchya>[image:
> > > > > > > >>>>>>>>>> https://twitter.com/ChaiBapchya] <
> > > > > > > >>>>> https://twitter.com/ChaiBapchya
> > > > > > > >>>>>>>>>> [image:
> > > > > > > >>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> > > > > > > >>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> --
> > > > > > > >>>>>>> Sandeep Krishnamurthy
> > > > > > > >>>>>>>
> > > > > > > >>>>>
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> --
> > > > > > > >>> *Chaitanya Prakash Bapat*
> > > > > > > >>> *+1 (973) 953-6299*
> > > > > > > >>>
> > > > > > > >>> [image: https://www.linkedin.com//in/chaibapat25]
> > > > > > > >>> <https://github.com/ChaiBapchya>[image:
> > > > > > > https://www.facebook.com/chaibapat
> > > > > > > >>> ]
> > > > > > > >>> <https://www.facebook.com/chaibapchya>[image:
> > > > > > > >>> https://twitter.com/ChaiBapchya] <
> > > > https://twitter.com/ChaiBapchya
> > > > > > > >[image:
> > > > > > > >>> https://www.linkedin.com//in/chaibapat25]
> > > > > > > >>> <https://www.linkedin.com//in/chaibapchya/>
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Chaitanya Prakash Bapat*
> > > > > *+1 (973) 953-6299*
> > > > >
> > > > > [image: https://www.linkedin.com//in/chaibapat25]
> > > > > <https://github.com/ChaiBapchya>[image:
> > > > https://www.facebook.com/chaibapat
> > > > > ]
> > > > > <https://www.facebook.com/chaibapchya>[image:
> > > > > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> > > > >[image:
> > > > > https://www.linkedin.com//in/chaibapat25]
> > > > > <https://www.linkedin.com//in/chaibapchya/>
> > > > >
> > > >
> > >
> > >
> > > --
> > > *Chaitanya Prakash Bapat*
> > > *+1 (973) 953-6299*
> > >
> > > [image: https://www.linkedin.com//in/chaibapat25]
> > > <https://github.com/ChaiBapchya>[image:
> > https://www.facebook.com/chaibapat
> > > ]
> > > <https://www.facebook.com/chaibapchya>[image:
> > > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> > >[image:
> > > https://www.linkedin.com//in/chaibapat25]
> > > <https://www.linkedin.com//in/chaibapchya/>
> > >
> >
>
>
> --
> *Chaitanya Prakash Bapat*
> *+1 (973) 953-6299*
>
> [image: https://www.linkedin.com//in/chaibapat25]
> <https://github.com/ChaiBapchya>[image: https://www.facebook.com/chaibapat
> ]
> <https://www.facebook.com/chaibapchya>[image:
> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
> https://www.linkedin.com//in/chaibapat25]
> <https://www.linkedin.com//in/chaibapchya/>
>
Re: MXNet Bot Demo
Posted by Chaitanya Bapat <ch...@gmail.com>.
@Marco Alright, it makes total sense to test out the Bot feature alongside
auto-trigger as a transition.
Path Forward:
1. Setup MXNet Bot on apache/incubator-mxnet repo (GitHub WebHook and Infra)
2. We don't turn off automatic trigger of PR builds for now.
3. Hopefully, bot is used by developers to trigger specific jobs
4. Later on (say around April 20), let's discuss the possibility of
switching off auto-trigger (with appropriate data) if it makes sense.
Thanks Marco for volunteering to help enable the web hook on
apache/incubator-mxnet. Let me know if we can sync up on Slack channel to
get the ball rolling.
Thanks once again for the entire community to step in and help try out this
Bot.
Chai
On Wed, 18 Mar 2020 at 17:07, Marco de Abreu <ma...@gmail.com>
wrote:
> Hi, that's correct. But as stated previously, it's not an option to remove
> the hook. For now, I'd like to see how the system behaves while it's
> optional. Later on, we can talk about revisiting this decision. But to me
> it's not an option to deploy an entirely new system and approach without
> having a transition or even a timeframe in which we are able to fall back.
>
> I'm happy to support the deployment of the bot and add an additional
> webhook to enable it's functionality to support selective triggering by PR
> authors and committers, but I will not support the disabling of automatic
> triggering of branches or PRs.
>
> -Marco
>
> Chaitanya Bapat <ch...@gmail.com> schrieb am Mi., 18. März 2020,
> 21:00:
>
> > Hey Marco,
> >
> > I thought currently every commit on PR and master triggers CI
> > because
> > a. github webhook points to Jenkins Server
> > b. GH Webhook events trigger builds on Jenkins for all commits to any
> > branch in apache/incubator-mxnet
> > may it be master/PR/non-PR
> > Reason:
> > Because all the 3 types of branches are discovered by Jenkins (non-PR
> > (including master) and PR)
> >
> > Proposal: Remove GitHub WebHook to Jenkins and replace with GH Webhook to
> > Lambda
> > But after I remove the github webhook that points to Jenkins : *N**o
> commit
> > will trigger Jenkins build by default* (as Jenkins wont receive GH
> events)
> > Only those that Bot deems fit will be triggered (using Jenkins API
> invoked
> > by Lambda).
> > Hence its needed to handle that case of master merge.
> > Am I understanding this correctly?
> >
> > On Wed, 18 Mar 2020 at 04:23, Marco de Abreu <ma...@gmail.com>
> > wrote:
> >
> > > Thanks Chai, sounds good to me. Could you elaborate a bit on the point
> > > about triggering a CI run after the PR has been merged? We already to
> > that
> > > automatically for the master, so what's the benefit to do it twice?
> > >
> > > -Marco
> > >
> > > Chaitanya Bapat <ch...@gmail.com> schrieb am Mi., 18. März 2020,
> > > 09:30:
> > >
> > > > Update:
> > > >
> > > > > we can ensure that all CI runs ran on the commit that will be
> merged
> > > > @Sam Skalicky <sa...@gmail.com> Branch Protection is added to
> > > public
> > > > MXNet repo. It ensures that for every PR to be merged, the CI passes.
> > All
> > > > the jobs selected "required" jobs will have to be green for the PR to
> > be
> > > > merged. Ofcourse, users with "Adminstrator" access can merge without
> it
> > > but
> > > > that's just a backdoor. It is the case now and will continue to be
> the
> > > case
> > > > with the inclusion of Bot.
> > > >
> > > > > easily verify that the CI has executed all runs on the commit that
> > will
> > > > be merged
> > > > GitHub UI shows all the jobs and the status corresponding to it on
> > every
> > > > commit. That should suffice. For the merged commits, Repo -> Commits
> ->
> > > > Commit ID (Status) can be tracked currently (only way that I know
> of).
> > > > Moreover, it is beyond the scope of this project (and possibly out of
> > our
> > > > control since this is purely GitHub UI specific use-case).
> > > >
> > > > Thanks @przemyslaw for supporting the opt-in.
> > > >
> > > > Thanks everyone in the community for sharing concerns, voicing your
> > > opinion
> > > > and participating in the discussion.
> > > > Thanks to those who attended the demo last Friday.
> > > >
> > > > Action items from that discussion
> > > > 1. Handle master merge builds [Done]
> > > > Bot runs entire CI suite after the PR is merged and comments on the
> PR
> > > > about the same.
> > > > Design decision :
> > > > MXNet Bot comment about master merge build on the *merge commit vs
> PR*.
> > > > After the PR is merged, Bot runs entire CI and comments the result of
> > CI
> > > > trigger on the PR (because it is easy to track on a PR rather than
> > > > commenting inside the merge commit)
> > > >
> > > > 2. Idempotent condition
> > > > In case of already running build, if an attempt is made to retrigger
> > the
> > > > job then what should be the response
> > > > a. Not to re-trigger, let the ongoing build continue till completion
> > > > b. End the ongoing build and re-trigger
> > > > c. Let the ongoing build continue, re-trigger new build
> > > >
> > > > From resource saving point of view, *c* looks costly and a can be
> > > > avoided/optimized by B.
> > > > In case when a re-trigger was started "erroneously" then killing
> > ongoing
> > > > build and re-trigger is a waste.
> > > > In case when ongoing build failed in one sub-part, then re-triggering
> > is
> > > > justified.
> > > > Erroneous re-triggers would be less often while conscious re-triggers
> > to
> > > > suppress failure is more common use-case. It looks like a safe
> > assumption
> > > > to make given the trade-off.
> > > > [Open to debate]
> > > >
> > > > 3. Add security consideration [Use of secret manager, but without
> > > > auto-rotation due to Jenkins manual config requirement] [Done]
> > > > 4. New PR Instruction message by the Bot [Done]
> > > > Thanks to the suggestion of Leonard, supported by others. I've now
> > added
> > > > the feature where the Bot comments a help message. [For reference -
> > > > https://github.com/ChaiBapchya/incubator-mxnet/pull/52]
> > > >
> > > > Barring the opt-in vs opt-out debate & idempotency, consensus was
> > quickly
> > > > reached for the rest.
> > > >
> > > > In the coming days, I hope to roll-out this feature into Prod (public
> > > > MXNet) for all devs to use.
> > > >
> > > > Thanks,
> > > > Chai
> > > >
> > > > On Mon, 16 Mar 2020 at 11:57, Marco de Abreu <
> marco.g.abreu@gmail.com>
> > > > wrote:
> > > >
> > > > > Well that's generally a problem with a deferred CI approach (CI is
> > run
> > > at
> > > > > commit and not at merge time). This can either be solved through
> the
> > > > other
> > > > > proposal that's currently on dev@, by having a bot which does
> merges
> > > by
> > > > > having a global lock and a merge queue or by accepting the issue.
> > > Reality
> > > > > right now is that we're running that model where two PRs which are
> > > merged
> > > > > in parallel might break one another. One thing to consider though
> is
> > > that
> > > > > this breakage would have to be introduced in two separate parts
> since
> > > > > otherwise there'd be merge conflicts. I think we had that situation
> > > twice
> > > > > so far and the result was a quick revert, so I'd say that it's a
> > > problem
> > > > > that can happily be accepted. All other solutions basically require
> > > some
> > > > > form of single-threaded and globally locked solution which limits
> us
> > in
> > > > > scalability. I'd recommend to just accept that risk and revert a PR
> > in
> > > > case
> > > > > it actually had a conflict.
> > > > >
> > > > > -Marco
> > > > >
> > > > > On Mon, Mar 16, 2020 at 6:29 PM Skalicky, Sam
> > > <sskalic@amazon.com.invalid
> > > > >
> > > > > wrote:
> > > > >
> > > > > > We probably need some way to track which CI runs ran for which
> > commit
> > > > > too,
> > > > > > that way we can ensure that all CI runs ran on the commit that
> will
> > > be
> > > > > > merged. Maybe the bot can comment with the commit hash when
> users
> > > > > command
> > > > > > it to do something. Although since users can trigger individual
> CI
> > > runs
> > > > > its
> > > > > > possible to have some commits run some CI runs but not others. We
> > > need
> > > > > some
> > > > > > way to easily verify that the CI has executed all runs on the
> > commit
> > > > that
> > > > > > will be merged.
> > > > > >
> > > > > > Sam
> > > > > >
> > > > > > > On Mar 13, 2020, at 8:28 PM, Przemysław Trędak <
> > ptrendx@apache.org
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > CAUTION: This email originated from outside of the
> organization.
> > Do
> > > > not
> > > > > > click links or open attachments unless you can confirm the sender
> > and
> > > > > know
> > > > > > the content is safe.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > I personally like the idea of opt-in more than opt-out:
> > > > > > > - ultimately PR author wants the PR to be merged so they (or
> > > > committer
> > > > > > reviewing the PR) will trigger the CI
> > > > > > > - if it is easy to trigger the PR via the bot command then the
> > > amount
> > > > > of
> > > > > > work per PR should be less than with opt-out (since most of the
> > > commits
> > > > > > should then be marked as [skip ci] or something similar
> > > > > > >
> > > > > > > +1 to the bot making a comment on each new PR with its commands
> > > (and
> > > > > > also explaining, or at least giving links to the general PR
> process
> > > so
> > > > > new
> > > > > > PR authors are not lost). Maybe we could make the bot recognize
> if
> > > the
> > > > PR
> > > > > > author is new or existing contributor and offer advice based on
> > that?
> > > > > > >
> > > > > > > Thanks
> > > > > > > Przemek
> > > > > > >
> > > > > > > On 2020/03/13 22:06:58, Marco de Abreu <
> marco.g.abreu@gmail.com>
> > > > > wrote:
> > > > > > >> Hi,
> > > > > > >>
> > > > > > >> since it's no longer necessary to push a new commit to trigger
> > CI,
> > > > it
> > > > > > will
> > > > > > >> already reduce the costs. But to me, requiring an action to
> > enable
> > > > CI
> > > > > > after
> > > > > > >> a PR has been created initially, is a no go. User can opt out
> of
> > > CI,
> > > > > but
> > > > > > >> the default has to be CI being triggered automatically for
> every
> > > > > commit
> > > > > > >> unless specifically disabled by a participant. I'm also fine
> > with
> > > > > > >> triggering certain additional jobs (think about running a
> > nightly
> > > > job
> > > > > > upon
> > > > > > >> request for a PR) to require a manual step, but the PR
> > validation
> > > > > > pipelines
> > > > > > >> have to run automatically. Every check that is marked as
> > > "Required"
> > > > in
> > > > > > >> GitHub has to be automatically kicked off.
> > > > > > >>
> > > > > > >> -Marco
> > > > > > >>
> > > > > > >> On Fri, Mar 13, 2020 at 9:50 PM Chaitanya Bapat <
> > > > chai.bapat@gmail.com
> > > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> Firstly,
> > > > > > >>> Sorry I missed out on attaching the mail thread that was sent
> > on
> > > > 12th
> > > > > > >>> February for notifying the community of the upcoming changes
> to
> > > the
> > > > > > MXNet
> > > > > > >>> CI
> > > > > > >>> For reference :
> > > > > > >>>
> > > > > > >>>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/r09a6ab2803a996fc80e00fe39ed312fa4865e8805e08df847f1addad%40%3Cdev.mxnet.apache.org%3E
> > > > > > >>>
> > > > > > >>> Now to the questions,
> > > > > > >>>> Is it possible for re-triggering a single job to be abused?
> > > > > > >>> @Tao In the case when a user re-triggers a single job
> multiple
> > > > times,
> > > > > > that
> > > > > > >>> will be visible in the PR conversation thread. A committer,
> > even
> > > > > after
> > > > > > he
> > > > > > >>> has approved the PR before, generally takes a look at the
> final
> > > > state
> > > > > > of
> > > > > > >>> the PR before merging. Would it be fair to assume the
> committer
> > > > could
> > > > > > take
> > > > > > >>> the multiple re-trigger of a single job into account before
> > > > merging?
> > > > > > The
> > > > > > >>> committer then has the option to invoke `@mxnet-bot run ci
> > [all]
> > > `
> > > > to
> > > > > > >>> trigger the entire build pipeline one last to counter the
> > abuse.
> > > > This
> > > > > > is
> > > > > > >>> aligned with what @Leonard said.
> > > > > > >>>
> > > > > > >>> @Sandeep Thanks a lot for collecting and sharing valuable
> data.
> > > I'd
> > > > > > concur
> > > > > > >>> with the opinion that given the existing things committers &
> PR
> > > > > Authors
> > > > > > >>> take care of, invoking CI shouldn't be that big of an
> > additional
> > > > > > burden.
> > > > > > >>>
> > > > > > >>> @Marco With the opt-out, the onus remains on the PR Author.
> It
> > > > > doesn't
> > > > > > help
> > > > > > >>> reduce the resource usage. Hence, it was suggested to switch
> to
> > > > > > >>> opt-in. @Leo's suggestion for proactive commenting on the
> part
> > of
> > > > bot
> > > > > > makes
> > > > > > >>> sense and is doable.
> > > > > > >>>
> > > > > > >>> Default : opt-out and User initiated opt-in (with addressing
> > > Leo's
> > > > > fix
> > > > > > for
> > > > > > >>> the usability issue you correctly pointed out )
> > > > > > >>> @Marco How does this sound to you?
> > > > > > >>>
> > > > > > >>> Again, thank you all for chiming in and voicing your opinion.
> > > > > > Appreciate
> > > > > > >>> it.
> > > > > > >>> We can take ahead these discussions in today's demo meeting.
> > > > [Design
> > > > > > Doc
> > > > > > >>> <
> > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > > >]
> > > > > > [Demo
> > > > > > >>> Video <https://www.youtube.com/watch?v=gfOGwZId8aU>]
> > > > > > >>>
> > > > > > >>> Thanks,
> > > > > > >>> Chai
> > > > > > >>>
> > > > > > >>> On Fri, 13 Mar 2020 at 12:34, Marco de Abreu <
> > > > > marco.g.abreu@gmail.com>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>>> I'd recommend that the bot makes an initial comment when a
> PR
> > > gets
> > > > > > opened
> > > > > > >>>> and informs the users of its commands. It then tells the
> user
> > > the
> > > > > > commend
> > > > > > >>>> to opt out of CI.
> > > > > > >>>>
> > > > > > >>>> -Marco
> > > > > > >>>>
> > > > > > >>>> Lausen, Leonard <la...@amazon.com.invalid> schrieb am Fr.,
> > 13.
> > > > > März
> > > > > > >>> 2020,
> > > > > > >>>> 20:27:
> > > > > > >>>>
> > > > > > >>>>> On opt-out: People may be unaware of opt-out would not use
> > it.
> > > > > There
> > > > > > is
> > > > > > >>>> no
> > > > > > >>>>> incentive to use opt-out, as the PR author doesn't pay any
> > > money
> > > > > for
> > > > > > CI
> > > > > > >>>>> run.
> > > > > > >>>>>
> > > > > > >>>>> I agree with Marco though that opt-in alone may cause
> > usability
> > > > > > issues,
> > > > > > >>>> as
> > > > > > >>>>> contributors may not be aware of how to trigger the CI.
> > > > > > >>>>> One solution is that the bot proactively comments on the PR
> > and
> > > > > > reminds
> > > > > > >>>> the
> > > > > > >>>>> author to trigger running CI once the author deems the PR
> > > ready.
> > > > > > >>>>>
> > > > > > >>>>> But even if we choose opt-out, the bot will still add a lot
> > of
> > > > > value,
> > > > > > >>> as
> > > > > > >>>> PR
> > > > > > >>>>> authors can retrigger single jobs that have failed due to
> > > > > flakiness.
> > > > > > >>>>>
> > > > > > >>>>>> Is it possible for re-triggering a single job to be
> abused?
> > > For
> > > > > > >>>> example,
> > > > > > >>>>>> the author spends two days re-triggering a flaky job to
> make
> > > it
> > > > > > pass.
> > > > > > >>>>>
> > > > > > >>>>> Yes, this is possible. I suggest the committer who likes to
> > > > merge a
> > > > > > PR
> > > > > > >>>>> needs to
> > > > > > >>>>> make a good judgement here if a PR is abusing the feature,
> > and
> > > if
> > > > > so,
> > > > > > >>>>> retrigger
> > > > > > >>>>> all CI runs.
> > > > > > >>>>>
> > > > > > >>>>> Best regards
> > > > > > >>>>> Leonard
> > > > > > >>>>>
> > > > > > >>>>> On Fri, 2020-03-13 at 08:07 +0100, Marco de Abreu wrote:
> > > > > > >>>>>> Thanks for the data Sandeep. In these cases it sounds like
> > it
> > > > > would
> > > > > > >>>> have
> > > > > > >>>>>> rather been better when people explicitly turned off CI in
> > > that
> > > > > > case.
> > > > > > >>>>>> What's the argument against an opt-out instead of an
> opt-in?
> > > > > > >>>>>>
> > > > > > >>>>>> My intention is that I consider it quite cumbersome to
> make
> > > it a
> > > > > > >>>>> *required*
> > > > > > >>>>>> step to always trigger CI manually, even if just
> submitting
> > a
> > > > > small
> > > > > > >>> PR.
> > > > > > >>>>> I'd
> > > > > > >>>>>> rather see people explicitly turning off CI if they
> wouldn't
> > > > like
> > > > > to
> > > > > > >>>> use
> > > > > > >>>>> it
> > > > > > >>>>>> - and there's also the "draft" stage for a PR which some
> > > > > > contributors
> > > > > > >>>> are
> > > > > > >>>>>> using.
> > > > > > >>>>>>
> > > > > > >>>>>> With regards to WIP and do not review: I think these are
> use
> > > > cases
> > > > > > >>>> where
> > > > > > >>>>>> you want CI feedback, as otherwise you wouldn't have
> opened
> > > the
> > > > > PR.
> > > > > > >>> If
> > > > > > >>>>> you
> > > > > > >>>>>> don't want human feedback and neither machine feedback,
> why
> > > open
> > > > > the
> > > > > > >>> PR
> > > > > > >>>>> at
> > > > > > >>>>>> all?
> > > > > > >>>>>>
> > > > > > >>>>>> -Marco
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> sandeep krishnamurthy <sa...@gmail.com>
> schrieb
> > > am
> > > > > Fr.,
> > > > > > >>>> 13.
> > > > > > >>>>>> März 2020, 05:24:
> > > > > > >>>>>>
> > > > > > >>>>>>> I tried to gather some data for us to discuss this topic
> in
> > > > this
> > > > > > >>>>> thread. I
> > > > > > >>>>>>> tried to count number of un-necessary builds by looking
> at
> > > most
> > > > > > >>>> recent
> > > > > > >>>>> (as
> > > > > > >>>>>>> of 12, March 9 PM PST) 50 PRs merged to master and 50
> PRs.
> > > > > > >>>>>>> Identifying un-necessary builds is bit subjective. I
> tried
> > to
> > > > be
> > > > > > >>> more
> > > > > > >>>>>>> conservative where I didn't count a build as un-necessary
> > if
> > > I
> > > > > was
> > > > > > >>> in
> > > > > > >>>>>>> doubt. Hence, I was not able to automate, but I made an
> > > effort
> > > > to
> > > > > > >>> go
> > > > > > >>>>>>> through PRs manually and use below criteria to identify
> > > > > > >>> un-necessary
> > > > > > >>>>>>> commits triggering the builds.
> > > > > > >>>>>>>
> > > > > > >>>>>>> 1. Explicitly marked as WIP / do not review PR
> > > > > > >>>>>>> 2. Incremental WIP commit and finally commenting a
> commit
> > > > > > >>> “trigger
> > > > > > >>>>> CI”
> > > > > > >>>>>>> 3. Multiple commits to address all comments from single
> > > > review.
> > > > > > >>>>> This is
> > > > > > >>>>>>> assuming we see a comment, address them, commit, next
> the
> > > > > > >>>> following
> > > > > > >>>>>>> comment
> > > > > > >>>>>>> 4. Sequence of documentation only changes
> > > > > > >>>>>>>
> > > > > > >>>>>>> I found there were around 42 avoidable builds from most
> > > recent
> > > > 50
> > > > > > >>>>> merged
> > > > > > >>>>>>> PRs and around 86 builds from recent 50 open PRs.
> > > > > > >>>>>>>
> > > > > > >>>>>>> I synced up with other contributors (Joe Evans, Chai)
> from
> > > > Amazon
> > > > > > >>> who
> > > > > > >>>>> is
> > > > > > >>>>>>> contributing to MXNet CI system. I was told that on an
> > > average
> > > > it
> > > > > > >>>> costs
> > > > > > >>>>>>> around $84 per build and on an average 6 commits per
> merged
> > > PR
> > > > > (for
> > > > > > >>>>> year
> > > > > > >>>>>>> 2019). Going by that, it is approximately 1/6 builds are
> > > > > avoidable.
> > > > > > >>>>> [100 /
> > > > > > >>>>>>> 300 + 300 ]
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>> Usability should be top most priority. But, since either
> a
> > > > > reviewer
> > > > > > >>>> or
> > > > > > >>>>> pr
> > > > > > >>>>>>> author can trigger the bot, is it really a hurdle for pr
> > > author
> > > > > or
> > > > > > >>>>> reviewer
> > > > > > >>>>>>> to call a bot to trigger CI? Given that PR author and
> > > reviewer
> > > > is
> > > > > > >>>>> already
> > > > > > >>>>>>> actively commenting various details such as - PR
> > description,
> > > > > > >>> review
> > > > > > >>>>>>> comments and responses, adding labels etc.
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>> Me too curious to know the behavior for Tao's above use
> > case.
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>> Best,
> > > > > > >>>>>>>
> > > > > > >>>>>>> Sandeep
> > > > > > >>>>>>>
> > > > > > >>>>>>> On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <
> mutouorz@gmail.com
> > >
> > > > > wrote:
> > > > > > >>>>>>>
> > > > > > >>>>>>>> Is it possible for re-triggering a single job to be
> > abused?
> > > > For
> > > > > > >>>>> example,
> > > > > > >>>>>>>> the author spends two days re-triggering a flaky job to
> > make
> > > > it
> > > > > > >>>>> pass. But
> > > > > > >>>>>>>> other jobs which have passed the validation may be
> broken
> > by
> > > > > > >>> other
> > > > > > >>>>>>> commits
> > > > > > >>>>>>>> during the two day without being noticed. And finally
> the
> > PR
> > > > is
> > > > > > >>>>> merged
> > > > > > >>>>>>> with
> > > > > > >>>>>>>> underlying problems.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <
> > > > > > >>>>> marco.g.abreu@gmail.com>
> > > > > > >>>>>>>> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> In the end it only comes down to money, considering
> that
> > > the
> > > > > > >>>>> system is
> > > > > > >>>>>>>> auto
> > > > > > >>>>>>>>> scaling, making the execution time constant.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> If we're trading money for usability, I certainly would
> > > > prefer
> > > > > > >>>>>>> usability.
> > > > > > >>>>>>>>> I'd rather recommend to spend time on parallelizing
> test
> > > > > > >>>> execution
> > > > > > >>>>> or
> > > > > > >>>>>>>>> getting rid of integration tests in the PR stage
> instead
> > > > > > >>> reducing
> > > > > > >>>>> the
> > > > > > >>>>>>>> costs
> > > > > > >>>>>>>>> by making people not use it. But taking a step back to
> > > > > > >>> requiring
> > > > > > >>>>> people
> > > > > > >>>>>>>> to
> > > > > > >>>>>>>>> manually trigger CI again doesn't feel right.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> I'm happy to see that bot deployed, but I do not agree
> > with
> > > > > > >>>>> removing
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> auto trigger functionality for new commits.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> -Marco
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Chaitanya Bapat <ch...@gmail.com> schrieb am Do.,
> > 12.
> > > > > > >>> März
> > > > > > >>>>> 2020,
> > > > > > >>>>>>>>> 22:47:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> @Marco Thanks for pointing that out.
> > > > > > >>>>>>>>>> Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM - 3:30
> > PM
> > > in
> > > > > > >>>>>>>> (UTC-08:00)
> > > > > > >>>>>>>>>> Pacific Time (US & Canada).
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>> When do we expect this bot to be deployed?
> > > > > > >>>>>>>>>> @Lin If all goes well in the next week I can deploy it
> > to
> > > > > > >>>> public
> > > > > > >>>>>>> Apache
> > > > > > >>>>>>>>>> (provided I get permissions from Apache Infra)
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> @Marco Thanks for your feedback.
> > > > > > >>>>>>>>>>> CI system has to support the community without
> > requiring
> > > > > > >>>>> people to
> > > > > > >>>>>>>>>> constantly shepherd every single run
> > > > > > >>>>>>>>>> We have data for the number of times CI was triggered
> > > > > > >>>>> unnecessarily
> > > > > > >>>>>>>> which
> > > > > > >>>>>>>>>> includes
> > > > > > >>>>>>>>>> - Entire build triggered instead of specific build
> > > > > > >>>>>>>>>> - CI triggered when PR is still work in progress or
> not
> > > yet
> > > > > > >>>> ready
> > > > > > >>>>>>> (say
> > > > > > >>>>>>>> -
> > > > > > >>>>>>>>>> intermediate commits)
> > > > > > >>>>>>>>>> At the end its a trade-off
> > > > > > >>>>>>>>>> Money, Resources, Time to build for each and every
> > commit
> > > vs
> > > > > > >>>>> Pain of
> > > > > > >>>>>>>>>> triggering builds
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>> Scan trigger plugin would poll SCM. Can we use plugin
> > at
> > > > > > >>>>> scale?
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> 1. I haven't tested it on scale. But I think with the
> > > > current
> > > > > > >>>>> scale
> > > > > > >>>>>>> of
> > > > > > >>>>>>>>>> MXNet repo (191 open PRs i.e. checking for changes to
> > 191
> > > > > > >>>>> branches -
> > > > > > >>>>>>> It
> > > > > > >>>>>>>>>> should be manageable)
> > > > > > >>>>>>>>>> 2. What's the purpose of the plugin? tldr; Branch
> > > discovery
> > > > > > >>> or
> > > > > > >>>>> branch
> > > > > > >>>>>>>>>> indexing.
> > > > > > >>>>>>>>>> Scan trigger plugin comes into the picture only once
> per
> > > PR
> > > > > > >>> per
> > > > > > >>>>> job
> > > > > > >>>>>>>>> (i.e. 8
> > > > > > >>>>>>>>>> times per PR for 8 jobs). It is basically done when a
> > new
> > > PR
> > > > > > >>> is
> > > > > > >>>>> made
> > > > > > >>>>>>>> and
> > > > > > >>>>>>>>>> the job (say unix-cpu hasn't discovered the new PR
> > branch
> > > > > > >>> yet).
> > > > > > >>>>>>> That's
> > > > > > >>>>>>>>> it.
> > > > > > >>>>>>>>>> So it shouldn't be a problem for public MXNet repo.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>> Chai
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <
> > > > > > >>>>>>> marco.g.abreu@gmail.com>
> > > > > > >>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>> Btw you forgot to set a date and time for the metting
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu <
> > > > > > >>>>>>>>> marco.g.abreu@gmail.com
> > > > > > >>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Thanks Chai, I generally like the idea of the bot.
> But
> > > > > > >>> I'm
> > > > > > >>>>> not a
> > > > > > >>>>>>>>>>> supporter
> > > > > > >>>>>>>>>>>> of the idea to disable any automatic triggering
> > > > > > >>> (disabling
> > > > > > >>>>> the
> > > > > > >>>>>>>>> webhook
> > > > > > >>>>>>>>>> is
> > > > > > >>>>>>>>>>>> also not an option, considering that this will
> disable
> > > > > > >>>> master
> > > > > > >>>>>>>>>> triggers).
> > > > > > >>>>>>>>>>>> The CI system has to support the community without
> > > > > > >>>> requiring
> > > > > > >>>>>>> people
> > > > > > >>>>>>>>> to
> > > > > > >>>>>>>>>>>> constantly shepherd every single run. Disabling
> > > automatic
> > > > > > >>>>>>>> triggering
> > > > > > >>>>>>>>>>> seems
> > > > > > >>>>>>>>>>>> like a step back to me.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Instead, I'd recommend that CI gets triggered upon
> > every
> > > > > > >>>>> commit
> > > > > > >>>>>>> as
> > > > > > >>>>>>>>>> usual,
> > > > > > >>>>>>>>>>>> but people have the possibility to call a "command"
> > > (i.e.
> > > > > > >>>>> make a
> > > > > > >>>>>>>>>> message
> > > > > > >>>>>>>>>>>> which results in the bot setting a label) to disable
> > CI
> > > > > > >>>> until
> > > > > > >>>>>>> they
> > > > > > >>>>>>>>>> revoke
> > > > > > >>>>>>>>>>>> it. But the standard should still be that a new
> commit
> > > > > > >>>>> triggers a
> > > > > > >>>>>>>> new
> > > > > > >>>>>>>>>> CI
> > > > > > >>>>>>>>>>>> run.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>
> https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
> > > > > > >>>>>>>> seems
> > > > > > >>>>>>>>>> like
> > > > > > >>>>>>>>>>>> this would poll SCM. This will incur high quota
> > > > > > >>>>> restrictions. Are
> > > > > > >>>>>>>> you
> > > > > > >>>>>>>>>>> sure
> > > > > > >>>>>>>>>>>> that you can use that plugin at scale?
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> -Marco
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <
> > > > > > >>>>> apeforest@gmail.com>
> > > > > > >>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>> Chai,
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Awesome work. When do we expect this bot to be
> > > > > > >>> deployed?
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Best,
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Lin
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <
> > > > > > >>>>>>>>> chai.bapat@gmail.com
> > > > > > >>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Hello MXNet community,
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> I have built an MXNet Bot <
> > > > > > >>>> https://github.com/mxnet-bot>
> > > > > > >>>>> that
> > > > > > >>>>>>>>>> allows
> > > > > > >>>>>>>>>>> PR
> > > > > > >>>>>>>>>>>>>> Authors, Committers and Jenkins Admins to trigger
> CI
> > > > > > >>>>> manually.
> > > > > > >>>>>>>>>>>>>> It handles 2 problems
> > > > > > >>>>>>>>>>>>>> 1. Manual CI trigger instead of existing automated
> > CI
> > > > > > >>>>> trigger
> > > > > > >>>>>>>>>>>>>> 2. Gives permissions to PR Authors (in addition to
> > > > > > >>>> MXNet
> > > > > > >>>>>>>>> Committers
> > > > > > >>>>>>>>>>> and
> > > > > > >>>>>>>>>>>>>> Jenkins Admins)
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Design Doc :
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>
> > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > > > > > >>>>>>>>>>>>>> I urge you all to attend the demonstration meeting
> > > > > > >>> and
> > > > > > >>>>> lend
> > > > > > >>>>>>> your
> > > > > > >>>>>>>>>> views
> > > > > > >>>>>>>>>>>>> on
> > > > > > >>>>>>>>>>>>>> the same.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Thank you,
> > > > > > >>>>>>>>>>>>>> Chai
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> *Meeting Details*:
> > > > > > >>>>>>>>>>>>>> ==============Conference Bridge
> > > > > > >>>> Information==============
> > > > > > >>>>>>>>>>>>>> You have been invited to an online meeting,
> powered
> > > > > > >>> by
> > > > > > >>>>> Amazon
> > > > > > >>>>>>>>> Chime.
> > > > > > >>>>>>>>>>>>>> *Chime meeting ID*: *9272158344*
> > > > > > >>>>>>>>>>>>>> Join via Chime clients (manually): Select
> 'Meetings
> > >
> > > > > > >>>>> Join a
> > > > > > >>>>>>>>>> Meeting',
> > > > > > >>>>>>>>>>>>> and
> > > > > > >>>>>>>>>>>>>> enter 9272158344
> > > > > > >>>>>>>>>>>>>> Join via Chime clients (auto-call): If you invite
> > > > > > >>>>> auto-call as
> > > > > > >>>>>>>>>>> attendee,
> > > > > > >>>>>>>>>>>>>> Chime will call you when the meeting starts,
> select
> > > > > > >>>>> 'Answer'
> > > > > > >>>>>>>>>>>>>> *Join via browser screen share*:
> > > > > > >>>>> https://chime.aws/9272158344
> > > > > > >>>>>>>>>>>>>> *Join via phone* (US):
> +1-929-432-4463,,,9272158344#
> > > > > > >>>>>>>>>>>>>> *Join via phone (US toll-free)*:
> > > > > > >>>>> +1-855-552-4463,,,9272158344#
> > > > > > >>>>>>>>>>>>>> International dial-in:
> > > > > > >>>> https://chime.aws/dialinnumbers/
> > > > > > >>>>>>>>>>>>>> In-room video system: Ext: 62000, Meeting PIN:
> > > > > > >>>>> 9272158344#
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> --
> > > > > > >>>>>>>>>>>>>> *Chaitanya Prakash Bapat*
> > > > > > >>>>>>>>>>>>>> *+1 (973) 953-6299*
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> [image: https://www.linkedin.com//in/chaibapat25]
> > > > > > >>>>>>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> > > > > > >>>>>>>>>>>>> https://www.facebook.com/chaibapat
> > > > > > >>>>>>>>>>>>>> ]
> > > > > > >>>>>>>>>>>>>> <https://www.facebook.com/chaibapchya>[image:
> > > > > > >>>>>>>>>>>>>> https://twitter.com/ChaiBapchya] <
> > > > > > >>>>>>>> https://twitter.com/ChaiBapchya
> > > > > > >>>>>>>>>>>>>> [image:
> > > > > > >>>>>>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> > > > > > >>>>>>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> --
> > > > > > >>>>>>>>>> *Chaitanya Prakash Bapat*
> > > > > > >>>>>>>>>> *+1 (973) 953-6299*
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> [image: https://www.linkedin.com//in/chaibapat25]
> > > > > > >>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> > > > > > >>>>>>>>> https://www.facebook.com/chaibapat
> > > > > > >>>>>>>>>> ]
> > > > > > >>>>>>>>>> <https://www.facebook.com/chaibapchya>[image:
> > > > > > >>>>>>>>>> https://twitter.com/ChaiBapchya] <
> > > > > > >>>>> https://twitter.com/ChaiBapchya
> > > > > > >>>>>>>>>> [image:
> > > > > > >>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> > > > > > >>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>> --
> > > > > > >>>>>>> Sandeep Krishnamurthy
> > > > > > >>>>>>>
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> --
> > > > > > >>> *Chaitanya Prakash Bapat*
> > > > > > >>> *+1 (973) 953-6299*
> > > > > > >>>
> > > > > > >>> [image: https://www.linkedin.com//in/chaibapat25]
> > > > > > >>> <https://github.com/ChaiBapchya>[image:
> > > > > > https://www.facebook.com/chaibapat
> > > > > > >>> ]
> > > > > > >>> <https://www.facebook.com/chaibapchya>[image:
> > > > > > >>> https://twitter.com/ChaiBapchya] <
> > > https://twitter.com/ChaiBapchya
> > > > > > >[image:
> > > > > > >>> https://www.linkedin.com//in/chaibapat25]
> > > > > > >>> <https://www.linkedin.com//in/chaibapchya/>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > *Chaitanya Prakash Bapat*
> > > > *+1 (973) 953-6299*
> > > >
> > > > [image: https://www.linkedin.com//in/chaibapat25]
> > > > <https://github.com/ChaiBapchya>[image:
> > > https://www.facebook.com/chaibapat
> > > > ]
> > > > <https://www.facebook.com/chaibapchya>[image:
> > > > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> > > >[image:
> > > > https://www.linkedin.com//in/chaibapat25]
> > > > <https://www.linkedin.com//in/chaibapchya/>
> > > >
> > >
> >
> >
> > --
> > *Chaitanya Prakash Bapat*
> > *+1 (973) 953-6299*
> >
> > [image: https://www.linkedin.com//in/chaibapat25]
> > <https://github.com/ChaiBapchya>[image:
> https://www.facebook.com/chaibapat
> > ]
> > <https://www.facebook.com/chaibapchya>[image:
> > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> >[image:
> > https://www.linkedin.com//in/chaibapat25]
> > <https://www.linkedin.com//in/chaibapchya/>
> >
>
--
*Chaitanya Prakash Bapat*
*+1 (973) 953-6299*
[image: https://www.linkedin.com//in/chaibapat25]
<https://github.com/ChaiBapchya>[image: https://www.facebook.com/chaibapat]
<https://www.facebook.com/chaibapchya>[image:
https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
https://www.linkedin.com//in/chaibapat25]
<https://www.linkedin.com//in/chaibapchya/>
Re: MXNet Bot Demo
Posted by Marco de Abreu <ma...@gmail.com>.
Hi, that's correct. But as stated previously, it's not an option to remove
the hook. For now, I'd like to see how the system behaves while it's
optional. Later on, we can talk about revisiting this decision. But to me
it's not an option to deploy an entirely new system and approach without
having a transition or even a timeframe in which we are able to fall back.
I'm happy to support the deployment of the bot and add an additional
webhook to enable it's functionality to support selective triggering by PR
authors and committers, but I will not support the disabling of automatic
triggering of branches or PRs.
-Marco
Chaitanya Bapat <ch...@gmail.com> schrieb am Mi., 18. März 2020, 21:00:
> Hey Marco,
>
> I thought currently every commit on PR and master triggers CI
> because
> a. github webhook points to Jenkins Server
> b. GH Webhook events trigger builds on Jenkins for all commits to any
> branch in apache/incubator-mxnet
> may it be master/PR/non-PR
> Reason:
> Because all the 3 types of branches are discovered by Jenkins (non-PR
> (including master) and PR)
>
> Proposal: Remove GitHub WebHook to Jenkins and replace with GH Webhook to
> Lambda
> But after I remove the github webhook that points to Jenkins : *N**o commit
> will trigger Jenkins build by default* (as Jenkins wont receive GH events)
> Only those that Bot deems fit will be triggered (using Jenkins API invoked
> by Lambda).
> Hence its needed to handle that case of master merge.
> Am I understanding this correctly?
>
> On Wed, 18 Mar 2020 at 04:23, Marco de Abreu <ma...@gmail.com>
> wrote:
>
> > Thanks Chai, sounds good to me. Could you elaborate a bit on the point
> > about triggering a CI run after the PR has been merged? We already to
> that
> > automatically for the master, so what's the benefit to do it twice?
> >
> > -Marco
> >
> > Chaitanya Bapat <ch...@gmail.com> schrieb am Mi., 18. März 2020,
> > 09:30:
> >
> > > Update:
> > >
> > > > we can ensure that all CI runs ran on the commit that will be merged
> > > @Sam Skalicky <sa...@gmail.com> Branch Protection is added to
> > public
> > > MXNet repo. It ensures that for every PR to be merged, the CI passes.
> All
> > > the jobs selected "required" jobs will have to be green for the PR to
> be
> > > merged. Ofcourse, users with "Adminstrator" access can merge without it
> > but
> > > that's just a backdoor. It is the case now and will continue to be the
> > case
> > > with the inclusion of Bot.
> > >
> > > > easily verify that the CI has executed all runs on the commit that
> will
> > > be merged
> > > GitHub UI shows all the jobs and the status corresponding to it on
> every
> > > commit. That should suffice. For the merged commits, Repo -> Commits ->
> > > Commit ID (Status) can be tracked currently (only way that I know of).
> > > Moreover, it is beyond the scope of this project (and possibly out of
> our
> > > control since this is purely GitHub UI specific use-case).
> > >
> > > Thanks @przemyslaw for supporting the opt-in.
> > >
> > > Thanks everyone in the community for sharing concerns, voicing your
> > opinion
> > > and participating in the discussion.
> > > Thanks to those who attended the demo last Friday.
> > >
> > > Action items from that discussion
> > > 1. Handle master merge builds [Done]
> > > Bot runs entire CI suite after the PR is merged and comments on the PR
> > > about the same.
> > > Design decision :
> > > MXNet Bot comment about master merge build on the *merge commit vs PR*.
> > > After the PR is merged, Bot runs entire CI and comments the result of
> CI
> > > trigger on the PR (because it is easy to track on a PR rather than
> > > commenting inside the merge commit)
> > >
> > > 2. Idempotent condition
> > > In case of already running build, if an attempt is made to retrigger
> the
> > > job then what should be the response
> > > a. Not to re-trigger, let the ongoing build continue till completion
> > > b. End the ongoing build and re-trigger
> > > c. Let the ongoing build continue, re-trigger new build
> > >
> > > From resource saving point of view, *c* looks costly and a can be
> > > avoided/optimized by B.
> > > In case when a re-trigger was started "erroneously" then killing
> ongoing
> > > build and re-trigger is a waste.
> > > In case when ongoing build failed in one sub-part, then re-triggering
> is
> > > justified.
> > > Erroneous re-triggers would be less often while conscious re-triggers
> to
> > > suppress failure is more common use-case. It looks like a safe
> assumption
> > > to make given the trade-off.
> > > [Open to debate]
> > >
> > > 3. Add security consideration [Use of secret manager, but without
> > > auto-rotation due to Jenkins manual config requirement] [Done]
> > > 4. New PR Instruction message by the Bot [Done]
> > > Thanks to the suggestion of Leonard, supported by others. I've now
> added
> > > the feature where the Bot comments a help message. [For reference -
> > > https://github.com/ChaiBapchya/incubator-mxnet/pull/52]
> > >
> > > Barring the opt-in vs opt-out debate & idempotency, consensus was
> quickly
> > > reached for the rest.
> > >
> > > In the coming days, I hope to roll-out this feature into Prod (public
> > > MXNet) for all devs to use.
> > >
> > > Thanks,
> > > Chai
> > >
> > > On Mon, 16 Mar 2020 at 11:57, Marco de Abreu <ma...@gmail.com>
> > > wrote:
> > >
> > > > Well that's generally a problem with a deferred CI approach (CI is
> run
> > at
> > > > commit and not at merge time). This can either be solved through the
> > > other
> > > > proposal that's currently on dev@, by having a bot which does merges
> > by
> > > > having a global lock and a merge queue or by accepting the issue.
> > Reality
> > > > right now is that we're running that model where two PRs which are
> > merged
> > > > in parallel might break one another. One thing to consider though is
> > that
> > > > this breakage would have to be introduced in two separate parts since
> > > > otherwise there'd be merge conflicts. I think we had that situation
> > twice
> > > > so far and the result was a quick revert, so I'd say that it's a
> > problem
> > > > that can happily be accepted. All other solutions basically require
> > some
> > > > form of single-threaded and globally locked solution which limits us
> in
> > > > scalability. I'd recommend to just accept that risk and revert a PR
> in
> > > case
> > > > it actually had a conflict.
> > > >
> > > > -Marco
> > > >
> > > > On Mon, Mar 16, 2020 at 6:29 PM Skalicky, Sam
> > <sskalic@amazon.com.invalid
> > > >
> > > > wrote:
> > > >
> > > > > We probably need some way to track which CI runs ran for which
> commit
> > > > too,
> > > > > that way we can ensure that all CI runs ran on the commit that will
> > be
> > > > > merged. Maybe the bot can comment with the commit hash when users
> > > > command
> > > > > it to do something. Although since users can trigger individual CI
> > runs
> > > > its
> > > > > possible to have some commits run some CI runs but not others. We
> > need
> > > > some
> > > > > way to easily verify that the CI has executed all runs on the
> commit
> > > that
> > > > > will be merged.
> > > > >
> > > > > Sam
> > > > >
> > > > > > On Mar 13, 2020, at 8:28 PM, Przemysław Trędak <
> ptrendx@apache.org
> > >
> > > > > wrote:
> > > > > >
> > > > > > CAUTION: This email originated from outside of the organization.
> Do
> > > not
> > > > > click links or open attachments unless you can confirm the sender
> and
> > > > know
> > > > > the content is safe.
> > > > > >
> > > > > >
> > > > > >
> > > > > > I personally like the idea of opt-in more than opt-out:
> > > > > > - ultimately PR author wants the PR to be merged so they (or
> > > committer
> > > > > reviewing the PR) will trigger the CI
> > > > > > - if it is easy to trigger the PR via the bot command then the
> > amount
> > > > of
> > > > > work per PR should be less than with opt-out (since most of the
> > commits
> > > > > should then be marked as [skip ci] or something similar
> > > > > >
> > > > > > +1 to the bot making a comment on each new PR with its commands
> > (and
> > > > > also explaining, or at least giving links to the general PR process
> > so
> > > > new
> > > > > PR authors are not lost). Maybe we could make the bot recognize if
> > the
> > > PR
> > > > > author is new or existing contributor and offer advice based on
> that?
> > > > > >
> > > > > > Thanks
> > > > > > Przemek
> > > > > >
> > > > > > On 2020/03/13 22:06:58, Marco de Abreu <ma...@gmail.com>
> > > > wrote:
> > > > > >> Hi,
> > > > > >>
> > > > > >> since it's no longer necessary to push a new commit to trigger
> CI,
> > > it
> > > > > will
> > > > > >> already reduce the costs. But to me, requiring an action to
> enable
> > > CI
> > > > > after
> > > > > >> a PR has been created initially, is a no go. User can opt out of
> > CI,
> > > > but
> > > > > >> the default has to be CI being triggered automatically for every
> > > > commit
> > > > > >> unless specifically disabled by a participant. I'm also fine
> with
> > > > > >> triggering certain additional jobs (think about running a
> nightly
> > > job
> > > > > upon
> > > > > >> request for a PR) to require a manual step, but the PR
> validation
> > > > > pipelines
> > > > > >> have to run automatically. Every check that is marked as
> > "Required"
> > > in
> > > > > >> GitHub has to be automatically kicked off.
> > > > > >>
> > > > > >> -Marco
> > > > > >>
> > > > > >> On Fri, Mar 13, 2020 at 9:50 PM Chaitanya Bapat <
> > > chai.bapat@gmail.com
> > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Firstly,
> > > > > >>> Sorry I missed out on attaching the mail thread that was sent
> on
> > > 12th
> > > > > >>> February for notifying the community of the upcoming changes to
> > the
> > > > > MXNet
> > > > > >>> CI
> > > > > >>> For reference :
> > > > > >>>
> > > > > >>>
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/r09a6ab2803a996fc80e00fe39ed312fa4865e8805e08df847f1addad%40%3Cdev.mxnet.apache.org%3E
> > > > > >>>
> > > > > >>> Now to the questions,
> > > > > >>>> Is it possible for re-triggering a single job to be abused?
> > > > > >>> @Tao In the case when a user re-triggers a single job multiple
> > > times,
> > > > > that
> > > > > >>> will be visible in the PR conversation thread. A committer,
> even
> > > > after
> > > > > he
> > > > > >>> has approved the PR before, generally takes a look at the final
> > > state
> > > > > of
> > > > > >>> the PR before merging. Would it be fair to assume the committer
> > > could
> > > > > take
> > > > > >>> the multiple re-trigger of a single job into account before
> > > merging?
> > > > > The
> > > > > >>> committer then has the option to invoke `@mxnet-bot run ci
> [all]
> > `
> > > to
> > > > > >>> trigger the entire build pipeline one last to counter the
> abuse.
> > > This
> > > > > is
> > > > > >>> aligned with what @Leonard said.
> > > > > >>>
> > > > > >>> @Sandeep Thanks a lot for collecting and sharing valuable data.
> > I'd
> > > > > concur
> > > > > >>> with the opinion that given the existing things committers & PR
> > > > Authors
> > > > > >>> take care of, invoking CI shouldn't be that big of an
> additional
> > > > > burden.
> > > > > >>>
> > > > > >>> @Marco With the opt-out, the onus remains on the PR Author. It
> > > > doesn't
> > > > > help
> > > > > >>> reduce the resource usage. Hence, it was suggested to switch to
> > > > > >>> opt-in. @Leo's suggestion for proactive commenting on the part
> of
> > > bot
> > > > > makes
> > > > > >>> sense and is doable.
> > > > > >>>
> > > > > >>> Default : opt-out and User initiated opt-in (with addressing
> > Leo's
> > > > fix
> > > > > for
> > > > > >>> the usability issue you correctly pointed out )
> > > > > >>> @Marco How does this sound to you?
> > > > > >>>
> > > > > >>> Again, thank you all for chiming in and voicing your opinion.
> > > > > Appreciate
> > > > > >>> it.
> > > > > >>> We can take ahead these discussions in today's demo meeting.
> > > [Design
> > > > > Doc
> > > > > >>> <
> https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > >]
> > > > > [Demo
> > > > > >>> Video <https://www.youtube.com/watch?v=gfOGwZId8aU>]
> > > > > >>>
> > > > > >>> Thanks,
> > > > > >>> Chai
> > > > > >>>
> > > > > >>> On Fri, 13 Mar 2020 at 12:34, Marco de Abreu <
> > > > marco.g.abreu@gmail.com>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> I'd recommend that the bot makes an initial comment when a PR
> > gets
> > > > > opened
> > > > > >>>> and informs the users of its commands. It then tells the user
> > the
> > > > > commend
> > > > > >>>> to opt out of CI.
> > > > > >>>>
> > > > > >>>> -Marco
> > > > > >>>>
> > > > > >>>> Lausen, Leonard <la...@amazon.com.invalid> schrieb am Fr.,
> 13.
> > > > März
> > > > > >>> 2020,
> > > > > >>>> 20:27:
> > > > > >>>>
> > > > > >>>>> On opt-out: People may be unaware of opt-out would not use
> it.
> > > > There
> > > > > is
> > > > > >>>> no
> > > > > >>>>> incentive to use opt-out, as the PR author doesn't pay any
> > money
> > > > for
> > > > > CI
> > > > > >>>>> run.
> > > > > >>>>>
> > > > > >>>>> I agree with Marco though that opt-in alone may cause
> usability
> > > > > issues,
> > > > > >>>> as
> > > > > >>>>> contributors may not be aware of how to trigger the CI.
> > > > > >>>>> One solution is that the bot proactively comments on the PR
> and
> > > > > reminds
> > > > > >>>> the
> > > > > >>>>> author to trigger running CI once the author deems the PR
> > ready.
> > > > > >>>>>
> > > > > >>>>> But even if we choose opt-out, the bot will still add a lot
> of
> > > > value,
> > > > > >>> as
> > > > > >>>> PR
> > > > > >>>>> authors can retrigger single jobs that have failed due to
> > > > flakiness.
> > > > > >>>>>
> > > > > >>>>>> Is it possible for re-triggering a single job to be abused?
> > For
> > > > > >>>> example,
> > > > > >>>>>> the author spends two days re-triggering a flaky job to make
> > it
> > > > > pass.
> > > > > >>>>>
> > > > > >>>>> Yes, this is possible. I suggest the committer who likes to
> > > merge a
> > > > > PR
> > > > > >>>>> needs to
> > > > > >>>>> make a good judgement here if a PR is abusing the feature,
> and
> > if
> > > > so,
> > > > > >>>>> retrigger
> > > > > >>>>> all CI runs.
> > > > > >>>>>
> > > > > >>>>> Best regards
> > > > > >>>>> Leonard
> > > > > >>>>>
> > > > > >>>>> On Fri, 2020-03-13 at 08:07 +0100, Marco de Abreu wrote:
> > > > > >>>>>> Thanks for the data Sandeep. In these cases it sounds like
> it
> > > > would
> > > > > >>>> have
> > > > > >>>>>> rather been better when people explicitly turned off CI in
> > that
> > > > > case.
> > > > > >>>>>> What's the argument against an opt-out instead of an opt-in?
> > > > > >>>>>>
> > > > > >>>>>> My intention is that I consider it quite cumbersome to make
> > it a
> > > > > >>>>> *required*
> > > > > >>>>>> step to always trigger CI manually, even if just submitting
> a
> > > > small
> > > > > >>> PR.
> > > > > >>>>> I'd
> > > > > >>>>>> rather see people explicitly turning off CI if they wouldn't
> > > like
> > > > to
> > > > > >>>> use
> > > > > >>>>> it
> > > > > >>>>>> - and there's also the "draft" stage for a PR which some
> > > > > contributors
> > > > > >>>> are
> > > > > >>>>>> using.
> > > > > >>>>>>
> > > > > >>>>>> With regards to WIP and do not review: I think these are use
> > > cases
> > > > > >>>> where
> > > > > >>>>>> you want CI feedback, as otherwise you wouldn't have opened
> > the
> > > > PR.
> > > > > >>> If
> > > > > >>>>> you
> > > > > >>>>>> don't want human feedback and neither machine feedback, why
> > open
> > > > the
> > > > > >>> PR
> > > > > >>>>> at
> > > > > >>>>>> all?
> > > > > >>>>>>
> > > > > >>>>>> -Marco
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> sandeep krishnamurthy <sa...@gmail.com> schrieb
> > am
> > > > Fr.,
> > > > > >>>> 13.
> > > > > >>>>>> März 2020, 05:24:
> > > > > >>>>>>
> > > > > >>>>>>> I tried to gather some data for us to discuss this topic in
> > > this
> > > > > >>>>> thread. I
> > > > > >>>>>>> tried to count number of un-necessary builds by looking at
> > most
> > > > > >>>> recent
> > > > > >>>>> (as
> > > > > >>>>>>> of 12, March 9 PM PST) 50 PRs merged to master and 50 PRs.
> > > > > >>>>>>> Identifying un-necessary builds is bit subjective. I tried
> to
> > > be
> > > > > >>> more
> > > > > >>>>>>> conservative where I didn't count a build as un-necessary
> if
> > I
> > > > was
> > > > > >>> in
> > > > > >>>>>>> doubt. Hence, I was not able to automate, but I made an
> > effort
> > > to
> > > > > >>> go
> > > > > >>>>>>> through PRs manually and use below criteria to identify
> > > > > >>> un-necessary
> > > > > >>>>>>> commits triggering the builds.
> > > > > >>>>>>>
> > > > > >>>>>>> 1. Explicitly marked as WIP / do not review PR
> > > > > >>>>>>> 2. Incremental WIP commit and finally commenting a commit
> > > > > >>> “trigger
> > > > > >>>>> CI”
> > > > > >>>>>>> 3. Multiple commits to address all comments from single
> > > review.
> > > > > >>>>> This is
> > > > > >>>>>>> assuming we see a comment, address them, commit, next the
> > > > > >>>> following
> > > > > >>>>>>> comment
> > > > > >>>>>>> 4. Sequence of documentation only changes
> > > > > >>>>>>>
> > > > > >>>>>>> I found there were around 42 avoidable builds from most
> > recent
> > > 50
> > > > > >>>>> merged
> > > > > >>>>>>> PRs and around 86 builds from recent 50 open PRs.
> > > > > >>>>>>>
> > > > > >>>>>>> I synced up with other contributors (Joe Evans, Chai) from
> > > Amazon
> > > > > >>> who
> > > > > >>>>> is
> > > > > >>>>>>> contributing to MXNet CI system. I was told that on an
> > average
> > > it
> > > > > >>>> costs
> > > > > >>>>>>> around $84 per build and on an average 6 commits per merged
> > PR
> > > > (for
> > > > > >>>>> year
> > > > > >>>>>>> 2019). Going by that, it is approximately 1/6 builds are
> > > > avoidable.
> > > > > >>>>> [100 /
> > > > > >>>>>>> 300 + 300 ]
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> Usability should be top most priority. But, since either a
> > > > reviewer
> > > > > >>>> or
> > > > > >>>>> pr
> > > > > >>>>>>> author can trigger the bot, is it really a hurdle for pr
> > author
> > > > or
> > > > > >>>>> reviewer
> > > > > >>>>>>> to call a bot to trigger CI? Given that PR author and
> > reviewer
> > > is
> > > > > >>>>> already
> > > > > >>>>>>> actively commenting various details such as - PR
> description,
> > > > > >>> review
> > > > > >>>>>>> comments and responses, adding labels etc.
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> Me too curious to know the behavior for Tao's above use
> case.
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> Best,
> > > > > >>>>>>>
> > > > > >>>>>>> Sandeep
> > > > > >>>>>>>
> > > > > >>>>>>> On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <mutouorz@gmail.com
> >
> > > > wrote:
> > > > > >>>>>>>
> > > > > >>>>>>>> Is it possible for re-triggering a single job to be
> abused?
> > > For
> > > > > >>>>> example,
> > > > > >>>>>>>> the author spends two days re-triggering a flaky job to
> make
> > > it
> > > > > >>>>> pass. But
> > > > > >>>>>>>> other jobs which have passed the validation may be broken
> by
> > > > > >>> other
> > > > > >>>>>>> commits
> > > > > >>>>>>>> during the two day without being noticed. And finally the
> PR
> > > is
> > > > > >>>>> merged
> > > > > >>>>>>> with
> > > > > >>>>>>>> underlying problems.
> > > > > >>>>>>>>
> > > > > >>>>>>>> On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <
> > > > > >>>>> marco.g.abreu@gmail.com>
> > > > > >>>>>>>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> In the end it only comes down to money, considering that
> > the
> > > > > >>>>> system is
> > > > > >>>>>>>> auto
> > > > > >>>>>>>>> scaling, making the execution time constant.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> If we're trading money for usability, I certainly would
> > > prefer
> > > > > >>>>>>> usability.
> > > > > >>>>>>>>> I'd rather recommend to spend time on parallelizing test
> > > > > >>>> execution
> > > > > >>>>> or
> > > > > >>>>>>>>> getting rid of integration tests in the PR stage instead
> > > > > >>> reducing
> > > > > >>>>> the
> > > > > >>>>>>>> costs
> > > > > >>>>>>>>> by making people not use it. But taking a step back to
> > > > > >>> requiring
> > > > > >>>>> people
> > > > > >>>>>>>> to
> > > > > >>>>>>>>> manually trigger CI again doesn't feel right.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> I'm happy to see that bot deployed, but I do not agree
> with
> > > > > >>>>> removing
> > > > > >>>>>>> the
> > > > > >>>>>>>>> auto trigger functionality for new commits.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> -Marco
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Chaitanya Bapat <ch...@gmail.com> schrieb am Do.,
> 12.
> > > > > >>> März
> > > > > >>>>> 2020,
> > > > > >>>>>>>>> 22:47:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> @Marco Thanks for pointing that out.
> > > > > >>>>>>>>>> Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM - 3:30
> PM
> > in
> > > > > >>>>>>>> (UTC-08:00)
> > > > > >>>>>>>>>> Pacific Time (US & Canada).
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> When do we expect this bot to be deployed?
> > > > > >>>>>>>>>> @Lin If all goes well in the next week I can deploy it
> to
> > > > > >>>> public
> > > > > >>>>>>> Apache
> > > > > >>>>>>>>>> (provided I get permissions from Apache Infra)
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> @Marco Thanks for your feedback.
> > > > > >>>>>>>>>>> CI system has to support the community without
> requiring
> > > > > >>>>> people to
> > > > > >>>>>>>>>> constantly shepherd every single run
> > > > > >>>>>>>>>> We have data for the number of times CI was triggered
> > > > > >>>>> unnecessarily
> > > > > >>>>>>>> which
> > > > > >>>>>>>>>> includes
> > > > > >>>>>>>>>> - Entire build triggered instead of specific build
> > > > > >>>>>>>>>> - CI triggered when PR is still work in progress or not
> > yet
> > > > > >>>> ready
> > > > > >>>>>>> (say
> > > > > >>>>>>>> -
> > > > > >>>>>>>>>> intermediate commits)
> > > > > >>>>>>>>>> At the end its a trade-off
> > > > > >>>>>>>>>> Money, Resources, Time to build for each and every
> commit
> > vs
> > > > > >>>>> Pain of
> > > > > >>>>>>>>>> triggering builds
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> Scan trigger plugin would poll SCM. Can we use plugin
> at
> > > > > >>>>> scale?
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> 1. I haven't tested it on scale. But I think with the
> > > current
> > > > > >>>>> scale
> > > > > >>>>>>> of
> > > > > >>>>>>>>>> MXNet repo (191 open PRs i.e. checking for changes to
> 191
> > > > > >>>>> branches -
> > > > > >>>>>>> It
> > > > > >>>>>>>>>> should be manageable)
> > > > > >>>>>>>>>> 2. What's the purpose of the plugin? tldr; Branch
> > discovery
> > > > > >>> or
> > > > > >>>>> branch
> > > > > >>>>>>>>>> indexing.
> > > > > >>>>>>>>>> Scan trigger plugin comes into the picture only once per
> > PR
> > > > > >>> per
> > > > > >>>>> job
> > > > > >>>>>>>>> (i.e. 8
> > > > > >>>>>>>>>> times per PR for 8 jobs). It is basically done when a
> new
> > PR
> > > > > >>> is
> > > > > >>>>> made
> > > > > >>>>>>>> and
> > > > > >>>>>>>>>> the job (say unix-cpu hasn't discovered the new PR
> branch
> > > > > >>> yet).
> > > > > >>>>>>> That's
> > > > > >>>>>>>>> it.
> > > > > >>>>>>>>>> So it shouldn't be a problem for public MXNet repo.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Thanks,
> > > > > >>>>>>>>>> Chai
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <
> > > > > >>>>>>> marco.g.abreu@gmail.com>
> > > > > >>>>>>>>>> wrote:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> Btw you forgot to set a date and time for the metting
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu <
> > > > > >>>>>>>>> marco.g.abreu@gmail.com
> > > > > >>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> Thanks Chai, I generally like the idea of the bot. But
> > > > > >>> I'm
> > > > > >>>>> not a
> > > > > >>>>>>>>>>> supporter
> > > > > >>>>>>>>>>>> of the idea to disable any automatic triggering
> > > > > >>> (disabling
> > > > > >>>>> the
> > > > > >>>>>>>>> webhook
> > > > > >>>>>>>>>> is
> > > > > >>>>>>>>>>>> also not an option, considering that this will disable
> > > > > >>>> master
> > > > > >>>>>>>>>> triggers).
> > > > > >>>>>>>>>>>> The CI system has to support the community without
> > > > > >>>> requiring
> > > > > >>>>>>> people
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>>>>> constantly shepherd every single run. Disabling
> > automatic
> > > > > >>>>>>>> triggering
> > > > > >>>>>>>>>>> seems
> > > > > >>>>>>>>>>>> like a step back to me.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Instead, I'd recommend that CI gets triggered upon
> every
> > > > > >>>>> commit
> > > > > >>>>>>> as
> > > > > >>>>>>>>>> usual,
> > > > > >>>>>>>>>>>> but people have the possibility to call a "command"
> > (i.e.
> > > > > >>>>> make a
> > > > > >>>>>>>>>> message
> > > > > >>>>>>>>>>>> which results in the bot setting a label) to disable
> CI
> > > > > >>>> until
> > > > > >>>>>>> they
> > > > > >>>>>>>>>> revoke
> > > > > >>>>>>>>>>>> it. But the standard should still be that a new commit
> > > > > >>>>> triggers a
> > > > > >>>>>>>> new
> > > > > >>>>>>>>>> CI
> > > > > >>>>>>>>>>>> run.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>> https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
> > > > > >>>>>>>> seems
> > > > > >>>>>>>>>> like
> > > > > >>>>>>>>>>>> this would poll SCM. This will incur high quota
> > > > > >>>>> restrictions. Are
> > > > > >>>>>>>> you
> > > > > >>>>>>>>>>> sure
> > > > > >>>>>>>>>>>> that you can use that plugin at scale?
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> -Marco
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <
> > > > > >>>>> apeforest@gmail.com>
> > > > > >>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>> Chai,
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Awesome work. When do we expect this bot to be
> > > > > >>> deployed?
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Best,
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Lin
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <
> > > > > >>>>>>>>> chai.bapat@gmail.com
> > > > > >>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Hello MXNet community,
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> I have built an MXNet Bot <
> > > > > >>>> https://github.com/mxnet-bot>
> > > > > >>>>> that
> > > > > >>>>>>>>>> allows
> > > > > >>>>>>>>>>> PR
> > > > > >>>>>>>>>>>>>> Authors, Committers and Jenkins Admins to trigger CI
> > > > > >>>>> manually.
> > > > > >>>>>>>>>>>>>> It handles 2 problems
> > > > > >>>>>>>>>>>>>> 1. Manual CI trigger instead of existing automated
> CI
> > > > > >>>>> trigger
> > > > > >>>>>>>>>>>>>> 2. Gives permissions to PR Authors (in addition to
> > > > > >>>> MXNet
> > > > > >>>>>>>>> Committers
> > > > > >>>>>>>>>>> and
> > > > > >>>>>>>>>>>>>> Jenkins Admins)
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Design Doc :
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>
> > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > > > > >>>>>>>>>>>>>> I urge you all to attend the demonstration meeting
> > > > > >>> and
> > > > > >>>>> lend
> > > > > >>>>>>> your
> > > > > >>>>>>>>>> views
> > > > > >>>>>>>>>>>>> on
> > > > > >>>>>>>>>>>>>> the same.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Thank you,
> > > > > >>>>>>>>>>>>>> Chai
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> *Meeting Details*:
> > > > > >>>>>>>>>>>>>> ==============Conference Bridge
> > > > > >>>> Information==============
> > > > > >>>>>>>>>>>>>> You have been invited to an online meeting, powered
> > > > > >>> by
> > > > > >>>>> Amazon
> > > > > >>>>>>>>> Chime.
> > > > > >>>>>>>>>>>>>> *Chime meeting ID*: *9272158344*
> > > > > >>>>>>>>>>>>>> Join via Chime clients (manually): Select 'Meetings
> >
> > > > > >>>>> Join a
> > > > > >>>>>>>>>> Meeting',
> > > > > >>>>>>>>>>>>> and
> > > > > >>>>>>>>>>>>>> enter 9272158344
> > > > > >>>>>>>>>>>>>> Join via Chime clients (auto-call): If you invite
> > > > > >>>>> auto-call as
> > > > > >>>>>>>>>>> attendee,
> > > > > >>>>>>>>>>>>>> Chime will call you when the meeting starts, select
> > > > > >>>>> 'Answer'
> > > > > >>>>>>>>>>>>>> *Join via browser screen share*:
> > > > > >>>>> https://chime.aws/9272158344
> > > > > >>>>>>>>>>>>>> *Join via phone* (US): +1-929-432-4463,,,9272158344#
> > > > > >>>>>>>>>>>>>> *Join via phone (US toll-free)*:
> > > > > >>>>> +1-855-552-4463,,,9272158344#
> > > > > >>>>>>>>>>>>>> International dial-in:
> > > > > >>>> https://chime.aws/dialinnumbers/
> > > > > >>>>>>>>>>>>>> In-room video system: Ext: 62000, Meeting PIN:
> > > > > >>>>> 9272158344#
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> --
> > > > > >>>>>>>>>>>>>> *Chaitanya Prakash Bapat*
> > > > > >>>>>>>>>>>>>> *+1 (973) 953-6299*
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> [image: https://www.linkedin.com//in/chaibapat25]
> > > > > >>>>>>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> > > > > >>>>>>>>>>>>> https://www.facebook.com/chaibapat
> > > > > >>>>>>>>>>>>>> ]
> > > > > >>>>>>>>>>>>>> <https://www.facebook.com/chaibapchya>[image:
> > > > > >>>>>>>>>>>>>> https://twitter.com/ChaiBapchya] <
> > > > > >>>>>>>> https://twitter.com/ChaiBapchya
> > > > > >>>>>>>>>>>>>> [image:
> > > > > >>>>>>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> > > > > >>>>>>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> --
> > > > > >>>>>>>>>> *Chaitanya Prakash Bapat*
> > > > > >>>>>>>>>> *+1 (973) 953-6299*
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> [image: https://www.linkedin.com//in/chaibapat25]
> > > > > >>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> > > > > >>>>>>>>> https://www.facebook.com/chaibapat
> > > > > >>>>>>>>>> ]
> > > > > >>>>>>>>>> <https://www.facebook.com/chaibapchya>[image:
> > > > > >>>>>>>>>> https://twitter.com/ChaiBapchya] <
> > > > > >>>>> https://twitter.com/ChaiBapchya
> > > > > >>>>>>>>>> [image:
> > > > > >>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> > > > > >>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> --
> > > > > >>>>>>> Sandeep Krishnamurthy
> > > > > >>>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>>
> > > > > >>> --
> > > > > >>> *Chaitanya Prakash Bapat*
> > > > > >>> *+1 (973) 953-6299*
> > > > > >>>
> > > > > >>> [image: https://www.linkedin.com//in/chaibapat25]
> > > > > >>> <https://github.com/ChaiBapchya>[image:
> > > > > https://www.facebook.com/chaibapat
> > > > > >>> ]
> > > > > >>> <https://www.facebook.com/chaibapchya>[image:
> > > > > >>> https://twitter.com/ChaiBapchya] <
> > https://twitter.com/ChaiBapchya
> > > > > >[image:
> > > > > >>> https://www.linkedin.com//in/chaibapat25]
> > > > > >>> <https://www.linkedin.com//in/chaibapchya/>
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > *Chaitanya Prakash Bapat*
> > > *+1 (973) 953-6299*
> > >
> > > [image: https://www.linkedin.com//in/chaibapat25]
> > > <https://github.com/ChaiBapchya>[image:
> > https://www.facebook.com/chaibapat
> > > ]
> > > <https://www.facebook.com/chaibapchya>[image:
> > > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> > >[image:
> > > https://www.linkedin.com//in/chaibapat25]
> > > <https://www.linkedin.com//in/chaibapchya/>
> > >
> >
>
>
> --
> *Chaitanya Prakash Bapat*
> *+1 (973) 953-6299*
>
> [image: https://www.linkedin.com//in/chaibapat25]
> <https://github.com/ChaiBapchya>[image: https://www.facebook.com/chaibapat
> ]
> <https://www.facebook.com/chaibapchya>[image:
> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
> https://www.linkedin.com//in/chaibapat25]
> <https://www.linkedin.com//in/chaibapchya/>
>
Re: MXNet Bot Demo
Posted by Chaitanya Bapat <ch...@gmail.com>.
Hey Marco,
I thought currently every commit on PR and master triggers CI
because
a. github webhook points to Jenkins Server
b. GH Webhook events trigger builds on Jenkins for all commits to any
branch in apache/incubator-mxnet
may it be master/PR/non-PR
Reason:
Because all the 3 types of branches are discovered by Jenkins (non-PR
(including master) and PR)
Proposal: Remove GitHub WebHook to Jenkins and replace with GH Webhook to
Lambda
But after I remove the github webhook that points to Jenkins : *N**o commit
will trigger Jenkins build by default* (as Jenkins wont receive GH events)
Only those that Bot deems fit will be triggered (using Jenkins API invoked
by Lambda).
Hence its needed to handle that case of master merge.
Am I understanding this correctly?
On Wed, 18 Mar 2020 at 04:23, Marco de Abreu <ma...@gmail.com>
wrote:
> Thanks Chai, sounds good to me. Could you elaborate a bit on the point
> about triggering a CI run after the PR has been merged? We already to that
> automatically for the master, so what's the benefit to do it twice?
>
> -Marco
>
> Chaitanya Bapat <ch...@gmail.com> schrieb am Mi., 18. März 2020,
> 09:30:
>
> > Update:
> >
> > > we can ensure that all CI runs ran on the commit that will be merged
> > @Sam Skalicky <sa...@gmail.com> Branch Protection is added to
> public
> > MXNet repo. It ensures that for every PR to be merged, the CI passes. All
> > the jobs selected "required" jobs will have to be green for the PR to be
> > merged. Ofcourse, users with "Adminstrator" access can merge without it
> but
> > that's just a backdoor. It is the case now and will continue to be the
> case
> > with the inclusion of Bot.
> >
> > > easily verify that the CI has executed all runs on the commit that will
> > be merged
> > GitHub UI shows all the jobs and the status corresponding to it on every
> > commit. That should suffice. For the merged commits, Repo -> Commits ->
> > Commit ID (Status) can be tracked currently (only way that I know of).
> > Moreover, it is beyond the scope of this project (and possibly out of our
> > control since this is purely GitHub UI specific use-case).
> >
> > Thanks @przemyslaw for supporting the opt-in.
> >
> > Thanks everyone in the community for sharing concerns, voicing your
> opinion
> > and participating in the discussion.
> > Thanks to those who attended the demo last Friday.
> >
> > Action items from that discussion
> > 1. Handle master merge builds [Done]
> > Bot runs entire CI suite after the PR is merged and comments on the PR
> > about the same.
> > Design decision :
> > MXNet Bot comment about master merge build on the *merge commit vs PR*.
> > After the PR is merged, Bot runs entire CI and comments the result of CI
> > trigger on the PR (because it is easy to track on a PR rather than
> > commenting inside the merge commit)
> >
> > 2. Idempotent condition
> > In case of already running build, if an attempt is made to retrigger the
> > job then what should be the response
> > a. Not to re-trigger, let the ongoing build continue till completion
> > b. End the ongoing build and re-trigger
> > c. Let the ongoing build continue, re-trigger new build
> >
> > From resource saving point of view, *c* looks costly and a can be
> > avoided/optimized by B.
> > In case when a re-trigger was started "erroneously" then killing ongoing
> > build and re-trigger is a waste.
> > In case when ongoing build failed in one sub-part, then re-triggering is
> > justified.
> > Erroneous re-triggers would be less often while conscious re-triggers to
> > suppress failure is more common use-case. It looks like a safe assumption
> > to make given the trade-off.
> > [Open to debate]
> >
> > 3. Add security consideration [Use of secret manager, but without
> > auto-rotation due to Jenkins manual config requirement] [Done]
> > 4. New PR Instruction message by the Bot [Done]
> > Thanks to the suggestion of Leonard, supported by others. I've now added
> > the feature where the Bot comments a help message. [For reference -
> > https://github.com/ChaiBapchya/incubator-mxnet/pull/52]
> >
> > Barring the opt-in vs opt-out debate & idempotency, consensus was quickly
> > reached for the rest.
> >
> > In the coming days, I hope to roll-out this feature into Prod (public
> > MXNet) for all devs to use.
> >
> > Thanks,
> > Chai
> >
> > On Mon, 16 Mar 2020 at 11:57, Marco de Abreu <ma...@gmail.com>
> > wrote:
> >
> > > Well that's generally a problem with a deferred CI approach (CI is run
> at
> > > commit and not at merge time). This can either be solved through the
> > other
> > > proposal that's currently on dev@, by having a bot which does merges
> by
> > > having a global lock and a merge queue or by accepting the issue.
> Reality
> > > right now is that we're running that model where two PRs which are
> merged
> > > in parallel might break one another. One thing to consider though is
> that
> > > this breakage would have to be introduced in two separate parts since
> > > otherwise there'd be merge conflicts. I think we had that situation
> twice
> > > so far and the result was a quick revert, so I'd say that it's a
> problem
> > > that can happily be accepted. All other solutions basically require
> some
> > > form of single-threaded and globally locked solution which limits us in
> > > scalability. I'd recommend to just accept that risk and revert a PR in
> > case
> > > it actually had a conflict.
> > >
> > > -Marco
> > >
> > > On Mon, Mar 16, 2020 at 6:29 PM Skalicky, Sam
> <sskalic@amazon.com.invalid
> > >
> > > wrote:
> > >
> > > > We probably need some way to track which CI runs ran for which commit
> > > too,
> > > > that way we can ensure that all CI runs ran on the commit that will
> be
> > > > merged. Maybe the bot can comment with the commit hash when users
> > > command
> > > > it to do something. Although since users can trigger individual CI
> runs
> > > its
> > > > possible to have some commits run some CI runs but not others. We
> need
> > > some
> > > > way to easily verify that the CI has executed all runs on the commit
> > that
> > > > will be merged.
> > > >
> > > > Sam
> > > >
> > > > > On Mar 13, 2020, at 8:28 PM, Przemysław Trędak <ptrendx@apache.org
> >
> > > > wrote:
> > > > >
> > > > > CAUTION: This email originated from outside of the organization. Do
> > not
> > > > click links or open attachments unless you can confirm the sender and
> > > know
> > > > the content is safe.
> > > > >
> > > > >
> > > > >
> > > > > I personally like the idea of opt-in more than opt-out:
> > > > > - ultimately PR author wants the PR to be merged so they (or
> > committer
> > > > reviewing the PR) will trigger the CI
> > > > > - if it is easy to trigger the PR via the bot command then the
> amount
> > > of
> > > > work per PR should be less than with opt-out (since most of the
> commits
> > > > should then be marked as [skip ci] or something similar
> > > > >
> > > > > +1 to the bot making a comment on each new PR with its commands
> (and
> > > > also explaining, or at least giving links to the general PR process
> so
> > > new
> > > > PR authors are not lost). Maybe we could make the bot recognize if
> the
> > PR
> > > > author is new or existing contributor and offer advice based on that?
> > > > >
> > > > > Thanks
> > > > > Przemek
> > > > >
> > > > > On 2020/03/13 22:06:58, Marco de Abreu <ma...@gmail.com>
> > > wrote:
> > > > >> Hi,
> > > > >>
> > > > >> since it's no longer necessary to push a new commit to trigger CI,
> > it
> > > > will
> > > > >> already reduce the costs. But to me, requiring an action to enable
> > CI
> > > > after
> > > > >> a PR has been created initially, is a no go. User can opt out of
> CI,
> > > but
> > > > >> the default has to be CI being triggered automatically for every
> > > commit
> > > > >> unless specifically disabled by a participant. I'm also fine with
> > > > >> triggering certain additional jobs (think about running a nightly
> > job
> > > > upon
> > > > >> request for a PR) to require a manual step, but the PR validation
> > > > pipelines
> > > > >> have to run automatically. Every check that is marked as
> "Required"
> > in
> > > > >> GitHub has to be automatically kicked off.
> > > > >>
> > > > >> -Marco
> > > > >>
> > > > >> On Fri, Mar 13, 2020 at 9:50 PM Chaitanya Bapat <
> > chai.bapat@gmail.com
> > > >
> > > > >> wrote:
> > > > >>
> > > > >>> Firstly,
> > > > >>> Sorry I missed out on attaching the mail thread that was sent on
> > 12th
> > > > >>> February for notifying the community of the upcoming changes to
> the
> > > > MXNet
> > > > >>> CI
> > > > >>> For reference :
> > > > >>>
> > > > >>>
> > > >
> > >
> >
> https://lists.apache.org/thread.html/r09a6ab2803a996fc80e00fe39ed312fa4865e8805e08df847f1addad%40%3Cdev.mxnet.apache.org%3E
> > > > >>>
> > > > >>> Now to the questions,
> > > > >>>> Is it possible for re-triggering a single job to be abused?
> > > > >>> @Tao In the case when a user re-triggers a single job multiple
> > times,
> > > > that
> > > > >>> will be visible in the PR conversation thread. A committer, even
> > > after
> > > > he
> > > > >>> has approved the PR before, generally takes a look at the final
> > state
> > > > of
> > > > >>> the PR before merging. Would it be fair to assume the committer
> > could
> > > > take
> > > > >>> the multiple re-trigger of a single job into account before
> > merging?
> > > > The
> > > > >>> committer then has the option to invoke `@mxnet-bot run ci [all]
> `
> > to
> > > > >>> trigger the entire build pipeline one last to counter the abuse.
> > This
> > > > is
> > > > >>> aligned with what @Leonard said.
> > > > >>>
> > > > >>> @Sandeep Thanks a lot for collecting and sharing valuable data.
> I'd
> > > > concur
> > > > >>> with the opinion that given the existing things committers & PR
> > > Authors
> > > > >>> take care of, invoking CI shouldn't be that big of an additional
> > > > burden.
> > > > >>>
> > > > >>> @Marco With the opt-out, the onus remains on the PR Author. It
> > > doesn't
> > > > help
> > > > >>> reduce the resource usage. Hence, it was suggested to switch to
> > > > >>> opt-in. @Leo's suggestion for proactive commenting on the part of
> > bot
> > > > makes
> > > > >>> sense and is doable.
> > > > >>>
> > > > >>> Default : opt-out and User initiated opt-in (with addressing
> Leo's
> > > fix
> > > > for
> > > > >>> the usability issue you correctly pointed out )
> > > > >>> @Marco How does this sound to you?
> > > > >>>
> > > > >>> Again, thank you all for chiming in and voicing your opinion.
> > > > Appreciate
> > > > >>> it.
> > > > >>> We can take ahead these discussions in today's demo meeting.
> > [Design
> > > > Doc
> > > > >>> <https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> >]
> > > > [Demo
> > > > >>> Video <https://www.youtube.com/watch?v=gfOGwZId8aU>]
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Chai
> > > > >>>
> > > > >>> On Fri, 13 Mar 2020 at 12:34, Marco de Abreu <
> > > marco.g.abreu@gmail.com>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> I'd recommend that the bot makes an initial comment when a PR
> gets
> > > > opened
> > > > >>>> and informs the users of its commands. It then tells the user
> the
> > > > commend
> > > > >>>> to opt out of CI.
> > > > >>>>
> > > > >>>> -Marco
> > > > >>>>
> > > > >>>> Lausen, Leonard <la...@amazon.com.invalid> schrieb am Fr., 13.
> > > März
> > > > >>> 2020,
> > > > >>>> 20:27:
> > > > >>>>
> > > > >>>>> On opt-out: People may be unaware of opt-out would not use it.
> > > There
> > > > is
> > > > >>>> no
> > > > >>>>> incentive to use opt-out, as the PR author doesn't pay any
> money
> > > for
> > > > CI
> > > > >>>>> run.
> > > > >>>>>
> > > > >>>>> I agree with Marco though that opt-in alone may cause usability
> > > > issues,
> > > > >>>> as
> > > > >>>>> contributors may not be aware of how to trigger the CI.
> > > > >>>>> One solution is that the bot proactively comments on the PR and
> > > > reminds
> > > > >>>> the
> > > > >>>>> author to trigger running CI once the author deems the PR
> ready.
> > > > >>>>>
> > > > >>>>> But even if we choose opt-out, the bot will still add a lot of
> > > value,
> > > > >>> as
> > > > >>>> PR
> > > > >>>>> authors can retrigger single jobs that have failed due to
> > > flakiness.
> > > > >>>>>
> > > > >>>>>> Is it possible for re-triggering a single job to be abused?
> For
> > > > >>>> example,
> > > > >>>>>> the author spends two days re-triggering a flaky job to make
> it
> > > > pass.
> > > > >>>>>
> > > > >>>>> Yes, this is possible. I suggest the committer who likes to
> > merge a
> > > > PR
> > > > >>>>> needs to
> > > > >>>>> make a good judgement here if a PR is abusing the feature, and
> if
> > > so,
> > > > >>>>> retrigger
> > > > >>>>> all CI runs.
> > > > >>>>>
> > > > >>>>> Best regards
> > > > >>>>> Leonard
> > > > >>>>>
> > > > >>>>> On Fri, 2020-03-13 at 08:07 +0100, Marco de Abreu wrote:
> > > > >>>>>> Thanks for the data Sandeep. In these cases it sounds like it
> > > would
> > > > >>>> have
> > > > >>>>>> rather been better when people explicitly turned off CI in
> that
> > > > case.
> > > > >>>>>> What's the argument against an opt-out instead of an opt-in?
> > > > >>>>>>
> > > > >>>>>> My intention is that I consider it quite cumbersome to make
> it a
> > > > >>>>> *required*
> > > > >>>>>> step to always trigger CI manually, even if just submitting a
> > > small
> > > > >>> PR.
> > > > >>>>> I'd
> > > > >>>>>> rather see people explicitly turning off CI if they wouldn't
> > like
> > > to
> > > > >>>> use
> > > > >>>>> it
> > > > >>>>>> - and there's also the "draft" stage for a PR which some
> > > > contributors
> > > > >>>> are
> > > > >>>>>> using.
> > > > >>>>>>
> > > > >>>>>> With regards to WIP and do not review: I think these are use
> > cases
> > > > >>>> where
> > > > >>>>>> you want CI feedback, as otherwise you wouldn't have opened
> the
> > > PR.
> > > > >>> If
> > > > >>>>> you
> > > > >>>>>> don't want human feedback and neither machine feedback, why
> open
> > > the
> > > > >>> PR
> > > > >>>>> at
> > > > >>>>>> all?
> > > > >>>>>>
> > > > >>>>>> -Marco
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> sandeep krishnamurthy <sa...@gmail.com> schrieb
> am
> > > Fr.,
> > > > >>>> 13.
> > > > >>>>>> März 2020, 05:24:
> > > > >>>>>>
> > > > >>>>>>> I tried to gather some data for us to discuss this topic in
> > this
> > > > >>>>> thread. I
> > > > >>>>>>> tried to count number of un-necessary builds by looking at
> most
> > > > >>>> recent
> > > > >>>>> (as
> > > > >>>>>>> of 12, March 9 PM PST) 50 PRs merged to master and 50 PRs.
> > > > >>>>>>> Identifying un-necessary builds is bit subjective. I tried to
> > be
> > > > >>> more
> > > > >>>>>>> conservative where I didn't count a build as un-necessary if
> I
> > > was
> > > > >>> in
> > > > >>>>>>> doubt. Hence, I was not able to automate, but I made an
> effort
> > to
> > > > >>> go
> > > > >>>>>>> through PRs manually and use below criteria to identify
> > > > >>> un-necessary
> > > > >>>>>>> commits triggering the builds.
> > > > >>>>>>>
> > > > >>>>>>> 1. Explicitly marked as WIP / do not review PR
> > > > >>>>>>> 2. Incremental WIP commit and finally commenting a commit
> > > > >>> “trigger
> > > > >>>>> CI”
> > > > >>>>>>> 3. Multiple commits to address all comments from single
> > review.
> > > > >>>>> This is
> > > > >>>>>>> assuming we see a comment, address them, commit, next the
> > > > >>>> following
> > > > >>>>>>> comment
> > > > >>>>>>> 4. Sequence of documentation only changes
> > > > >>>>>>>
> > > > >>>>>>> I found there were around 42 avoidable builds from most
> recent
> > 50
> > > > >>>>> merged
> > > > >>>>>>> PRs and around 86 builds from recent 50 open PRs.
> > > > >>>>>>>
> > > > >>>>>>> I synced up with other contributors (Joe Evans, Chai) from
> > Amazon
> > > > >>> who
> > > > >>>>> is
> > > > >>>>>>> contributing to MXNet CI system. I was told that on an
> average
> > it
> > > > >>>> costs
> > > > >>>>>>> around $84 per build and on an average 6 commits per merged
> PR
> > > (for
> > > > >>>>> year
> > > > >>>>>>> 2019). Going by that, it is approximately 1/6 builds are
> > > avoidable.
> > > > >>>>> [100 /
> > > > >>>>>>> 300 + 300 ]
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> Usability should be top most priority. But, since either a
> > > reviewer
> > > > >>>> or
> > > > >>>>> pr
> > > > >>>>>>> author can trigger the bot, is it really a hurdle for pr
> author
> > > or
> > > > >>>>> reviewer
> > > > >>>>>>> to call a bot to trigger CI? Given that PR author and
> reviewer
> > is
> > > > >>>>> already
> > > > >>>>>>> actively commenting various details such as - PR description,
> > > > >>> review
> > > > >>>>>>> comments and responses, adding labels etc.
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> Me too curious to know the behavior for Tao's above use case.
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> Best,
> > > > >>>>>>>
> > > > >>>>>>> Sandeep
> > > > >>>>>>>
> > > > >>>>>>> On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <mu...@gmail.com>
> > > wrote:
> > > > >>>>>>>
> > > > >>>>>>>> Is it possible for re-triggering a single job to be abused?
> > For
> > > > >>>>> example,
> > > > >>>>>>>> the author spends two days re-triggering a flaky job to make
> > it
> > > > >>>>> pass. But
> > > > >>>>>>>> other jobs which have passed the validation may be broken by
> > > > >>> other
> > > > >>>>>>> commits
> > > > >>>>>>>> during the two day without being noticed. And finally the PR
> > is
> > > > >>>>> merged
> > > > >>>>>>> with
> > > > >>>>>>>> underlying problems.
> > > > >>>>>>>>
> > > > >>>>>>>> On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <
> > > > >>>>> marco.g.abreu@gmail.com>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> In the end it only comes down to money, considering that
> the
> > > > >>>>> system is
> > > > >>>>>>>> auto
> > > > >>>>>>>>> scaling, making the execution time constant.
> > > > >>>>>>>>>
> > > > >>>>>>>>> If we're trading money for usability, I certainly would
> > prefer
> > > > >>>>>>> usability.
> > > > >>>>>>>>> I'd rather recommend to spend time on parallelizing test
> > > > >>>> execution
> > > > >>>>> or
> > > > >>>>>>>>> getting rid of integration tests in the PR stage instead
> > > > >>> reducing
> > > > >>>>> the
> > > > >>>>>>>> costs
> > > > >>>>>>>>> by making people not use it. But taking a step back to
> > > > >>> requiring
> > > > >>>>> people
> > > > >>>>>>>> to
> > > > >>>>>>>>> manually trigger CI again doesn't feel right.
> > > > >>>>>>>>>
> > > > >>>>>>>>> I'm happy to see that bot deployed, but I do not agree with
> > > > >>>>> removing
> > > > >>>>>>> the
> > > > >>>>>>>>> auto trigger functionality for new commits.
> > > > >>>>>>>>>
> > > > >>>>>>>>> -Marco
> > > > >>>>>>>>>
> > > > >>>>>>>>> Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 12.
> > > > >>> März
> > > > >>>>> 2020,
> > > > >>>>>>>>> 22:47:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> @Marco Thanks for pointing that out.
> > > > >>>>>>>>>> Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM - 3:30 PM
> in
> > > > >>>>>>>> (UTC-08:00)
> > > > >>>>>>>>>> Pacific Time (US & Canada).
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> When do we expect this bot to be deployed?
> > > > >>>>>>>>>> @Lin If all goes well in the next week I can deploy it to
> > > > >>>> public
> > > > >>>>>>> Apache
> > > > >>>>>>>>>> (provided I get permissions from Apache Infra)
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> @Marco Thanks for your feedback.
> > > > >>>>>>>>>>> CI system has to support the community without requiring
> > > > >>>>> people to
> > > > >>>>>>>>>> constantly shepherd every single run
> > > > >>>>>>>>>> We have data for the number of times CI was triggered
> > > > >>>>> unnecessarily
> > > > >>>>>>>> which
> > > > >>>>>>>>>> includes
> > > > >>>>>>>>>> - Entire build triggered instead of specific build
> > > > >>>>>>>>>> - CI triggered when PR is still work in progress or not
> yet
> > > > >>>> ready
> > > > >>>>>>> (say
> > > > >>>>>>>> -
> > > > >>>>>>>>>> intermediate commits)
> > > > >>>>>>>>>> At the end its a trade-off
> > > > >>>>>>>>>> Money, Resources, Time to build for each and every commit
> vs
> > > > >>>>> Pain of
> > > > >>>>>>>>>> triggering builds
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Scan trigger plugin would poll SCM. Can we use plugin at
> > > > >>>>> scale?
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> 1. I haven't tested it on scale. But I think with the
> > current
> > > > >>>>> scale
> > > > >>>>>>> of
> > > > >>>>>>>>>> MXNet repo (191 open PRs i.e. checking for changes to 191
> > > > >>>>> branches -
> > > > >>>>>>> It
> > > > >>>>>>>>>> should be manageable)
> > > > >>>>>>>>>> 2. What's the purpose of the plugin? tldr; Branch
> discovery
> > > > >>> or
> > > > >>>>> branch
> > > > >>>>>>>>>> indexing.
> > > > >>>>>>>>>> Scan trigger plugin comes into the picture only once per
> PR
> > > > >>> per
> > > > >>>>> job
> > > > >>>>>>>>> (i.e. 8
> > > > >>>>>>>>>> times per PR for 8 jobs). It is basically done when a new
> PR
> > > > >>> is
> > > > >>>>> made
> > > > >>>>>>>> and
> > > > >>>>>>>>>> the job (say unix-cpu hasn't discovered the new PR branch
> > > > >>> yet).
> > > > >>>>>>> That's
> > > > >>>>>>>>> it.
> > > > >>>>>>>>>> So it shouldn't be a problem for public MXNet repo.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Thanks,
> > > > >>>>>>>>>> Chai
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <
> > > > >>>>>>> marco.g.abreu@gmail.com>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Btw you forgot to set a date and time for the metting
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu <
> > > > >>>>>>>>> marco.g.abreu@gmail.com
> > > > >>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Thanks Chai, I generally like the idea of the bot. But
> > > > >>> I'm
> > > > >>>>> not a
> > > > >>>>>>>>>>> supporter
> > > > >>>>>>>>>>>> of the idea to disable any automatic triggering
> > > > >>> (disabling
> > > > >>>>> the
> > > > >>>>>>>>> webhook
> > > > >>>>>>>>>> is
> > > > >>>>>>>>>>>> also not an option, considering that this will disable
> > > > >>>> master
> > > > >>>>>>>>>> triggers).
> > > > >>>>>>>>>>>> The CI system has to support the community without
> > > > >>>> requiring
> > > > >>>>>>> people
> > > > >>>>>>>>> to
> > > > >>>>>>>>>>>> constantly shepherd every single run. Disabling
> automatic
> > > > >>>>>>>> triggering
> > > > >>>>>>>>>>> seems
> > > > >>>>>>>>>>>> like a step back to me.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Instead, I'd recommend that CI gets triggered upon every
> > > > >>>>> commit
> > > > >>>>>>> as
> > > > >>>>>>>>>> usual,
> > > > >>>>>>>>>>>> but people have the possibility to call a "command"
> (i.e.
> > > > >>>>> make a
> > > > >>>>>>>>>> message
> > > > >>>>>>>>>>>> which results in the bot setting a label) to disable CI
> > > > >>>> until
> > > > >>>>>>> they
> > > > >>>>>>>>>> revoke
> > > > >>>>>>>>>>>> it. But the standard should still be that a new commit
> > > > >>>>> triggers a
> > > > >>>>>>>> new
> > > > >>>>>>>>>> CI
> > > > >>>>>>>>>>>> run.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>> https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
> > > > >>>>>>>> seems
> > > > >>>>>>>>>> like
> > > > >>>>>>>>>>>> this would poll SCM. This will incur high quota
> > > > >>>>> restrictions. Are
> > > > >>>>>>>> you
> > > > >>>>>>>>>>> sure
> > > > >>>>>>>>>>>> that you can use that plugin at scale?
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> -Marco
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <
> > > > >>>>> apeforest@gmail.com>
> > > > >>>>>>>>> wrote:
> > > > >>>>>>>>>>>>> Chai,
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Awesome work. When do we expect this bot to be
> > > > >>> deployed?
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Lin
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <
> > > > >>>>>>>>> chai.bapat@gmail.com
> > > > >>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Hello MXNet community,
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> I have built an MXNet Bot <
> > > > >>>> https://github.com/mxnet-bot>
> > > > >>>>> that
> > > > >>>>>>>>>> allows
> > > > >>>>>>>>>>> PR
> > > > >>>>>>>>>>>>>> Authors, Committers and Jenkins Admins to trigger CI
> > > > >>>>> manually.
> > > > >>>>>>>>>>>>>> It handles 2 problems
> > > > >>>>>>>>>>>>>> 1. Manual CI trigger instead of existing automated CI
> > > > >>>>> trigger
> > > > >>>>>>>>>>>>>> 2. Gives permissions to PR Authors (in addition to
> > > > >>>> MXNet
> > > > >>>>>>>>> Committers
> > > > >>>>>>>>>>> and
> > > > >>>>>>>>>>>>>> Jenkins Admins)
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Design Doc :
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>
> https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > > > >>>>>>>>>>>>>> I urge you all to attend the demonstration meeting
> > > > >>> and
> > > > >>>>> lend
> > > > >>>>>>> your
> > > > >>>>>>>>>> views
> > > > >>>>>>>>>>>>> on
> > > > >>>>>>>>>>>>>> the same.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Thank you,
> > > > >>>>>>>>>>>>>> Chai
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> *Meeting Details*:
> > > > >>>>>>>>>>>>>> ==============Conference Bridge
> > > > >>>> Information==============
> > > > >>>>>>>>>>>>>> You have been invited to an online meeting, powered
> > > > >>> by
> > > > >>>>> Amazon
> > > > >>>>>>>>> Chime.
> > > > >>>>>>>>>>>>>> *Chime meeting ID*: *9272158344*
> > > > >>>>>>>>>>>>>> Join via Chime clients (manually): Select 'Meetings >
> > > > >>>>> Join a
> > > > >>>>>>>>>> Meeting',
> > > > >>>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>> enter 9272158344
> > > > >>>>>>>>>>>>>> Join via Chime clients (auto-call): If you invite
> > > > >>>>> auto-call as
> > > > >>>>>>>>>>> attendee,
> > > > >>>>>>>>>>>>>> Chime will call you when the meeting starts, select
> > > > >>>>> 'Answer'
> > > > >>>>>>>>>>>>>> *Join via browser screen share*:
> > > > >>>>> https://chime.aws/9272158344
> > > > >>>>>>>>>>>>>> *Join via phone* (US): +1-929-432-4463,,,9272158344#
> > > > >>>>>>>>>>>>>> *Join via phone (US toll-free)*:
> > > > >>>>> +1-855-552-4463,,,9272158344#
> > > > >>>>>>>>>>>>>> International dial-in:
> > > > >>>> https://chime.aws/dialinnumbers/
> > > > >>>>>>>>>>>>>> In-room video system: Ext: 62000, Meeting PIN:
> > > > >>>>> 9272158344#
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> --
> > > > >>>>>>>>>>>>>> *Chaitanya Prakash Bapat*
> > > > >>>>>>>>>>>>>> *+1 (973) 953-6299*
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> [image: https://www.linkedin.com//in/chaibapat25]
> > > > >>>>>>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> > > > >>>>>>>>>>>>> https://www.facebook.com/chaibapat
> > > > >>>>>>>>>>>>>> ]
> > > > >>>>>>>>>>>>>> <https://www.facebook.com/chaibapchya>[image:
> > > > >>>>>>>>>>>>>> https://twitter.com/ChaiBapchya] <
> > > > >>>>>>>> https://twitter.com/ChaiBapchya
> > > > >>>>>>>>>>>>>> [image:
> > > > >>>>>>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> > > > >>>>>>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> --
> > > > >>>>>>>>>> *Chaitanya Prakash Bapat*
> > > > >>>>>>>>>> *+1 (973) 953-6299*
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> [image: https://www.linkedin.com//in/chaibapat25]
> > > > >>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> > > > >>>>>>>>> https://www.facebook.com/chaibapat
> > > > >>>>>>>>>> ]
> > > > >>>>>>>>>> <https://www.facebook.com/chaibapchya>[image:
> > > > >>>>>>>>>> https://twitter.com/ChaiBapchya] <
> > > > >>>>> https://twitter.com/ChaiBapchya
> > > > >>>>>>>>>> [image:
> > > > >>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> > > > >>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
> > > > >>>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> --
> > > > >>>>>>> Sandeep Krishnamurthy
> > > > >>>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> *Chaitanya Prakash Bapat*
> > > > >>> *+1 (973) 953-6299*
> > > > >>>
> > > > >>> [image: https://www.linkedin.com//in/chaibapat25]
> > > > >>> <https://github.com/ChaiBapchya>[image:
> > > > https://www.facebook.com/chaibapat
> > > > >>> ]
> > > > >>> <https://www.facebook.com/chaibapchya>[image:
> > > > >>> https://twitter.com/ChaiBapchya] <
> https://twitter.com/ChaiBapchya
> > > > >[image:
> > > > >>> https://www.linkedin.com//in/chaibapat25]
> > > > >>> <https://www.linkedin.com//in/chaibapchya/>
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
> >
> > --
> > *Chaitanya Prakash Bapat*
> > *+1 (973) 953-6299*
> >
> > [image: https://www.linkedin.com//in/chaibapat25]
> > <https://github.com/ChaiBapchya>[image:
> https://www.facebook.com/chaibapat
> > ]
> > <https://www.facebook.com/chaibapchya>[image:
> > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> >[image:
> > https://www.linkedin.com//in/chaibapat25]
> > <https://www.linkedin.com//in/chaibapchya/>
> >
>
--
*Chaitanya Prakash Bapat*
*+1 (973) 953-6299*
[image: https://www.linkedin.com//in/chaibapat25]
<https://github.com/ChaiBapchya>[image: https://www.facebook.com/chaibapat]
<https://www.facebook.com/chaibapchya>[image:
https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
https://www.linkedin.com//in/chaibapat25]
<https://www.linkedin.com//in/chaibapchya/>
Re: MXNet Bot Demo
Posted by Marco de Abreu <ma...@gmail.com>.
Thanks Chai, sounds good to me. Could you elaborate a bit on the point
about triggering a CI run after the PR has been merged? We already to that
automatically for the master, so what's the benefit to do it twice?
-Marco
Chaitanya Bapat <ch...@gmail.com> schrieb am Mi., 18. März 2020, 09:30:
> Update:
>
> > we can ensure that all CI runs ran on the commit that will be merged
> @Sam Skalicky <sa...@gmail.com> Branch Protection is added to public
> MXNet repo. It ensures that for every PR to be merged, the CI passes. All
> the jobs selected "required" jobs will have to be green for the PR to be
> merged. Ofcourse, users with "Adminstrator" access can merge without it but
> that's just a backdoor. It is the case now and will continue to be the case
> with the inclusion of Bot.
>
> > easily verify that the CI has executed all runs on the commit that will
> be merged
> GitHub UI shows all the jobs and the status corresponding to it on every
> commit. That should suffice. For the merged commits, Repo -> Commits ->
> Commit ID (Status) can be tracked currently (only way that I know of).
> Moreover, it is beyond the scope of this project (and possibly out of our
> control since this is purely GitHub UI specific use-case).
>
> Thanks @przemyslaw for supporting the opt-in.
>
> Thanks everyone in the community for sharing concerns, voicing your opinion
> and participating in the discussion.
> Thanks to those who attended the demo last Friday.
>
> Action items from that discussion
> 1. Handle master merge builds [Done]
> Bot runs entire CI suite after the PR is merged and comments on the PR
> about the same.
> Design decision :
> MXNet Bot comment about master merge build on the *merge commit vs PR*.
> After the PR is merged, Bot runs entire CI and comments the result of CI
> trigger on the PR (because it is easy to track on a PR rather than
> commenting inside the merge commit)
>
> 2. Idempotent condition
> In case of already running build, if an attempt is made to retrigger the
> job then what should be the response
> a. Not to re-trigger, let the ongoing build continue till completion
> b. End the ongoing build and re-trigger
> c. Let the ongoing build continue, re-trigger new build
>
> From resource saving point of view, *c* looks costly and a can be
> avoided/optimized by B.
> In case when a re-trigger was started "erroneously" then killing ongoing
> build and re-trigger is a waste.
> In case when ongoing build failed in one sub-part, then re-triggering is
> justified.
> Erroneous re-triggers would be less often while conscious re-triggers to
> suppress failure is more common use-case. It looks like a safe assumption
> to make given the trade-off.
> [Open to debate]
>
> 3. Add security consideration [Use of secret manager, but without
> auto-rotation due to Jenkins manual config requirement] [Done]
> 4. New PR Instruction message by the Bot [Done]
> Thanks to the suggestion of Leonard, supported by others. I've now added
> the feature where the Bot comments a help message. [For reference -
> https://github.com/ChaiBapchya/incubator-mxnet/pull/52]
>
> Barring the opt-in vs opt-out debate & idempotency, consensus was quickly
> reached for the rest.
>
> In the coming days, I hope to roll-out this feature into Prod (public
> MXNet) for all devs to use.
>
> Thanks,
> Chai
>
> On Mon, 16 Mar 2020 at 11:57, Marco de Abreu <ma...@gmail.com>
> wrote:
>
> > Well that's generally a problem with a deferred CI approach (CI is run at
> > commit and not at merge time). This can either be solved through the
> other
> > proposal that's currently on dev@, by having a bot which does merges by
> > having a global lock and a merge queue or by accepting the issue. Reality
> > right now is that we're running that model where two PRs which are merged
> > in parallel might break one another. One thing to consider though is that
> > this breakage would have to be introduced in two separate parts since
> > otherwise there'd be merge conflicts. I think we had that situation twice
> > so far and the result was a quick revert, so I'd say that it's a problem
> > that can happily be accepted. All other solutions basically require some
> > form of single-threaded and globally locked solution which limits us in
> > scalability. I'd recommend to just accept that risk and revert a PR in
> case
> > it actually had a conflict.
> >
> > -Marco
> >
> > On Mon, Mar 16, 2020 at 6:29 PM Skalicky, Sam <sskalic@amazon.com.invalid
> >
> > wrote:
> >
> > > We probably need some way to track which CI runs ran for which commit
> > too,
> > > that way we can ensure that all CI runs ran on the commit that will be
> > > merged. Maybe the bot can comment with the commit hash when users
> > command
> > > it to do something. Although since users can trigger individual CI runs
> > its
> > > possible to have some commits run some CI runs but not others. We need
> > some
> > > way to easily verify that the CI has executed all runs on the commit
> that
> > > will be merged.
> > >
> > > Sam
> > >
> > > > On Mar 13, 2020, at 8:28 PM, Przemysław Trędak <pt...@apache.org>
> > > wrote:
> > > >
> > > > CAUTION: This email originated from outside of the organization. Do
> not
> > > click links or open attachments unless you can confirm the sender and
> > know
> > > the content is safe.
> > > >
> > > >
> > > >
> > > > I personally like the idea of opt-in more than opt-out:
> > > > - ultimately PR author wants the PR to be merged so they (or
> committer
> > > reviewing the PR) will trigger the CI
> > > > - if it is easy to trigger the PR via the bot command then the amount
> > of
> > > work per PR should be less than with opt-out (since most of the commits
> > > should then be marked as [skip ci] or something similar
> > > >
> > > > +1 to the bot making a comment on each new PR with its commands (and
> > > also explaining, or at least giving links to the general PR process so
> > new
> > > PR authors are not lost). Maybe we could make the bot recognize if the
> PR
> > > author is new or existing contributor and offer advice based on that?
> > > >
> > > > Thanks
> > > > Przemek
> > > >
> > > > On 2020/03/13 22:06:58, Marco de Abreu <ma...@gmail.com>
> > wrote:
> > > >> Hi,
> > > >>
> > > >> since it's no longer necessary to push a new commit to trigger CI,
> it
> > > will
> > > >> already reduce the costs. But to me, requiring an action to enable
> CI
> > > after
> > > >> a PR has been created initially, is a no go. User can opt out of CI,
> > but
> > > >> the default has to be CI being triggered automatically for every
> > commit
> > > >> unless specifically disabled by a participant. I'm also fine with
> > > >> triggering certain additional jobs (think about running a nightly
> job
> > > upon
> > > >> request for a PR) to require a manual step, but the PR validation
> > > pipelines
> > > >> have to run automatically. Every check that is marked as "Required"
> in
> > > >> GitHub has to be automatically kicked off.
> > > >>
> > > >> -Marco
> > > >>
> > > >> On Fri, Mar 13, 2020 at 9:50 PM Chaitanya Bapat <
> chai.bapat@gmail.com
> > >
> > > >> wrote:
> > > >>
> > > >>> Firstly,
> > > >>> Sorry I missed out on attaching the mail thread that was sent on
> 12th
> > > >>> February for notifying the community of the upcoming changes to the
> > > MXNet
> > > >>> CI
> > > >>> For reference :
> > > >>>
> > > >>>
> > >
> >
> https://lists.apache.org/thread.html/r09a6ab2803a996fc80e00fe39ed312fa4865e8805e08df847f1addad%40%3Cdev.mxnet.apache.org%3E
> > > >>>
> > > >>> Now to the questions,
> > > >>>> Is it possible for re-triggering a single job to be abused?
> > > >>> @Tao In the case when a user re-triggers a single job multiple
> times,
> > > that
> > > >>> will be visible in the PR conversation thread. A committer, even
> > after
> > > he
> > > >>> has approved the PR before, generally takes a look at the final
> state
> > > of
> > > >>> the PR before merging. Would it be fair to assume the committer
> could
> > > take
> > > >>> the multiple re-trigger of a single job into account before
> merging?
> > > The
> > > >>> committer then has the option to invoke `@mxnet-bot run ci [all] `
> to
> > > >>> trigger the entire build pipeline one last to counter the abuse.
> This
> > > is
> > > >>> aligned with what @Leonard said.
> > > >>>
> > > >>> @Sandeep Thanks a lot for collecting and sharing valuable data. I'd
> > > concur
> > > >>> with the opinion that given the existing things committers & PR
> > Authors
> > > >>> take care of, invoking CI shouldn't be that big of an additional
> > > burden.
> > > >>>
> > > >>> @Marco With the opt-out, the onus remains on the PR Author. It
> > doesn't
> > > help
> > > >>> reduce the resource usage. Hence, it was suggested to switch to
> > > >>> opt-in. @Leo's suggestion for proactive commenting on the part of
> bot
> > > makes
> > > >>> sense and is doable.
> > > >>>
> > > >>> Default : opt-out and User initiated opt-in (with addressing Leo's
> > fix
> > > for
> > > >>> the usability issue you correctly pointed out )
> > > >>> @Marco How does this sound to you?
> > > >>>
> > > >>> Again, thank you all for chiming in and voicing your opinion.
> > > Appreciate
> > > >>> it.
> > > >>> We can take ahead these discussions in today's demo meeting.
> [Design
> > > Doc
> > > >>> <https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot>]
> > > [Demo
> > > >>> Video <https://www.youtube.com/watch?v=gfOGwZId8aU>]
> > > >>>
> > > >>> Thanks,
> > > >>> Chai
> > > >>>
> > > >>> On Fri, 13 Mar 2020 at 12:34, Marco de Abreu <
> > marco.g.abreu@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>>> I'd recommend that the bot makes an initial comment when a PR gets
> > > opened
> > > >>>> and informs the users of its commands. It then tells the user the
> > > commend
> > > >>>> to opt out of CI.
> > > >>>>
> > > >>>> -Marco
> > > >>>>
> > > >>>> Lausen, Leonard <la...@amazon.com.invalid> schrieb am Fr., 13.
> > März
> > > >>> 2020,
> > > >>>> 20:27:
> > > >>>>
> > > >>>>> On opt-out: People may be unaware of opt-out would not use it.
> > There
> > > is
> > > >>>> no
> > > >>>>> incentive to use opt-out, as the PR author doesn't pay any money
> > for
> > > CI
> > > >>>>> run.
> > > >>>>>
> > > >>>>> I agree with Marco though that opt-in alone may cause usability
> > > issues,
> > > >>>> as
> > > >>>>> contributors may not be aware of how to trigger the CI.
> > > >>>>> One solution is that the bot proactively comments on the PR and
> > > reminds
> > > >>>> the
> > > >>>>> author to trigger running CI once the author deems the PR ready.
> > > >>>>>
> > > >>>>> But even if we choose opt-out, the bot will still add a lot of
> > value,
> > > >>> as
> > > >>>> PR
> > > >>>>> authors can retrigger single jobs that have failed due to
> > flakiness.
> > > >>>>>
> > > >>>>>> Is it possible for re-triggering a single job to be abused? For
> > > >>>> example,
> > > >>>>>> the author spends two days re-triggering a flaky job to make it
> > > pass.
> > > >>>>>
> > > >>>>> Yes, this is possible. I suggest the committer who likes to
> merge a
> > > PR
> > > >>>>> needs to
> > > >>>>> make a good judgement here if a PR is abusing the feature, and if
> > so,
> > > >>>>> retrigger
> > > >>>>> all CI runs.
> > > >>>>>
> > > >>>>> Best regards
> > > >>>>> Leonard
> > > >>>>>
> > > >>>>> On Fri, 2020-03-13 at 08:07 +0100, Marco de Abreu wrote:
> > > >>>>>> Thanks for the data Sandeep. In these cases it sounds like it
> > would
> > > >>>> have
> > > >>>>>> rather been better when people explicitly turned off CI in that
> > > case.
> > > >>>>>> What's the argument against an opt-out instead of an opt-in?
> > > >>>>>>
> > > >>>>>> My intention is that I consider it quite cumbersome to make it a
> > > >>>>> *required*
> > > >>>>>> step to always trigger CI manually, even if just submitting a
> > small
> > > >>> PR.
> > > >>>>> I'd
> > > >>>>>> rather see people explicitly turning off CI if they wouldn't
> like
> > to
> > > >>>> use
> > > >>>>> it
> > > >>>>>> - and there's also the "draft" stage for a PR which some
> > > contributors
> > > >>>> are
> > > >>>>>> using.
> > > >>>>>>
> > > >>>>>> With regards to WIP and do not review: I think these are use
> cases
> > > >>>> where
> > > >>>>>> you want CI feedback, as otherwise you wouldn't have opened the
> > PR.
> > > >>> If
> > > >>>>> you
> > > >>>>>> don't want human feedback and neither machine feedback, why open
> > the
> > > >>> PR
> > > >>>>> at
> > > >>>>>> all?
> > > >>>>>>
> > > >>>>>> -Marco
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> sandeep krishnamurthy <sa...@gmail.com> schrieb am
> > Fr.,
> > > >>>> 13.
> > > >>>>>> März 2020, 05:24:
> > > >>>>>>
> > > >>>>>>> I tried to gather some data for us to discuss this topic in
> this
> > > >>>>> thread. I
> > > >>>>>>> tried to count number of un-necessary builds by looking at most
> > > >>>> recent
> > > >>>>> (as
> > > >>>>>>> of 12, March 9 PM PST) 50 PRs merged to master and 50 PRs.
> > > >>>>>>> Identifying un-necessary builds is bit subjective. I tried to
> be
> > > >>> more
> > > >>>>>>> conservative where I didn't count a build as un-necessary if I
> > was
> > > >>> in
> > > >>>>>>> doubt. Hence, I was not able to automate, but I made an effort
> to
> > > >>> go
> > > >>>>>>> through PRs manually and use below criteria to identify
> > > >>> un-necessary
> > > >>>>>>> commits triggering the builds.
> > > >>>>>>>
> > > >>>>>>> 1. Explicitly marked as WIP / do not review PR
> > > >>>>>>> 2. Incremental WIP commit and finally commenting a commit
> > > >>> “trigger
> > > >>>>> CI”
> > > >>>>>>> 3. Multiple commits to address all comments from single
> review.
> > > >>>>> This is
> > > >>>>>>> assuming we see a comment, address them, commit, next the
> > > >>>> following
> > > >>>>>>> comment
> > > >>>>>>> 4. Sequence of documentation only changes
> > > >>>>>>>
> > > >>>>>>> I found there were around 42 avoidable builds from most recent
> 50
> > > >>>>> merged
> > > >>>>>>> PRs and around 86 builds from recent 50 open PRs.
> > > >>>>>>>
> > > >>>>>>> I synced up with other contributors (Joe Evans, Chai) from
> Amazon
> > > >>> who
> > > >>>>> is
> > > >>>>>>> contributing to MXNet CI system. I was told that on an average
> it
> > > >>>> costs
> > > >>>>>>> around $84 per build and on an average 6 commits per merged PR
> > (for
> > > >>>>> year
> > > >>>>>>> 2019). Going by that, it is approximately 1/6 builds are
> > avoidable.
> > > >>>>> [100 /
> > > >>>>>>> 300 + 300 ]
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Usability should be top most priority. But, since either a
> > reviewer
> > > >>>> or
> > > >>>>> pr
> > > >>>>>>> author can trigger the bot, is it really a hurdle for pr author
> > or
> > > >>>>> reviewer
> > > >>>>>>> to call a bot to trigger CI? Given that PR author and reviewer
> is
> > > >>>>> already
> > > >>>>>>> actively commenting various details such as - PR description,
> > > >>> review
> > > >>>>>>> comments and responses, adding labels etc.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Me too curious to know the behavior for Tao's above use case.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Best,
> > > >>>>>>>
> > > >>>>>>> Sandeep
> > > >>>>>>>
> > > >>>>>>> On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <mu...@gmail.com>
> > wrote:
> > > >>>>>>>
> > > >>>>>>>> Is it possible for re-triggering a single job to be abused?
> For
> > > >>>>> example,
> > > >>>>>>>> the author spends two days re-triggering a flaky job to make
> it
> > > >>>>> pass. But
> > > >>>>>>>> other jobs which have passed the validation may be broken by
> > > >>> other
> > > >>>>>>> commits
> > > >>>>>>>> during the two day without being noticed. And finally the PR
> is
> > > >>>>> merged
> > > >>>>>>> with
> > > >>>>>>>> underlying problems.
> > > >>>>>>>>
> > > >>>>>>>> On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <
> > > >>>>> marco.g.abreu@gmail.com>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> In the end it only comes down to money, considering that the
> > > >>>>> system is
> > > >>>>>>>> auto
> > > >>>>>>>>> scaling, making the execution time constant.
> > > >>>>>>>>>
> > > >>>>>>>>> If we're trading money for usability, I certainly would
> prefer
> > > >>>>>>> usability.
> > > >>>>>>>>> I'd rather recommend to spend time on parallelizing test
> > > >>>> execution
> > > >>>>> or
> > > >>>>>>>>> getting rid of integration tests in the PR stage instead
> > > >>> reducing
> > > >>>>> the
> > > >>>>>>>> costs
> > > >>>>>>>>> by making people not use it. But taking a step back to
> > > >>> requiring
> > > >>>>> people
> > > >>>>>>>> to
> > > >>>>>>>>> manually trigger CI again doesn't feel right.
> > > >>>>>>>>>
> > > >>>>>>>>> I'm happy to see that bot deployed, but I do not agree with
> > > >>>>> removing
> > > >>>>>>> the
> > > >>>>>>>>> auto trigger functionality for new commits.
> > > >>>>>>>>>
> > > >>>>>>>>> -Marco
> > > >>>>>>>>>
> > > >>>>>>>>> Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 12.
> > > >>> März
> > > >>>>> 2020,
> > > >>>>>>>>> 22:47:
> > > >>>>>>>>>
> > > >>>>>>>>>> @Marco Thanks for pointing that out.
> > > >>>>>>>>>> Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM - 3:30 PM in
> > > >>>>>>>> (UTC-08:00)
> > > >>>>>>>>>> Pacific Time (US & Canada).
> > > >>>>>>>>>>
> > > >>>>>>>>>>> When do we expect this bot to be deployed?
> > > >>>>>>>>>> @Lin If all goes well in the next week I can deploy it to
> > > >>>> public
> > > >>>>>>> Apache
> > > >>>>>>>>>> (provided I get permissions from Apache Infra)
> > > >>>>>>>>>>
> > > >>>>>>>>>> @Marco Thanks for your feedback.
> > > >>>>>>>>>>> CI system has to support the community without requiring
> > > >>>>> people to
> > > >>>>>>>>>> constantly shepherd every single run
> > > >>>>>>>>>> We have data for the number of times CI was triggered
> > > >>>>> unnecessarily
> > > >>>>>>>> which
> > > >>>>>>>>>> includes
> > > >>>>>>>>>> - Entire build triggered instead of specific build
> > > >>>>>>>>>> - CI triggered when PR is still work in progress or not yet
> > > >>>> ready
> > > >>>>>>> (say
> > > >>>>>>>> -
> > > >>>>>>>>>> intermediate commits)
> > > >>>>>>>>>> At the end its a trade-off
> > > >>>>>>>>>> Money, Resources, Time to build for each and every commit vs
> > > >>>>> Pain of
> > > >>>>>>>>>> triggering builds
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Scan trigger plugin would poll SCM. Can we use plugin at
> > > >>>>> scale?
> > > >>>>>>>>>>
> > > >>>>>>>>>> 1. I haven't tested it on scale. But I think with the
> current
> > > >>>>> scale
> > > >>>>>>> of
> > > >>>>>>>>>> MXNet repo (191 open PRs i.e. checking for changes to 191
> > > >>>>> branches -
> > > >>>>>>> It
> > > >>>>>>>>>> should be manageable)
> > > >>>>>>>>>> 2. What's the purpose of the plugin? tldr; Branch discovery
> > > >>> or
> > > >>>>> branch
> > > >>>>>>>>>> indexing.
> > > >>>>>>>>>> Scan trigger plugin comes into the picture only once per PR
> > > >>> per
> > > >>>>> job
> > > >>>>>>>>> (i.e. 8
> > > >>>>>>>>>> times per PR for 8 jobs). It is basically done when a new PR
> > > >>> is
> > > >>>>> made
> > > >>>>>>>> and
> > > >>>>>>>>>> the job (say unix-cpu hasn't discovered the new PR branch
> > > >>> yet).
> > > >>>>>>> That's
> > > >>>>>>>>> it.
> > > >>>>>>>>>> So it shouldn't be a problem for public MXNet repo.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks,
> > > >>>>>>>>>> Chai
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <
> > > >>>>>>> marco.g.abreu@gmail.com>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Btw you forgot to set a date and time for the metting
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu <
> > > >>>>>>>>> marco.g.abreu@gmail.com
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Thanks Chai, I generally like the idea of the bot. But
> > > >>> I'm
> > > >>>>> not a
> > > >>>>>>>>>>> supporter
> > > >>>>>>>>>>>> of the idea to disable any automatic triggering
> > > >>> (disabling
> > > >>>>> the
> > > >>>>>>>>> webhook
> > > >>>>>>>>>> is
> > > >>>>>>>>>>>> also not an option, considering that this will disable
> > > >>>> master
> > > >>>>>>>>>> triggers).
> > > >>>>>>>>>>>> The CI system has to support the community without
> > > >>>> requiring
> > > >>>>>>> people
> > > >>>>>>>>> to
> > > >>>>>>>>>>>> constantly shepherd every single run. Disabling automatic
> > > >>>>>>>> triggering
> > > >>>>>>>>>>> seems
> > > >>>>>>>>>>>> like a step back to me.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Instead, I'd recommend that CI gets triggered upon every
> > > >>>>> commit
> > > >>>>>>> as
> > > >>>>>>>>>> usual,
> > > >>>>>>>>>>>> but people have the possibility to call a "command" (i.e.
> > > >>>>> make a
> > > >>>>>>>>>> message
> > > >>>>>>>>>>>> which results in the bot setting a label) to disable CI
> > > >>>> until
> > > >>>>>>> they
> > > >>>>>>>>>> revoke
> > > >>>>>>>>>>>> it. But the standard should still be that a new commit
> > > >>>>> triggers a
> > > >>>>>>>> new
> > > >>>>>>>>>> CI
> > > >>>>>>>>>>>> run.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>> https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
> > > >>>>>>>> seems
> > > >>>>>>>>>> like
> > > >>>>>>>>>>>> this would poll SCM. This will incur high quota
> > > >>>>> restrictions. Are
> > > >>>>>>>> you
> > > >>>>>>>>>>> sure
> > > >>>>>>>>>>>> that you can use that plugin at scale?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> -Marco
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <
> > > >>>>> apeforest@gmail.com>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>>>> Chai,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Awesome work. When do we expect this bot to be
> > > >>> deployed?
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Lin
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <
> > > >>>>>>>>> chai.bapat@gmail.com
> > > >>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Hello MXNet community,
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> I have built an MXNet Bot <
> > > >>>> https://github.com/mxnet-bot>
> > > >>>>> that
> > > >>>>>>>>>> allows
> > > >>>>>>>>>>> PR
> > > >>>>>>>>>>>>>> Authors, Committers and Jenkins Admins to trigger CI
> > > >>>>> manually.
> > > >>>>>>>>>>>>>> It handles 2 problems
> > > >>>>>>>>>>>>>> 1. Manual CI trigger instead of existing automated CI
> > > >>>>> trigger
> > > >>>>>>>>>>>>>> 2. Gives permissions to PR Authors (in addition to
> > > >>>> MXNet
> > > >>>>>>>>> Committers
> > > >>>>>>>>>>> and
> > > >>>>>>>>>>>>>> Jenkins Admins)
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Design Doc :
> > > >>>>>>>>>>>>>>
> > > >>>>>>> https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > > >>>>>>>>>>>>>> I urge you all to attend the demonstration meeting
> > > >>> and
> > > >>>>> lend
> > > >>>>>>> your
> > > >>>>>>>>>> views
> > > >>>>>>>>>>>>> on
> > > >>>>>>>>>>>>>> the same.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Thank you,
> > > >>>>>>>>>>>>>> Chai
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> *Meeting Details*:
> > > >>>>>>>>>>>>>> ==============Conference Bridge
> > > >>>> Information==============
> > > >>>>>>>>>>>>>> You have been invited to an online meeting, powered
> > > >>> by
> > > >>>>> Amazon
> > > >>>>>>>>> Chime.
> > > >>>>>>>>>>>>>> *Chime meeting ID*: *9272158344*
> > > >>>>>>>>>>>>>> Join via Chime clients (manually): Select 'Meetings >
> > > >>>>> Join a
> > > >>>>>>>>>> Meeting',
> > > >>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>> enter 9272158344
> > > >>>>>>>>>>>>>> Join via Chime clients (auto-call): If you invite
> > > >>>>> auto-call as
> > > >>>>>>>>>>> attendee,
> > > >>>>>>>>>>>>>> Chime will call you when the meeting starts, select
> > > >>>>> 'Answer'
> > > >>>>>>>>>>>>>> *Join via browser screen share*:
> > > >>>>> https://chime.aws/9272158344
> > > >>>>>>>>>>>>>> *Join via phone* (US): +1-929-432-4463,,,9272158344#
> > > >>>>>>>>>>>>>> *Join via phone (US toll-free)*:
> > > >>>>> +1-855-552-4463,,,9272158344#
> > > >>>>>>>>>>>>>> International dial-in:
> > > >>>> https://chime.aws/dialinnumbers/
> > > >>>>>>>>>>>>>> In-room video system: Ext: 62000, Meeting PIN:
> > > >>>>> 9272158344#
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> --
> > > >>>>>>>>>>>>>> *Chaitanya Prakash Bapat*
> > > >>>>>>>>>>>>>> *+1 (973) 953-6299*
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> [image: https://www.linkedin.com//in/chaibapat25]
> > > >>>>>>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> > > >>>>>>>>>>>>> https://www.facebook.com/chaibapat
> > > >>>>>>>>>>>>>> ]
> > > >>>>>>>>>>>>>> <https://www.facebook.com/chaibapchya>[image:
> > > >>>>>>>>>>>>>> https://twitter.com/ChaiBapchya] <
> > > >>>>>>>> https://twitter.com/ChaiBapchya
> > > >>>>>>>>>>>>>> [image:
> > > >>>>>>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> > > >>>>>>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> --
> > > >>>>>>>>>> *Chaitanya Prakash Bapat*
> > > >>>>>>>>>> *+1 (973) 953-6299*
> > > >>>>>>>>>>
> > > >>>>>>>>>> [image: https://www.linkedin.com//in/chaibapat25]
> > > >>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> > > >>>>>>>>> https://www.facebook.com/chaibapat
> > > >>>>>>>>>> ]
> > > >>>>>>>>>> <https://www.facebook.com/chaibapchya>[image:
> > > >>>>>>>>>> https://twitter.com/ChaiBapchya] <
> > > >>>>> https://twitter.com/ChaiBapchya
> > > >>>>>>>>>> [image:
> > > >>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> > > >>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
> > > >>>>>>>>>>
> > > >>>>>>>
> > > >>>>>>> --
> > > >>>>>>> Sandeep Krishnamurthy
> > > >>>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> *Chaitanya Prakash Bapat*
> > > >>> *+1 (973) 953-6299*
> > > >>>
> > > >>> [image: https://www.linkedin.com//in/chaibapat25]
> > > >>> <https://github.com/ChaiBapchya>[image:
> > > https://www.facebook.com/chaibapat
> > > >>> ]
> > > >>> <https://www.facebook.com/chaibapchya>[image:
> > > >>> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> > > >[image:
> > > >>> https://www.linkedin.com//in/chaibapat25]
> > > >>> <https://www.linkedin.com//in/chaibapchya/>
> > > >>>
> > > >>
> > >
> > >
> >
>
>
> --
> *Chaitanya Prakash Bapat*
> *+1 (973) 953-6299*
>
> [image: https://www.linkedin.com//in/chaibapat25]
> <https://github.com/ChaiBapchya>[image: https://www.facebook.com/chaibapat
> ]
> <https://www.facebook.com/chaibapchya>[image:
> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
> https://www.linkedin.com//in/chaibapat25]
> <https://www.linkedin.com//in/chaibapchya/>
>
Re: MXNet Bot Demo
Posted by Chaitanya Bapat <ch...@gmail.com>.
Update:
> we can ensure that all CI runs ran on the commit that will be merged
@Sam Skalicky <sa...@gmail.com> Branch Protection is added to public
MXNet repo. It ensures that for every PR to be merged, the CI passes. All
the jobs selected "required" jobs will have to be green for the PR to be
merged. Ofcourse, users with "Adminstrator" access can merge without it but
that's just a backdoor. It is the case now and will continue to be the case
with the inclusion of Bot.
> easily verify that the CI has executed all runs on the commit that will
be merged
GitHub UI shows all the jobs and the status corresponding to it on every
commit. That should suffice. For the merged commits, Repo -> Commits ->
Commit ID (Status) can be tracked currently (only way that I know of).
Moreover, it is beyond the scope of this project (and possibly out of our
control since this is purely GitHub UI specific use-case).
Thanks @przemyslaw for supporting the opt-in.
Thanks everyone in the community for sharing concerns, voicing your opinion
and participating in the discussion.
Thanks to those who attended the demo last Friday.
Action items from that discussion
1. Handle master merge builds [Done]
Bot runs entire CI suite after the PR is merged and comments on the PR
about the same.
Design decision :
MXNet Bot comment about master merge build on the *merge commit vs PR*.
After the PR is merged, Bot runs entire CI and comments the result of CI
trigger on the PR (because it is easy to track on a PR rather than
commenting inside the merge commit)
2. Idempotent condition
In case of already running build, if an attempt is made to retrigger the
job then what should be the response
a. Not to re-trigger, let the ongoing build continue till completion
b. End the ongoing build and re-trigger
c. Let the ongoing build continue, re-trigger new build
From resource saving point of view, *c* looks costly and a can be
avoided/optimized by B.
In case when a re-trigger was started "erroneously" then killing ongoing
build and re-trigger is a waste.
In case when ongoing build failed in one sub-part, then re-triggering is
justified.
Erroneous re-triggers would be less often while conscious re-triggers to
suppress failure is more common use-case. It looks like a safe assumption
to make given the trade-off.
[Open to debate]
3. Add security consideration [Use of secret manager, but without
auto-rotation due to Jenkins manual config requirement] [Done]
4. New PR Instruction message by the Bot [Done]
Thanks to the suggestion of Leonard, supported by others. I've now added
the feature where the Bot comments a help message. [For reference -
https://github.com/ChaiBapchya/incubator-mxnet/pull/52]
Barring the opt-in vs opt-out debate & idempotency, consensus was quickly
reached for the rest.
In the coming days, I hope to roll-out this feature into Prod (public
MXNet) for all devs to use.
Thanks,
Chai
On Mon, 16 Mar 2020 at 11:57, Marco de Abreu <ma...@gmail.com>
wrote:
> Well that's generally a problem with a deferred CI approach (CI is run at
> commit and not at merge time). This can either be solved through the other
> proposal that's currently on dev@, by having a bot which does merges by
> having a global lock and a merge queue or by accepting the issue. Reality
> right now is that we're running that model where two PRs which are merged
> in parallel might break one another. One thing to consider though is that
> this breakage would have to be introduced in two separate parts since
> otherwise there'd be merge conflicts. I think we had that situation twice
> so far and the result was a quick revert, so I'd say that it's a problem
> that can happily be accepted. All other solutions basically require some
> form of single-threaded and globally locked solution which limits us in
> scalability. I'd recommend to just accept that risk and revert a PR in case
> it actually had a conflict.
>
> -Marco
>
> On Mon, Mar 16, 2020 at 6:29 PM Skalicky, Sam <ss...@amazon.com.invalid>
> wrote:
>
> > We probably need some way to track which CI runs ran for which commit
> too,
> > that way we can ensure that all CI runs ran on the commit that will be
> > merged. Maybe the bot can comment with the commit hash when users
> command
> > it to do something. Although since users can trigger individual CI runs
> its
> > possible to have some commits run some CI runs but not others. We need
> some
> > way to easily verify that the CI has executed all runs on the commit that
> > will be merged.
> >
> > Sam
> >
> > > On Mar 13, 2020, at 8:28 PM, Przemysław Trędak <pt...@apache.org>
> > wrote:
> > >
> > > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> > >
> > >
> > >
> > > I personally like the idea of opt-in more than opt-out:
> > > - ultimately PR author wants the PR to be merged so they (or committer
> > reviewing the PR) will trigger the CI
> > > - if it is easy to trigger the PR via the bot command then the amount
> of
> > work per PR should be less than with opt-out (since most of the commits
> > should then be marked as [skip ci] or something similar
> > >
> > > +1 to the bot making a comment on each new PR with its commands (and
> > also explaining, or at least giving links to the general PR process so
> new
> > PR authors are not lost). Maybe we could make the bot recognize if the PR
> > author is new or existing contributor and offer advice based on that?
> > >
> > > Thanks
> > > Przemek
> > >
> > > On 2020/03/13 22:06:58, Marco de Abreu <ma...@gmail.com>
> wrote:
> > >> Hi,
> > >>
> > >> since it's no longer necessary to push a new commit to trigger CI, it
> > will
> > >> already reduce the costs. But to me, requiring an action to enable CI
> > after
> > >> a PR has been created initially, is a no go. User can opt out of CI,
> but
> > >> the default has to be CI being triggered automatically for every
> commit
> > >> unless specifically disabled by a participant. I'm also fine with
> > >> triggering certain additional jobs (think about running a nightly job
> > upon
> > >> request for a PR) to require a manual step, but the PR validation
> > pipelines
> > >> have to run automatically. Every check that is marked as "Required" in
> > >> GitHub has to be automatically kicked off.
> > >>
> > >> -Marco
> > >>
> > >> On Fri, Mar 13, 2020 at 9:50 PM Chaitanya Bapat <chai.bapat@gmail.com
> >
> > >> wrote:
> > >>
> > >>> Firstly,
> > >>> Sorry I missed out on attaching the mail thread that was sent on 12th
> > >>> February for notifying the community of the upcoming changes to the
> > MXNet
> > >>> CI
> > >>> For reference :
> > >>>
> > >>>
> >
> https://lists.apache.org/thread.html/r09a6ab2803a996fc80e00fe39ed312fa4865e8805e08df847f1addad%40%3Cdev.mxnet.apache.org%3E
> > >>>
> > >>> Now to the questions,
> > >>>> Is it possible for re-triggering a single job to be abused?
> > >>> @Tao In the case when a user re-triggers a single job multiple times,
> > that
> > >>> will be visible in the PR conversation thread. A committer, even
> after
> > he
> > >>> has approved the PR before, generally takes a look at the final state
> > of
> > >>> the PR before merging. Would it be fair to assume the committer could
> > take
> > >>> the multiple re-trigger of a single job into account before merging?
> > The
> > >>> committer then has the option to invoke `@mxnet-bot run ci [all] ` to
> > >>> trigger the entire build pipeline one last to counter the abuse. This
> > is
> > >>> aligned with what @Leonard said.
> > >>>
> > >>> @Sandeep Thanks a lot for collecting and sharing valuable data. I'd
> > concur
> > >>> with the opinion that given the existing things committers & PR
> Authors
> > >>> take care of, invoking CI shouldn't be that big of an additional
> > burden.
> > >>>
> > >>> @Marco With the opt-out, the onus remains on the PR Author. It
> doesn't
> > help
> > >>> reduce the resource usage. Hence, it was suggested to switch to
> > >>> opt-in. @Leo's suggestion for proactive commenting on the part of bot
> > makes
> > >>> sense and is doable.
> > >>>
> > >>> Default : opt-out and User initiated opt-in (with addressing Leo's
> fix
> > for
> > >>> the usability issue you correctly pointed out )
> > >>> @Marco How does this sound to you?
> > >>>
> > >>> Again, thank you all for chiming in and voicing your opinion.
> > Appreciate
> > >>> it.
> > >>> We can take ahead these discussions in today's demo meeting. [Design
> > Doc
> > >>> <https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot>]
> > [Demo
> > >>> Video <https://www.youtube.com/watch?v=gfOGwZId8aU>]
> > >>>
> > >>> Thanks,
> > >>> Chai
> > >>>
> > >>> On Fri, 13 Mar 2020 at 12:34, Marco de Abreu <
> marco.g.abreu@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> I'd recommend that the bot makes an initial comment when a PR gets
> > opened
> > >>>> and informs the users of its commands. It then tells the user the
> > commend
> > >>>> to opt out of CI.
> > >>>>
> > >>>> -Marco
> > >>>>
> > >>>> Lausen, Leonard <la...@amazon.com.invalid> schrieb am Fr., 13.
> März
> > >>> 2020,
> > >>>> 20:27:
> > >>>>
> > >>>>> On opt-out: People may be unaware of opt-out would not use it.
> There
> > is
> > >>>> no
> > >>>>> incentive to use opt-out, as the PR author doesn't pay any money
> for
> > CI
> > >>>>> run.
> > >>>>>
> > >>>>> I agree with Marco though that opt-in alone may cause usability
> > issues,
> > >>>> as
> > >>>>> contributors may not be aware of how to trigger the CI.
> > >>>>> One solution is that the bot proactively comments on the PR and
> > reminds
> > >>>> the
> > >>>>> author to trigger running CI once the author deems the PR ready.
> > >>>>>
> > >>>>> But even if we choose opt-out, the bot will still add a lot of
> value,
> > >>> as
> > >>>> PR
> > >>>>> authors can retrigger single jobs that have failed due to
> flakiness.
> > >>>>>
> > >>>>>> Is it possible for re-triggering a single job to be abused? For
> > >>>> example,
> > >>>>>> the author spends two days re-triggering a flaky job to make it
> > pass.
> > >>>>>
> > >>>>> Yes, this is possible. I suggest the committer who likes to merge a
> > PR
> > >>>>> needs to
> > >>>>> make a good judgement here if a PR is abusing the feature, and if
> so,
> > >>>>> retrigger
> > >>>>> all CI runs.
> > >>>>>
> > >>>>> Best regards
> > >>>>> Leonard
> > >>>>>
> > >>>>> On Fri, 2020-03-13 at 08:07 +0100, Marco de Abreu wrote:
> > >>>>>> Thanks for the data Sandeep. In these cases it sounds like it
> would
> > >>>> have
> > >>>>>> rather been better when people explicitly turned off CI in that
> > case.
> > >>>>>> What's the argument against an opt-out instead of an opt-in?
> > >>>>>>
> > >>>>>> My intention is that I consider it quite cumbersome to make it a
> > >>>>> *required*
> > >>>>>> step to always trigger CI manually, even if just submitting a
> small
> > >>> PR.
> > >>>>> I'd
> > >>>>>> rather see people explicitly turning off CI if they wouldn't like
> to
> > >>>> use
> > >>>>> it
> > >>>>>> - and there's also the "draft" stage for a PR which some
> > contributors
> > >>>> are
> > >>>>>> using.
> > >>>>>>
> > >>>>>> With regards to WIP and do not review: I think these are use cases
> > >>>> where
> > >>>>>> you want CI feedback, as otherwise you wouldn't have opened the
> PR.
> > >>> If
> > >>>>> you
> > >>>>>> don't want human feedback and neither machine feedback, why open
> the
> > >>> PR
> > >>>>> at
> > >>>>>> all?
> > >>>>>>
> > >>>>>> -Marco
> > >>>>>>
> > >>>>>>
> > >>>>>> sandeep krishnamurthy <sa...@gmail.com> schrieb am
> Fr.,
> > >>>> 13.
> > >>>>>> März 2020, 05:24:
> > >>>>>>
> > >>>>>>> I tried to gather some data for us to discuss this topic in this
> > >>>>> thread. I
> > >>>>>>> tried to count number of un-necessary builds by looking at most
> > >>>> recent
> > >>>>> (as
> > >>>>>>> of 12, March 9 PM PST) 50 PRs merged to master and 50 PRs.
> > >>>>>>> Identifying un-necessary builds is bit subjective. I tried to be
> > >>> more
> > >>>>>>> conservative where I didn't count a build as un-necessary if I
> was
> > >>> in
> > >>>>>>> doubt. Hence, I was not able to automate, but I made an effort to
> > >>> go
> > >>>>>>> through PRs manually and use below criteria to identify
> > >>> un-necessary
> > >>>>>>> commits triggering the builds.
> > >>>>>>>
> > >>>>>>> 1. Explicitly marked as WIP / do not review PR
> > >>>>>>> 2. Incremental WIP commit and finally commenting a commit
> > >>> “trigger
> > >>>>> CI”
> > >>>>>>> 3. Multiple commits to address all comments from single review.
> > >>>>> This is
> > >>>>>>> assuming we see a comment, address them, commit, next the
> > >>>> following
> > >>>>>>> comment
> > >>>>>>> 4. Sequence of documentation only changes
> > >>>>>>>
> > >>>>>>> I found there were around 42 avoidable builds from most recent 50
> > >>>>> merged
> > >>>>>>> PRs and around 86 builds from recent 50 open PRs.
> > >>>>>>>
> > >>>>>>> I synced up with other contributors (Joe Evans, Chai) from Amazon
> > >>> who
> > >>>>> is
> > >>>>>>> contributing to MXNet CI system. I was told that on an average it
> > >>>> costs
> > >>>>>>> around $84 per build and on an average 6 commits per merged PR
> (for
> > >>>>> year
> > >>>>>>> 2019). Going by that, it is approximately 1/6 builds are
> avoidable.
> > >>>>> [100 /
> > >>>>>>> 300 + 300 ]
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Usability should be top most priority. But, since either a
> reviewer
> > >>>> or
> > >>>>> pr
> > >>>>>>> author can trigger the bot, is it really a hurdle for pr author
> or
> > >>>>> reviewer
> > >>>>>>> to call a bot to trigger CI? Given that PR author and reviewer is
> > >>>>> already
> > >>>>>>> actively commenting various details such as - PR description,
> > >>> review
> > >>>>>>> comments and responses, adding labels etc.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Me too curious to know the behavior for Tao's above use case.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>>
> > >>>>>>> Sandeep
> > >>>>>>>
> > >>>>>>> On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <mu...@gmail.com>
> wrote:
> > >>>>>>>
> > >>>>>>>> Is it possible for re-triggering a single job to be abused? For
> > >>>>> example,
> > >>>>>>>> the author spends two days re-triggering a flaky job to make it
> > >>>>> pass. But
> > >>>>>>>> other jobs which have passed the validation may be broken by
> > >>> other
> > >>>>>>> commits
> > >>>>>>>> during the two day without being noticed. And finally the PR is
> > >>>>> merged
> > >>>>>>> with
> > >>>>>>>> underlying problems.
> > >>>>>>>>
> > >>>>>>>> On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <
> > >>>>> marco.g.abreu@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> In the end it only comes down to money, considering that the
> > >>>>> system is
> > >>>>>>>> auto
> > >>>>>>>>> scaling, making the execution time constant.
> > >>>>>>>>>
> > >>>>>>>>> If we're trading money for usability, I certainly would prefer
> > >>>>>>> usability.
> > >>>>>>>>> I'd rather recommend to spend time on parallelizing test
> > >>>> execution
> > >>>>> or
> > >>>>>>>>> getting rid of integration tests in the PR stage instead
> > >>> reducing
> > >>>>> the
> > >>>>>>>> costs
> > >>>>>>>>> by making people not use it. But taking a step back to
> > >>> requiring
> > >>>>> people
> > >>>>>>>> to
> > >>>>>>>>> manually trigger CI again doesn't feel right.
> > >>>>>>>>>
> > >>>>>>>>> I'm happy to see that bot deployed, but I do not agree with
> > >>>>> removing
> > >>>>>>> the
> > >>>>>>>>> auto trigger functionality for new commits.
> > >>>>>>>>>
> > >>>>>>>>> -Marco
> > >>>>>>>>>
> > >>>>>>>>> Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 12.
> > >>> März
> > >>>>> 2020,
> > >>>>>>>>> 22:47:
> > >>>>>>>>>
> > >>>>>>>>>> @Marco Thanks for pointing that out.
> > >>>>>>>>>> Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM - 3:30 PM in
> > >>>>>>>> (UTC-08:00)
> > >>>>>>>>>> Pacific Time (US & Canada).
> > >>>>>>>>>>
> > >>>>>>>>>>> When do we expect this bot to be deployed?
> > >>>>>>>>>> @Lin If all goes well in the next week I can deploy it to
> > >>>> public
> > >>>>>>> Apache
> > >>>>>>>>>> (provided I get permissions from Apache Infra)
> > >>>>>>>>>>
> > >>>>>>>>>> @Marco Thanks for your feedback.
> > >>>>>>>>>>> CI system has to support the community without requiring
> > >>>>> people to
> > >>>>>>>>>> constantly shepherd every single run
> > >>>>>>>>>> We have data for the number of times CI was triggered
> > >>>>> unnecessarily
> > >>>>>>>> which
> > >>>>>>>>>> includes
> > >>>>>>>>>> - Entire build triggered instead of specific build
> > >>>>>>>>>> - CI triggered when PR is still work in progress or not yet
> > >>>> ready
> > >>>>>>> (say
> > >>>>>>>> -
> > >>>>>>>>>> intermediate commits)
> > >>>>>>>>>> At the end its a trade-off
> > >>>>>>>>>> Money, Resources, Time to build for each and every commit vs
> > >>>>> Pain of
> > >>>>>>>>>> triggering builds
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>> Scan trigger plugin would poll SCM. Can we use plugin at
> > >>>>> scale?
> > >>>>>>>>>>
> > >>>>>>>>>> 1. I haven't tested it on scale. But I think with the current
> > >>>>> scale
> > >>>>>>> of
> > >>>>>>>>>> MXNet repo (191 open PRs i.e. checking for changes to 191
> > >>>>> branches -
> > >>>>>>> It
> > >>>>>>>>>> should be manageable)
> > >>>>>>>>>> 2. What's the purpose of the plugin? tldr; Branch discovery
> > >>> or
> > >>>>> branch
> > >>>>>>>>>> indexing.
> > >>>>>>>>>> Scan trigger plugin comes into the picture only once per PR
> > >>> per
> > >>>>> job
> > >>>>>>>>> (i.e. 8
> > >>>>>>>>>> times per PR for 8 jobs). It is basically done when a new PR
> > >>> is
> > >>>>> made
> > >>>>>>>> and
> > >>>>>>>>>> the job (say unix-cpu hasn't discovered the new PR branch
> > >>> yet).
> > >>>>>>> That's
> > >>>>>>>>> it.
> > >>>>>>>>>> So it shouldn't be a problem for public MXNet repo.
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks,
> > >>>>>>>>>> Chai
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <
> > >>>>>>> marco.g.abreu@gmail.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Btw you forgot to set a date and time for the metting
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu <
> > >>>>>>>>> marco.g.abreu@gmail.com
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Thanks Chai, I generally like the idea of the bot. But
> > >>> I'm
> > >>>>> not a
> > >>>>>>>>>>> supporter
> > >>>>>>>>>>>> of the idea to disable any automatic triggering
> > >>> (disabling
> > >>>>> the
> > >>>>>>>>> webhook
> > >>>>>>>>>> is
> > >>>>>>>>>>>> also not an option, considering that this will disable
> > >>>> master
> > >>>>>>>>>> triggers).
> > >>>>>>>>>>>> The CI system has to support the community without
> > >>>> requiring
> > >>>>>>> people
> > >>>>>>>>> to
> > >>>>>>>>>>>> constantly shepherd every single run. Disabling automatic
> > >>>>>>>> triggering
> > >>>>>>>>>>> seems
> > >>>>>>>>>>>> like a step back to me.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Instead, I'd recommend that CI gets triggered upon every
> > >>>>> commit
> > >>>>>>> as
> > >>>>>>>>>> usual,
> > >>>>>>>>>>>> but people have the possibility to call a "command" (i.e.
> > >>>>> make a
> > >>>>>>>>>> message
> > >>>>>>>>>>>> which results in the bot setting a label) to disable CI
> > >>>> until
> > >>>>>>> they
> > >>>>>>>>>> revoke
> > >>>>>>>>>>>> it. But the standard should still be that a new commit
> > >>>>> triggers a
> > >>>>>>>> new
> > >>>>>>>>>> CI
> > >>>>>>>>>>>> run.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>> https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
> > >>>>>>>> seems
> > >>>>>>>>>> like
> > >>>>>>>>>>>> this would poll SCM. This will incur high quota
> > >>>>> restrictions. Are
> > >>>>>>>> you
> > >>>>>>>>>>> sure
> > >>>>>>>>>>>> that you can use that plugin at scale?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> -Marco
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <
> > >>>>> apeforest@gmail.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>>> Chai,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Awesome work. When do we expect this bot to be
> > >>> deployed?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Lin
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <
> > >>>>>>>>> chai.bapat@gmail.com
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hello MXNet community,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I have built an MXNet Bot <
> > >>>> https://github.com/mxnet-bot>
> > >>>>> that
> > >>>>>>>>>> allows
> > >>>>>>>>>>> PR
> > >>>>>>>>>>>>>> Authors, Committers and Jenkins Admins to trigger CI
> > >>>>> manually.
> > >>>>>>>>>>>>>> It handles 2 problems
> > >>>>>>>>>>>>>> 1. Manual CI trigger instead of existing automated CI
> > >>>>> trigger
> > >>>>>>>>>>>>>> 2. Gives permissions to PR Authors (in addition to
> > >>>> MXNet
> > >>>>>>>>> Committers
> > >>>>>>>>>>> and
> > >>>>>>>>>>>>>> Jenkins Admins)
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Design Doc :
> > >>>>>>>>>>>>>>
> > >>>>>>> https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > >>>>>>>>>>>>>> I urge you all to attend the demonstration meeting
> > >>> and
> > >>>>> lend
> > >>>>>>> your
> > >>>>>>>>>> views
> > >>>>>>>>>>>>> on
> > >>>>>>>>>>>>>> the same.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thank you,
> > >>>>>>>>>>>>>> Chai
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> *Meeting Details*:
> > >>>>>>>>>>>>>> ==============Conference Bridge
> > >>>> Information==============
> > >>>>>>>>>>>>>> You have been invited to an online meeting, powered
> > >>> by
> > >>>>> Amazon
> > >>>>>>>>> Chime.
> > >>>>>>>>>>>>>> *Chime meeting ID*: *9272158344*
> > >>>>>>>>>>>>>> Join via Chime clients (manually): Select 'Meetings >
> > >>>>> Join a
> > >>>>>>>>>> Meeting',
> > >>>>>>>>>>>>> and
> > >>>>>>>>>>>>>> enter 9272158344
> > >>>>>>>>>>>>>> Join via Chime clients (auto-call): If you invite
> > >>>>> auto-call as
> > >>>>>>>>>>> attendee,
> > >>>>>>>>>>>>>> Chime will call you when the meeting starts, select
> > >>>>> 'Answer'
> > >>>>>>>>>>>>>> *Join via browser screen share*:
> > >>>>> https://chime.aws/9272158344
> > >>>>>>>>>>>>>> *Join via phone* (US): +1-929-432-4463,,,9272158344#
> > >>>>>>>>>>>>>> *Join via phone (US toll-free)*:
> > >>>>> +1-855-552-4463,,,9272158344#
> > >>>>>>>>>>>>>> International dial-in:
> > >>>> https://chime.aws/dialinnumbers/
> > >>>>>>>>>>>>>> In-room video system: Ext: 62000, Meeting PIN:
> > >>>>> 9272158344#
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>> *Chaitanya Prakash Bapat*
> > >>>>>>>>>>>>>> *+1 (973) 953-6299*
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> [image: https://www.linkedin.com//in/chaibapat25]
> > >>>>>>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> > >>>>>>>>>>>>> https://www.facebook.com/chaibapat
> > >>>>>>>>>>>>>> ]
> > >>>>>>>>>>>>>> <https://www.facebook.com/chaibapchya>[image:
> > >>>>>>>>>>>>>> https://twitter.com/ChaiBapchya] <
> > >>>>>>>> https://twitter.com/ChaiBapchya
> > >>>>>>>>>>>>>> [image:
> > >>>>>>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> > >>>>>>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> --
> > >>>>>>>>>> *Chaitanya Prakash Bapat*
> > >>>>>>>>>> *+1 (973) 953-6299*
> > >>>>>>>>>>
> > >>>>>>>>>> [image: https://www.linkedin.com//in/chaibapat25]
> > >>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> > >>>>>>>>> https://www.facebook.com/chaibapat
> > >>>>>>>>>> ]
> > >>>>>>>>>> <https://www.facebook.com/chaibapchya>[image:
> > >>>>>>>>>> https://twitter.com/ChaiBapchya] <
> > >>>>> https://twitter.com/ChaiBapchya
> > >>>>>>>>>> [image:
> > >>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> > >>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
> > >>>>>>>>>>
> > >>>>>>>
> > >>>>>>> --
> > >>>>>>> Sandeep Krishnamurthy
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>> --
> > >>> *Chaitanya Prakash Bapat*
> > >>> *+1 (973) 953-6299*
> > >>>
> > >>> [image: https://www.linkedin.com//in/chaibapat25]
> > >>> <https://github.com/ChaiBapchya>[image:
> > https://www.facebook.com/chaibapat
> > >>> ]
> > >>> <https://www.facebook.com/chaibapchya>[image:
> > >>> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> > >[image:
> > >>> https://www.linkedin.com//in/chaibapat25]
> > >>> <https://www.linkedin.com//in/chaibapchya/>
> > >>>
> > >>
> >
> >
>
--
*Chaitanya Prakash Bapat*
*+1 (973) 953-6299*
[image: https://www.linkedin.com//in/chaibapat25]
<https://github.com/ChaiBapchya>[image: https://www.facebook.com/chaibapat]
<https://www.facebook.com/chaibapchya>[image:
https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
https://www.linkedin.com//in/chaibapat25]
<https://www.linkedin.com//in/chaibapchya/>
Re: MXNet Bot Demo
Posted by Marco de Abreu <ma...@gmail.com>.
Well that's generally a problem with a deferred CI approach (CI is run at
commit and not at merge time). This can either be solved through the other
proposal that's currently on dev@, by having a bot which does merges by
having a global lock and a merge queue or by accepting the issue. Reality
right now is that we're running that model where two PRs which are merged
in parallel might break one another. One thing to consider though is that
this breakage would have to be introduced in two separate parts since
otherwise there'd be merge conflicts. I think we had that situation twice
so far and the result was a quick revert, so I'd say that it's a problem
that can happily be accepted. All other solutions basically require some
form of single-threaded and globally locked solution which limits us in
scalability. I'd recommend to just accept that risk and revert a PR in case
it actually had a conflict.
-Marco
On Mon, Mar 16, 2020 at 6:29 PM Skalicky, Sam <ss...@amazon.com.invalid>
wrote:
> We probably need some way to track which CI runs ran for which commit too,
> that way we can ensure that all CI runs ran on the commit that will be
> merged. Maybe the bot can comment with the commit hash when users command
> it to do something. Although since users can trigger individual CI runs its
> possible to have some commits run some CI runs but not others. We need some
> way to easily verify that the CI has executed all runs on the commit that
> will be merged.
>
> Sam
>
> > On Mar 13, 2020, at 8:28 PM, Przemysław Trędak <pt...@apache.org>
> wrote:
> >
> > CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
> >
> >
> >
> > I personally like the idea of opt-in more than opt-out:
> > - ultimately PR author wants the PR to be merged so they (or committer
> reviewing the PR) will trigger the CI
> > - if it is easy to trigger the PR via the bot command then the amount of
> work per PR should be less than with opt-out (since most of the commits
> should then be marked as [skip ci] or something similar
> >
> > +1 to the bot making a comment on each new PR with its commands (and
> also explaining, or at least giving links to the general PR process so new
> PR authors are not lost). Maybe we could make the bot recognize if the PR
> author is new or existing contributor and offer advice based on that?
> >
> > Thanks
> > Przemek
> >
> > On 2020/03/13 22:06:58, Marco de Abreu <ma...@gmail.com> wrote:
> >> Hi,
> >>
> >> since it's no longer necessary to push a new commit to trigger CI, it
> will
> >> already reduce the costs. But to me, requiring an action to enable CI
> after
> >> a PR has been created initially, is a no go. User can opt out of CI, but
> >> the default has to be CI being triggered automatically for every commit
> >> unless specifically disabled by a participant. I'm also fine with
> >> triggering certain additional jobs (think about running a nightly job
> upon
> >> request for a PR) to require a manual step, but the PR validation
> pipelines
> >> have to run automatically. Every check that is marked as "Required" in
> >> GitHub has to be automatically kicked off.
> >>
> >> -Marco
> >>
> >> On Fri, Mar 13, 2020 at 9:50 PM Chaitanya Bapat <ch...@gmail.com>
> >> wrote:
> >>
> >>> Firstly,
> >>> Sorry I missed out on attaching the mail thread that was sent on 12th
> >>> February for notifying the community of the upcoming changes to the
> MXNet
> >>> CI
> >>> For reference :
> >>>
> >>>
> https://lists.apache.org/thread.html/r09a6ab2803a996fc80e00fe39ed312fa4865e8805e08df847f1addad%40%3Cdev.mxnet.apache.org%3E
> >>>
> >>> Now to the questions,
> >>>> Is it possible for re-triggering a single job to be abused?
> >>> @Tao In the case when a user re-triggers a single job multiple times,
> that
> >>> will be visible in the PR conversation thread. A committer, even after
> he
> >>> has approved the PR before, generally takes a look at the final state
> of
> >>> the PR before merging. Would it be fair to assume the committer could
> take
> >>> the multiple re-trigger of a single job into account before merging?
> The
> >>> committer then has the option to invoke `@mxnet-bot run ci [all] ` to
> >>> trigger the entire build pipeline one last to counter the abuse. This
> is
> >>> aligned with what @Leonard said.
> >>>
> >>> @Sandeep Thanks a lot for collecting and sharing valuable data. I'd
> concur
> >>> with the opinion that given the existing things committers & PR Authors
> >>> take care of, invoking CI shouldn't be that big of an additional
> burden.
> >>>
> >>> @Marco With the opt-out, the onus remains on the PR Author. It doesn't
> help
> >>> reduce the resource usage. Hence, it was suggested to switch to
> >>> opt-in. @Leo's suggestion for proactive commenting on the part of bot
> makes
> >>> sense and is doable.
> >>>
> >>> Default : opt-out and User initiated opt-in (with addressing Leo's fix
> for
> >>> the usability issue you correctly pointed out )
> >>> @Marco How does this sound to you?
> >>>
> >>> Again, thank you all for chiming in and voicing your opinion.
> Appreciate
> >>> it.
> >>> We can take ahead these discussions in today's demo meeting. [Design
> Doc
> >>> <https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot>]
> [Demo
> >>> Video <https://www.youtube.com/watch?v=gfOGwZId8aU>]
> >>>
> >>> Thanks,
> >>> Chai
> >>>
> >>> On Fri, 13 Mar 2020 at 12:34, Marco de Abreu <ma...@gmail.com>
> >>> wrote:
> >>>
> >>>> I'd recommend that the bot makes an initial comment when a PR gets
> opened
> >>>> and informs the users of its commands. It then tells the user the
> commend
> >>>> to opt out of CI.
> >>>>
> >>>> -Marco
> >>>>
> >>>> Lausen, Leonard <la...@amazon.com.invalid> schrieb am Fr., 13. März
> >>> 2020,
> >>>> 20:27:
> >>>>
> >>>>> On opt-out: People may be unaware of opt-out would not use it. There
> is
> >>>> no
> >>>>> incentive to use opt-out, as the PR author doesn't pay any money for
> CI
> >>>>> run.
> >>>>>
> >>>>> I agree with Marco though that opt-in alone may cause usability
> issues,
> >>>> as
> >>>>> contributors may not be aware of how to trigger the CI.
> >>>>> One solution is that the bot proactively comments on the PR and
> reminds
> >>>> the
> >>>>> author to trigger running CI once the author deems the PR ready.
> >>>>>
> >>>>> But even if we choose opt-out, the bot will still add a lot of value,
> >>> as
> >>>> PR
> >>>>> authors can retrigger single jobs that have failed due to flakiness.
> >>>>>
> >>>>>> Is it possible for re-triggering a single job to be abused? For
> >>>> example,
> >>>>>> the author spends two days re-triggering a flaky job to make it
> pass.
> >>>>>
> >>>>> Yes, this is possible. I suggest the committer who likes to merge a
> PR
> >>>>> needs to
> >>>>> make a good judgement here if a PR is abusing the feature, and if so,
> >>>>> retrigger
> >>>>> all CI runs.
> >>>>>
> >>>>> Best regards
> >>>>> Leonard
> >>>>>
> >>>>> On Fri, 2020-03-13 at 08:07 +0100, Marco de Abreu wrote:
> >>>>>> Thanks for the data Sandeep. In these cases it sounds like it would
> >>>> have
> >>>>>> rather been better when people explicitly turned off CI in that
> case.
> >>>>>> What's the argument against an opt-out instead of an opt-in?
> >>>>>>
> >>>>>> My intention is that I consider it quite cumbersome to make it a
> >>>>> *required*
> >>>>>> step to always trigger CI manually, even if just submitting a small
> >>> PR.
> >>>>> I'd
> >>>>>> rather see people explicitly turning off CI if they wouldn't like to
> >>>> use
> >>>>> it
> >>>>>> - and there's also the "draft" stage for a PR which some
> contributors
> >>>> are
> >>>>>> using.
> >>>>>>
> >>>>>> With regards to WIP and do not review: I think these are use cases
> >>>> where
> >>>>>> you want CI feedback, as otherwise you wouldn't have opened the PR.
> >>> If
> >>>>> you
> >>>>>> don't want human feedback and neither machine feedback, why open the
> >>> PR
> >>>>> at
> >>>>>> all?
> >>>>>>
> >>>>>> -Marco
> >>>>>>
> >>>>>>
> >>>>>> sandeep krishnamurthy <sa...@gmail.com> schrieb am Fr.,
> >>>> 13.
> >>>>>> März 2020, 05:24:
> >>>>>>
> >>>>>>> I tried to gather some data for us to discuss this topic in this
> >>>>> thread. I
> >>>>>>> tried to count number of un-necessary builds by looking at most
> >>>> recent
> >>>>> (as
> >>>>>>> of 12, March 9 PM PST) 50 PRs merged to master and 50 PRs.
> >>>>>>> Identifying un-necessary builds is bit subjective. I tried to be
> >>> more
> >>>>>>> conservative where I didn't count a build as un-necessary if I was
> >>> in
> >>>>>>> doubt. Hence, I was not able to automate, but I made an effort to
> >>> go
> >>>>>>> through PRs manually and use below criteria to identify
> >>> un-necessary
> >>>>>>> commits triggering the builds.
> >>>>>>>
> >>>>>>> 1. Explicitly marked as WIP / do not review PR
> >>>>>>> 2. Incremental WIP commit and finally commenting a commit
> >>> “trigger
> >>>>> CI”
> >>>>>>> 3. Multiple commits to address all comments from single review.
> >>>>> This is
> >>>>>>> assuming we see a comment, address them, commit, next the
> >>>> following
> >>>>>>> comment
> >>>>>>> 4. Sequence of documentation only changes
> >>>>>>>
> >>>>>>> I found there were around 42 avoidable builds from most recent 50
> >>>>> merged
> >>>>>>> PRs and around 86 builds from recent 50 open PRs.
> >>>>>>>
> >>>>>>> I synced up with other contributors (Joe Evans, Chai) from Amazon
> >>> who
> >>>>> is
> >>>>>>> contributing to MXNet CI system. I was told that on an average it
> >>>> costs
> >>>>>>> around $84 per build and on an average 6 commits per merged PR (for
> >>>>> year
> >>>>>>> 2019). Going by that, it is approximately 1/6 builds are avoidable.
> >>>>> [100 /
> >>>>>>> 300 + 300 ]
> >>>>>>>
> >>>>>>>
> >>>>>>> Usability should be top most priority. But, since either a reviewer
> >>>> or
> >>>>> pr
> >>>>>>> author can trigger the bot, is it really a hurdle for pr author or
> >>>>> reviewer
> >>>>>>> to call a bot to trigger CI? Given that PR author and reviewer is
> >>>>> already
> >>>>>>> actively commenting various details such as - PR description,
> >>> review
> >>>>>>> comments and responses, adding labels etc.
> >>>>>>>
> >>>>>>>
> >>>>>>> Me too curious to know the behavior for Tao's above use case.
> >>>>>>>
> >>>>>>>
> >>>>>>> Best,
> >>>>>>>
> >>>>>>> Sandeep
> >>>>>>>
> >>>>>>> On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <mu...@gmail.com> wrote:
> >>>>>>>
> >>>>>>>> Is it possible for re-triggering a single job to be abused? For
> >>>>> example,
> >>>>>>>> the author spends two days re-triggering a flaky job to make it
> >>>>> pass. But
> >>>>>>>> other jobs which have passed the validation may be broken by
> >>> other
> >>>>>>> commits
> >>>>>>>> during the two day without being noticed. And finally the PR is
> >>>>> merged
> >>>>>>> with
> >>>>>>>> underlying problems.
> >>>>>>>>
> >>>>>>>> On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <
> >>>>> marco.g.abreu@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> In the end it only comes down to money, considering that the
> >>>>> system is
> >>>>>>>> auto
> >>>>>>>>> scaling, making the execution time constant.
> >>>>>>>>>
> >>>>>>>>> If we're trading money for usability, I certainly would prefer
> >>>>>>> usability.
> >>>>>>>>> I'd rather recommend to spend time on parallelizing test
> >>>> execution
> >>>>> or
> >>>>>>>>> getting rid of integration tests in the PR stage instead
> >>> reducing
> >>>>> the
> >>>>>>>> costs
> >>>>>>>>> by making people not use it. But taking a step back to
> >>> requiring
> >>>>> people
> >>>>>>>> to
> >>>>>>>>> manually trigger CI again doesn't feel right.
> >>>>>>>>>
> >>>>>>>>> I'm happy to see that bot deployed, but I do not agree with
> >>>>> removing
> >>>>>>> the
> >>>>>>>>> auto trigger functionality for new commits.
> >>>>>>>>>
> >>>>>>>>> -Marco
> >>>>>>>>>
> >>>>>>>>> Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 12.
> >>> März
> >>>>> 2020,
> >>>>>>>>> 22:47:
> >>>>>>>>>
> >>>>>>>>>> @Marco Thanks for pointing that out.
> >>>>>>>>>> Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM - 3:30 PM in
> >>>>>>>> (UTC-08:00)
> >>>>>>>>>> Pacific Time (US & Canada).
> >>>>>>>>>>
> >>>>>>>>>>> When do we expect this bot to be deployed?
> >>>>>>>>>> @Lin If all goes well in the next week I can deploy it to
> >>>> public
> >>>>>>> Apache
> >>>>>>>>>> (provided I get permissions from Apache Infra)
> >>>>>>>>>>
> >>>>>>>>>> @Marco Thanks for your feedback.
> >>>>>>>>>>> CI system has to support the community without requiring
> >>>>> people to
> >>>>>>>>>> constantly shepherd every single run
> >>>>>>>>>> We have data for the number of times CI was triggered
> >>>>> unnecessarily
> >>>>>>>> which
> >>>>>>>>>> includes
> >>>>>>>>>> - Entire build triggered instead of specific build
> >>>>>>>>>> - CI triggered when PR is still work in progress or not yet
> >>>> ready
> >>>>>>> (say
> >>>>>>>> -
> >>>>>>>>>> intermediate commits)
> >>>>>>>>>> At the end its a trade-off
> >>>>>>>>>> Money, Resources, Time to build for each and every commit vs
> >>>>> Pain of
> >>>>>>>>>> triggering builds
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Scan trigger plugin would poll SCM. Can we use plugin at
> >>>>> scale?
> >>>>>>>>>>
> >>>>>>>>>> 1. I haven't tested it on scale. But I think with the current
> >>>>> scale
> >>>>>>> of
> >>>>>>>>>> MXNet repo (191 open PRs i.e. checking for changes to 191
> >>>>> branches -
> >>>>>>> It
> >>>>>>>>>> should be manageable)
> >>>>>>>>>> 2. What's the purpose of the plugin? tldr; Branch discovery
> >>> or
> >>>>> branch
> >>>>>>>>>> indexing.
> >>>>>>>>>> Scan trigger plugin comes into the picture only once per PR
> >>> per
> >>>>> job
> >>>>>>>>> (i.e. 8
> >>>>>>>>>> times per PR for 8 jobs). It is basically done when a new PR
> >>> is
> >>>>> made
> >>>>>>>> and
> >>>>>>>>>> the job (say unix-cpu hasn't discovered the new PR branch
> >>> yet).
> >>>>>>> That's
> >>>>>>>>> it.
> >>>>>>>>>> So it shouldn't be a problem for public MXNet repo.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Chai
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <
> >>>>>>> marco.g.abreu@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Btw you forgot to set a date and time for the metting
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu <
> >>>>>>>>> marco.g.abreu@gmail.com
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Thanks Chai, I generally like the idea of the bot. But
> >>> I'm
> >>>>> not a
> >>>>>>>>>>> supporter
> >>>>>>>>>>>> of the idea to disable any automatic triggering
> >>> (disabling
> >>>>> the
> >>>>>>>>> webhook
> >>>>>>>>>> is
> >>>>>>>>>>>> also not an option, considering that this will disable
> >>>> master
> >>>>>>>>>> triggers).
> >>>>>>>>>>>> The CI system has to support the community without
> >>>> requiring
> >>>>>>> people
> >>>>>>>>> to
> >>>>>>>>>>>> constantly shepherd every single run. Disabling automatic
> >>>>>>>> triggering
> >>>>>>>>>>> seems
> >>>>>>>>>>>> like a step back to me.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Instead, I'd recommend that CI gets triggered upon every
> >>>>> commit
> >>>>>>> as
> >>>>>>>>>> usual,
> >>>>>>>>>>>> but people have the possibility to call a "command" (i.e.
> >>>>> make a
> >>>>>>>>>> message
> >>>>>>>>>>>> which results in the bot setting a label) to disable CI
> >>>> until
> >>>>>>> they
> >>>>>>>>>> revoke
> >>>>>>>>>>>> it. But the standard should still be that a new commit
> >>>>> triggers a
> >>>>>>>> new
> >>>>>>>>>> CI
> >>>>>>>>>>>> run.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>> https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
> >>>>>>>> seems
> >>>>>>>>>> like
> >>>>>>>>>>>> this would poll SCM. This will incur high quota
> >>>>> restrictions. Are
> >>>>>>>> you
> >>>>>>>>>>> sure
> >>>>>>>>>>>> that you can use that plugin at scale?
> >>>>>>>>>>>>
> >>>>>>>>>>>> -Marco
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <
> >>>>> apeforest@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>> Chai,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Awesome work. When do we expect this bot to be
> >>> deployed?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Lin
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <
> >>>>>>>>> chai.bapat@gmail.com
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hello MXNet community,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I have built an MXNet Bot <
> >>>> https://github.com/mxnet-bot>
> >>>>> that
> >>>>>>>>>> allows
> >>>>>>>>>>> PR
> >>>>>>>>>>>>>> Authors, Committers and Jenkins Admins to trigger CI
> >>>>> manually.
> >>>>>>>>>>>>>> It handles 2 problems
> >>>>>>>>>>>>>> 1. Manual CI trigger instead of existing automated CI
> >>>>> trigger
> >>>>>>>>>>>>>> 2. Gives permissions to PR Authors (in addition to
> >>>> MXNet
> >>>>>>>>> Committers
> >>>>>>>>>>> and
> >>>>>>>>>>>>>> Jenkins Admins)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Design Doc :
> >>>>>>>>>>>>>>
> >>>>>>> https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> >>>>>>>>>>>>>> I urge you all to attend the demonstration meeting
> >>> and
> >>>>> lend
> >>>>>>> your
> >>>>>>>>>> views
> >>>>>>>>>>>>> on
> >>>>>>>>>>>>>> the same.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thank you,
> >>>>>>>>>>>>>> Chai
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *Meeting Details*:
> >>>>>>>>>>>>>> ==============Conference Bridge
> >>>> Information==============
> >>>>>>>>>>>>>> You have been invited to an online meeting, powered
> >>> by
> >>>>> Amazon
> >>>>>>>>> Chime.
> >>>>>>>>>>>>>> *Chime meeting ID*: *9272158344*
> >>>>>>>>>>>>>> Join via Chime clients (manually): Select 'Meetings >
> >>>>> Join a
> >>>>>>>>>> Meeting',
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>> enter 9272158344
> >>>>>>>>>>>>>> Join via Chime clients (auto-call): If you invite
> >>>>> auto-call as
> >>>>>>>>>>> attendee,
> >>>>>>>>>>>>>> Chime will call you when the meeting starts, select
> >>>>> 'Answer'
> >>>>>>>>>>>>>> *Join via browser screen share*:
> >>>>> https://chime.aws/9272158344
> >>>>>>>>>>>>>> *Join via phone* (US): +1-929-432-4463,,,9272158344#
> >>>>>>>>>>>>>> *Join via phone (US toll-free)*:
> >>>>> +1-855-552-4463,,,9272158344#
> >>>>>>>>>>>>>> International dial-in:
> >>>> https://chime.aws/dialinnumbers/
> >>>>>>>>>>>>>> In-room video system: Ext: 62000, Meeting PIN:
> >>>>> 9272158344#
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>> *Chaitanya Prakash Bapat*
> >>>>>>>>>>>>>> *+1 (973) 953-6299*
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [image: https://www.linkedin.com//in/chaibapat25]
> >>>>>>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> >>>>>>>>>>>>> https://www.facebook.com/chaibapat
> >>>>>>>>>>>>>> ]
> >>>>>>>>>>>>>> <https://www.facebook.com/chaibapchya>[image:
> >>>>>>>>>>>>>> https://twitter.com/ChaiBapchya] <
> >>>>>>>> https://twitter.com/ChaiBapchya
> >>>>>>>>>>>>>> [image:
> >>>>>>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> >>>>>>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
> >>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> *Chaitanya Prakash Bapat*
> >>>>>>>>>> *+1 (973) 953-6299*
> >>>>>>>>>>
> >>>>>>>>>> [image: https://www.linkedin.com//in/chaibapat25]
> >>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> >>>>>>>>> https://www.facebook.com/chaibapat
> >>>>>>>>>> ]
> >>>>>>>>>> <https://www.facebook.com/chaibapchya>[image:
> >>>>>>>>>> https://twitter.com/ChaiBapchya] <
> >>>>> https://twitter.com/ChaiBapchya
> >>>>>>>>>> [image:
> >>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> >>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
> >>>>>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Sandeep Krishnamurthy
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> *Chaitanya Prakash Bapat*
> >>> *+1 (973) 953-6299*
> >>>
> >>> [image: https://www.linkedin.com//in/chaibapat25]
> >>> <https://github.com/ChaiBapchya>[image:
> https://www.facebook.com/chaibapat
> >>> ]
> >>> <https://www.facebook.com/chaibapchya>[image:
> >>> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> >[image:
> >>> https://www.linkedin.com//in/chaibapat25]
> >>> <https://www.linkedin.com//in/chaibapchya/>
> >>>
> >>
>
>
Re: MXNet Bot Demo
Posted by "Skalicky, Sam" <ss...@amazon.com.INVALID>.
We probably need some way to track which CI runs ran for which commit too, that way we can ensure that all CI runs ran on the commit that will be merged. Maybe the bot can comment with the commit hash when users command it to do something. Although since users can trigger individual CI runs its possible to have some commits run some CI runs but not others. We need some way to easily verify that the CI has executed all runs on the commit that will be merged.
Sam
> On Mar 13, 2020, at 8:28 PM, Przemysław Trędak <pt...@apache.org> wrote:
>
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
> I personally like the idea of opt-in more than opt-out:
> - ultimately PR author wants the PR to be merged so they (or committer reviewing the PR) will trigger the CI
> - if it is easy to trigger the PR via the bot command then the amount of work per PR should be less than with opt-out (since most of the commits should then be marked as [skip ci] or something similar
>
> +1 to the bot making a comment on each new PR with its commands (and also explaining, or at least giving links to the general PR process so new PR authors are not lost). Maybe we could make the bot recognize if the PR author is new or existing contributor and offer advice based on that?
>
> Thanks
> Przemek
>
> On 2020/03/13 22:06:58, Marco de Abreu <ma...@gmail.com> wrote:
>> Hi,
>>
>> since it's no longer necessary to push a new commit to trigger CI, it will
>> already reduce the costs. But to me, requiring an action to enable CI after
>> a PR has been created initially, is a no go. User can opt out of CI, but
>> the default has to be CI being triggered automatically for every commit
>> unless specifically disabled by a participant. I'm also fine with
>> triggering certain additional jobs (think about running a nightly job upon
>> request for a PR) to require a manual step, but the PR validation pipelines
>> have to run automatically. Every check that is marked as "Required" in
>> GitHub has to be automatically kicked off.
>>
>> -Marco
>>
>> On Fri, Mar 13, 2020 at 9:50 PM Chaitanya Bapat <ch...@gmail.com>
>> wrote:
>>
>>> Firstly,
>>> Sorry I missed out on attaching the mail thread that was sent on 12th
>>> February for notifying the community of the upcoming changes to the MXNet
>>> CI
>>> For reference :
>>>
>>> https://lists.apache.org/thread.html/r09a6ab2803a996fc80e00fe39ed312fa4865e8805e08df847f1addad%40%3Cdev.mxnet.apache.org%3E
>>>
>>> Now to the questions,
>>>> Is it possible for re-triggering a single job to be abused?
>>> @Tao In the case when a user re-triggers a single job multiple times, that
>>> will be visible in the PR conversation thread. A committer, even after he
>>> has approved the PR before, generally takes a look at the final state of
>>> the PR before merging. Would it be fair to assume the committer could take
>>> the multiple re-trigger of a single job into account before merging? The
>>> committer then has the option to invoke `@mxnet-bot run ci [all] ` to
>>> trigger the entire build pipeline one last to counter the abuse. This is
>>> aligned with what @Leonard said.
>>>
>>> @Sandeep Thanks a lot for collecting and sharing valuable data. I'd concur
>>> with the opinion that given the existing things committers & PR Authors
>>> take care of, invoking CI shouldn't be that big of an additional burden.
>>>
>>> @Marco With the opt-out, the onus remains on the PR Author. It doesn't help
>>> reduce the resource usage. Hence, it was suggested to switch to
>>> opt-in. @Leo's suggestion for proactive commenting on the part of bot makes
>>> sense and is doable.
>>>
>>> Default : opt-out and User initiated opt-in (with addressing Leo's fix for
>>> the usability issue you correctly pointed out )
>>> @Marco How does this sound to you?
>>>
>>> Again, thank you all for chiming in and voicing your opinion. Appreciate
>>> it.
>>> We can take ahead these discussions in today's demo meeting. [Design Doc
>>> <https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot>] [Demo
>>> Video <https://www.youtube.com/watch?v=gfOGwZId8aU>]
>>>
>>> Thanks,
>>> Chai
>>>
>>> On Fri, 13 Mar 2020 at 12:34, Marco de Abreu <ma...@gmail.com>
>>> wrote:
>>>
>>>> I'd recommend that the bot makes an initial comment when a PR gets opened
>>>> and informs the users of its commands. It then tells the user the commend
>>>> to opt out of CI.
>>>>
>>>> -Marco
>>>>
>>>> Lausen, Leonard <la...@amazon.com.invalid> schrieb am Fr., 13. März
>>> 2020,
>>>> 20:27:
>>>>
>>>>> On opt-out: People may be unaware of opt-out would not use it. There is
>>>> no
>>>>> incentive to use opt-out, as the PR author doesn't pay any money for CI
>>>>> run.
>>>>>
>>>>> I agree with Marco though that opt-in alone may cause usability issues,
>>>> as
>>>>> contributors may not be aware of how to trigger the CI.
>>>>> One solution is that the bot proactively comments on the PR and reminds
>>>> the
>>>>> author to trigger running CI once the author deems the PR ready.
>>>>>
>>>>> But even if we choose opt-out, the bot will still add a lot of value,
>>> as
>>>> PR
>>>>> authors can retrigger single jobs that have failed due to flakiness.
>>>>>
>>>>>> Is it possible for re-triggering a single job to be abused? For
>>>> example,
>>>>>> the author spends two days re-triggering a flaky job to make it pass.
>>>>>
>>>>> Yes, this is possible. I suggest the committer who likes to merge a PR
>>>>> needs to
>>>>> make a good judgement here if a PR is abusing the feature, and if so,
>>>>> retrigger
>>>>> all CI runs.
>>>>>
>>>>> Best regards
>>>>> Leonard
>>>>>
>>>>> On Fri, 2020-03-13 at 08:07 +0100, Marco de Abreu wrote:
>>>>>> Thanks for the data Sandeep. In these cases it sounds like it would
>>>> have
>>>>>> rather been better when people explicitly turned off CI in that case.
>>>>>> What's the argument against an opt-out instead of an opt-in?
>>>>>>
>>>>>> My intention is that I consider it quite cumbersome to make it a
>>>>> *required*
>>>>>> step to always trigger CI manually, even if just submitting a small
>>> PR.
>>>>> I'd
>>>>>> rather see people explicitly turning off CI if they wouldn't like to
>>>> use
>>>>> it
>>>>>> - and there's also the "draft" stage for a PR which some contributors
>>>> are
>>>>>> using.
>>>>>>
>>>>>> With regards to WIP and do not review: I think these are use cases
>>>> where
>>>>>> you want CI feedback, as otherwise you wouldn't have opened the PR.
>>> If
>>>>> you
>>>>>> don't want human feedback and neither machine feedback, why open the
>>> PR
>>>>> at
>>>>>> all?
>>>>>>
>>>>>> -Marco
>>>>>>
>>>>>>
>>>>>> sandeep krishnamurthy <sa...@gmail.com> schrieb am Fr.,
>>>> 13.
>>>>>> März 2020, 05:24:
>>>>>>
>>>>>>> I tried to gather some data for us to discuss this topic in this
>>>>> thread. I
>>>>>>> tried to count number of un-necessary builds by looking at most
>>>> recent
>>>>> (as
>>>>>>> of 12, March 9 PM PST) 50 PRs merged to master and 50 PRs.
>>>>>>> Identifying un-necessary builds is bit subjective. I tried to be
>>> more
>>>>>>> conservative where I didn't count a build as un-necessary if I was
>>> in
>>>>>>> doubt. Hence, I was not able to automate, but I made an effort to
>>> go
>>>>>>> through PRs manually and use below criteria to identify
>>> un-necessary
>>>>>>> commits triggering the builds.
>>>>>>>
>>>>>>> 1. Explicitly marked as WIP / do not review PR
>>>>>>> 2. Incremental WIP commit and finally commenting a commit
>>> “trigger
>>>>> CI”
>>>>>>> 3. Multiple commits to address all comments from single review.
>>>>> This is
>>>>>>> assuming we see a comment, address them, commit, next the
>>>> following
>>>>>>> comment
>>>>>>> 4. Sequence of documentation only changes
>>>>>>>
>>>>>>> I found there were around 42 avoidable builds from most recent 50
>>>>> merged
>>>>>>> PRs and around 86 builds from recent 50 open PRs.
>>>>>>>
>>>>>>> I synced up with other contributors (Joe Evans, Chai) from Amazon
>>> who
>>>>> is
>>>>>>> contributing to MXNet CI system. I was told that on an average it
>>>> costs
>>>>>>> around $84 per build and on an average 6 commits per merged PR (for
>>>>> year
>>>>>>> 2019). Going by that, it is approximately 1/6 builds are avoidable.
>>>>> [100 /
>>>>>>> 300 + 300 ]
>>>>>>>
>>>>>>>
>>>>>>> Usability should be top most priority. But, since either a reviewer
>>>> or
>>>>> pr
>>>>>>> author can trigger the bot, is it really a hurdle for pr author or
>>>>> reviewer
>>>>>>> to call a bot to trigger CI? Given that PR author and reviewer is
>>>>> already
>>>>>>> actively commenting various details such as - PR description,
>>> review
>>>>>>> comments and responses, adding labels etc.
>>>>>>>
>>>>>>>
>>>>>>> Me too curious to know the behavior for Tao's above use case.
>>>>>>>
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Sandeep
>>>>>>>
>>>>>>> On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <mu...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Is it possible for re-triggering a single job to be abused? For
>>>>> example,
>>>>>>>> the author spends two days re-triggering a flaky job to make it
>>>>> pass. But
>>>>>>>> other jobs which have passed the validation may be broken by
>>> other
>>>>>>> commits
>>>>>>>> during the two day without being noticed. And finally the PR is
>>>>> merged
>>>>>>> with
>>>>>>>> underlying problems.
>>>>>>>>
>>>>>>>> On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <
>>>>> marco.g.abreu@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> In the end it only comes down to money, considering that the
>>>>> system is
>>>>>>>> auto
>>>>>>>>> scaling, making the execution time constant.
>>>>>>>>>
>>>>>>>>> If we're trading money for usability, I certainly would prefer
>>>>>>> usability.
>>>>>>>>> I'd rather recommend to spend time on parallelizing test
>>>> execution
>>>>> or
>>>>>>>>> getting rid of integration tests in the PR stage instead
>>> reducing
>>>>> the
>>>>>>>> costs
>>>>>>>>> by making people not use it. But taking a step back to
>>> requiring
>>>>> people
>>>>>>>> to
>>>>>>>>> manually trigger CI again doesn't feel right.
>>>>>>>>>
>>>>>>>>> I'm happy to see that bot deployed, but I do not agree with
>>>>> removing
>>>>>>> the
>>>>>>>>> auto trigger functionality for new commits.
>>>>>>>>>
>>>>>>>>> -Marco
>>>>>>>>>
>>>>>>>>> Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 12.
>>> März
>>>>> 2020,
>>>>>>>>> 22:47:
>>>>>>>>>
>>>>>>>>>> @Marco Thanks for pointing that out.
>>>>>>>>>> Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM - 3:30 PM in
>>>>>>>> (UTC-08:00)
>>>>>>>>>> Pacific Time (US & Canada).
>>>>>>>>>>
>>>>>>>>>>> When do we expect this bot to be deployed?
>>>>>>>>>> @Lin If all goes well in the next week I can deploy it to
>>>> public
>>>>>>> Apache
>>>>>>>>>> (provided I get permissions from Apache Infra)
>>>>>>>>>>
>>>>>>>>>> @Marco Thanks for your feedback.
>>>>>>>>>>> CI system has to support the community without requiring
>>>>> people to
>>>>>>>>>> constantly shepherd every single run
>>>>>>>>>> We have data for the number of times CI was triggered
>>>>> unnecessarily
>>>>>>>> which
>>>>>>>>>> includes
>>>>>>>>>> - Entire build triggered instead of specific build
>>>>>>>>>> - CI triggered when PR is still work in progress or not yet
>>>> ready
>>>>>>> (say
>>>>>>>> -
>>>>>>>>>> intermediate commits)
>>>>>>>>>> At the end its a trade-off
>>>>>>>>>> Money, Resources, Time to build for each and every commit vs
>>>>> Pain of
>>>>>>>>>> triggering builds
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Scan trigger plugin would poll SCM. Can we use plugin at
>>>>> scale?
>>>>>>>>>>
>>>>>>>>>> 1. I haven't tested it on scale. But I think with the current
>>>>> scale
>>>>>>> of
>>>>>>>>>> MXNet repo (191 open PRs i.e. checking for changes to 191
>>>>> branches -
>>>>>>> It
>>>>>>>>>> should be manageable)
>>>>>>>>>> 2. What's the purpose of the plugin? tldr; Branch discovery
>>> or
>>>>> branch
>>>>>>>>>> indexing.
>>>>>>>>>> Scan trigger plugin comes into the picture only once per PR
>>> per
>>>>> job
>>>>>>>>> (i.e. 8
>>>>>>>>>> times per PR for 8 jobs). It is basically done when a new PR
>>> is
>>>>> made
>>>>>>>> and
>>>>>>>>>> the job (say unix-cpu hasn't discovered the new PR branch
>>> yet).
>>>>>>> That's
>>>>>>>>> it.
>>>>>>>>>> So it shouldn't be a problem for public MXNet repo.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Chai
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <
>>>>>>> marco.g.abreu@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Btw you forgot to set a date and time for the metting
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu <
>>>>>>>>> marco.g.abreu@gmail.com
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks Chai, I generally like the idea of the bot. But
>>> I'm
>>>>> not a
>>>>>>>>>>> supporter
>>>>>>>>>>>> of the idea to disable any automatic triggering
>>> (disabling
>>>>> the
>>>>>>>>> webhook
>>>>>>>>>> is
>>>>>>>>>>>> also not an option, considering that this will disable
>>>> master
>>>>>>>>>> triggers).
>>>>>>>>>>>> The CI system has to support the community without
>>>> requiring
>>>>>>> people
>>>>>>>>> to
>>>>>>>>>>>> constantly shepherd every single run. Disabling automatic
>>>>>>>> triggering
>>>>>>>>>>> seems
>>>>>>>>>>>> like a step back to me.
>>>>>>>>>>>>
>>>>>>>>>>>> Instead, I'd recommend that CI gets triggered upon every
>>>>> commit
>>>>>>> as
>>>>>>>>>> usual,
>>>>>>>>>>>> but people have the possibility to call a "command" (i.e.
>>>>> make a
>>>>>>>>>> message
>>>>>>>>>>>> which results in the bot setting a label) to disable CI
>>>> until
>>>>>>> they
>>>>>>>>>> revoke
>>>>>>>>>>>> it. But the standard should still be that a new commit
>>>>> triggers a
>>>>>>>> new
>>>>>>>>>> CI
>>>>>>>>>>>> run.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>> https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
>>>>>>>> seems
>>>>>>>>>> like
>>>>>>>>>>>> this would poll SCM. This will incur high quota
>>>>> restrictions. Are
>>>>>>>> you
>>>>>>>>>>> sure
>>>>>>>>>>>> that you can use that plugin at scale?
>>>>>>>>>>>>
>>>>>>>>>>>> -Marco
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <
>>>>> apeforest@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>> Chai,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Awesome work. When do we expect this bot to be
>>> deployed?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Lin
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <
>>>>>>>>> chai.bapat@gmail.com
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello MXNet community,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have built an MXNet Bot <
>>>> https://github.com/mxnet-bot>
>>>>> that
>>>>>>>>>> allows
>>>>>>>>>>> PR
>>>>>>>>>>>>>> Authors, Committers and Jenkins Admins to trigger CI
>>>>> manually.
>>>>>>>>>>>>>> It handles 2 problems
>>>>>>>>>>>>>> 1. Manual CI trigger instead of existing automated CI
>>>>> trigger
>>>>>>>>>>>>>> 2. Gives permissions to PR Authors (in addition to
>>>> MXNet
>>>>>>>>> Committers
>>>>>>>>>>> and
>>>>>>>>>>>>>> Jenkins Admins)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Design Doc :
>>>>>>>>>>>>>>
>>>>>>> https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
>>>>>>>>>>>>>> I urge you all to attend the demonstration meeting
>>> and
>>>>> lend
>>>>>>> your
>>>>>>>>>> views
>>>>>>>>>>>>> on
>>>>>>>>>>>>>> the same.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>> Chai
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Meeting Details*:
>>>>>>>>>>>>>> ==============Conference Bridge
>>>> Information==============
>>>>>>>>>>>>>> You have been invited to an online meeting, powered
>>> by
>>>>> Amazon
>>>>>>>>> Chime.
>>>>>>>>>>>>>> *Chime meeting ID*: *9272158344*
>>>>>>>>>>>>>> Join via Chime clients (manually): Select 'Meetings >
>>>>> Join a
>>>>>>>>>> Meeting',
>>>>>>>>>>>>> and
>>>>>>>>>>>>>> enter 9272158344
>>>>>>>>>>>>>> Join via Chime clients (auto-call): If you invite
>>>>> auto-call as
>>>>>>>>>>> attendee,
>>>>>>>>>>>>>> Chime will call you when the meeting starts, select
>>>>> 'Answer'
>>>>>>>>>>>>>> *Join via browser screen share*:
>>>>> https://chime.aws/9272158344
>>>>>>>>>>>>>> *Join via phone* (US): +1-929-432-4463,,,9272158344#
>>>>>>>>>>>>>> *Join via phone (US toll-free)*:
>>>>> +1-855-552-4463,,,9272158344#
>>>>>>>>>>>>>> International dial-in:
>>>> https://chime.aws/dialinnumbers/
>>>>>>>>>>>>>> In-room video system: Ext: 62000, Meeting PIN:
>>>>> 9272158344#
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> *Chaitanya Prakash Bapat*
>>>>>>>>>>>>>> *+1 (973) 953-6299*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [image: https://www.linkedin.com//in/chaibapat25]
>>>>>>>>>>>>>> <https://github.com/ChaiBapchya>[image:
>>>>>>>>>>>>> https://www.facebook.com/chaibapat
>>>>>>>>>>>>>> ]
>>>>>>>>>>>>>> <https://www.facebook.com/chaibapchya>[image:
>>>>>>>>>>>>>> https://twitter.com/ChaiBapchya] <
>>>>>>>> https://twitter.com/ChaiBapchya
>>>>>>>>>>>>>> [image:
>>>>>>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
>>>>>>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> *Chaitanya Prakash Bapat*
>>>>>>>>>> *+1 (973) 953-6299*
>>>>>>>>>>
>>>>>>>>>> [image: https://www.linkedin.com//in/chaibapat25]
>>>>>>>>>> <https://github.com/ChaiBapchya>[image:
>>>>>>>>> https://www.facebook.com/chaibapat
>>>>>>>>>> ]
>>>>>>>>>> <https://www.facebook.com/chaibapchya>[image:
>>>>>>>>>> https://twitter.com/ChaiBapchya] <
>>>>> https://twitter.com/ChaiBapchya
>>>>>>>>>> [image:
>>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
>>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
>>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Sandeep Krishnamurthy
>>>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> *Chaitanya Prakash Bapat*
>>> *+1 (973) 953-6299*
>>>
>>> [image: https://www.linkedin.com//in/chaibapat25]
>>> <https://github.com/ChaiBapchya>[image: https://www.facebook.com/chaibapat
>>> ]
>>> <https://www.facebook.com/chaibapchya>[image:
>>> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
>>> https://www.linkedin.com//in/chaibapat25]
>>> <https://www.linkedin.com//in/chaibapchya/>
>>>
>>
Re: MXNet Bot Demo
Posted by Przemys��aw Tr��dak <pt...@apache.org>.
I personally like the idea of opt-in more than opt-out:
- ultimately PR author wants the PR to be merged so they (or committer reviewing the PR) will trigger the CI
- if it is easy to trigger the PR via the bot command then the amount of work per PR should be less than with opt-out (since most of the commits should then be marked as [skip ci] or something similar
+1 to the bot making a comment on each new PR with its commands (and also explaining, or at least giving links to the general PR process so new PR authors are not lost). Maybe we could make the bot recognize if the PR author is new or existing contributor and offer advice based on that?
Thanks
Przemek
On 2020/03/13 22:06:58, Marco de Abreu <ma...@gmail.com> wrote:
> Hi,
>
> since it's no longer necessary to push a new commit to trigger CI, it will
> already reduce the costs. But to me, requiring an action to enable CI after
> a PR has been created initially, is a no go. User can opt out of CI, but
> the default has to be CI being triggered automatically for every commit
> unless specifically disabled by a participant. I'm also fine with
> triggering certain additional jobs (think about running a nightly job upon
> request for a PR) to require a manual step, but the PR validation pipelines
> have to run automatically. Every check that is marked as "Required" in
> GitHub has to be automatically kicked off.
>
> -Marco
>
> On Fri, Mar 13, 2020 at 9:50 PM Chaitanya Bapat <ch...@gmail.com>
> wrote:
>
> > Firstly,
> > Sorry I missed out on attaching the mail thread that was sent on 12th
> > February for notifying the community of the upcoming changes to the MXNet
> > CI
> > For reference :
> >
> > https://lists.apache.org/thread.html/r09a6ab2803a996fc80e00fe39ed312fa4865e8805e08df847f1addad%40%3Cdev.mxnet.apache.org%3E
> >
> > Now to the questions,
> > > Is it possible for re-triggering a single job to be abused?
> > @Tao In the case when a user re-triggers a single job multiple times, that
> > will be visible in the PR conversation thread. A committer, even after he
> > has approved the PR before, generally takes a look at the final state of
> > the PR before merging. Would it be fair to assume the committer could take
> > the multiple re-trigger of a single job into account before merging? The
> > committer then has the option to invoke `@mxnet-bot run ci [all] ` to
> > trigger the entire build pipeline one last to counter the abuse. This is
> > aligned with what @Leonard said.
> >
> > @Sandeep Thanks a lot for collecting and sharing valuable data. I'd concur
> > with the opinion that given the existing things committers & PR Authors
> > take care of, invoking CI shouldn't be that big of an additional burden.
> >
> > @Marco With the opt-out, the onus remains on the PR Author. It doesn't help
> > reduce the resource usage. Hence, it was suggested to switch to
> > opt-in. @Leo's suggestion for proactive commenting on the part of bot makes
> > sense and is doable.
> >
> > Default : opt-out and User initiated opt-in (with addressing Leo's fix for
> > the usability issue you correctly pointed out )
> > @Marco How does this sound to you?
> >
> > Again, thank you all for chiming in and voicing your opinion. Appreciate
> > it.
> > We can take ahead these discussions in today's demo meeting. [Design Doc
> > <https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot>] [Demo
> > Video <https://www.youtube.com/watch?v=gfOGwZId8aU>]
> >
> > Thanks,
> > Chai
> >
> > On Fri, 13 Mar 2020 at 12:34, Marco de Abreu <ma...@gmail.com>
> > wrote:
> >
> > > I'd recommend that the bot makes an initial comment when a PR gets opened
> > > and informs the users of its commands. It then tells the user the commend
> > > to opt out of CI.
> > >
> > > -Marco
> > >
> > > Lausen, Leonard <la...@amazon.com.invalid> schrieb am Fr., 13. März
> > 2020,
> > > 20:27:
> > >
> > > > On opt-out: People may be unaware of opt-out would not use it. There is
> > > no
> > > > incentive to use opt-out, as the PR author doesn't pay any money for CI
> > > > run.
> > > >
> > > > I agree with Marco though that opt-in alone may cause usability issues,
> > > as
> > > > contributors may not be aware of how to trigger the CI.
> > > > One solution is that the bot proactively comments on the PR and reminds
> > > the
> > > > author to trigger running CI once the author deems the PR ready.
> > > >
> > > > But even if we choose opt-out, the bot will still add a lot of value,
> > as
> > > PR
> > > > authors can retrigger single jobs that have failed due to flakiness.
> > > >
> > > > > Is it possible for re-triggering a single job to be abused? For
> > > example,
> > > > > the author spends two days re-triggering a flaky job to make it pass.
> > > >
> > > > Yes, this is possible. I suggest the committer who likes to merge a PR
> > > > needs to
> > > > make a good judgement here if a PR is abusing the feature, and if so,
> > > > retrigger
> > > > all CI runs.
> > > >
> > > > Best regards
> > > > Leonard
> > > >
> > > > On Fri, 2020-03-13 at 08:07 +0100, Marco de Abreu wrote:
> > > > > Thanks for the data Sandeep. In these cases it sounds like it would
> > > have
> > > > > rather been better when people explicitly turned off CI in that case.
> > > > > What's the argument against an opt-out instead of an opt-in?
> > > > >
> > > > > My intention is that I consider it quite cumbersome to make it a
> > > > *required*
> > > > > step to always trigger CI manually, even if just submitting a small
> > PR.
> > > > I'd
> > > > > rather see people explicitly turning off CI if they wouldn't like to
> > > use
> > > > it
> > > > > - and there's also the "draft" stage for a PR which some contributors
> > > are
> > > > > using.
> > > > >
> > > > > With regards to WIP and do not review: I think these are use cases
> > > where
> > > > > you want CI feedback, as otherwise you wouldn't have opened the PR.
> > If
> > > > you
> > > > > don't want human feedback and neither machine feedback, why open the
> > PR
> > > > at
> > > > > all?
> > > > >
> > > > > -Marco
> > > > >
> > > > >
> > > > > sandeep krishnamurthy <sa...@gmail.com> schrieb am Fr.,
> > > 13.
> > > > > März 2020, 05:24:
> > > > >
> > > > > > I tried to gather some data for us to discuss this topic in this
> > > > thread. I
> > > > > > tried to count number of un-necessary builds by looking at most
> > > recent
> > > > (as
> > > > > > of 12, March 9 PM PST) 50 PRs merged to master and 50 PRs.
> > > > > > Identifying un-necessary builds is bit subjective. I tried to be
> > more
> > > > > > conservative where I didn't count a build as un-necessary if I was
> > in
> > > > > > doubt. Hence, I was not able to automate, but I made an effort to
> > go
> > > > > > through PRs manually and use below criteria to identify
> > un-necessary
> > > > > > commits triggering the builds.
> > > > > >
> > > > > > 1. Explicitly marked as WIP / do not review PR
> > > > > > 2. Incremental WIP commit and finally commenting a commit
> > “trigger
> > > > CI”
> > > > > > 3. Multiple commits to address all comments from single review.
> > > > This is
> > > > > > assuming we see a comment, address them, commit, next the
> > > following
> > > > > > comment
> > > > > > 4. Sequence of documentation only changes
> > > > > >
> > > > > > I found there were around 42 avoidable builds from most recent 50
> > > > merged
> > > > > > PRs and around 86 builds from recent 50 open PRs.
> > > > > >
> > > > > > I synced up with other contributors (Joe Evans, Chai) from Amazon
> > who
> > > > is
> > > > > > contributing to MXNet CI system. I was told that on an average it
> > > costs
> > > > > > around $84 per build and on an average 6 commits per merged PR (for
> > > > year
> > > > > > 2019). Going by that, it is approximately 1/6 builds are avoidable.
> > > > [100 /
> > > > > > 300 + 300 ]
> > > > > >
> > > > > >
> > > > > > Usability should be top most priority. But, since either a reviewer
> > > or
> > > > pr
> > > > > > author can trigger the bot, is it really a hurdle for pr author or
> > > > reviewer
> > > > > > to call a bot to trigger CI? Given that PR author and reviewer is
> > > > already
> > > > > > actively commenting various details such as - PR description,
> > review
> > > > > > comments and responses, adding labels etc.
> > > > > >
> > > > > >
> > > > > > Me too curious to know the behavior for Tao's above use case.
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Sandeep
> > > > > >
> > > > > > On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <mu...@gmail.com> wrote:
> > > > > >
> > > > > > > Is it possible for re-triggering a single job to be abused? For
> > > > example,
> > > > > > > the author spends two days re-triggering a flaky job to make it
> > > > pass. But
> > > > > > > other jobs which have passed the validation may be broken by
> > other
> > > > > > commits
> > > > > > > during the two day without being noticed. And finally the PR is
> > > > merged
> > > > > > with
> > > > > > > underlying problems.
> > > > > > >
> > > > > > > On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <
> > > > marco.g.abreu@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > In the end it only comes down to money, considering that the
> > > > system is
> > > > > > > auto
> > > > > > > > scaling, making the execution time constant.
> > > > > > > >
> > > > > > > > If we're trading money for usability, I certainly would prefer
> > > > > > usability.
> > > > > > > > I'd rather recommend to spend time on parallelizing test
> > > execution
> > > > or
> > > > > > > > getting rid of integration tests in the PR stage instead
> > reducing
> > > > the
> > > > > > > costs
> > > > > > > > by making people not use it. But taking a step back to
> > requiring
> > > > people
> > > > > > > to
> > > > > > > > manually trigger CI again doesn't feel right.
> > > > > > > >
> > > > > > > > I'm happy to see that bot deployed, but I do not agree with
> > > > removing
> > > > > > the
> > > > > > > > auto trigger functionality for new commits.
> > > > > > > >
> > > > > > > > -Marco
> > > > > > > >
> > > > > > > > Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 12.
> > März
> > > > 2020,
> > > > > > > > 22:47:
> > > > > > > >
> > > > > > > > > @Marco Thanks for pointing that out.
> > > > > > > > > Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM - 3:30 PM in
> > > > > > > (UTC-08:00)
> > > > > > > > > Pacific Time (US & Canada).
> > > > > > > > >
> > > > > > > > > > When do we expect this bot to be deployed?
> > > > > > > > > @Lin If all goes well in the next week I can deploy it to
> > > public
> > > > > > Apache
> > > > > > > > > (provided I get permissions from Apache Infra)
> > > > > > > > >
> > > > > > > > > @Marco Thanks for your feedback.
> > > > > > > > > > CI system has to support the community without requiring
> > > > people to
> > > > > > > > > constantly shepherd every single run
> > > > > > > > > We have data for the number of times CI was triggered
> > > > unnecessarily
> > > > > > > which
> > > > > > > > > includes
> > > > > > > > > - Entire build triggered instead of specific build
> > > > > > > > > - CI triggered when PR is still work in progress or not yet
> > > ready
> > > > > > (say
> > > > > > > -
> > > > > > > > > intermediate commits)
> > > > > > > > > At the end its a trade-off
> > > > > > > > > Money, Resources, Time to build for each and every commit vs
> > > > Pain of
> > > > > > > > > triggering builds
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > Scan trigger plugin would poll SCM. Can we use plugin at
> > > > scale?
> > > > > > > > >
> > > > > > > > > 1. I haven't tested it on scale. But I think with the current
> > > > scale
> > > > > > of
> > > > > > > > > MXNet repo (191 open PRs i.e. checking for changes to 191
> > > > branches -
> > > > > > It
> > > > > > > > > should be manageable)
> > > > > > > > > 2. What's the purpose of the plugin? tldr; Branch discovery
> > or
> > > > branch
> > > > > > > > > indexing.
> > > > > > > > > Scan trigger plugin comes into the picture only once per PR
> > per
> > > > job
> > > > > > > > (i.e. 8
> > > > > > > > > times per PR for 8 jobs). It is basically done when a new PR
> > is
> > > > made
> > > > > > > and
> > > > > > > > > the job (say unix-cpu hasn't discovered the new PR branch
> > yet).
> > > > > > That's
> > > > > > > > it.
> > > > > > > > > So it shouldn't be a problem for public MXNet repo.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Chai
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <
> > > > > > marco.g.abreu@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Btw you forgot to set a date and time for the metting
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu <
> > > > > > > > marco.g.abreu@gmail.com
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks Chai, I generally like the idea of the bot. But
> > I'm
> > > > not a
> > > > > > > > > > supporter
> > > > > > > > > > > of the idea to disable any automatic triggering
> > (disabling
> > > > the
> > > > > > > > webhook
> > > > > > > > > is
> > > > > > > > > > > also not an option, considering that this will disable
> > > master
> > > > > > > > > triggers).
> > > > > > > > > > > The CI system has to support the community without
> > > requiring
> > > > > > people
> > > > > > > > to
> > > > > > > > > > > constantly shepherd every single run. Disabling automatic
> > > > > > > triggering
> > > > > > > > > > seems
> > > > > > > > > > > like a step back to me.
> > > > > > > > > > >
> > > > > > > > > > > Instead, I'd recommend that CI gets triggered upon every
> > > > commit
> > > > > > as
> > > > > > > > > usual,
> > > > > > > > > > > but people have the possibility to call a "command" (i.e.
> > > > make a
> > > > > > > > > message
> > > > > > > > > > > which results in the bot setting a label) to disable CI
> > > until
> > > > > > they
> > > > > > > > > revoke
> > > > > > > > > > > it. But the standard should still be that a new commit
> > > > triggers a
> > > > > > > new
> > > > > > > > > CI
> > > > > > > > > > > run.
> > > > > > > > > > >
> > > > > > > > > > >
> > > https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
> > > > > > > seems
> > > > > > > > > like
> > > > > > > > > > > this would poll SCM. This will incur high quota
> > > > restrictions. Are
> > > > > > > you
> > > > > > > > > > sure
> > > > > > > > > > > that you can use that plugin at scale?
> > > > > > > > > > >
> > > > > > > > > > > -Marco
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <
> > > > apeforest@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > > > Chai,
> > > > > > > > > > > >
> > > > > > > > > > > > Awesome work. When do we expect this bot to be
> > deployed?
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > >
> > > > > > > > > > > > Lin
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <
> > > > > > > > chai.bapat@gmail.com
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hello MXNet community,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I have built an MXNet Bot <
> > > https://github.com/mxnet-bot>
> > > > that
> > > > > > > > > allows
> > > > > > > > > > PR
> > > > > > > > > > > > > Authors, Committers and Jenkins Admins to trigger CI
> > > > manually.
> > > > > > > > > > > > > It handles 2 problems
> > > > > > > > > > > > > 1. Manual CI trigger instead of existing automated CI
> > > > trigger
> > > > > > > > > > > > > 2. Gives permissions to PR Authors (in addition to
> > > MXNet
> > > > > > > > Committers
> > > > > > > > > > and
> > > > > > > > > > > > > Jenkins Admins)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Design Doc :
> > > > > > > > > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > > > > > > > > > > > > I urge you all to attend the demonstration meeting
> > and
> > > > lend
> > > > > > your
> > > > > > > > > views
> > > > > > > > > > > > on
> > > > > > > > > > > > > the same.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thank you,
> > > > > > > > > > > > > Chai
> > > > > > > > > > > > >
> > > > > > > > > > > > > *Meeting Details*:
> > > > > > > > > > > > > ==============Conference Bridge
> > > Information==============
> > > > > > > > > > > > > You have been invited to an online meeting, powered
> > by
> > > > Amazon
> > > > > > > > Chime.
> > > > > > > > > > > > > *Chime meeting ID*: *9272158344*
> > > > > > > > > > > > > Join via Chime clients (manually): Select 'Meetings >
> > > > Join a
> > > > > > > > > Meeting',
> > > > > > > > > > > > and
> > > > > > > > > > > > > enter 9272158344
> > > > > > > > > > > > > Join via Chime clients (auto-call): If you invite
> > > > auto-call as
> > > > > > > > > > attendee,
> > > > > > > > > > > > > Chime will call you when the meeting starts, select
> > > > 'Answer'
> > > > > > > > > > > > > *Join via browser screen share*:
> > > > https://chime.aws/9272158344
> > > > > > > > > > > > > *Join via phone* (US): +1-929-432-4463,,,9272158344#
> > > > > > > > > > > > > *Join via phone (US toll-free)*:
> > > > +1-855-552-4463,,,9272158344#
> > > > > > > > > > > > > International dial-in:
> > > https://chime.aws/dialinnumbers/
> > > > > > > > > > > > > In-room video system: Ext: 62000, Meeting PIN:
> > > > 9272158344#
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > *Chaitanya Prakash Bapat*
> > > > > > > > > > > > > *+1 (973) 953-6299*
> > > > > > > > > > > > >
> > > > > > > > > > > > > [image: https://www.linkedin.com//in/chaibapat25]
> > > > > > > > > > > > > <https://github.com/ChaiBapchya>[image:
> > > > > > > > > > > > https://www.facebook.com/chaibapat
> > > > > > > > > > > > > ]
> > > > > > > > > > > > > <https://www.facebook.com/chaibapchya>[image:
> > > > > > > > > > > > > https://twitter.com/ChaiBapchya] <
> > > > > > > https://twitter.com/ChaiBapchya
> > > > > > > > > > > > > [image:
> > > > > > > > > > > > > https://www.linkedin.com//in/chaibapat25]
> > > > > > > > > > > > > <https://www.linkedin.com//in/chaibapchya/>
> > > > > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > *Chaitanya Prakash Bapat*
> > > > > > > > > *+1 (973) 953-6299*
> > > > > > > > >
> > > > > > > > > [image: https://www.linkedin.com//in/chaibapat25]
> > > > > > > > > <https://github.com/ChaiBapchya>[image:
> > > > > > > > https://www.facebook.com/chaibapat
> > > > > > > > > ]
> > > > > > > > > <https://www.facebook.com/chaibapchya>[image:
> > > > > > > > > https://twitter.com/ChaiBapchya] <
> > > > https://twitter.com/ChaiBapchya
> > > > > > > > > [image:
> > > > > > > > > https://www.linkedin.com//in/chaibapat25]
> > > > > > > > > <https://www.linkedin.com//in/chaibapchya/>
> > > > > > > > >
> > > > > >
> > > > > > --
> > > > > > Sandeep Krishnamurthy
> > > > > >
> > > >
> > >
> >
> >
> > --
> > *Chaitanya Prakash Bapat*
> > *+1 (973) 953-6299*
> >
> > [image: https://www.linkedin.com//in/chaibapat25]
> > <https://github.com/ChaiBapchya>[image: https://www.facebook.com/chaibapat
> > ]
> > <https://www.facebook.com/chaibapchya>[image:
> > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
> > https://www.linkedin.com//in/chaibapat25]
> > <https://www.linkedin.com//in/chaibapchya/>
> >
>
Re: MXNet Bot Demo
Posted by Marco de Abreu <ma...@gmail.com>.
Hi,
since it's no longer necessary to push a new commit to trigger CI, it will
already reduce the costs. But to me, requiring an action to enable CI after
a PR has been created initially, is a no go. User can opt out of CI, but
the default has to be CI being triggered automatically for every commit
unless specifically disabled by a participant. I'm also fine with
triggering certain additional jobs (think about running a nightly job upon
request for a PR) to require a manual step, but the PR validation pipelines
have to run automatically. Every check that is marked as "Required" in
GitHub has to be automatically kicked off.
-Marco
On Fri, Mar 13, 2020 at 9:50 PM Chaitanya Bapat <ch...@gmail.com>
wrote:
> Firstly,
> Sorry I missed out on attaching the mail thread that was sent on 12th
> February for notifying the community of the upcoming changes to the MXNet
> CI
> For reference :
>
> https://lists.apache.org/thread.html/r09a6ab2803a996fc80e00fe39ed312fa4865e8805e08df847f1addad%40%3Cdev.mxnet.apache.org%3E
>
> Now to the questions,
> > Is it possible for re-triggering a single job to be abused?
> @Tao In the case when a user re-triggers a single job multiple times, that
> will be visible in the PR conversation thread. A committer, even after he
> has approved the PR before, generally takes a look at the final state of
> the PR before merging. Would it be fair to assume the committer could take
> the multiple re-trigger of a single job into account before merging? The
> committer then has the option to invoke `@mxnet-bot run ci [all] ` to
> trigger the entire build pipeline one last to counter the abuse. This is
> aligned with what @Leonard said.
>
> @Sandeep Thanks a lot for collecting and sharing valuable data. I'd concur
> with the opinion that given the existing things committers & PR Authors
> take care of, invoking CI shouldn't be that big of an additional burden.
>
> @Marco With the opt-out, the onus remains on the PR Author. It doesn't help
> reduce the resource usage. Hence, it was suggested to switch to
> opt-in. @Leo's suggestion for proactive commenting on the part of bot makes
> sense and is doable.
>
> Default : opt-out and User initiated opt-in (with addressing Leo's fix for
> the usability issue you correctly pointed out )
> @Marco How does this sound to you?
>
> Again, thank you all for chiming in and voicing your opinion. Appreciate
> it.
> We can take ahead these discussions in today's demo meeting. [Design Doc
> <https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot>] [Demo
> Video <https://www.youtube.com/watch?v=gfOGwZId8aU>]
>
> Thanks,
> Chai
>
> On Fri, 13 Mar 2020 at 12:34, Marco de Abreu <ma...@gmail.com>
> wrote:
>
> > I'd recommend that the bot makes an initial comment when a PR gets opened
> > and informs the users of its commands. It then tells the user the commend
> > to opt out of CI.
> >
> > -Marco
> >
> > Lausen, Leonard <la...@amazon.com.invalid> schrieb am Fr., 13. März
> 2020,
> > 20:27:
> >
> > > On opt-out: People may be unaware of opt-out would not use it. There is
> > no
> > > incentive to use opt-out, as the PR author doesn't pay any money for CI
> > > run.
> > >
> > > I agree with Marco though that opt-in alone may cause usability issues,
> > as
> > > contributors may not be aware of how to trigger the CI.
> > > One solution is that the bot proactively comments on the PR and reminds
> > the
> > > author to trigger running CI once the author deems the PR ready.
> > >
> > > But even if we choose opt-out, the bot will still add a lot of value,
> as
> > PR
> > > authors can retrigger single jobs that have failed due to flakiness.
> > >
> > > > Is it possible for re-triggering a single job to be abused? For
> > example,
> > > > the author spends two days re-triggering a flaky job to make it pass.
> > >
> > > Yes, this is possible. I suggest the committer who likes to merge a PR
> > > needs to
> > > make a good judgement here if a PR is abusing the feature, and if so,
> > > retrigger
> > > all CI runs.
> > >
> > > Best regards
> > > Leonard
> > >
> > > On Fri, 2020-03-13 at 08:07 +0100, Marco de Abreu wrote:
> > > > Thanks for the data Sandeep. In these cases it sounds like it would
> > have
> > > > rather been better when people explicitly turned off CI in that case.
> > > > What's the argument against an opt-out instead of an opt-in?
> > > >
> > > > My intention is that I consider it quite cumbersome to make it a
> > > *required*
> > > > step to always trigger CI manually, even if just submitting a small
> PR.
> > > I'd
> > > > rather see people explicitly turning off CI if they wouldn't like to
> > use
> > > it
> > > > - and there's also the "draft" stage for a PR which some contributors
> > are
> > > > using.
> > > >
> > > > With regards to WIP and do not review: I think these are use cases
> > where
> > > > you want CI feedback, as otherwise you wouldn't have opened the PR.
> If
> > > you
> > > > don't want human feedback and neither machine feedback, why open the
> PR
> > > at
> > > > all?
> > > >
> > > > -Marco
> > > >
> > > >
> > > > sandeep krishnamurthy <sa...@gmail.com> schrieb am Fr.,
> > 13.
> > > > März 2020, 05:24:
> > > >
> > > > > I tried to gather some data for us to discuss this topic in this
> > > thread. I
> > > > > tried to count number of un-necessary builds by looking at most
> > recent
> > > (as
> > > > > of 12, March 9 PM PST) 50 PRs merged to master and 50 PRs.
> > > > > Identifying un-necessary builds is bit subjective. I tried to be
> more
> > > > > conservative where I didn't count a build as un-necessary if I was
> in
> > > > > doubt. Hence, I was not able to automate, but I made an effort to
> go
> > > > > through PRs manually and use below criteria to identify
> un-necessary
> > > > > commits triggering the builds.
> > > > >
> > > > > 1. Explicitly marked as WIP / do not review PR
> > > > > 2. Incremental WIP commit and finally commenting a commit
> “trigger
> > > CI”
> > > > > 3. Multiple commits to address all comments from single review.
> > > This is
> > > > > assuming we see a comment, address them, commit, next the
> > following
> > > > > comment
> > > > > 4. Sequence of documentation only changes
> > > > >
> > > > > I found there were around 42 avoidable builds from most recent 50
> > > merged
> > > > > PRs and around 86 builds from recent 50 open PRs.
> > > > >
> > > > > I synced up with other contributors (Joe Evans, Chai) from Amazon
> who
> > > is
> > > > > contributing to MXNet CI system. I was told that on an average it
> > costs
> > > > > around $84 per build and on an average 6 commits per merged PR (for
> > > year
> > > > > 2019). Going by that, it is approximately 1/6 builds are avoidable.
> > > [100 /
> > > > > 300 + 300 ]
> > > > >
> > > > >
> > > > > Usability should be top most priority. But, since either a reviewer
> > or
> > > pr
> > > > > author can trigger the bot, is it really a hurdle for pr author or
> > > reviewer
> > > > > to call a bot to trigger CI? Given that PR author and reviewer is
> > > already
> > > > > actively commenting various details such as - PR description,
> review
> > > > > comments and responses, adding labels etc.
> > > > >
> > > > >
> > > > > Me too curious to know the behavior for Tao's above use case.
> > > > >
> > > > >
> > > > > Best,
> > > > >
> > > > > Sandeep
> > > > >
> > > > > On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <mu...@gmail.com> wrote:
> > > > >
> > > > > > Is it possible for re-triggering a single job to be abused? For
> > > example,
> > > > > > the author spends two days re-triggering a flaky job to make it
> > > pass. But
> > > > > > other jobs which have passed the validation may be broken by
> other
> > > > > commits
> > > > > > during the two day without being noticed. And finally the PR is
> > > merged
> > > > > with
> > > > > > underlying problems.
> > > > > >
> > > > > > On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <
> > > marco.g.abreu@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > In the end it only comes down to money, considering that the
> > > system is
> > > > > > auto
> > > > > > > scaling, making the execution time constant.
> > > > > > >
> > > > > > > If we're trading money for usability, I certainly would prefer
> > > > > usability.
> > > > > > > I'd rather recommend to spend time on parallelizing test
> > execution
> > > or
> > > > > > > getting rid of integration tests in the PR stage instead
> reducing
> > > the
> > > > > > costs
> > > > > > > by making people not use it. But taking a step back to
> requiring
> > > people
> > > > > > to
> > > > > > > manually trigger CI again doesn't feel right.
> > > > > > >
> > > > > > > I'm happy to see that bot deployed, but I do not agree with
> > > removing
> > > > > the
> > > > > > > auto trigger functionality for new commits.
> > > > > > >
> > > > > > > -Marco
> > > > > > >
> > > > > > > Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 12.
> März
> > > 2020,
> > > > > > > 22:47:
> > > > > > >
> > > > > > > > @Marco Thanks for pointing that out.
> > > > > > > > Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM - 3:30 PM in
> > > > > > (UTC-08:00)
> > > > > > > > Pacific Time (US & Canada).
> > > > > > > >
> > > > > > > > > When do we expect this bot to be deployed?
> > > > > > > > @Lin If all goes well in the next week I can deploy it to
> > public
> > > > > Apache
> > > > > > > > (provided I get permissions from Apache Infra)
> > > > > > > >
> > > > > > > > @Marco Thanks for your feedback.
> > > > > > > > > CI system has to support the community without requiring
> > > people to
> > > > > > > > constantly shepherd every single run
> > > > > > > > We have data for the number of times CI was triggered
> > > unnecessarily
> > > > > > which
> > > > > > > > includes
> > > > > > > > - Entire build triggered instead of specific build
> > > > > > > > - CI triggered when PR is still work in progress or not yet
> > ready
> > > > > (say
> > > > > > -
> > > > > > > > intermediate commits)
> > > > > > > > At the end its a trade-off
> > > > > > > > Money, Resources, Time to build for each and every commit vs
> > > Pain of
> > > > > > > > triggering builds
> > > > > > > >
> > > > > > > >
> > > > > > > > > Scan trigger plugin would poll SCM. Can we use plugin at
> > > scale?
> > > > > > > >
> > > > > > > > 1. I haven't tested it on scale. But I think with the current
> > > scale
> > > > > of
> > > > > > > > MXNet repo (191 open PRs i.e. checking for changes to 191
> > > branches -
> > > > > It
> > > > > > > > should be manageable)
> > > > > > > > 2. What's the purpose of the plugin? tldr; Branch discovery
> or
> > > branch
> > > > > > > > indexing.
> > > > > > > > Scan trigger plugin comes into the picture only once per PR
> per
> > > job
> > > > > > > (i.e. 8
> > > > > > > > times per PR for 8 jobs). It is basically done when a new PR
> is
> > > made
> > > > > > and
> > > > > > > > the job (say unix-cpu hasn't discovered the new PR branch
> yet).
> > > > > That's
> > > > > > > it.
> > > > > > > > So it shouldn't be a problem for public MXNet repo.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Chai
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <
> > > > > marco.g.abreu@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Btw you forgot to set a date and time for the metting
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu <
> > > > > > > marco.g.abreu@gmail.com
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks Chai, I generally like the idea of the bot. But
> I'm
> > > not a
> > > > > > > > > supporter
> > > > > > > > > > of the idea to disable any automatic triggering
> (disabling
> > > the
> > > > > > > webhook
> > > > > > > > is
> > > > > > > > > > also not an option, considering that this will disable
> > master
> > > > > > > > triggers).
> > > > > > > > > > The CI system has to support the community without
> > requiring
> > > > > people
> > > > > > > to
> > > > > > > > > > constantly shepherd every single run. Disabling automatic
> > > > > > triggering
> > > > > > > > > seems
> > > > > > > > > > like a step back to me.
> > > > > > > > > >
> > > > > > > > > > Instead, I'd recommend that CI gets triggered upon every
> > > commit
> > > > > as
> > > > > > > > usual,
> > > > > > > > > > but people have the possibility to call a "command" (i.e.
> > > make a
> > > > > > > > message
> > > > > > > > > > which results in the bot setting a label) to disable CI
> > until
> > > > > they
> > > > > > > > revoke
> > > > > > > > > > it. But the standard should still be that a new commit
> > > triggers a
> > > > > > new
> > > > > > > > CI
> > > > > > > > > > run.
> > > > > > > > > >
> > > > > > > > > >
> > https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
> > > > > > seems
> > > > > > > > like
> > > > > > > > > > this would poll SCM. This will incur high quota
> > > restrictions. Are
> > > > > > you
> > > > > > > > > sure
> > > > > > > > > > that you can use that plugin at scale?
> > > > > > > > > >
> > > > > > > > > > -Marco
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <
> > > apeforest@gmail.com>
> > > > > > > wrote:
> > > > > > > > > > > Chai,
> > > > > > > > > > >
> > > > > > > > > > > Awesome work. When do we expect this bot to be
> deployed?
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > >
> > > > > > > > > > > Lin
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <
> > > > > > > chai.bapat@gmail.com
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hello MXNet community,
> > > > > > > > > > > >
> > > > > > > > > > > > I have built an MXNet Bot <
> > https://github.com/mxnet-bot>
> > > that
> > > > > > > > allows
> > > > > > > > > PR
> > > > > > > > > > > > Authors, Committers and Jenkins Admins to trigger CI
> > > manually.
> > > > > > > > > > > > It handles 2 problems
> > > > > > > > > > > > 1. Manual CI trigger instead of existing automated CI
> > > trigger
> > > > > > > > > > > > 2. Gives permissions to PR Authors (in addition to
> > MXNet
> > > > > > > Committers
> > > > > > > > > and
> > > > > > > > > > > > Jenkins Admins)
> > > > > > > > > > > >
> > > > > > > > > > > > Design Doc :
> > > > > > > > > > > >
> > > > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > > > > > > > > > > > I urge you all to attend the demonstration meeting
> and
> > > lend
> > > > > your
> > > > > > > > views
> > > > > > > > > > > on
> > > > > > > > > > > > the same.
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you,
> > > > > > > > > > > > Chai
> > > > > > > > > > > >
> > > > > > > > > > > > *Meeting Details*:
> > > > > > > > > > > > ==============Conference Bridge
> > Information==============
> > > > > > > > > > > > You have been invited to an online meeting, powered
> by
> > > Amazon
> > > > > > > Chime.
> > > > > > > > > > > > *Chime meeting ID*: *9272158344*
> > > > > > > > > > > > Join via Chime clients (manually): Select 'Meetings >
> > > Join a
> > > > > > > > Meeting',
> > > > > > > > > > > and
> > > > > > > > > > > > enter 9272158344
> > > > > > > > > > > > Join via Chime clients (auto-call): If you invite
> > > auto-call as
> > > > > > > > > attendee,
> > > > > > > > > > > > Chime will call you when the meeting starts, select
> > > 'Answer'
> > > > > > > > > > > > *Join via browser screen share*:
> > > https://chime.aws/9272158344
> > > > > > > > > > > > *Join via phone* (US): +1-929-432-4463,,,9272158344#
> > > > > > > > > > > > *Join via phone (US toll-free)*:
> > > +1-855-552-4463,,,9272158344#
> > > > > > > > > > > > International dial-in:
> > https://chime.aws/dialinnumbers/
> > > > > > > > > > > > In-room video system: Ext: 62000, Meeting PIN:
> > > 9272158344#
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > *Chaitanya Prakash Bapat*
> > > > > > > > > > > > *+1 (973) 953-6299*
> > > > > > > > > > > >
> > > > > > > > > > > > [image: https://www.linkedin.com//in/chaibapat25]
> > > > > > > > > > > > <https://github.com/ChaiBapchya>[image:
> > > > > > > > > > > https://www.facebook.com/chaibapat
> > > > > > > > > > > > ]
> > > > > > > > > > > > <https://www.facebook.com/chaibapchya>[image:
> > > > > > > > > > > > https://twitter.com/ChaiBapchya] <
> > > > > > https://twitter.com/ChaiBapchya
> > > > > > > > > > > > [image:
> > > > > > > > > > > > https://www.linkedin.com//in/chaibapat25]
> > > > > > > > > > > > <https://www.linkedin.com//in/chaibapchya/>
> > > > > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *Chaitanya Prakash Bapat*
> > > > > > > > *+1 (973) 953-6299*
> > > > > > > >
> > > > > > > > [image: https://www.linkedin.com//in/chaibapat25]
> > > > > > > > <https://github.com/ChaiBapchya>[image:
> > > > > > > https://www.facebook.com/chaibapat
> > > > > > > > ]
> > > > > > > > <https://www.facebook.com/chaibapchya>[image:
> > > > > > > > https://twitter.com/ChaiBapchya] <
> > > https://twitter.com/ChaiBapchya
> > > > > > > > [image:
> > > > > > > > https://www.linkedin.com//in/chaibapat25]
> > > > > > > > <https://www.linkedin.com//in/chaibapchya/>
> > > > > > > >
> > > > >
> > > > > --
> > > > > Sandeep Krishnamurthy
> > > > >
> > >
> >
>
>
> --
> *Chaitanya Prakash Bapat*
> *+1 (973) 953-6299*
>
> [image: https://www.linkedin.com//in/chaibapat25]
> <https://github.com/ChaiBapchya>[image: https://www.facebook.com/chaibapat
> ]
> <https://www.facebook.com/chaibapchya>[image:
> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
> https://www.linkedin.com//in/chaibapat25]
> <https://www.linkedin.com//in/chaibapchya/>
>
Re: MXNet Bot Demo
Posted by Chaitanya Bapat <ch...@gmail.com>.
Firstly,
Sorry I missed out on attaching the mail thread that was sent on 12th
February for notifying the community of the upcoming changes to the MXNet CI
For reference :
https://lists.apache.org/thread.html/r09a6ab2803a996fc80e00fe39ed312fa4865e8805e08df847f1addad%40%3Cdev.mxnet.apache.org%3E
Now to the questions,
> Is it possible for re-triggering a single job to be abused?
@Tao In the case when a user re-triggers a single job multiple times, that
will be visible in the PR conversation thread. A committer, even after he
has approved the PR before, generally takes a look at the final state of
the PR before merging. Would it be fair to assume the committer could take
the multiple re-trigger of a single job into account before merging? The
committer then has the option to invoke `@mxnet-bot run ci [all] ` to
trigger the entire build pipeline one last to counter the abuse. This is
aligned with what @Leonard said.
@Sandeep Thanks a lot for collecting and sharing valuable data. I'd concur
with the opinion that given the existing things committers & PR Authors
take care of, invoking CI shouldn't be that big of an additional burden.
@Marco With the opt-out, the onus remains on the PR Author. It doesn't help
reduce the resource usage. Hence, it was suggested to switch to
opt-in. @Leo's suggestion for proactive commenting on the part of bot makes
sense and is doable.
Default : opt-out and User initiated opt-in (with addressing Leo's fix for
the usability issue you correctly pointed out )
@Marco How does this sound to you?
Again, thank you all for chiming in and voicing your opinion. Appreciate it.
We can take ahead these discussions in today's demo meeting. [Design Doc
<https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot>] [Demo
Video <https://www.youtube.com/watch?v=gfOGwZId8aU>]
Thanks,
Chai
On Fri, 13 Mar 2020 at 12:34, Marco de Abreu <ma...@gmail.com>
wrote:
> I'd recommend that the bot makes an initial comment when a PR gets opened
> and informs the users of its commands. It then tells the user the commend
> to opt out of CI.
>
> -Marco
>
> Lausen, Leonard <la...@amazon.com.invalid> schrieb am Fr., 13. März 2020,
> 20:27:
>
> > On opt-out: People may be unaware of opt-out would not use it. There is
> no
> > incentive to use opt-out, as the PR author doesn't pay any money for CI
> > run.
> >
> > I agree with Marco though that opt-in alone may cause usability issues,
> as
> > contributors may not be aware of how to trigger the CI.
> > One solution is that the bot proactively comments on the PR and reminds
> the
> > author to trigger running CI once the author deems the PR ready.
> >
> > But even if we choose opt-out, the bot will still add a lot of value, as
> PR
> > authors can retrigger single jobs that have failed due to flakiness.
> >
> > > Is it possible for re-triggering a single job to be abused? For
> example,
> > > the author spends two days re-triggering a flaky job to make it pass.
> >
> > Yes, this is possible. I suggest the committer who likes to merge a PR
> > needs to
> > make a good judgement here if a PR is abusing the feature, and if so,
> > retrigger
> > all CI runs.
> >
> > Best regards
> > Leonard
> >
> > On Fri, 2020-03-13 at 08:07 +0100, Marco de Abreu wrote:
> > > Thanks for the data Sandeep. In these cases it sounds like it would
> have
> > > rather been better when people explicitly turned off CI in that case.
> > > What's the argument against an opt-out instead of an opt-in?
> > >
> > > My intention is that I consider it quite cumbersome to make it a
> > *required*
> > > step to always trigger CI manually, even if just submitting a small PR.
> > I'd
> > > rather see people explicitly turning off CI if they wouldn't like to
> use
> > it
> > > - and there's also the "draft" stage for a PR which some contributors
> are
> > > using.
> > >
> > > With regards to WIP and do not review: I think these are use cases
> where
> > > you want CI feedback, as otherwise you wouldn't have opened the PR. If
> > you
> > > don't want human feedback and neither machine feedback, why open the PR
> > at
> > > all?
> > >
> > > -Marco
> > >
> > >
> > > sandeep krishnamurthy <sa...@gmail.com> schrieb am Fr.,
> 13.
> > > März 2020, 05:24:
> > >
> > > > I tried to gather some data for us to discuss this topic in this
> > thread. I
> > > > tried to count number of un-necessary builds by looking at most
> recent
> > (as
> > > > of 12, March 9 PM PST) 50 PRs merged to master and 50 PRs.
> > > > Identifying un-necessary builds is bit subjective. I tried to be more
> > > > conservative where I didn't count a build as un-necessary if I was in
> > > > doubt. Hence, I was not able to automate, but I made an effort to go
> > > > through PRs manually and use below criteria to identify un-necessary
> > > > commits triggering the builds.
> > > >
> > > > 1. Explicitly marked as WIP / do not review PR
> > > > 2. Incremental WIP commit and finally commenting a commit “trigger
> > CI”
> > > > 3. Multiple commits to address all comments from single review.
> > This is
> > > > assuming we see a comment, address them, commit, next the
> following
> > > > comment
> > > > 4. Sequence of documentation only changes
> > > >
> > > > I found there were around 42 avoidable builds from most recent 50
> > merged
> > > > PRs and around 86 builds from recent 50 open PRs.
> > > >
> > > > I synced up with other contributors (Joe Evans, Chai) from Amazon who
> > is
> > > > contributing to MXNet CI system. I was told that on an average it
> costs
> > > > around $84 per build and on an average 6 commits per merged PR (for
> > year
> > > > 2019). Going by that, it is approximately 1/6 builds are avoidable.
> > [100 /
> > > > 300 + 300 ]
> > > >
> > > >
> > > > Usability should be top most priority. But, since either a reviewer
> or
> > pr
> > > > author can trigger the bot, is it really a hurdle for pr author or
> > reviewer
> > > > to call a bot to trigger CI? Given that PR author and reviewer is
> > already
> > > > actively commenting various details such as - PR description, review
> > > > comments and responses, adding labels etc.
> > > >
> > > >
> > > > Me too curious to know the behavior for Tao's above use case.
> > > >
> > > >
> > > > Best,
> > > >
> > > > Sandeep
> > > >
> > > > On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <mu...@gmail.com> wrote:
> > > >
> > > > > Is it possible for re-triggering a single job to be abused? For
> > example,
> > > > > the author spends two days re-triggering a flaky job to make it
> > pass. But
> > > > > other jobs which have passed the validation may be broken by other
> > > > commits
> > > > > during the two day without being noticed. And finally the PR is
> > merged
> > > > with
> > > > > underlying problems.
> > > > >
> > > > > On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <
> > marco.g.abreu@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > In the end it only comes down to money, considering that the
> > system is
> > > > > auto
> > > > > > scaling, making the execution time constant.
> > > > > >
> > > > > > If we're trading money for usability, I certainly would prefer
> > > > usability.
> > > > > > I'd rather recommend to spend time on parallelizing test
> execution
> > or
> > > > > > getting rid of integration tests in the PR stage instead reducing
> > the
> > > > > costs
> > > > > > by making people not use it. But taking a step back to requiring
> > people
> > > > > to
> > > > > > manually trigger CI again doesn't feel right.
> > > > > >
> > > > > > I'm happy to see that bot deployed, but I do not agree with
> > removing
> > > > the
> > > > > > auto trigger functionality for new commits.
> > > > > >
> > > > > > -Marco
> > > > > >
> > > > > > Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 12. März
> > 2020,
> > > > > > 22:47:
> > > > > >
> > > > > > > @Marco Thanks for pointing that out.
> > > > > > > Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM - 3:30 PM in
> > > > > (UTC-08:00)
> > > > > > > Pacific Time (US & Canada).
> > > > > > >
> > > > > > > > When do we expect this bot to be deployed?
> > > > > > > @Lin If all goes well in the next week I can deploy it to
> public
> > > > Apache
> > > > > > > (provided I get permissions from Apache Infra)
> > > > > > >
> > > > > > > @Marco Thanks for your feedback.
> > > > > > > > CI system has to support the community without requiring
> > people to
> > > > > > > constantly shepherd every single run
> > > > > > > We have data for the number of times CI was triggered
> > unnecessarily
> > > > > which
> > > > > > > includes
> > > > > > > - Entire build triggered instead of specific build
> > > > > > > - CI triggered when PR is still work in progress or not yet
> ready
> > > > (say
> > > > > -
> > > > > > > intermediate commits)
> > > > > > > At the end its a trade-off
> > > > > > > Money, Resources, Time to build for each and every commit vs
> > Pain of
> > > > > > > triggering builds
> > > > > > >
> > > > > > >
> > > > > > > > Scan trigger plugin would poll SCM. Can we use plugin at
> > scale?
> > > > > > >
> > > > > > > 1. I haven't tested it on scale. But I think with the current
> > scale
> > > > of
> > > > > > > MXNet repo (191 open PRs i.e. checking for changes to 191
> > branches -
> > > > It
> > > > > > > should be manageable)
> > > > > > > 2. What's the purpose of the plugin? tldr; Branch discovery or
> > branch
> > > > > > > indexing.
> > > > > > > Scan trigger plugin comes into the picture only once per PR per
> > job
> > > > > > (i.e. 8
> > > > > > > times per PR for 8 jobs). It is basically done when a new PR is
> > made
> > > > > and
> > > > > > > the job (say unix-cpu hasn't discovered the new PR branch yet).
> > > > That's
> > > > > > it.
> > > > > > > So it shouldn't be a problem for public MXNet repo.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Chai
> > > > > > >
> > > > > > >
> > > > > > > On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <
> > > > marco.g.abreu@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Btw you forgot to set a date and time for the metting
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu <
> > > > > > marco.g.abreu@gmail.com
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks Chai, I generally like the idea of the bot. But I'm
> > not a
> > > > > > > > supporter
> > > > > > > > > of the idea to disable any automatic triggering (disabling
> > the
> > > > > > webhook
> > > > > > > is
> > > > > > > > > also not an option, considering that this will disable
> master
> > > > > > > triggers).
> > > > > > > > > The CI system has to support the community without
> requiring
> > > > people
> > > > > > to
> > > > > > > > > constantly shepherd every single run. Disabling automatic
> > > > > triggering
> > > > > > > > seems
> > > > > > > > > like a step back to me.
> > > > > > > > >
> > > > > > > > > Instead, I'd recommend that CI gets triggered upon every
> > commit
> > > > as
> > > > > > > usual,
> > > > > > > > > but people have the possibility to call a "command" (i.e.
> > make a
> > > > > > > message
> > > > > > > > > which results in the bot setting a label) to disable CI
> until
> > > > they
> > > > > > > revoke
> > > > > > > > > it. But the standard should still be that a new commit
> > triggers a
> > > > > new
> > > > > > > CI
> > > > > > > > > run.
> > > > > > > > >
> > > > > > > > >
> https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
> > > > > seems
> > > > > > > like
> > > > > > > > > this would poll SCM. This will incur high quota
> > restrictions. Are
> > > > > you
> > > > > > > > sure
> > > > > > > > > that you can use that plugin at scale?
> > > > > > > > >
> > > > > > > > > -Marco
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <
> > apeforest@gmail.com>
> > > > > > wrote:
> > > > > > > > > > Chai,
> > > > > > > > > >
> > > > > > > > > > Awesome work. When do we expect this bot to be deployed?
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > >
> > > > > > > > > > Lin
> > > > > > > > > >
> > > > > > > > > > On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <
> > > > > > chai.bapat@gmail.com
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hello MXNet community,
> > > > > > > > > > >
> > > > > > > > > > > I have built an MXNet Bot <
> https://github.com/mxnet-bot>
> > that
> > > > > > > allows
> > > > > > > > PR
> > > > > > > > > > > Authors, Committers and Jenkins Admins to trigger CI
> > manually.
> > > > > > > > > > > It handles 2 problems
> > > > > > > > > > > 1. Manual CI trigger instead of existing automated CI
> > trigger
> > > > > > > > > > > 2. Gives permissions to PR Authors (in addition to
> MXNet
> > > > > > Committers
> > > > > > > > and
> > > > > > > > > > > Jenkins Admins)
> > > > > > > > > > >
> > > > > > > > > > > Design Doc :
> > > > > > > > > > >
> > > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > > > > > > > > > > I urge you all to attend the demonstration meeting and
> > lend
> > > > your
> > > > > > > views
> > > > > > > > > > on
> > > > > > > > > > > the same.
> > > > > > > > > > >
> > > > > > > > > > > Thank you,
> > > > > > > > > > > Chai
> > > > > > > > > > >
> > > > > > > > > > > *Meeting Details*:
> > > > > > > > > > > ==============Conference Bridge
> Information==============
> > > > > > > > > > > You have been invited to an online meeting, powered by
> > Amazon
> > > > > > Chime.
> > > > > > > > > > > *Chime meeting ID*: *9272158344*
> > > > > > > > > > > Join via Chime clients (manually): Select 'Meetings >
> > Join a
> > > > > > > Meeting',
> > > > > > > > > > and
> > > > > > > > > > > enter 9272158344
> > > > > > > > > > > Join via Chime clients (auto-call): If you invite
> > auto-call as
> > > > > > > > attendee,
> > > > > > > > > > > Chime will call you when the meeting starts, select
> > 'Answer'
> > > > > > > > > > > *Join via browser screen share*:
> > https://chime.aws/9272158344
> > > > > > > > > > > *Join via phone* (US): +1-929-432-4463,,,9272158344#
> > > > > > > > > > > *Join via phone (US toll-free)*:
> > +1-855-552-4463,,,9272158344#
> > > > > > > > > > > International dial-in:
> https://chime.aws/dialinnumbers/
> > > > > > > > > > > In-room video system: Ext: 62000, Meeting PIN:
> > 9272158344#
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > *Chaitanya Prakash Bapat*
> > > > > > > > > > > *+1 (973) 953-6299*
> > > > > > > > > > >
> > > > > > > > > > > [image: https://www.linkedin.com//in/chaibapat25]
> > > > > > > > > > > <https://github.com/ChaiBapchya>[image:
> > > > > > > > > > https://www.facebook.com/chaibapat
> > > > > > > > > > > ]
> > > > > > > > > > > <https://www.facebook.com/chaibapchya>[image:
> > > > > > > > > > > https://twitter.com/ChaiBapchya] <
> > > > > https://twitter.com/ChaiBapchya
> > > > > > > > > > > [image:
> > > > > > > > > > > https://www.linkedin.com//in/chaibapat25]
> > > > > > > > > > > <https://www.linkedin.com//in/chaibapchya/>
> > > > > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *Chaitanya Prakash Bapat*
> > > > > > > *+1 (973) 953-6299*
> > > > > > >
> > > > > > > [image: https://www.linkedin.com//in/chaibapat25]
> > > > > > > <https://github.com/ChaiBapchya>[image:
> > > > > > https://www.facebook.com/chaibapat
> > > > > > > ]
> > > > > > > <https://www.facebook.com/chaibapchya>[image:
> > > > > > > https://twitter.com/ChaiBapchya] <
> > https://twitter.com/ChaiBapchya
> > > > > > > [image:
> > > > > > > https://www.linkedin.com//in/chaibapat25]
> > > > > > > <https://www.linkedin.com//in/chaibapchya/>
> > > > > > >
> > > >
> > > > --
> > > > Sandeep Krishnamurthy
> > > >
> >
>
--
*Chaitanya Prakash Bapat*
*+1 (973) 953-6299*
[image: https://www.linkedin.com//in/chaibapat25]
<https://github.com/ChaiBapchya>[image: https://www.facebook.com/chaibapat]
<https://www.facebook.com/chaibapchya>[image:
https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
https://www.linkedin.com//in/chaibapat25]
<https://www.linkedin.com//in/chaibapchya/>
Re: MXNet Bot Demo
Posted by Marco de Abreu <ma...@gmail.com>.
I'd recommend that the bot makes an initial comment when a PR gets opened
and informs the users of its commands. It then tells the user the commend
to opt out of CI.
-Marco
Lausen, Leonard <la...@amazon.com.invalid> schrieb am Fr., 13. März 2020,
20:27:
> On opt-out: People may be unaware of opt-out would not use it. There is no
> incentive to use opt-out, as the PR author doesn't pay any money for CI
> run.
>
> I agree with Marco though that opt-in alone may cause usability issues, as
> contributors may not be aware of how to trigger the CI.
> One solution is that the bot proactively comments on the PR and reminds the
> author to trigger running CI once the author deems the PR ready.
>
> But even if we choose opt-out, the bot will still add a lot of value, as PR
> authors can retrigger single jobs that have failed due to flakiness.
>
> > Is it possible for re-triggering a single job to be abused? For example,
> > the author spends two days re-triggering a flaky job to make it pass.
>
> Yes, this is possible. I suggest the committer who likes to merge a PR
> needs to
> make a good judgement here if a PR is abusing the feature, and if so,
> retrigger
> all CI runs.
>
> Best regards
> Leonard
>
> On Fri, 2020-03-13 at 08:07 +0100, Marco de Abreu wrote:
> > Thanks for the data Sandeep. In these cases it sounds like it would have
> > rather been better when people explicitly turned off CI in that case.
> > What's the argument against an opt-out instead of an opt-in?
> >
> > My intention is that I consider it quite cumbersome to make it a
> *required*
> > step to always trigger CI manually, even if just submitting a small PR.
> I'd
> > rather see people explicitly turning off CI if they wouldn't like to use
> it
> > - and there's also the "draft" stage for a PR which some contributors are
> > using.
> >
> > With regards to WIP and do not review: I think these are use cases where
> > you want CI feedback, as otherwise you wouldn't have opened the PR. If
> you
> > don't want human feedback and neither machine feedback, why open the PR
> at
> > all?
> >
> > -Marco
> >
> >
> > sandeep krishnamurthy <sa...@gmail.com> schrieb am Fr., 13.
> > März 2020, 05:24:
> >
> > > I tried to gather some data for us to discuss this topic in this
> thread. I
> > > tried to count number of un-necessary builds by looking at most recent
> (as
> > > of 12, March 9 PM PST) 50 PRs merged to master and 50 PRs.
> > > Identifying un-necessary builds is bit subjective. I tried to be more
> > > conservative where I didn't count a build as un-necessary if I was in
> > > doubt. Hence, I was not able to automate, but I made an effort to go
> > > through PRs manually and use below criteria to identify un-necessary
> > > commits triggering the builds.
> > >
> > > 1. Explicitly marked as WIP / do not review PR
> > > 2. Incremental WIP commit and finally commenting a commit “trigger
> CI”
> > > 3. Multiple commits to address all comments from single review.
> This is
> > > assuming we see a comment, address them, commit, next the following
> > > comment
> > > 4. Sequence of documentation only changes
> > >
> > > I found there were around 42 avoidable builds from most recent 50
> merged
> > > PRs and around 86 builds from recent 50 open PRs.
> > >
> > > I synced up with other contributors (Joe Evans, Chai) from Amazon who
> is
> > > contributing to MXNet CI system. I was told that on an average it costs
> > > around $84 per build and on an average 6 commits per merged PR (for
> year
> > > 2019). Going by that, it is approximately 1/6 builds are avoidable.
> [100 /
> > > 300 + 300 ]
> > >
> > >
> > > Usability should be top most priority. But, since either a reviewer or
> pr
> > > author can trigger the bot, is it really a hurdle for pr author or
> reviewer
> > > to call a bot to trigger CI? Given that PR author and reviewer is
> already
> > > actively commenting various details such as - PR description, review
> > > comments and responses, adding labels etc.
> > >
> > >
> > > Me too curious to know the behavior for Tao's above use case.
> > >
> > >
> > > Best,
> > >
> > > Sandeep
> > >
> > > On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <mu...@gmail.com> wrote:
> > >
> > > > Is it possible for re-triggering a single job to be abused? For
> example,
> > > > the author spends two days re-triggering a flaky job to make it
> pass. But
> > > > other jobs which have passed the validation may be broken by other
> > > commits
> > > > during the two day without being noticed. And finally the PR is
> merged
> > > with
> > > > underlying problems.
> > > >
> > > > On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <
> marco.g.abreu@gmail.com>
> > > > wrote:
> > > >
> > > > > In the end it only comes down to money, considering that the
> system is
> > > > auto
> > > > > scaling, making the execution time constant.
> > > > >
> > > > > If we're trading money for usability, I certainly would prefer
> > > usability.
> > > > > I'd rather recommend to spend time on parallelizing test execution
> or
> > > > > getting rid of integration tests in the PR stage instead reducing
> the
> > > > costs
> > > > > by making people not use it. But taking a step back to requiring
> people
> > > > to
> > > > > manually trigger CI again doesn't feel right.
> > > > >
> > > > > I'm happy to see that bot deployed, but I do not agree with
> removing
> > > the
> > > > > auto trigger functionality for new commits.
> > > > >
> > > > > -Marco
> > > > >
> > > > > Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 12. März
> 2020,
> > > > > 22:47:
> > > > >
> > > > > > @Marco Thanks for pointing that out.
> > > > > > Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM - 3:30 PM in
> > > > (UTC-08:00)
> > > > > > Pacific Time (US & Canada).
> > > > > >
> > > > > > > When do we expect this bot to be deployed?
> > > > > > @Lin If all goes well in the next week I can deploy it to public
> > > Apache
> > > > > > (provided I get permissions from Apache Infra)
> > > > > >
> > > > > > @Marco Thanks for your feedback.
> > > > > > > CI system has to support the community without requiring
> people to
> > > > > > constantly shepherd every single run
> > > > > > We have data for the number of times CI was triggered
> unnecessarily
> > > > which
> > > > > > includes
> > > > > > - Entire build triggered instead of specific build
> > > > > > - CI triggered when PR is still work in progress or not yet ready
> > > (say
> > > > -
> > > > > > intermediate commits)
> > > > > > At the end its a trade-off
> > > > > > Money, Resources, Time to build for each and every commit vs
> Pain of
> > > > > > triggering builds
> > > > > >
> > > > > >
> > > > > > > Scan trigger plugin would poll SCM. Can we use plugin at
> scale?
> > > > > >
> > > > > > 1. I haven't tested it on scale. But I think with the current
> scale
> > > of
> > > > > > MXNet repo (191 open PRs i.e. checking for changes to 191
> branches -
> > > It
> > > > > > should be manageable)
> > > > > > 2. What's the purpose of the plugin? tldr; Branch discovery or
> branch
> > > > > > indexing.
> > > > > > Scan trigger plugin comes into the picture only once per PR per
> job
> > > > > (i.e. 8
> > > > > > times per PR for 8 jobs). It is basically done when a new PR is
> made
> > > > and
> > > > > > the job (say unix-cpu hasn't discovered the new PR branch yet).
> > > That's
> > > > > it.
> > > > > > So it shouldn't be a problem for public MXNet repo.
> > > > > >
> > > > > > Thanks,
> > > > > > Chai
> > > > > >
> > > > > >
> > > > > > On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <
> > > marco.g.abreu@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Btw you forgot to set a date and time for the metting
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu <
> > > > > marco.g.abreu@gmail.com
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Chai, I generally like the idea of the bot. But I'm
> not a
> > > > > > > supporter
> > > > > > > > of the idea to disable any automatic triggering (disabling
> the
> > > > > webhook
> > > > > > is
> > > > > > > > also not an option, considering that this will disable master
> > > > > > triggers).
> > > > > > > > The CI system has to support the community without requiring
> > > people
> > > > > to
> > > > > > > > constantly shepherd every single run. Disabling automatic
> > > > triggering
> > > > > > > seems
> > > > > > > > like a step back to me.
> > > > > > > >
> > > > > > > > Instead, I'd recommend that CI gets triggered upon every
> commit
> > > as
> > > > > > usual,
> > > > > > > > but people have the possibility to call a "command" (i.e.
> make a
> > > > > > message
> > > > > > > > which results in the bot setting a label) to disable CI until
> > > they
> > > > > > revoke
> > > > > > > > it. But the standard should still be that a new commit
> triggers a
> > > > new
> > > > > > CI
> > > > > > > > run.
> > > > > > > >
> > > > > > > > https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
> > > > seems
> > > > > > like
> > > > > > > > this would poll SCM. This will incur high quota
> restrictions. Are
> > > > you
> > > > > > > sure
> > > > > > > > that you can use that plugin at scale?
> > > > > > > >
> > > > > > > > -Marco
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <
> apeforest@gmail.com>
> > > > > wrote:
> > > > > > > > > Chai,
> > > > > > > > >
> > > > > > > > > Awesome work. When do we expect this bot to be deployed?
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > >
> > > > > > > > > Lin
> > > > > > > > >
> > > > > > > > > On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <
> > > > > chai.bapat@gmail.com
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello MXNet community,
> > > > > > > > > >
> > > > > > > > > > I have built an MXNet Bot <https://github.com/mxnet-bot>
> that
> > > > > > allows
> > > > > > > PR
> > > > > > > > > > Authors, Committers and Jenkins Admins to trigger CI
> manually.
> > > > > > > > > > It handles 2 problems
> > > > > > > > > > 1. Manual CI trigger instead of existing automated CI
> trigger
> > > > > > > > > > 2. Gives permissions to PR Authors (in addition to MXNet
> > > > > Committers
> > > > > > > and
> > > > > > > > > > Jenkins Admins)
> > > > > > > > > >
> > > > > > > > > > Design Doc :
> > > > > > > > > >
> > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > > > > > > > > > I urge you all to attend the demonstration meeting and
> lend
> > > your
> > > > > > views
> > > > > > > > > on
> > > > > > > > > > the same.
> > > > > > > > > >
> > > > > > > > > > Thank you,
> > > > > > > > > > Chai
> > > > > > > > > >
> > > > > > > > > > *Meeting Details*:
> > > > > > > > > > ==============Conference Bridge Information==============
> > > > > > > > > > You have been invited to an online meeting, powered by
> Amazon
> > > > > Chime.
> > > > > > > > > > *Chime meeting ID*: *9272158344*
> > > > > > > > > > Join via Chime clients (manually): Select 'Meetings >
> Join a
> > > > > > Meeting',
> > > > > > > > > and
> > > > > > > > > > enter 9272158344
> > > > > > > > > > Join via Chime clients (auto-call): If you invite
> auto-call as
> > > > > > > attendee,
> > > > > > > > > > Chime will call you when the meeting starts, select
> 'Answer'
> > > > > > > > > > *Join via browser screen share*:
> https://chime.aws/9272158344
> > > > > > > > > > *Join via phone* (US): +1-929-432-4463,,,9272158344#
> > > > > > > > > > *Join via phone (US toll-free)*:
> +1-855-552-4463,,,9272158344#
> > > > > > > > > > International dial-in: https://chime.aws/dialinnumbers/
> > > > > > > > > > In-room video system: Ext: 62000, Meeting PIN:
> 9272158344#
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > *Chaitanya Prakash Bapat*
> > > > > > > > > > *+1 (973) 953-6299*
> > > > > > > > > >
> > > > > > > > > > [image: https://www.linkedin.com//in/chaibapat25]
> > > > > > > > > > <https://github.com/ChaiBapchya>[image:
> > > > > > > > > https://www.facebook.com/chaibapat
> > > > > > > > > > ]
> > > > > > > > > > <https://www.facebook.com/chaibapchya>[image:
> > > > > > > > > > https://twitter.com/ChaiBapchya] <
> > > > https://twitter.com/ChaiBapchya
> > > > > > > > > > [image:
> > > > > > > > > > https://www.linkedin.com//in/chaibapat25]
> > > > > > > > > > <https://www.linkedin.com//in/chaibapchya/>
> > > > > > > > > >
> > > > > >
> > > > > > --
> > > > > > *Chaitanya Prakash Bapat*
> > > > > > *+1 (973) 953-6299*
> > > > > >
> > > > > > [image: https://www.linkedin.com//in/chaibapat25]
> > > > > > <https://github.com/ChaiBapchya>[image:
> > > > > https://www.facebook.com/chaibapat
> > > > > > ]
> > > > > > <https://www.facebook.com/chaibapchya>[image:
> > > > > > https://twitter.com/ChaiBapchya] <
> https://twitter.com/ChaiBapchya
> > > > > > [image:
> > > > > > https://www.linkedin.com//in/chaibapat25]
> > > > > > <https://www.linkedin.com//in/chaibapchya/>
> > > > > >
> > >
> > > --
> > > Sandeep Krishnamurthy
> > >
>
Re: MXNet Bot Demo
Posted by "Lausen, Leonard" <la...@amazon.com.INVALID>.
On opt-out: People may be unaware of opt-out would not use it. There is no
incentive to use opt-out, as the PR author doesn't pay any money for CI run.
I agree with Marco though that opt-in alone may cause usability issues, as
contributors may not be aware of how to trigger the CI.
One solution is that the bot proactively comments on the PR and reminds the
author to trigger running CI once the author deems the PR ready.
But even if we choose opt-out, the bot will still add a lot of value, as PR
authors can retrigger single jobs that have failed due to flakiness.
> Is it possible for re-triggering a single job to be abused? For example,
> the author spends two days re-triggering a flaky job to make it pass.
Yes, this is possible. I suggest the committer who likes to merge a PR needs to
make a good judgement here if a PR is abusing the feature, and if so, retrigger
all CI runs.
Best regards
Leonard
On Fri, 2020-03-13 at 08:07 +0100, Marco de Abreu wrote:
> Thanks for the data Sandeep. In these cases it sounds like it would have
> rather been better when people explicitly turned off CI in that case.
> What's the argument against an opt-out instead of an opt-in?
>
> My intention is that I consider it quite cumbersome to make it a *required*
> step to always trigger CI manually, even if just submitting a small PR. I'd
> rather see people explicitly turning off CI if they wouldn't like to use it
> - and there's also the "draft" stage for a PR which some contributors are
> using.
>
> With regards to WIP and do not review: I think these are use cases where
> you want CI feedback, as otherwise you wouldn't have opened the PR. If you
> don't want human feedback and neither machine feedback, why open the PR at
> all?
>
> -Marco
>
>
> sandeep krishnamurthy <sa...@gmail.com> schrieb am Fr., 13.
> März 2020, 05:24:
>
> > I tried to gather some data for us to discuss this topic in this thread. I
> > tried to count number of un-necessary builds by looking at most recent (as
> > of 12, March 9 PM PST) 50 PRs merged to master and 50 PRs.
> > Identifying un-necessary builds is bit subjective. I tried to be more
> > conservative where I didn't count a build as un-necessary if I was in
> > doubt. Hence, I was not able to automate, but I made an effort to go
> > through PRs manually and use below criteria to identify un-necessary
> > commits triggering the builds.
> >
> > 1. Explicitly marked as WIP / do not review PR
> > 2. Incremental WIP commit and finally commenting a commit “trigger CI”
> > 3. Multiple commits to address all comments from single review. This is
> > assuming we see a comment, address them, commit, next the following
> > comment
> > 4. Sequence of documentation only changes
> >
> > I found there were around 42 avoidable builds from most recent 50 merged
> > PRs and around 86 builds from recent 50 open PRs.
> >
> > I synced up with other contributors (Joe Evans, Chai) from Amazon who is
> > contributing to MXNet CI system. I was told that on an average it costs
> > around $84 per build and on an average 6 commits per merged PR (for year
> > 2019). Going by that, it is approximately 1/6 builds are avoidable. [100 /
> > 300 + 300 ]
> >
> >
> > Usability should be top most priority. But, since either a reviewer or pr
> > author can trigger the bot, is it really a hurdle for pr author or reviewer
> > to call a bot to trigger CI? Given that PR author and reviewer is already
> > actively commenting various details such as - PR description, review
> > comments and responses, adding labels etc.
> >
> >
> > Me too curious to know the behavior for Tao's above use case.
> >
> >
> > Best,
> >
> > Sandeep
> >
> > On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <mu...@gmail.com> wrote:
> >
> > > Is it possible for re-triggering a single job to be abused? For example,
> > > the author spends two days re-triggering a flaky job to make it pass. But
> > > other jobs which have passed the validation may be broken by other
> > commits
> > > during the two day without being noticed. And finally the PR is merged
> > with
> > > underlying problems.
> > >
> > > On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <ma...@gmail.com>
> > > wrote:
> > >
> > > > In the end it only comes down to money, considering that the system is
> > > auto
> > > > scaling, making the execution time constant.
> > > >
> > > > If we're trading money for usability, I certainly would prefer
> > usability.
> > > > I'd rather recommend to spend time on parallelizing test execution or
> > > > getting rid of integration tests in the PR stage instead reducing the
> > > costs
> > > > by making people not use it. But taking a step back to requiring people
> > > to
> > > > manually trigger CI again doesn't feel right.
> > > >
> > > > I'm happy to see that bot deployed, but I do not agree with removing
> > the
> > > > auto trigger functionality for new commits.
> > > >
> > > > -Marco
> > > >
> > > > Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 12. März 2020,
> > > > 22:47:
> > > >
> > > > > @Marco Thanks for pointing that out.
> > > > > Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM - 3:30 PM in
> > > (UTC-08:00)
> > > > > Pacific Time (US & Canada).
> > > > >
> > > > > > When do we expect this bot to be deployed?
> > > > > @Lin If all goes well in the next week I can deploy it to public
> > Apache
> > > > > (provided I get permissions from Apache Infra)
> > > > >
> > > > > @Marco Thanks for your feedback.
> > > > > > CI system has to support the community without requiring people to
> > > > > constantly shepherd every single run
> > > > > We have data for the number of times CI was triggered unnecessarily
> > > which
> > > > > includes
> > > > > - Entire build triggered instead of specific build
> > > > > - CI triggered when PR is still work in progress or not yet ready
> > (say
> > > -
> > > > > intermediate commits)
> > > > > At the end its a trade-off
> > > > > Money, Resources, Time to build for each and every commit vs Pain of
> > > > > triggering builds
> > > > >
> > > > >
> > > > > > Scan trigger plugin would poll SCM. Can we use plugin at scale?
> > > > >
> > > > > 1. I haven't tested it on scale. But I think with the current scale
> > of
> > > > > MXNet repo (191 open PRs i.e. checking for changes to 191 branches -
> > It
> > > > > should be manageable)
> > > > > 2. What's the purpose of the plugin? tldr; Branch discovery or branch
> > > > > indexing.
> > > > > Scan trigger plugin comes into the picture only once per PR per job
> > > > (i.e. 8
> > > > > times per PR for 8 jobs). It is basically done when a new PR is made
> > > and
> > > > > the job (say unix-cpu hasn't discovered the new PR branch yet).
> > That's
> > > > it.
> > > > > So it shouldn't be a problem for public MXNet repo.
> > > > >
> > > > > Thanks,
> > > > > Chai
> > > > >
> > > > >
> > > > > On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <
> > marco.g.abreu@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Btw you forgot to set a date and time for the metting
> > > > > >
> > > > > >
> > > > > > On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu <
> > > > marco.g.abreu@gmail.com
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Chai, I generally like the idea of the bot. But I'm not a
> > > > > > supporter
> > > > > > > of the idea to disable any automatic triggering (disabling the
> > > > webhook
> > > > > is
> > > > > > > also not an option, considering that this will disable master
> > > > > triggers).
> > > > > > > The CI system has to support the community without requiring
> > people
> > > > to
> > > > > > > constantly shepherd every single run. Disabling automatic
> > > triggering
> > > > > > seems
> > > > > > > like a step back to me.
> > > > > > >
> > > > > > > Instead, I'd recommend that CI gets triggered upon every commit
> > as
> > > > > usual,
> > > > > > > but people have the possibility to call a "command" (i.e. make a
> > > > > message
> > > > > > > which results in the bot setting a label) to disable CI until
> > they
> > > > > revoke
> > > > > > > it. But the standard should still be that a new commit triggers a
> > > new
> > > > > CI
> > > > > > > run.
> > > > > > >
> > > > > > > https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
> > > seems
> > > > > like
> > > > > > > this would poll SCM. This will incur high quota restrictions. Are
> > > you
> > > > > > sure
> > > > > > > that you can use that plugin at scale?
> > > > > > >
> > > > > > > -Marco
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <ap...@gmail.com>
> > > > wrote:
> > > > > > > > Chai,
> > > > > > > >
> > > > > > > > Awesome work. When do we expect this bot to be deployed?
> > > > > > > >
> > > > > > > > Best,
> > > > > > > >
> > > > > > > > Lin
> > > > > > > >
> > > > > > > > On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <
> > > > chai.bapat@gmail.com
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello MXNet community,
> > > > > > > > >
> > > > > > > > > I have built an MXNet Bot <https://github.com/mxnet-bot> that
> > > > > allows
> > > > > > PR
> > > > > > > > > Authors, Committers and Jenkins Admins to trigger CI manually.
> > > > > > > > > It handles 2 problems
> > > > > > > > > 1. Manual CI trigger instead of existing automated CI trigger
> > > > > > > > > 2. Gives permissions to PR Authors (in addition to MXNet
> > > > Committers
> > > > > > and
> > > > > > > > > Jenkins Admins)
> > > > > > > > >
> > > > > > > > > Design Doc :
> > > > > > > > >
> > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > > > > > > > > I urge you all to attend the demonstration meeting and lend
> > your
> > > > > views
> > > > > > > > on
> > > > > > > > > the same.
> > > > > > > > >
> > > > > > > > > Thank you,
> > > > > > > > > Chai
> > > > > > > > >
> > > > > > > > > *Meeting Details*:
> > > > > > > > > ==============Conference Bridge Information==============
> > > > > > > > > You have been invited to an online meeting, powered by Amazon
> > > > Chime.
> > > > > > > > > *Chime meeting ID*: *9272158344*
> > > > > > > > > Join via Chime clients (manually): Select 'Meetings > Join a
> > > > > Meeting',
> > > > > > > > and
> > > > > > > > > enter 9272158344
> > > > > > > > > Join via Chime clients (auto-call): If you invite auto-call as
> > > > > > attendee,
> > > > > > > > > Chime will call you when the meeting starts, select 'Answer'
> > > > > > > > > *Join via browser screen share*: https://chime.aws/9272158344
> > > > > > > > > *Join via phone* (US): +1-929-432-4463,,,9272158344#
> > > > > > > > > *Join via phone (US toll-free)*: +1-855-552-4463,,,9272158344#
> > > > > > > > > International dial-in: https://chime.aws/dialinnumbers/
> > > > > > > > > In-room video system: Ext: 62000, Meeting PIN: 9272158344#
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > *Chaitanya Prakash Bapat*
> > > > > > > > > *+1 (973) 953-6299*
> > > > > > > > >
> > > > > > > > > [image: https://www.linkedin.com//in/chaibapat25]
> > > > > > > > > <https://github.com/ChaiBapchya>[image:
> > > > > > > > https://www.facebook.com/chaibapat
> > > > > > > > > ]
> > > > > > > > > <https://www.facebook.com/chaibapchya>[image:
> > > > > > > > > https://twitter.com/ChaiBapchya] <
> > > https://twitter.com/ChaiBapchya
> > > > > > > > > [image:
> > > > > > > > > https://www.linkedin.com//in/chaibapat25]
> > > > > > > > > <https://www.linkedin.com//in/chaibapchya/>
> > > > > > > > >
> > > > >
> > > > > --
> > > > > *Chaitanya Prakash Bapat*
> > > > > *+1 (973) 953-6299*
> > > > >
> > > > > [image: https://www.linkedin.com//in/chaibapat25]
> > > > > <https://github.com/ChaiBapchya>[image:
> > > > https://www.facebook.com/chaibapat
> > > > > ]
> > > > > <https://www.facebook.com/chaibapchya>[image:
> > > > > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> > > > > [image:
> > > > > https://www.linkedin.com//in/chaibapat25]
> > > > > <https://www.linkedin.com//in/chaibapchya/>
> > > > >
> >
> > --
> > Sandeep Krishnamurthy
> >
Re: MXNet Bot Demo
Posted by Marco de Abreu <ma...@gmail.com>.
Thanks for the data Sandeep. In these cases it sounds like it would have
rather been better when people explicitly turned off CI in that case.
What's the argument against an opt-out instead of an opt-in?
My intention is that I consider it quite cumbersome to make it a *required*
step to always trigger CI manually, even if just submitting a small PR. I'd
rather see people explicitly turning off CI if they wouldn't like to use it
- and there's also the "draft" stage for a PR which some contributors are
using.
With regards to WIP and do not review: I think these are use cases where
you want CI feedback, as otherwise you wouldn't have opened the PR. If you
don't want human feedback and neither machine feedback, why open the PR at
all?
-Marco
sandeep krishnamurthy <sa...@gmail.com> schrieb am Fr., 13.
März 2020, 05:24:
> I tried to gather some data for us to discuss this topic in this thread. I
> tried to count number of un-necessary builds by looking at most recent (as
> of 12, March 9 PM PST) 50 PRs merged to master and 50 PRs.
> Identifying un-necessary builds is bit subjective. I tried to be more
> conservative where I didn't count a build as un-necessary if I was in
> doubt. Hence, I was not able to automate, but I made an effort to go
> through PRs manually and use below criteria to identify un-necessary
> commits triggering the builds.
>
> 1. Explicitly marked as WIP / do not review PR
> 2. Incremental WIP commit and finally commenting a commit “trigger CI”
> 3. Multiple commits to address all comments from single review. This is
> assuming we see a comment, address them, commit, next the following
> comment
> 4. Sequence of documentation only changes
>
> I found there were around 42 avoidable builds from most recent 50 merged
> PRs and around 86 builds from recent 50 open PRs.
>
> I synced up with other contributors (Joe Evans, Chai) from Amazon who is
> contributing to MXNet CI system. I was told that on an average it costs
> around $84 per build and on an average 6 commits per merged PR (for year
> 2019). Going by that, it is approximately 1/6 builds are avoidable. [100 /
> 300 + 300 ]
>
>
> Usability should be top most priority. But, since either a reviewer or pr
> author can trigger the bot, is it really a hurdle for pr author or reviewer
> to call a bot to trigger CI? Given that PR author and reviewer is already
> actively commenting various details such as - PR description, review
> comments and responses, adding labels etc.
>
>
> Me too curious to know the behavior for Tao's above use case.
>
>
> Best,
>
> Sandeep
>
> On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <mu...@gmail.com> wrote:
>
> > Is it possible for re-triggering a single job to be abused? For example,
> > the author spends two days re-triggering a flaky job to make it pass. But
> > other jobs which have passed the validation may be broken by other
> commits
> > during the two day without being noticed. And finally the PR is merged
> with
> > underlying problems.
> >
> > On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <ma...@gmail.com>
> > wrote:
> >
> > > In the end it only comes down to money, considering that the system is
> > auto
> > > scaling, making the execution time constant.
> > >
> > > If we're trading money for usability, I certainly would prefer
> usability.
> > > I'd rather recommend to spend time on parallelizing test execution or
> > > getting rid of integration tests in the PR stage instead reducing the
> > costs
> > > by making people not use it. But taking a step back to requiring people
> > to
> > > manually trigger CI again doesn't feel right.
> > >
> > > I'm happy to see that bot deployed, but I do not agree with removing
> the
> > > auto trigger functionality for new commits.
> > >
> > > -Marco
> > >
> > > Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 12. März 2020,
> > > 22:47:
> > >
> > > > @Marco Thanks for pointing that out.
> > > > Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM - 3:30 PM in
> > (UTC-08:00)
> > > > Pacific Time (US & Canada).
> > > >
> > > > > When do we expect this bot to be deployed?
> > > > @Lin If all goes well in the next week I can deploy it to public
> Apache
> > > > (provided I get permissions from Apache Infra)
> > > >
> > > > @Marco Thanks for your feedback.
> > > > > CI system has to support the community without requiring people to
> > > > constantly shepherd every single run
> > > > We have data for the number of times CI was triggered unnecessarily
> > which
> > > > includes
> > > > - Entire build triggered instead of specific build
> > > > - CI triggered when PR is still work in progress or not yet ready
> (say
> > -
> > > > intermediate commits)
> > > > At the end its a trade-off
> > > > Money, Resources, Time to build for each and every commit vs Pain of
> > > > triggering builds
> > > >
> > > >
> > > > > Scan trigger plugin would poll SCM. Can we use plugin at scale?
> > > >
> > > > 1. I haven't tested it on scale. But I think with the current scale
> of
> > > > MXNet repo (191 open PRs i.e. checking for changes to 191 branches -
> It
> > > > should be manageable)
> > > > 2. What's the purpose of the plugin? tldr; Branch discovery or branch
> > > > indexing.
> > > > Scan trigger plugin comes into the picture only once per PR per job
> > > (i.e. 8
> > > > times per PR for 8 jobs). It is basically done when a new PR is made
> > and
> > > > the job (say unix-cpu hasn't discovered the new PR branch yet).
> That's
> > > it.
> > > > So it shouldn't be a problem for public MXNet repo.
> > > >
> > > > Thanks,
> > > > Chai
> > > >
> > > >
> > > > On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <
> marco.g.abreu@gmail.com>
> > > > wrote:
> > > >
> > > > > Btw you forgot to set a date and time for the metting
> > > > >
> > > > >
> > > > > On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu <
> > > marco.g.abreu@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Thanks Chai, I generally like the idea of the bot. But I'm not a
> > > > > supporter
> > > > > > of the idea to disable any automatic triggering (disabling the
> > > webhook
> > > > is
> > > > > > also not an option, considering that this will disable master
> > > > triggers).
> > > > > > The CI system has to support the community without requiring
> people
> > > to
> > > > > > constantly shepherd every single run. Disabling automatic
> > triggering
> > > > > seems
> > > > > > like a step back to me.
> > > > > >
> > > > > > Instead, I'd recommend that CI gets triggered upon every commit
> as
> > > > usual,
> > > > > > but people have the possibility to call a "command" (i.e. make a
> > > > message
> > > > > > which results in the bot setting a label) to disable CI until
> they
> > > > revoke
> > > > > > it. But the standard should still be that a new commit triggers a
> > new
> > > > CI
> > > > > > run.
> > > > > >
> > > > > > https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
> > seems
> > > > like
> > > > > > this would poll SCM. This will incur high quota restrictions. Are
> > you
> > > > > sure
> > > > > > that you can use that plugin at scale?
> > > > > >
> > > > > > -Marco
> > > > > >
> > > > > >
> > > > > > On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <ap...@gmail.com>
> > > wrote:
> > > > > >
> > > > > >> Chai,
> > > > > >>
> > > > > >> Awesome work. When do we expect this bot to be deployed?
> > > > > >>
> > > > > >> Best,
> > > > > >>
> > > > > >> Lin
> > > > > >>
> > > > > >> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <
> > > chai.bapat@gmail.com
> > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hello MXNet community,
> > > > > >> >
> > > > > >> > I have built an MXNet Bot <https://github.com/mxnet-bot> that
> > > > allows
> > > > > PR
> > > > > >> > Authors, Committers and Jenkins Admins to trigger CI manually.
> > > > > >> > It handles 2 problems
> > > > > >> > 1. Manual CI trigger instead of existing automated CI trigger
> > > > > >> > 2. Gives permissions to PR Authors (in addition to MXNet
> > > Committers
> > > > > and
> > > > > >> > Jenkins Admins)
> > > > > >> >
> > > > > >> > Design Doc :
> > > > > >> >
> https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > > > > >> >
> > > > > >> > I urge you all to attend the demonstration meeting and lend
> your
> > > > views
> > > > > >> on
> > > > > >> > the same.
> > > > > >> >
> > > > > >> > Thank you,
> > > > > >> > Chai
> > > > > >> >
> > > > > >> > *Meeting Details*:
> > > > > >> > ==============Conference Bridge Information==============
> > > > > >> > You have been invited to an online meeting, powered by Amazon
> > > Chime.
> > > > > >> > *Chime meeting ID*: *9272158344*
> > > > > >> > Join via Chime clients (manually): Select 'Meetings > Join a
> > > > Meeting',
> > > > > >> and
> > > > > >> > enter 9272158344
> > > > > >> > Join via Chime clients (auto-call): If you invite auto-call as
> > > > > attendee,
> > > > > >> > Chime will call you when the meeting starts, select 'Answer'
> > > > > >> > *Join via browser screen share*: https://chime.aws/9272158344
> > > > > >> > *Join via phone* (US): +1-929-432-4463,,,9272158344#
> > > > > >> > *Join via phone (US toll-free)*: +1-855-552-4463,,,9272158344#
> > > > > >> > International dial-in: https://chime.aws/dialinnumbers/
> > > > > >> > In-room video system: Ext: 62000, Meeting PIN: 9272158344#
> > > > > >> >
> > > > > >> > --
> > > > > >> > *Chaitanya Prakash Bapat*
> > > > > >> > *+1 (973) 953-6299*
> > > > > >> >
> > > > > >> > [image: https://www.linkedin.com//in/chaibapat25]
> > > > > >> > <https://github.com/ChaiBapchya>[image:
> > > > > >> https://www.facebook.com/chaibapat
> > > > > >> > ]
> > > > > >> > <https://www.facebook.com/chaibapchya>[image:
> > > > > >> > https://twitter.com/ChaiBapchya] <
> > https://twitter.com/ChaiBapchya
> > > > > >> >[image:
> > > > > >> > https://www.linkedin.com//in/chaibapat25]
> > > > > >> > <https://www.linkedin.com//in/chaibapchya/>
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > *Chaitanya Prakash Bapat*
> > > > *+1 (973) 953-6299*
> > > >
> > > > [image: https://www.linkedin.com//in/chaibapat25]
> > > > <https://github.com/ChaiBapchya>[image:
> > > https://www.facebook.com/chaibapat
> > > > ]
> > > > <https://www.facebook.com/chaibapchya>[image:
> > > > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> > > >[image:
> > > > https://www.linkedin.com//in/chaibapat25]
> > > > <https://www.linkedin.com//in/chaibapchya/>
> > > >
> > >
> >
>
>
> --
> Sandeep Krishnamurthy
>
Re: MXNet Bot Demo
Posted by sandeep krishnamurthy <sa...@gmail.com>.
I tried to gather some data for us to discuss this topic in this thread. I
tried to count number of un-necessary builds by looking at most recent (as
of 12, March 9 PM PST) 50 PRs merged to master and 50 PRs.
Identifying un-necessary builds is bit subjective. I tried to be more
conservative where I didn't count a build as un-necessary if I was in
doubt. Hence, I was not able to automate, but I made an effort to go
through PRs manually and use below criteria to identify un-necessary
commits triggering the builds.
1. Explicitly marked as WIP / do not review PR
2. Incremental WIP commit and finally commenting a commit “trigger CI”
3. Multiple commits to address all comments from single review. This is
assuming we see a comment, address them, commit, next the following comment
4. Sequence of documentation only changes
I found there were around 42 avoidable builds from most recent 50 merged
PRs and around 86 builds from recent 50 open PRs.
I synced up with other contributors (Joe Evans, Chai) from Amazon who is
contributing to MXNet CI system. I was told that on an average it costs
around $84 per build and on an average 6 commits per merged PR (for year
2019). Going by that, it is approximately 1/6 builds are avoidable. [100 /
300 + 300 ]
Usability should be top most priority. But, since either a reviewer or pr
author can trigger the bot, is it really a hurdle for pr author or reviewer
to call a bot to trigger CI? Given that PR author and reviewer is already
actively commenting various details such as - PR description, review
comments and responses, adding labels etc.
Me too curious to know the behavior for Tao's above use case.
Best,
Sandeep
On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <mu...@gmail.com> wrote:
> Is it possible for re-triggering a single job to be abused? For example,
> the author spends two days re-triggering a flaky job to make it pass. But
> other jobs which have passed the validation may be broken by other commits
> during the two day without being noticed. And finally the PR is merged with
> underlying problems.
>
> On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <ma...@gmail.com>
> wrote:
>
> > In the end it only comes down to money, considering that the system is
> auto
> > scaling, making the execution time constant.
> >
> > If we're trading money for usability, I certainly would prefer usability.
> > I'd rather recommend to spend time on parallelizing test execution or
> > getting rid of integration tests in the PR stage instead reducing the
> costs
> > by making people not use it. But taking a step back to requiring people
> to
> > manually trigger CI again doesn't feel right.
> >
> > I'm happy to see that bot deployed, but I do not agree with removing the
> > auto trigger functionality for new commits.
> >
> > -Marco
> >
> > Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 12. März 2020,
> > 22:47:
> >
> > > @Marco Thanks for pointing that out.
> > > Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM - 3:30 PM in
> (UTC-08:00)
> > > Pacific Time (US & Canada).
> > >
> > > > When do we expect this bot to be deployed?
> > > @Lin If all goes well in the next week I can deploy it to public Apache
> > > (provided I get permissions from Apache Infra)
> > >
> > > @Marco Thanks for your feedback.
> > > > CI system has to support the community without requiring people to
> > > constantly shepherd every single run
> > > We have data for the number of times CI was triggered unnecessarily
> which
> > > includes
> > > - Entire build triggered instead of specific build
> > > - CI triggered when PR is still work in progress or not yet ready (say
> -
> > > intermediate commits)
> > > At the end its a trade-off
> > > Money, Resources, Time to build for each and every commit vs Pain of
> > > triggering builds
> > >
> > >
> > > > Scan trigger plugin would poll SCM. Can we use plugin at scale?
> > >
> > > 1. I haven't tested it on scale. But I think with the current scale of
> > > MXNet repo (191 open PRs i.e. checking for changes to 191 branches - It
> > > should be manageable)
> > > 2. What's the purpose of the plugin? tldr; Branch discovery or branch
> > > indexing.
> > > Scan trigger plugin comes into the picture only once per PR per job
> > (i.e. 8
> > > times per PR for 8 jobs). It is basically done when a new PR is made
> and
> > > the job (say unix-cpu hasn't discovered the new PR branch yet). That's
> > it.
> > > So it shouldn't be a problem for public MXNet repo.
> > >
> > > Thanks,
> > > Chai
> > >
> > >
> > > On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <ma...@gmail.com>
> > > wrote:
> > >
> > > > Btw you forgot to set a date and time for the metting
> > > >
> > > >
> > > > On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu <
> > marco.g.abreu@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Thanks Chai, I generally like the idea of the bot. But I'm not a
> > > > supporter
> > > > > of the idea to disable any automatic triggering (disabling the
> > webhook
> > > is
> > > > > also not an option, considering that this will disable master
> > > triggers).
> > > > > The CI system has to support the community without requiring people
> > to
> > > > > constantly shepherd every single run. Disabling automatic
> triggering
> > > > seems
> > > > > like a step back to me.
> > > > >
> > > > > Instead, I'd recommend that CI gets triggered upon every commit as
> > > usual,
> > > > > but people have the possibility to call a "command" (i.e. make a
> > > message
> > > > > which results in the bot setting a label) to disable CI until they
> > > revoke
> > > > > it. But the standard should still be that a new commit triggers a
> new
> > > CI
> > > > > run.
> > > > >
> > > > > https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
> seems
> > > like
> > > > > this would poll SCM. This will incur high quota restrictions. Are
> you
> > > > sure
> > > > > that you can use that plugin at scale?
> > > > >
> > > > > -Marco
> > > > >
> > > > >
> > > > > On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <ap...@gmail.com>
> > wrote:
> > > > >
> > > > >> Chai,
> > > > >>
> > > > >> Awesome work. When do we expect this bot to be deployed?
> > > > >>
> > > > >> Best,
> > > > >>
> > > > >> Lin
> > > > >>
> > > > >> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <
> > chai.bapat@gmail.com
> > > >
> > > > >> wrote:
> > > > >>
> > > > >> > Hello MXNet community,
> > > > >> >
> > > > >> > I have built an MXNet Bot <https://github.com/mxnet-bot> that
> > > allows
> > > > PR
> > > > >> > Authors, Committers and Jenkins Admins to trigger CI manually.
> > > > >> > It handles 2 problems
> > > > >> > 1. Manual CI trigger instead of existing automated CI trigger
> > > > >> > 2. Gives permissions to PR Authors (in addition to MXNet
> > Committers
> > > > and
> > > > >> > Jenkins Admins)
> > > > >> >
> > > > >> > Design Doc :
> > > > >> > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > > > >> >
> > > > >> > I urge you all to attend the demonstration meeting and lend your
> > > views
> > > > >> on
> > > > >> > the same.
> > > > >> >
> > > > >> > Thank you,
> > > > >> > Chai
> > > > >> >
> > > > >> > *Meeting Details*:
> > > > >> > ==============Conference Bridge Information==============
> > > > >> > You have been invited to an online meeting, powered by Amazon
> > Chime.
> > > > >> > *Chime meeting ID*: *9272158344*
> > > > >> > Join via Chime clients (manually): Select 'Meetings > Join a
> > > Meeting',
> > > > >> and
> > > > >> > enter 9272158344
> > > > >> > Join via Chime clients (auto-call): If you invite auto-call as
> > > > attendee,
> > > > >> > Chime will call you when the meeting starts, select 'Answer'
> > > > >> > *Join via browser screen share*: https://chime.aws/9272158344
> > > > >> > *Join via phone* (US): +1-929-432-4463,,,9272158344#
> > > > >> > *Join via phone (US toll-free)*: +1-855-552-4463,,,9272158344#
> > > > >> > International dial-in: https://chime.aws/dialinnumbers/
> > > > >> > In-room video system: Ext: 62000, Meeting PIN: 9272158344#
> > > > >> >
> > > > >> > --
> > > > >> > *Chaitanya Prakash Bapat*
> > > > >> > *+1 (973) 953-6299*
> > > > >> >
> > > > >> > [image: https://www.linkedin.com//in/chaibapat25]
> > > > >> > <https://github.com/ChaiBapchya>[image:
> > > > >> https://www.facebook.com/chaibapat
> > > > >> > ]
> > > > >> > <https://www.facebook.com/chaibapchya>[image:
> > > > >> > https://twitter.com/ChaiBapchya] <
> https://twitter.com/ChaiBapchya
> > > > >> >[image:
> > > > >> > https://www.linkedin.com//in/chaibapat25]
> > > > >> > <https://www.linkedin.com//in/chaibapchya/>
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> > >
> > > --
> > > *Chaitanya Prakash Bapat*
> > > *+1 (973) 953-6299*
> > >
> > > [image: https://www.linkedin.com//in/chaibapat25]
> > > <https://github.com/ChaiBapchya>[image:
> > https://www.facebook.com/chaibapat
> > > ]
> > > <https://www.facebook.com/chaibapchya>[image:
> > > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> > >[image:
> > > https://www.linkedin.com//in/chaibapat25]
> > > <https://www.linkedin.com//in/chaibapchya/>
> > >
> >
>
--
Sandeep Krishnamurthy
Re: MXNet Bot Demo
Posted by Tao Lv <mu...@gmail.com>.
Is it possible for re-triggering a single job to be abused? For example,
the author spends two days re-triggering a flaky job to make it pass. But
other jobs which have passed the validation may be broken by other commits
during the two day without being noticed. And finally the PR is merged with
underlying problems.
On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <ma...@gmail.com>
wrote:
> In the end it only comes down to money, considering that the system is auto
> scaling, making the execution time constant.
>
> If we're trading money for usability, I certainly would prefer usability.
> I'd rather recommend to spend time on parallelizing test execution or
> getting rid of integration tests in the PR stage instead reducing the costs
> by making people not use it. But taking a step back to requiring people to
> manually trigger CI again doesn't feel right.
>
> I'm happy to see that bot deployed, but I do not agree with removing the
> auto trigger functionality for new commits.
>
> -Marco
>
> Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 12. März 2020,
> 22:47:
>
> > @Marco Thanks for pointing that out.
> > Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM - 3:30 PM in (UTC-08:00)
> > Pacific Time (US & Canada).
> >
> > > When do we expect this bot to be deployed?
> > @Lin If all goes well in the next week I can deploy it to public Apache
> > (provided I get permissions from Apache Infra)
> >
> > @Marco Thanks for your feedback.
> > > CI system has to support the community without requiring people to
> > constantly shepherd every single run
> > We have data for the number of times CI was triggered unnecessarily which
> > includes
> > - Entire build triggered instead of specific build
> > - CI triggered when PR is still work in progress or not yet ready (say -
> > intermediate commits)
> > At the end its a trade-off
> > Money, Resources, Time to build for each and every commit vs Pain of
> > triggering builds
> >
> >
> > > Scan trigger plugin would poll SCM. Can we use plugin at scale?
> >
> > 1. I haven't tested it on scale. But I think with the current scale of
> > MXNet repo (191 open PRs i.e. checking for changes to 191 branches - It
> > should be manageable)
> > 2. What's the purpose of the plugin? tldr; Branch discovery or branch
> > indexing.
> > Scan trigger plugin comes into the picture only once per PR per job
> (i.e. 8
> > times per PR for 8 jobs). It is basically done when a new PR is made and
> > the job (say unix-cpu hasn't discovered the new PR branch yet). That's
> it.
> > So it shouldn't be a problem for public MXNet repo.
> >
> > Thanks,
> > Chai
> >
> >
> > On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <ma...@gmail.com>
> > wrote:
> >
> > > Btw you forgot to set a date and time for the metting
> > >
> > >
> > > On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu <
> marco.g.abreu@gmail.com
> > >
> > > wrote:
> > >
> > > > Thanks Chai, I generally like the idea of the bot. But I'm not a
> > > supporter
> > > > of the idea to disable any automatic triggering (disabling the
> webhook
> > is
> > > > also not an option, considering that this will disable master
> > triggers).
> > > > The CI system has to support the community without requiring people
> to
> > > > constantly shepherd every single run. Disabling automatic triggering
> > > seems
> > > > like a step back to me.
> > > >
> > > > Instead, I'd recommend that CI gets triggered upon every commit as
> > usual,
> > > > but people have the possibility to call a "command" (i.e. make a
> > message
> > > > which results in the bot setting a label) to disable CI until they
> > revoke
> > > > it. But the standard should still be that a new commit triggers a new
> > CI
> > > > run.
> > > >
> > > > https://plugins.jenkins.io/multibranch-scan-webhook-trigger/ seems
> > like
> > > > this would poll SCM. This will incur high quota restrictions. Are you
> > > sure
> > > > that you can use that plugin at scale?
> > > >
> > > > -Marco
> > > >
> > > >
> > > > On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <ap...@gmail.com>
> wrote:
> > > >
> > > >> Chai,
> > > >>
> > > >> Awesome work. When do we expect this bot to be deployed?
> > > >>
> > > >> Best,
> > > >>
> > > >> Lin
> > > >>
> > > >> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <
> chai.bapat@gmail.com
> > >
> > > >> wrote:
> > > >>
> > > >> > Hello MXNet community,
> > > >> >
> > > >> > I have built an MXNet Bot <https://github.com/mxnet-bot> that
> > allows
> > > PR
> > > >> > Authors, Committers and Jenkins Admins to trigger CI manually.
> > > >> > It handles 2 problems
> > > >> > 1. Manual CI trigger instead of existing automated CI trigger
> > > >> > 2. Gives permissions to PR Authors (in addition to MXNet
> Committers
> > > and
> > > >> > Jenkins Admins)
> > > >> >
> > > >> > Design Doc :
> > > >> > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > > >> >
> > > >> > I urge you all to attend the demonstration meeting and lend your
> > views
> > > >> on
> > > >> > the same.
> > > >> >
> > > >> > Thank you,
> > > >> > Chai
> > > >> >
> > > >> > *Meeting Details*:
> > > >> > ==============Conference Bridge Information==============
> > > >> > You have been invited to an online meeting, powered by Amazon
> Chime.
> > > >> > *Chime meeting ID*: *9272158344*
> > > >> > Join via Chime clients (manually): Select 'Meetings > Join a
> > Meeting',
> > > >> and
> > > >> > enter 9272158344
> > > >> > Join via Chime clients (auto-call): If you invite auto-call as
> > > attendee,
> > > >> > Chime will call you when the meeting starts, select 'Answer'
> > > >> > *Join via browser screen share*: https://chime.aws/9272158344
> > > >> > *Join via phone* (US): +1-929-432-4463,,,9272158344#
> > > >> > *Join via phone (US toll-free)*: +1-855-552-4463,,,9272158344#
> > > >> > International dial-in: https://chime.aws/dialinnumbers/
> > > >> > In-room video system: Ext: 62000, Meeting PIN: 9272158344#
> > > >> >
> > > >> > --
> > > >> > *Chaitanya Prakash Bapat*
> > > >> > *+1 (973) 953-6299*
> > > >> >
> > > >> > [image: https://www.linkedin.com//in/chaibapat25]
> > > >> > <https://github.com/ChaiBapchya>[image:
> > > >> https://www.facebook.com/chaibapat
> > > >> > ]
> > > >> > <https://www.facebook.com/chaibapchya>[image:
> > > >> > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> > > >> >[image:
> > > >> > https://www.linkedin.com//in/chaibapat25]
> > > >> > <https://www.linkedin.com//in/chaibapchya/>
> > > >> >
> > > >>
> > > >
> > >
> >
> >
> > --
> > *Chaitanya Prakash Bapat*
> > *+1 (973) 953-6299*
> >
> > [image: https://www.linkedin.com//in/chaibapat25]
> > <https://github.com/ChaiBapchya>[image:
> https://www.facebook.com/chaibapat
> > ]
> > <https://www.facebook.com/chaibapchya>[image:
> > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> >[image:
> > https://www.linkedin.com//in/chaibapat25]
> > <https://www.linkedin.com//in/chaibapchya/>
> >
>
Re: MXNet Bot Demo
Posted by Marco de Abreu <ma...@gmail.com>.
In the end it only comes down to money, considering that the system is auto
scaling, making the execution time constant.
If we're trading money for usability, I certainly would prefer usability.
I'd rather recommend to spend time on parallelizing test execution or
getting rid of integration tests in the PR stage instead reducing the costs
by making people not use it. But taking a step back to requiring people to
manually trigger CI again doesn't feel right.
I'm happy to see that bot deployed, but I do not agree with removing the
auto trigger functionality for new commits.
-Marco
Chaitanya Bapat <ch...@gmail.com> schrieb am Do., 12. März 2020, 22:47:
> @Marco Thanks for pointing that out.
> Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM - 3:30 PM in (UTC-08:00)
> Pacific Time (US & Canada).
>
> > When do we expect this bot to be deployed?
> @Lin If all goes well in the next week I can deploy it to public Apache
> (provided I get permissions from Apache Infra)
>
> @Marco Thanks for your feedback.
> > CI system has to support the community without requiring people to
> constantly shepherd every single run
> We have data for the number of times CI was triggered unnecessarily which
> includes
> - Entire build triggered instead of specific build
> - CI triggered when PR is still work in progress or not yet ready (say -
> intermediate commits)
> At the end its a trade-off
> Money, Resources, Time to build for each and every commit vs Pain of
> triggering builds
>
>
> > Scan trigger plugin would poll SCM. Can we use plugin at scale?
>
> 1. I haven't tested it on scale. But I think with the current scale of
> MXNet repo (191 open PRs i.e. checking for changes to 191 branches - It
> should be manageable)
> 2. What's the purpose of the plugin? tldr; Branch discovery or branch
> indexing.
> Scan trigger plugin comes into the picture only once per PR per job (i.e. 8
> times per PR for 8 jobs). It is basically done when a new PR is made and
> the job (say unix-cpu hasn't discovered the new PR branch yet). That's it.
> So it shouldn't be a problem for public MXNet repo.
>
> Thanks,
> Chai
>
>
> On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <ma...@gmail.com>
> wrote:
>
> > Btw you forgot to set a date and time for the metting
> >
> >
> > On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu <marco.g.abreu@gmail.com
> >
> > wrote:
> >
> > > Thanks Chai, I generally like the idea of the bot. But I'm not a
> > supporter
> > > of the idea to disable any automatic triggering (disabling the webhook
> is
> > > also not an option, considering that this will disable master
> triggers).
> > > The CI system has to support the community without requiring people to
> > > constantly shepherd every single run. Disabling automatic triggering
> > seems
> > > like a step back to me.
> > >
> > > Instead, I'd recommend that CI gets triggered upon every commit as
> usual,
> > > but people have the possibility to call a "command" (i.e. make a
> message
> > > which results in the bot setting a label) to disable CI until they
> revoke
> > > it. But the standard should still be that a new commit triggers a new
> CI
> > > run.
> > >
> > > https://plugins.jenkins.io/multibranch-scan-webhook-trigger/ seems
> like
> > > this would poll SCM. This will incur high quota restrictions. Are you
> > sure
> > > that you can use that plugin at scale?
> > >
> > > -Marco
> > >
> > >
> > > On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <ap...@gmail.com> wrote:
> > >
> > >> Chai,
> > >>
> > >> Awesome work. When do we expect this bot to be deployed?
> > >>
> > >> Best,
> > >>
> > >> Lin
> > >>
> > >> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <chai.bapat@gmail.com
> >
> > >> wrote:
> > >>
> > >> > Hello MXNet community,
> > >> >
> > >> > I have built an MXNet Bot <https://github.com/mxnet-bot> that
> allows
> > PR
> > >> > Authors, Committers and Jenkins Admins to trigger CI manually.
> > >> > It handles 2 problems
> > >> > 1. Manual CI trigger instead of existing automated CI trigger
> > >> > 2. Gives permissions to PR Authors (in addition to MXNet Committers
> > and
> > >> > Jenkins Admins)
> > >> >
> > >> > Design Doc :
> > >> > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > >> >
> > >> > I urge you all to attend the demonstration meeting and lend your
> views
> > >> on
> > >> > the same.
> > >> >
> > >> > Thank you,
> > >> > Chai
> > >> >
> > >> > *Meeting Details*:
> > >> > ==============Conference Bridge Information==============
> > >> > You have been invited to an online meeting, powered by Amazon Chime.
> > >> > *Chime meeting ID*: *9272158344*
> > >> > Join via Chime clients (manually): Select 'Meetings > Join a
> Meeting',
> > >> and
> > >> > enter 9272158344
> > >> > Join via Chime clients (auto-call): If you invite auto-call as
> > attendee,
> > >> > Chime will call you when the meeting starts, select 'Answer'
> > >> > *Join via browser screen share*: https://chime.aws/9272158344
> > >> > *Join via phone* (US): +1-929-432-4463,,,9272158344#
> > >> > *Join via phone (US toll-free)*: +1-855-552-4463,,,9272158344#
> > >> > International dial-in: https://chime.aws/dialinnumbers/
> > >> > In-room video system: Ext: 62000, Meeting PIN: 9272158344#
> > >> >
> > >> > --
> > >> > *Chaitanya Prakash Bapat*
> > >> > *+1 (973) 953-6299*
> > >> >
> > >> > [image: https://www.linkedin.com//in/chaibapat25]
> > >> > <https://github.com/ChaiBapchya>[image:
> > >> https://www.facebook.com/chaibapat
> > >> > ]
> > >> > <https://www.facebook.com/chaibapchya>[image:
> > >> > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> > >> >[image:
> > >> > https://www.linkedin.com//in/chaibapat25]
> > >> > <https://www.linkedin.com//in/chaibapchya/>
> > >> >
> > >>
> > >
> >
>
>
> --
> *Chaitanya Prakash Bapat*
> *+1 (973) 953-6299*
>
> [image: https://www.linkedin.com//in/chaibapat25]
> <https://github.com/ChaiBapchya>[image: https://www.facebook.com/chaibapat
> ]
> <https://www.facebook.com/chaibapchya>[image:
> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
> https://www.linkedin.com//in/chaibapat25]
> <https://www.linkedin.com//in/chaibapchya/>
>
Re: MXNet Bot Demo
Posted by Chaitanya Bapat <ch...@gmail.com>.
@Marco Thanks for pointing that out.
Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM - 3:30 PM in (UTC-08:00)
Pacific Time (US & Canada).
> When do we expect this bot to be deployed?
@Lin If all goes well in the next week I can deploy it to public Apache
(provided I get permissions from Apache Infra)
@Marco Thanks for your feedback.
> CI system has to support the community without requiring people to
constantly shepherd every single run
We have data for the number of times CI was triggered unnecessarily which
includes
- Entire build triggered instead of specific build
- CI triggered when PR is still work in progress or not yet ready (say -
intermediate commits)
At the end its a trade-off
Money, Resources, Time to build for each and every commit vs Pain of
triggering builds
> Scan trigger plugin would poll SCM. Can we use plugin at scale?
1. I haven't tested it on scale. But I think with the current scale of
MXNet repo (191 open PRs i.e. checking for changes to 191 branches - It
should be manageable)
2. What's the purpose of the plugin? tldr; Branch discovery or branch
indexing.
Scan trigger plugin comes into the picture only once per PR per job (i.e. 8
times per PR for 8 jobs). It is basically done when a new PR is made and
the job (say unix-cpu hasn't discovered the new PR branch yet). That's it.
So it shouldn't be a problem for public MXNet repo.
Thanks,
Chai
On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <ma...@gmail.com>
wrote:
> Btw you forgot to set a date and time for the metting
>
>
> On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu <ma...@gmail.com>
> wrote:
>
> > Thanks Chai, I generally like the idea of the bot. But I'm not a
> supporter
> > of the idea to disable any automatic triggering (disabling the webhook is
> > also not an option, considering that this will disable master triggers).
> > The CI system has to support the community without requiring people to
> > constantly shepherd every single run. Disabling automatic triggering
> seems
> > like a step back to me.
> >
> > Instead, I'd recommend that CI gets triggered upon every commit as usual,
> > but people have the possibility to call a "command" (i.e. make a message
> > which results in the bot setting a label) to disable CI until they revoke
> > it. But the standard should still be that a new commit triggers a new CI
> > run.
> >
> > https://plugins.jenkins.io/multibranch-scan-webhook-trigger/ seems like
> > this would poll SCM. This will incur high quota restrictions. Are you
> sure
> > that you can use that plugin at scale?
> >
> > -Marco
> >
> >
> > On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <ap...@gmail.com> wrote:
> >
> >> Chai,
> >>
> >> Awesome work. When do we expect this bot to be deployed?
> >>
> >> Best,
> >>
> >> Lin
> >>
> >> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <ch...@gmail.com>
> >> wrote:
> >>
> >> > Hello MXNet community,
> >> >
> >> > I have built an MXNet Bot <https://github.com/mxnet-bot> that allows
> PR
> >> > Authors, Committers and Jenkins Admins to trigger CI manually.
> >> > It handles 2 problems
> >> > 1. Manual CI trigger instead of existing automated CI trigger
> >> > 2. Gives permissions to PR Authors (in addition to MXNet Committers
> and
> >> > Jenkins Admins)
> >> >
> >> > Design Doc :
> >> > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> >> >
> >> > I urge you all to attend the demonstration meeting and lend your views
> >> on
> >> > the same.
> >> >
> >> > Thank you,
> >> > Chai
> >> >
> >> > *Meeting Details*:
> >> > ==============Conference Bridge Information==============
> >> > You have been invited to an online meeting, powered by Amazon Chime.
> >> > *Chime meeting ID*: *9272158344*
> >> > Join via Chime clients (manually): Select 'Meetings > Join a Meeting',
> >> and
> >> > enter 9272158344
> >> > Join via Chime clients (auto-call): If you invite auto-call as
> attendee,
> >> > Chime will call you when the meeting starts, select 'Answer'
> >> > *Join via browser screen share*: https://chime.aws/9272158344
> >> > *Join via phone* (US): +1-929-432-4463,,,9272158344#
> >> > *Join via phone (US toll-free)*: +1-855-552-4463,,,9272158344#
> >> > International dial-in: https://chime.aws/dialinnumbers/
> >> > In-room video system: Ext: 62000, Meeting PIN: 9272158344#
> >> >
> >> > --
> >> > *Chaitanya Prakash Bapat*
> >> > *+1 (973) 953-6299*
> >> >
> >> > [image: https://www.linkedin.com//in/chaibapat25]
> >> > <https://github.com/ChaiBapchya>[image:
> >> https://www.facebook.com/chaibapat
> >> > ]
> >> > <https://www.facebook.com/chaibapchya>[image:
> >> > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> >> >[image:
> >> > https://www.linkedin.com//in/chaibapat25]
> >> > <https://www.linkedin.com//in/chaibapchya/>
> >> >
> >>
> >
>
--
*Chaitanya Prakash Bapat*
*+1 (973) 953-6299*
[image: https://www.linkedin.com//in/chaibapat25]
<https://github.com/ChaiBapchya>[image: https://www.facebook.com/chaibapat]
<https://www.facebook.com/chaibapchya>[image:
https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
https://www.linkedin.com//in/chaibapat25]
<https://www.linkedin.com//in/chaibapchya/>
Re: MXNet Bot Demo
Posted by Marco de Abreu <ma...@gmail.com>.
Btw you forgot to set a date and time for the metting
On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu <ma...@gmail.com>
wrote:
> Thanks Chai, I generally like the idea of the bot. But I'm not a supporter
> of the idea to disable any automatic triggering (disabling the webhook is
> also not an option, considering that this will disable master triggers).
> The CI system has to support the community without requiring people to
> constantly shepherd every single run. Disabling automatic triggering seems
> like a step back to me.
>
> Instead, I'd recommend that CI gets triggered upon every commit as usual,
> but people have the possibility to call a "command" (i.e. make a message
> which results in the bot setting a label) to disable CI until they revoke
> it. But the standard should still be that a new commit triggers a new CI
> run.
>
> https://plugins.jenkins.io/multibranch-scan-webhook-trigger/ seems like
> this would poll SCM. This will incur high quota restrictions. Are you sure
> that you can use that plugin at scale?
>
> -Marco
>
>
> On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <ap...@gmail.com> wrote:
>
>> Chai,
>>
>> Awesome work. When do we expect this bot to be deployed?
>>
>> Best,
>>
>> Lin
>>
>> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <ch...@gmail.com>
>> wrote:
>>
>> > Hello MXNet community,
>> >
>> > I have built an MXNet Bot <https://github.com/mxnet-bot> that allows PR
>> > Authors, Committers and Jenkins Admins to trigger CI manually.
>> > It handles 2 problems
>> > 1. Manual CI trigger instead of existing automated CI trigger
>> > 2. Gives permissions to PR Authors (in addition to MXNet Committers and
>> > Jenkins Admins)
>> >
>> > Design Doc :
>> > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
>> >
>> > I urge you all to attend the demonstration meeting and lend your views
>> on
>> > the same.
>> >
>> > Thank you,
>> > Chai
>> >
>> > *Meeting Details*:
>> > ==============Conference Bridge Information==============
>> > You have been invited to an online meeting, powered by Amazon Chime.
>> > *Chime meeting ID*: *9272158344*
>> > Join via Chime clients (manually): Select 'Meetings > Join a Meeting',
>> and
>> > enter 9272158344
>> > Join via Chime clients (auto-call): If you invite auto-call as attendee,
>> > Chime will call you when the meeting starts, select 'Answer'
>> > *Join via browser screen share*: https://chime.aws/9272158344
>> > *Join via phone* (US): +1-929-432-4463,,,9272158344#
>> > *Join via phone (US toll-free)*: +1-855-552-4463,,,9272158344#
>> > International dial-in: https://chime.aws/dialinnumbers/
>> > In-room video system: Ext: 62000, Meeting PIN: 9272158344#
>> >
>> > --
>> > *Chaitanya Prakash Bapat*
>> > *+1 (973) 953-6299*
>> >
>> > [image: https://www.linkedin.com//in/chaibapat25]
>> > <https://github.com/ChaiBapchya>[image:
>> https://www.facebook.com/chaibapat
>> > ]
>> > <https://www.facebook.com/chaibapchya>[image:
>> > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
>> >[image:
>> > https://www.linkedin.com//in/chaibapat25]
>> > <https://www.linkedin.com//in/chaibapchya/>
>> >
>>
>
Re: MXNet Bot Demo
Posted by Marco de Abreu <ma...@gmail.com>.
Thanks Chai, I generally like the idea of the bot. But I'm not a supporter
of the idea to disable any automatic triggering (disabling the webhook is
also not an option, considering that this will disable master triggers).
The CI system has to support the community without requiring people to
constantly shepherd every single run. Disabling automatic triggering seems
like a step back to me.
Instead, I'd recommend that CI gets triggered upon every commit as usual,
but people have the possibility to call a "command" (i.e. make a message
which results in the bot setting a label) to disable CI until they revoke
it. But the standard should still be that a new commit triggers a new CI
run.
https://plugins.jenkins.io/multibranch-scan-webhook-trigger/ seems like
this would poll SCM. This will incur high quota restrictions. Are you sure
that you can use that plugin at scale?
-Marco
On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <ap...@gmail.com> wrote:
> Chai,
>
> Awesome work. When do we expect this bot to be deployed?
>
> Best,
>
> Lin
>
> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <ch...@gmail.com>
> wrote:
>
> > Hello MXNet community,
> >
> > I have built an MXNet Bot <https://github.com/mxnet-bot> that allows PR
> > Authors, Committers and Jenkins Admins to trigger CI manually.
> > It handles 2 problems
> > 1. Manual CI trigger instead of existing automated CI trigger
> > 2. Gives permissions to PR Authors (in addition to MXNet Committers and
> > Jenkins Admins)
> >
> > Design Doc :
> > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> >
> > I urge you all to attend the demonstration meeting and lend your views on
> > the same.
> >
> > Thank you,
> > Chai
> >
> > *Meeting Details*:
> > ==============Conference Bridge Information==============
> > You have been invited to an online meeting, powered by Amazon Chime.
> > *Chime meeting ID*: *9272158344*
> > Join via Chime clients (manually): Select 'Meetings > Join a Meeting',
> and
> > enter 9272158344
> > Join via Chime clients (auto-call): If you invite auto-call as attendee,
> > Chime will call you when the meeting starts, select 'Answer'
> > *Join via browser screen share*: https://chime.aws/9272158344
> > *Join via phone* (US): +1-929-432-4463,,,9272158344#
> > *Join via phone (US toll-free)*: +1-855-552-4463,,,9272158344#
> > International dial-in: https://chime.aws/dialinnumbers/
> > In-room video system: Ext: 62000, Meeting PIN: 9272158344#
> >
> > --
> > *Chaitanya Prakash Bapat*
> > *+1 (973) 953-6299*
> >
> > [image: https://www.linkedin.com//in/chaibapat25]
> > <https://github.com/ChaiBapchya>[image:
> https://www.facebook.com/chaibapat
> > ]
> > <https://www.facebook.com/chaibapchya>[image:
> > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> >[image:
> > https://www.linkedin.com//in/chaibapat25]
> > <https://www.linkedin.com//in/chaibapchya/>
> >
>
Re: MXNet Bot Demo
Posted by Lin Yuan <ap...@gmail.com>.
Chai,
Awesome work. When do we expect this bot to be deployed?
Best,
Lin
On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <ch...@gmail.com>
wrote:
> Hello MXNet community,
>
> I have built an MXNet Bot <https://github.com/mxnet-bot> that allows PR
> Authors, Committers and Jenkins Admins to trigger CI manually.
> It handles 2 problems
> 1. Manual CI trigger instead of existing automated CI trigger
> 2. Gives permissions to PR Authors (in addition to MXNet Committers and
> Jenkins Admins)
>
> Design Doc :
> https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
>
> I urge you all to attend the demonstration meeting and lend your views on
> the same.
>
> Thank you,
> Chai
>
> *Meeting Details*:
> ==============Conference Bridge Information==============
> You have been invited to an online meeting, powered by Amazon Chime.
> *Chime meeting ID*: *9272158344*
> Join via Chime clients (manually): Select 'Meetings > Join a Meeting', and
> enter 9272158344
> Join via Chime clients (auto-call): If you invite auto-call as attendee,
> Chime will call you when the meeting starts, select 'Answer'
> *Join via browser screen share*: https://chime.aws/9272158344
> *Join via phone* (US): +1-929-432-4463,,,9272158344#
> *Join via phone (US toll-free)*: +1-855-552-4463,,,9272158344#
> International dial-in: https://chime.aws/dialinnumbers/
> In-room video system: Ext: 62000, Meeting PIN: 9272158344#
>
> --
> *Chaitanya Prakash Bapat*
> *+1 (973) 953-6299*
>
> [image: https://www.linkedin.com//in/chaibapat25]
> <https://github.com/ChaiBapchya>[image: https://www.facebook.com/chaibapat
> ]
> <https://www.facebook.com/chaibapchya>[image:
> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
> https://www.linkedin.com//in/chaibapat25]
> <https://www.linkedin.com//in/chaibapchya/>
>