Pull Request review bot

[Marco de Abreu <ma...@apache.org>]

Hello dev@,

at the moment, the user experience when submitting a Pull Request has quite
a few issues and discourages people from further contributions. To improve
the situation, I made a design for a Pull Request review bot that will
introduce a process for PRs and also assist maintaining it. I'd appreciate
it if everyone could review my design [1] and give some feedback.

Thanks in advance!

Best regards,
Marco

[1]:
https://cwiki.apache.org/confluence/display/MXNET/Pull+Request+review+bot

Re: Pull Request review bot

[Aaron Markham <aa...@gmail.com>]

I like this idea. Some thoughts:

* Disable CI checks when a PR is first submitted. Trigger it only when the
PR is flagged for that stage of review.
* I have WIP PRs that waste CI resources because of the automatic trigger,
but then I need CI to run on them sometimes. Changing the title could be a
trigger too... Like removing [WIP] or "NO CI". Something like this that any
contributor can do. The bot can tell you what to add or remove to change
your pr state.
* A variation of this is to have it only run sanity checks until the PR has
moved to it's final phase.
* Use GitHub API in CI to check the labels and/or the configs in your
proposal to route the kind of checks that happen.
* Labels could be the main functional operators so that committers can
nudge things along without the config files and the bot being the only way.
* I report a lot of broken tests. The bot could find and report these
faster. If x number of PRs fail the same test there could be an alarm.
* I'd like it if when something gets fixed, if CI was retriggered
automatically. When I have a lot of PRs I have to spend a lot of time
shepherding them through CI landmines of flaky tests. And I usually don't
know if things are better or not. Nudge and pray is wasteful of time and
resources.
* At a bare minimum, I really like the ideas around helping people know
what's next and moving things along.
* Leaderboards are cool too. Sure, they can be manipulated and create their
own problems, but I think we should start somewhere with incentives and
recognition
* The metrics that can come out this would be insightful

On Fri, Aug 23, 2019, 22:39 Marco de Abreu <ma...@apache.org> wrote:

> Hello dev@,
>
> at the moment, the user experience when submitting a Pull Request has quite
> a few issues and discourages people from further contributions. To improve
> the situation, I made a design for a Pull Request review bot that will
> introduce a process for PRs and also assist maintaining it. I'd appreciate
> it if everyone could review my design [1] and give some feedback.
>
> Thanks in advance!
>
> Best regards,
> Marco
>
> [1]:
> https://cwiki.apache.org/confluence/display/MXNET/Pull+Request+review+bot
>