[DISCUSS] Automatic Dependency Update PRs

[Ismaël Mejía <ie...@gmail.com>]

Hi everyone,

Github has a bot to create Dependency Update PRs and report security issues
called dependabot. I requested INFRA to enable it for Avro so we can benefit of
more automation. I am enthusiastic in particular about the multiple language
support (so far we can get automatic updates for Java/C#/Python/Ruby/Js. For an
example of what it does in practice you can look at the PRs it created
automatically on my personal fork of Avro.
https://github.com/iemejia/avro/pulls?q=is%3Apr+is%3Aclosed

We might be getting extra PRs (lots at the beginning) and we have to be cautious
about updates that might have unintended consequences for example we should not
merge non stable dependency updates (those ending on -rc1 or -beta on Java) that
might be proposed or dependencies that committers are aware we
should not update to for example there are projects that their main stable
version is not the most recent one like Hadoop or dependencies that do not
support our ongoing language target version (e.g. Java 11 only deps).

Another issue is that these updates might not get a JIRA associated with it so
we need to decide if (1) we create one and rename/associate the PR with it, or
(2) we just decide not to have JIRAs for dependency updates. I am in the (1)
camp but I also can see that it is a lot of extra work for not much in return
apart of the nice looking JIRA release notes.

Any other issues I might be missing? Other comments?

Re: [DISCUSS] Automatic Dependency Update PRs

[Ismaël Mejía <ie...@gmail.com>]

Dependabot is enabled now, time to validate all these upgrades.
The more eyes we can have the better !
Notice that these upgrades won't be taken for the ongoing release (1.10.1)
but probably catched up and eventually backported for future ones.

On Fri, Nov 13, 2020 at 3:12 PM Ismaël Mejía <ie...@gmail.com> wrote:
>
> Yes Michael you are right. Things have evolved. There were two open
> issues at the time:
>
> 1. Support from Infra
> Infra did not allow this because of strict requirements on github
> extensions NOT having write permissions on the repo. This has been
> fixed by them and dependabot now is even used by other Apache
> projects.
>
> 2. The question of authorship (do bots have to sign a CLA?)
> I opened a question on this on the private Apache members list and the
> consensus was that since the bot is not committing the code the
> responsible of the 'authorship' would be the committer since we
> already set up the bot and the example given was that this is like
> having a script to generate code, so only the person who commits the
> code is responsible.
>
> So both are covered now.
>
> On Fri, Nov 13, 2020 at 2:31 PM Michael A. Smith <mi...@smith-li.com> wrote:
> >
> > There was a thread on this list in May 2019 headed "Automate python
> > formatting" that touched on dependabot. At the time, Fokko, and you,
> > Ismaël, were discussing that dependabot might violate Apache rules about
> > modifying the code. Has that been worked out?
> >
> > I'm otherwise totally in favor of this.
> >
> > On Fri, Nov 13, 2020 at 04:36 Ismaël Mejía <ie...@gmail.com> wrote:
> >
> > > Hi everyone,
> > >
> > > Github has a bot to create Dependency Update PRs and report security issues
> > > called dependabot. I requested INFRA to enable it for Avro so we can
> > > benefit of
> > > more automation. I am enthusiastic in particular about the multiple
> > > language
> > > support (so far we can get automatic updates for Java/C#/Python/Ruby/Js.
> > > For an
> > > example of what it does in practice you can look at the PRs it created
> > > automatically on my personal fork of Avro.
> > > https://github.com/iemejia/avro/pulls?q=is%3Apr+is%3Aclosed
> > >
> > > We might be getting extra PRs (lots at the beginning) and we have to be
> > > cautious
> > > about updates that might have unintended consequences for example we
> > > should not
> > > merge non stable dependency updates (those ending on -rc1 or -beta on
> > > Java) that
> > > might be proposed or dependencies that committers are aware we
> > > should not update to for example there are projects that their main stable
> > > version is not the most recent one like Hadoop or dependencies that do not
> > > support our ongoing language target version (e.g. Java 11 only deps).
> > >
> > > Another issue is that these updates might not get a JIRA associated with
> > > it so
> > > we need to decide if (1) we create one and rename/associate the PR with
> > > it, or
> > > (2) we just decide not to have JIRAs for dependency updates. I am in the
> > > (1)
> > > camp but I also can see that it is a lot of extra work for not much in
> > > return
> > > apart of the nice looking JIRA release notes.
> > >
> > > Any other issues I might be missing? Other comments?
> > >

Re: [DISCUSS] Automatic Dependency Update PRs

[Ismaël Mejía <ie...@gmail.com>]

Yes Michael you are right. Things have evolved. There were two open
issues at the time:

1. Support from Infra
Infra did not allow this because of strict requirements on github
extensions NOT having write permissions on the repo. This has been
fixed by them and dependabot now is even used by other Apache
projects.

2. The question of authorship (do bots have to sign a CLA?)
I opened a question on this on the private Apache members list and the
consensus was that since the bot is not committing the code the
responsible of the 'authorship' would be the committer since we
already set up the bot and the example given was that this is like
having a script to generate code, so only the person who commits the
code is responsible.

So both are covered now.

On Fri, Nov 13, 2020 at 2:31 PM Michael A. Smith <mi...@smith-li.com> wrote:
>
> There was a thread on this list in May 2019 headed "Automate python
> formatting" that touched on dependabot. At the time, Fokko, and you,
> Ismaël, were discussing that dependabot might violate Apache rules about
> modifying the code. Has that been worked out?
>
> I'm otherwise totally in favor of this.
>
> On Fri, Nov 13, 2020 at 04:36 Ismaël Mejía <ie...@gmail.com> wrote:
>
> > Hi everyone,
> >
> > Github has a bot to create Dependency Update PRs and report security issues
> > called dependabot. I requested INFRA to enable it for Avro so we can
> > benefit of
> > more automation. I am enthusiastic in particular about the multiple
> > language
> > support (so far we can get automatic updates for Java/C#/Python/Ruby/Js.
> > For an
> > example of what it does in practice you can look at the PRs it created
> > automatically on my personal fork of Avro.
> > https://github.com/iemejia/avro/pulls?q=is%3Apr+is%3Aclosed
> >
> > We might be getting extra PRs (lots at the beginning) and we have to be
> > cautious
> > about updates that might have unintended consequences for example we
> > should not
> > merge non stable dependency updates (those ending on -rc1 or -beta on
> > Java) that
> > might be proposed or dependencies that committers are aware we
> > should not update to for example there are projects that their main stable
> > version is not the most recent one like Hadoop or dependencies that do not
> > support our ongoing language target version (e.g. Java 11 only deps).
> >
> > Another issue is that these updates might not get a JIRA associated with
> > it so
> > we need to decide if (1) we create one and rename/associate the PR with
> > it, or
> > (2) we just decide not to have JIRAs for dependency updates. I am in the
> > (1)
> > camp but I also can see that it is a lot of extra work for not much in
> > return
> > apart of the nice looking JIRA release notes.
> >
> > Any other issues I might be missing? Other comments?
> >

Re: [DISCUSS] Automatic Dependency Update PRs

["Michael A. Smith" <mi...@smith-li.com>]

There was a thread on this list in May 2019 headed "Automate python
formatting" that touched on dependabot. At the time, Fokko, and you,
Ismaël, were discussing that dependabot might violate Apache rules about
modifying the code. Has that been worked out?

I'm otherwise totally in favor of this.

On Fri, Nov 13, 2020 at 04:36 Ismaël Mejía <ie...@gmail.com> wrote:

> Hi everyone,
>
> Github has a bot to create Dependency Update PRs and report security issues
> called dependabot. I requested INFRA to enable it for Avro so we can
> benefit of
> more automation. I am enthusiastic in particular about the multiple
> language
> support (so far we can get automatic updates for Java/C#/Python/Ruby/Js.
> For an
> example of what it does in practice you can look at the PRs it created
> automatically on my personal fork of Avro.
> https://github.com/iemejia/avro/pulls?q=is%3Apr+is%3Aclosed
>
> We might be getting extra PRs (lots at the beginning) and we have to be
> cautious
> about updates that might have unintended consequences for example we
> should not
> merge non stable dependency updates (those ending on -rc1 or -beta on
> Java) that
> might be proposed or dependencies that committers are aware we
> should not update to for example there are projects that their main stable
> version is not the most recent one like Hadoop or dependencies that do not
> support our ongoing language target version (e.g. Java 11 only deps).
>
> Another issue is that these updates might not get a JIRA associated with
> it so
> we need to decide if (1) we create one and rename/associate the PR with
> it, or
> (2) we just decide not to have JIRAs for dependency updates. I am in the
> (1)
> camp but I also can see that it is a lot of extra work for not much in
> return
> apart of the nice looking JIRA release notes.
>
> Any other issues I might be missing? Other comments?
>

Re: [DISCUSS] Automatic Dependency Update PRs

[Ryan Skraba <ry...@skraba.com>]

I like dependabot, and it takes out a lot of manual work -- the automatic
PRs are really quite nice, with all of the readily available links.

I don't really need or want to see one-liner JIRAs per minor bump,
especially if it's a manual task -- I don't think it adds much value, even
for automatically generating release notes!  I'd rather write this part of
the release notes manually by watching the git log.

On the other hand, if there is additional context (like protobuf this
time), or if the version bump requires discussion, or additional changes,
yeah, that's appropriate for a new JIRA!

Thanks for taking this task!  Ryan



On Fri, Nov 13, 2020 at 10:36 AM Ismaël Mejía <ie...@gmail.com> wrote:

> Hi everyone,
>
> Github has a bot to create Dependency Update PRs and report security issues
> called dependabot. I requested INFRA to enable it for Avro so we can
> benefit of
> more automation. I am enthusiastic in particular about the multiple
> language
> support (so far we can get automatic updates for Java/C#/Python/Ruby/Js.
> For an
> example of what it does in practice you can look at the PRs it created
> automatically on my personal fork of Avro.
> https://github.com/iemejia/avro/pulls?q=is%3Apr+is%3Aclosed
>
> We might be getting extra PRs (lots at the beginning) and we have to be
> cautious
> about updates that might have unintended consequences for example we
> should not
> merge non stable dependency updates (those ending on -rc1 or -beta on
> Java) that
> might be proposed or dependencies that committers are aware we
> should not update to for example there are projects that their main stable
> version is not the most recent one like Hadoop or dependencies that do not
> support our ongoing language target version (e.g. Java 11 only deps).
>
> Another issue is that these updates might not get a JIRA associated with
> it so
> we need to decide if (1) we create one and rename/associate the PR with
> it, or
> (2) we just decide not to have JIRAs for dependency updates. I am in the
> (1)
> camp but I also can see that it is a lot of extra work for not much in
> return
> apart of the nice looking JIRA release notes.
>
> Any other issues I might be missing? Other comments?
>