Using a bot to keep dependencies up to date

[Florian Hockmann <fh...@florian-hockmann.de>]

Hi,

we have a lot of dependencies in TinkerPop in different projects and
even across different languages. That makes it hard to keep them updated
which sometimes has security implications.

I recently noticed that other open source projects use a bot that
regularly checks whether any updates are available for their
dependencies and then creates one PR per dependency. Just to try it out
with TinkerPop, I activated such a bot on my fork:

https://github.com/florianhockmann/tinkerpop/pulls

and the overall result looks quite good in my opinion. It created a lot
of PRs* and most could probably be directly merged. The bot can also be
easily configured just by adding comments to its PR, for example to
ignore a certain (major/minor/patch) version of a dependency:

https://github.com/FlorianHockmann/tinkerpop/pull/24#issuecomment-473936360

What do you think about adding such a bot for our repo?


* This is limited to only 5 PRs per day at first to not overwhelm a
project with PRs.

Re: Using a bot to keep dependencies up to date

[Stephen Mallette <sp...@gmail.com>]

Thanks again for investigating.

> I can try that out and if it's actually helpful add something to our
> development docs. Ideally we have simple commands to run for all GLVs to
> keep them updated in the future.

+1 for this approach for sure though :)

On Fri, Apr 26, 2019 at 11:49 AM Florian Hockmann <fh...@florian-hockmann.de>
wrote:

> I just reached out to ASF Infra and that basically concluded this
> discussion as they simply don't allow any GitHub bots that require write
> access to the repo which includes dependabot unfortunately.
>
> We could only enable it on a fork and then forward the PRs from there to
> the main repo which we could also automate in principal, but at least
> for now this sounds like too much work.
>
> So, I guess we have to find other ways to keep the dependencies up to
> date, especially for the GLVs.
>
> At least for .NET, I know that there is a nice tool we could use that
> can list outdated dependencies and it should also be able to perform an
> update: https://github.com/jerriep/dotnet-outdated
>
> I can try that out and if it's actually helpful add something to our
> development docs. Ideally we have simple commands to run for all GLVs to
> keep them updated in the future.
>
> Am 16.04.2019 um 13:39 schrieb Stephen Mallette:
> >>  We can define the target branch for the PRs. So, we could set that to
> > tp33 and then merge though to master like we usually do it for other PRs.
> > (Maybe it's also possible to create different configurations for the
> > different branches if we want to get one PR per branch.)
> >
> > There are different dependencies on tp33 vs master so it would be nice to
> > know for sure how they handle that. Of course, you allude to a
> > proliferation of PRs that would result....ugh. Anyway, I guess I'd like
> to
> > know what the options really are there...
> >
> >> I assume that Apacha Infra will have to activate GitHub bots like
> > dependabot for us as that requires ownership of the GitHub organization.
> I
> > hope that they can afterwards give us permissions to change the settings,
> > but I'm not sure about that. Maybe these permissions also only work on an
> > organization basis (which would include all ASF repos). Worst case would
> be
> > that ASF Infra has to configure such a bot for us.
> >
> > all good questions to answer. no one seem to really be objecting to this
> > direction (i only want to know more about the details of how it will
> work)
> > so I guess we're at a point where you could try to move things forward
> with
> > infra and the bot folks to see how these final issues we're discussing
> > would work out. please let us know what you find out. thanks.
> >
> > On Wed, Apr 10, 2019 at 9:52 AM Florian Hockmann <fh@florian-hockmann.de
> >
> > wrote:
> >
> >>> always using the most recent has been disastrous in python. our build
> >> breaks all the time with no changes from us because of that style where
> we
> >> don't pin to specific dependencies. i don't understand that model at
> all.
> >> i
> >> know you're not saying that we blindly upgrade, i was just making a
> point
> >> about python that is semi-related.
> >>
> >> Understood and I agree completely that we should pin versions.
> >>
> >>> we're not always sure of what a change in dependency will bring in
> terms
> >> of change to the API but i agree that upgrades can take place (and as
> you
> >> pointed out at the start, are already taking place in a more manual
> >> fashion).
> >>
> >> IF our dependencies use semantic versioning, then this shouldn't be a
> >> problem as long as we don't do major version updates in our patch
> >> versions.
> >>
> >>> Why no PR for spark, hadoop, etc? not that they would pass compilation
> -
> >> i'd expect failures, but I'm just wondering if you knew the reason it
> >> didn't catch those?
> >>
> >> I also wondered about that as I had exactly the same expectation of
> >> immediately getting a failed build for these dependencies. Maybe they
> have
> >> a list of dependencies where they don't attempt an update? We can of
> >> course
> >> ask them about this if we decide to use this bot.
> >>
> >>> it looks like it submits all PRs against master. what about tp33? will
> >> we have to cherry-pick out of the PR to tp33, test and then merge
> forward
> >> to master?
> >>
> >> We can define the target branch for the PRs. So, we could set that to
> tp33
> >> and then merge though to master like we usually do it for other PRs.
> >> (Maybe
> >> it's also possible to create different configurations for the different
> >> branches if we want to get one PR per branch.)
> >>
> >>> i assume that you setup an account for dependabot that gets it
> >> configured at our repo? I assume that the account is bound to your
> github
> >> account?
> >>
> >> Yes, I enabled dependabot for my account and could then give it access
> to
> >> my own repos which is a fork of our TinkerPop repo in this case.
> >>
> >>> can that dependabot dashboard for "tinkerpop" be accessed by others or
> >> just your account? how will that work?
> >>
> >> I assume that Apacha Infra will have to activate GitHub bots like
> >> dependabot for us as that requires ownership of the GitHub
> organization. I
> >> hope that they can afterwards give us permissions to change the
> settings,
> >> but I'm not sure about that. Maybe these permissions also only work on
> an
> >> organization basis (which would include all ASF repos). Worst case
> would
> >> be
> >> that ASF Infra has to configure such a bot for us.
> >>
> >>> i don't see much information about what the dashboard allows users to
> do
> >> - could you summarize some of the options available and what we see
> there?
> >>
> >> It allows to add different configurations for different languages or
> >> directories in a repo. We can then configure the following aspects per
> >> directory / language:
> >>
> >> Update schedule: daily / weekly / monthly
> >> Directory (important for the GLVs as that directory should contain the
> >> project file, like the .sln file for .NET)
> >> Target branch
> >> Filters: They currently only have a filter to only get security updates.
> >> Automatic PR merging: Can be configured for only some versions (like
> patch
> >> versions, only security fixes, ...) and only for specific dependencies.
> (I
> >> don't think that we want automatic merging.)
> >> For the GitHub PRs, we can also add default reviewers, assignees, and
> >> labels.
> >>
> >> On top of that, the UI also allows to manually trigger an update run,
> >> irrespective of the configured update schedule. I used that on the first
> >> day as dependabot by default only creates 5 PRs per directory/language
> and
> >> per update run, but I wanted to get more PRs without having to wait
> >> several
> >> days.
> >>
> >> -----Ursprüngliche Nachricht-----
> >> Von: Stephen Mallette <sp...@gmail.com>
> >> Gesendet: Mittwoch, 10. April 2019 14:41
> >> An: dev@tinkerpop.apache.org
> >> Betreff: Re: Using a bot to keep dependencies up to date
> >>
> >>> I'm still not that experienced with Java/Maven, but the reasoning is
> >>> that we want to avoid dependency version conflicts for our users,
> right?
> >>>
> >> well, we just try to keep things stable as possible without introducing
> a
> >> breaking change by way of a dependency. it's not to say that we never do
> >> it, but it's always with done with careful consideration.
> >>
> >>
> >>> In that case, it's specific to Java I'd say. At least for .NET, I
> >>> would expect a library to use recent versions of its dependencies.
> >>>
> >> always using the most recent has been disastrous in python. our build
> >> breaks all the time with no changes from us because of that style where
> we
> >> don't pin to specific dependencies. i don't understand that model at
> all.
> >> i
> >> know you're not saying that we blindly upgrade, i was just making a
> point
> >> about python that is semi-related.
> >>
> >>
> >>> Since we add new features in our "patch versions" (we aren't doing
> >>> semantic versioning so those aren't actually patch versions), I think
> >>> it's also OK in general to update dependencies in those versions, at
> >>> least to a higher patch version, but I think a higher minor version
> >>> should also be OK if we also add features ourselves in those versions.
> >>>
> >> We allow new features into patch versions because we control the new
> >> features and make choices as to whether they are "safe" or not. we're
> not
> >> always sure of what a change in dependency will bring in terms of
> change
> >> to
> >> the API but i agree that upgrades can take place (and as you pointed
> out
> >> at
> >> the start, are already taking place in a more manual fashion). Again, I
> >> just feel like we need to take each upgrade on a case by case basis.
> >>
> >>
> >>>> We also typically have JIRAs for version bumps of dependencies - not
> >>> always plugins - so if we continued with that those would be need to
> >>> be created and CHANGELOG entries added manually after a merge i guess.
> >>>
> >>> Do we need a JIRA + CHANGELOG entry for all dependency updates or only
> >>> for important ones?
> >>> If we need them, then the bot at least notifies us of an available
> >>> update and provides a PR for the update itself. We can then still
> >>> create a JIRA issue + CHANGELOG entry manually if necessary.
> >>> The notification should also include fixes for security problems in
> >>> which case the bot sometimes also includes this information directly in
> >> the PR.
> >> maybe we just need it for "important" ones. i guess that's how we've
> been
> >> doing it so far.......
> >>
> >>
> >>>> kuppitz makes the good point of the mess Travis has been in lately.
> >>> Is that a reason against using a bot for automation or am I
> >>> misunderstanding your/Daniel's point here? If a Travis build fails
> >>> non-deterministically for a PR created by a bot, then I would simply
> >>> trigger a rebuild manually.
> >>>
> >> i guess that's just us railing about travis........not a reason to avoid
> >> automation.
> >>
> >>
> >>> We could also use such a bot only for the GLVs / some of the GLVs if
> >>> we don't want to update dependencies in general outside of major
> >>> releases (like 3.5.0). At least for .NET I would appreciate such a
> >>> bot, but I can't say much for Python / JavaScript as I don't know
> >>> enough about how they handle dependency updates.
> >>
> >> if we use this, i think we should use it across the project. i looked at
> >> the list of PRs that were generated by it in your fork more carefully
> and
> >> it was curious that the larger version bumps weren't present. Why no PR
> >> for
> >> spark, hadoop, etc? not that they would pass compilation - i'd expect
> >> failures, but I'm just wondering if you knew the reason it didn't catch
> >> those?
> >>
> >> some more questions:
> >>
> >> 1. I see the list of commands that dependabot has - it looks like it
> >> submits all PRs against master. what about tp33? will we have to
> >> cherry-pick out of the PR to tp33, test and then merge forward to
> master?
> >> or is there something easier?
> >> 2. i assume that you setup an account for dependabot that gets it
> >> configured at our repo? I assume that the account is bound to your
> github
> >> account? can that dependabot dashboard for "tinkerpop" be accessed by
> >> others or just your account? how will that work?
> >> 3. i don't see much information about what the dashboard allows users
> to
> >> do
> >> - could you summarize some of the options available and what we see
> there?
> >>
> >> thanks!
> >>
> >>
> >> On Wed, Apr 10, 2019 at 7:59 AM Florian Hockmann <
> fh@florian-hockmann.de>
> >> wrote:
> >>
> >>>> we don't usually upgrade them in a release line unless
> >>> I'm still not that experienced with Java/Maven, but the reasoning is
> >>> that we want to avoid dependency version conflicts for our users,
> right?
> >>>
> >>> In that case, it's specific to Java I'd say. At least for .NET, I
> >>> would expect a library to use recent versions of its dependencies.
> >>>
> >>> Since we add new features in our "patch versions" (we aren't doing
> >>> semantic versioning so those aren't actually patch versions), I think
> >>> it's also OK in general to update dependencies in those versions, at
> >>> least to a higher patch version, but I think a higher minor version
> >>> should also be OK if we also add features ourselves in those versions.
> >>>
> >>>> We also typically have JIRAs for version bumps of dependencies - not
> >>> always plugins - so if we continued with that those would be need to
> >>> be created and CHANGELOG entries added manually after a merge i guess.
> >>>
> >>> Do we need a JIRA + CHANGELOG entry for all dependency updates or only
> >>> for important ones?
> >>> If we need them, then the bot at least notifies us of an available
> >>> update and provides a PR for the update itself. We can then still
> >>> create a JIRA issue + CHANGELOG entry manually if necessary.
> >>> The notification should also include fixes for security problems in
> >>> which case the bot sometimes also includes this information directly in
> >> the PR.
> >>>> kuppitz makes the good point of the mess Travis has been in lately.
> >>> Is that a reason against using a bot for automation or am I
> >>> misunderstanding your/Daniel's point here? If a Travis build fails
> >>> non-deterministically for a PR created by a bot, then I would simply
> >>> trigger a rebuild manually.
> >>>
> >>> We could also use such a bot only for the GLVs / some of the GLVs if
> >>> we don't want to update dependencies in general outside of major
> >>> releases (like 3.5.0). At least for .NET I would appreciate such a
> >>> bot, but I can't say much for Python / JavaScript as I don't know
> >>> enough about how they handle dependency updates.
> >>>
> >>> -----Ursprüngliche Nachricht-----
> >>> Von: Stephen Mallette <sp...@gmail.com>
> >>> Gesendet: Mittwoch, 3. April 2019 20:27
> >>> An: dev@tinkerpop.apache.org
> >>> Betreff: Re: Using a bot to keep dependencies up to date
> >>>
> >>> hmm - so far i've been the bot. i run maven commands to find out what
> >>> needs upgrading and then make decisions on when to do those upgrades
> >>> as we don't usually upgrade them in a release line unless:
> >>>
> >>> 1. they are maven plugins
> >>> 2. they are fixes for security problems 3. they introduce something
> >>> important for our code base - like a performance fix or other
> >>> enhancements
> >>>
> >>> That said, my maven commands only deal with Java and not the GLVs so
> >>> not much attention has been paid there unfortunately now that I think
> >> about it.
> >>> We also typically have JIRAs for version bumps of dependencies - not
> >>> always plugins - so if we continued with that those would be need to
> >>> be created and CHANGELOG entries added manually after a merge i guess.
> >>>
> >>> I'm not completely against the idea, just pointing out some points of
> >>> friction with what we currently do and kuppitz makes the good point of
> >>> the mess Travis has been in lately. Utterly useless the last few
> >>> weeks. If we didn't use a bot, at minimum it would be nice to document
> >>> the manual methods for getting reports for the GLVs that show upgrade
> >>> paths (I assume that those exist for other language ecosystems as they
> >>> do for maven). it would be nice to hear additional thoughts on the
> >> matter.
> >>> Florian, thanks for taking the time to look into improving our build
> >>> process.
> >>>
> >>>
> >>>
> >>> On Wed, Apr 3, 2019 at 1:29 PM Daniel Kuppitz <me...@gremlin.guru> wrote:
> >>>
> >>>> Pretty cool, I like that (if only Travis would be a little more
> >>> reliable).
> >>>> Cheers,
> >>>> Daniel
> >>>>
> >>>>
> >>>> On Wed, Apr 3, 2019 at 9:43 AM Florian Hockmann
> >>>> <fh...@florian-hockmann.de>
> >>>> wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> we have a lot of dependencies in TinkerPop in different projects
> >>>>> and even across different languages. That makes it hard to keep
> >>>>> them updated which sometimes has security implications.
> >>>>>
> >>>>> I recently noticed that other open source projects use a bot that
> >>>>> regularly checks whether any updates are available for their
> >>>>> dependencies and then creates one PR per dependency. Just to try
> >>>>> it out with TinkerPop, I activated such a bot on my fork:
> >>>>>
> >>>>> https://github.com/florianhockmann/tinkerpop/pulls
> >>>>>
> >>>>> and the overall result looks quite good in my opinion. It created
> >>>>> a lot of PRs* and most could probably be directly merged. The bot
> >>>>> can also be easily configured just by adding comments to its PR,
> >>>>> for example to ignore a certain (major/minor/patch) version of a
> >>> dependency:
> >>>>>
> >>>> https://github.com/FlorianHockmann/tinkerpop/pull/24#issuecomment-47
> >>>> 39
> >>>> 36360
> >>>>> What do you think about adding such a bot for our repo?
> >>>>>
> >>>>>
> >>>>> * This is limited to only 5 PRs per day at first to not overwhelm
> >>>>> a project with PRs.
> >>>>>
> >>>>>
> >>>>>
> >>>
> >>
>
>

Re: Using a bot to keep dependencies up to date

[Florian Hockmann <fh...@florian-hockmann.de>]

I just reached out to ASF Infra and that basically concluded this
discussion as they simply don't allow any GitHub bots that require write
access to the repo which includes dependabot unfortunately.

We could only enable it on a fork and then forward the PRs from there to
the main repo which we could also automate in principal, but at least
for now this sounds like too much work.

So, I guess we have to find other ways to keep the dependencies up to
date, especially for the GLVs.

At least for .NET, I know that there is a nice tool we could use that
can list outdated dependencies and it should also be able to perform an
update: https://github.com/jerriep/dotnet-outdated

I can try that out and if it's actually helpful add something to our
development docs. Ideally we have simple commands to run for all GLVs to
keep them updated in the future.

Am 16.04.2019 um 13:39 schrieb Stephen Mallette:
>>  We can define the target branch for the PRs. So, we could set that to
> tp33 and then merge though to master like we usually do it for other PRs.
> (Maybe it's also possible to create different configurations for the
> different branches if we want to get one PR per branch.)
>
> There are different dependencies on tp33 vs master so it would be nice to
> know for sure how they handle that. Of course, you allude to a
> proliferation of PRs that would result....ugh. Anyway, I guess I'd like to
> know what the options really are there...
>
>> I assume that Apacha Infra will have to activate GitHub bots like
> dependabot for us as that requires ownership of the GitHub organization. I
> hope that they can afterwards give us permissions to change the settings,
> but I'm not sure about that. Maybe these permissions also only work on an
> organization basis (which would include all ASF repos). Worst case would be
> that ASF Infra has to configure such a bot for us.
>
> all good questions to answer. no one seem to really be objecting to this
> direction (i only want to know more about the details of how it will work)
> so I guess we're at a point where you could try to move things forward with
> infra and the bot folks to see how these final issues we're discussing
> would work out. please let us know what you find out. thanks.
>
> On Wed, Apr 10, 2019 at 9:52 AM Florian Hockmann <fh...@florian-hockmann.de>
> wrote:
>
>>> always using the most recent has been disastrous in python. our build
>> breaks all the time with no changes from us because of that style where we
>> don't pin to specific dependencies. i don't understand that model at all. 
>> i
>> know you're not saying that we blindly upgrade, i was just making a point
>> about python that is semi-related.
>>
>> Understood and I agree completely that we should pin versions.
>>
>>> we're not always sure of what a change in dependency will bring in terms
>> of change to the API but i agree that upgrades can take place (and as you
>> pointed out at the start, are already taking place in a more manual
>> fashion).
>>
>> IF our dependencies use semantic versioning, then this shouldn't be a
>> problem as long as we don't do major version updates in our patch 
>> versions.
>>
>>> Why no PR for spark, hadoop, etc? not that they would pass compilation -
>> i'd expect failures, but I'm just wondering if you knew the reason it
>> didn't catch those?
>>
>> I also wondered about that as I had exactly the same expectation of
>> immediately getting a failed build for these dependencies. Maybe they have
>> a list of dependencies where they don't attempt an update? We can of 
>> course
>> ask them about this if we decide to use this bot.
>>
>>> it looks like it submits all PRs against master. what about tp33? will
>> we have to cherry-pick out of the PR to tp33, test and then merge forward
>> to master?
>>
>> We can define the target branch for the PRs. So, we could set that to tp33
>> and then merge though to master like we usually do it for other PRs. 
>> (Maybe
>> it's also possible to create different configurations for the different
>> branches if we want to get one PR per branch.)
>>
>>> i assume that you setup an account for dependabot that gets it
>> configured at our repo? I assume that the account is bound to your github
>> account?
>>
>> Yes, I enabled dependabot for my account and could then give it access to
>> my own repos which is a fork of our TinkerPop repo in this case.
>>
>>> can that dependabot dashboard for "tinkerpop" be accessed by others or
>> just your account? how will that work?
>>
>> I assume that Apacha Infra will have to activate GitHub bots like
>> dependabot for us as that requires ownership of the GitHub organization. I
>> hope that they can afterwards give us permissions to change the settings,
>> but I'm not sure about that. Maybe these permissions also only work on an
>> organization basis (which would include all ASF repos). Worst case would 
>> be
>> that ASF Infra has to configure such a bot for us.
>>
>>> i don't see much information about what the dashboard allows users to do
>> - could you summarize some of the options available and what we see there?
>>
>> It allows to add different configurations for different languages or
>> directories in a repo. We can then configure the following aspects per
>> directory / language:
>>
>> Update schedule: daily / weekly / monthly
>> Directory (important for the GLVs as that directory should contain the
>> project file, like the .sln file for .NET)
>> Target branch
>> Filters: They currently only have a filter to only get security updates.
>> Automatic PR merging: Can be configured for only some versions (like patch
>> versions, only security fixes, ...) and only for specific dependencies. (I
>> don't think that we want automatic merging.)
>> For the GitHub PRs, we can also add default reviewers, assignees, and
>> labels.
>>
>> On top of that, the UI also allows to manually trigger an update run,
>> irrespective of the configured update schedule. I used that on the first
>> day as dependabot by default only creates 5 PRs per directory/language and
>> per update run, but I wanted to get more PRs without having to wait 
>> several
>> days.
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Stephen Mallette <sp...@gmail.com>
>> Gesendet: Mittwoch, 10. April 2019 14:41
>> An: dev@tinkerpop.apache.org
>> Betreff: Re: Using a bot to keep dependencies up to date
>>
>>> I'm still not that experienced with Java/Maven, but the reasoning is
>>> that we want to avoid dependency version conflicts for our users, right?
>>>
>> well, we just try to keep things stable as possible without introducing a
>> breaking change by way of a dependency. it's not to say that we never do
>> it, but it's always with done with careful consideration.
>>
>>
>>> In that case, it's specific to Java I'd say. At least for .NET, I
>>> would expect a library to use recent versions of its dependencies.
>>>
>> always using the most recent has been disastrous in python. our build
>> breaks all the time with no changes from us because of that style where we
>> don't pin to specific dependencies. i don't understand that model at all. 
>> i
>> know you're not saying that we blindly upgrade, i was just making a point
>> about python that is semi-related.
>>
>>
>>> Since we add new features in our "patch versions" (we aren't doing
>>> semantic versioning so those aren't actually patch versions), I think
>>> it's also OK in general to update dependencies in those versions, at
>>> least to a higher patch version, but I think a higher minor version
>>> should also be OK if we also add features ourselves in those versions.
>>>
>> We allow new features into patch versions because we control the new
>> features and make choices as to whether they are "safe" or not. we're not
>> always sure of what a change in dependency will bring in terms of change 
>> to
>> the API but i agree that upgrades can take place (and as you pointed out 
>> at
>> the start, are already taking place in a more manual fashion). Again, I
>> just feel like we need to take each upgrade on a case by case basis.
>>
>>
>>>> We also typically have JIRAs for version bumps of dependencies - not
>>> always plugins - so if we continued with that those would be need to
>>> be created and CHANGELOG entries added manually after a merge i guess.
>>>
>>> Do we need a JIRA + CHANGELOG entry for all dependency updates or only
>>> for important ones?
>>> If we need them, then the bot at least notifies us of an available
>>> update and provides a PR for the update itself. We can then still
>>> create a JIRA issue + CHANGELOG entry manually if necessary.
>>> The notification should also include fixes for security problems in
>>> which case the bot sometimes also includes this information directly in
>> the PR.
>> maybe we just need it for "important" ones. i guess that's how we've been
>> doing it so far.......
>>
>>
>>>> kuppitz makes the good point of the mess Travis has been in lately.
>>> Is that a reason against using a bot for automation or am I
>>> misunderstanding your/Daniel's point here? If a Travis build fails
>>> non-deterministically for a PR created by a bot, then I would simply
>>> trigger a rebuild manually.
>>>
>> i guess that's just us railing about travis........not a reason to avoid
>> automation.
>>
>>
>>> We could also use such a bot only for the GLVs / some of the GLVs if
>>> we don't want to update dependencies in general outside of major
>>> releases (like 3.5.0). At least for .NET I would appreciate such a
>>> bot, but I can't say much for Python / JavaScript as I don't know
>>> enough about how they handle dependency updates.
>>
>> if we use this, i think we should use it across the project. i looked at
>> the list of PRs that were generated by it in your fork more carefully and
>> it was curious that the larger version bumps weren't present. Why no PR 
>> for
>> spark, hadoop, etc? not that they would pass compilation - i'd expect
>> failures, but I'm just wondering if you knew the reason it didn't catch
>> those?
>>
>> some more questions:
>>
>> 1. I see the list of commands that dependabot has - it looks like it
>> submits all PRs against master. what about tp33? will we have to
>> cherry-pick out of the PR to tp33, test and then merge forward to master?
>> or is there something easier?
>> 2. i assume that you setup an account for dependabot that gets it
>> configured at our repo? I assume that the account is bound to your github
>> account? can that dependabot dashboard for "tinkerpop" be accessed by
>> others or just your account? how will that work?
>> 3. i don't see much information about what the dashboard allows users to 
>> do
>> - could you summarize some of the options available and what we see there?
>>
>> thanks!
>>
>>
>> On Wed, Apr 10, 2019 at 7:59 AM Florian Hockmann <fh...@florian-hockmann.de>
>> wrote:
>>
>>>> we don't usually upgrade them in a release line unless
>>> I'm still not that experienced with Java/Maven, but the reasoning is
>>> that we want to avoid dependency version conflicts for our users, right?
>>>
>>> In that case, it's specific to Java I'd say. At least for .NET, I
>>> would expect a library to use recent versions of its dependencies.
>>>
>>> Since we add new features in our "patch versions" (we aren't doing
>>> semantic versioning so those aren't actually patch versions), I think
>>> it's also OK in general to update dependencies in those versions, at
>>> least to a higher patch version, but I think a higher minor version
>>> should also be OK if we also add features ourselves in those versions.
>>>
>>>> We also typically have JIRAs for version bumps of dependencies - not
>>> always plugins - so if we continued with that those would be need to
>>> be created and CHANGELOG entries added manually after a merge i guess.
>>>
>>> Do we need a JIRA + CHANGELOG entry for all dependency updates or only
>>> for important ones?
>>> If we need them, then the bot at least notifies us of an available
>>> update and provides a PR for the update itself. We can then still
>>> create a JIRA issue + CHANGELOG entry manually if necessary.
>>> The notification should also include fixes for security problems in
>>> which case the bot sometimes also includes this information directly in
>> the PR.
>>>> kuppitz makes the good point of the mess Travis has been in lately.
>>> Is that a reason against using a bot for automation or am I
>>> misunderstanding your/Daniel's point here? If a Travis build fails
>>> non-deterministically for a PR created by a bot, then I would simply
>>> trigger a rebuild manually.
>>>
>>> We could also use such a bot only for the GLVs / some of the GLVs if
>>> we don't want to update dependencies in general outside of major
>>> releases (like 3.5.0). At least for .NET I would appreciate such a
>>> bot, but I can't say much for Python / JavaScript as I don't know
>>> enough about how they handle dependency updates.
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Stephen Mallette <sp...@gmail.com>
>>> Gesendet: Mittwoch, 3. April 2019 20:27
>>> An: dev@tinkerpop.apache.org
>>> Betreff: Re: Using a bot to keep dependencies up to date
>>>
>>> hmm - so far i've been the bot. i run maven commands to find out what
>>> needs upgrading and then make decisions on when to do those upgrades
>>> as we don't usually upgrade them in a release line unless:
>>>
>>> 1. they are maven plugins
>>> 2. they are fixes for security problems 3. they introduce something
>>> important for our code base - like a performance fix or other
>>> enhancements
>>>
>>> That said, my maven commands only deal with Java and not the GLVs so
>>> not much attention has been paid there unfortunately now that I think
>> about it.
>>> We also typically have JIRAs for version bumps of dependencies - not
>>> always plugins - so if we continued with that those would be need to
>>> be created and CHANGELOG entries added manually after a merge i guess.
>>>
>>> I'm not completely against the idea, just pointing out some points of
>>> friction with what we currently do and kuppitz makes the good point of
>>> the mess Travis has been in lately. Utterly useless the last few
>>> weeks. If we didn't use a bot, at minimum it would be nice to document
>>> the manual methods for getting reports for the GLVs that show upgrade
>>> paths (I assume that those exist for other language ecosystems as they
>>> do for maven). it would be nice to hear additional thoughts on the
>> matter.
>>> Florian, thanks for taking the time to look into improving our build
>>> process.
>>>
>>>
>>>
>>> On Wed, Apr 3, 2019 at 1:29 PM Daniel Kuppitz <me...@gremlin.guru> wrote:
>>>
>>>> Pretty cool, I like that (if only Travis would be a little more
>>> reliable).
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>>> On Wed, Apr 3, 2019 at 9:43 AM Florian Hockmann
>>>> <fh...@florian-hockmann.de>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> we have a lot of dependencies in TinkerPop in different projects
>>>>> and even across different languages. That makes it hard to keep
>>>>> them updated which sometimes has security implications.
>>>>>
>>>>> I recently noticed that other open source projects use a bot that
>>>>> regularly checks whether any updates are available for their
>>>>> dependencies and then creates one PR per dependency. Just to try
>>>>> it out with TinkerPop, I activated such a bot on my fork:
>>>>>
>>>>> https://github.com/florianhockmann/tinkerpop/pulls
>>>>>
>>>>> and the overall result looks quite good in my opinion. It created
>>>>> a lot of PRs* and most could probably be directly merged. The bot
>>>>> can also be easily configured just by adding comments to its PR,
>>>>> for example to ignore a certain (major/minor/patch) version of a
>>> dependency:
>>>>>
>>>> https://github.com/FlorianHockmann/tinkerpop/pull/24#issuecomment-47
>>>> 39
>>>> 36360
>>>>> What do you think about adding such a bot for our repo?
>>>>>
>>>>>
>>>>> * This is limited to only 5 PRs per day at first to not overwhelm
>>>>> a project with PRs.
>>>>>
>>>>>
>>>>>
>>>
>>

Re: Using a bot to keep dependencies up to date

[Stephen Mallette <sp...@gmail.com>]

>  We can define the target branch for the PRs. So, we could set that to
tp33 and then merge though to master like we usually do it for other PRs.
(Maybe it's also possible to create different configurations for the
different branches if we want to get one PR per branch.)

There are different dependencies on tp33 vs master so it would be nice to
know for sure how they handle that. Of course, you allude to a
proliferation of PRs that would result....ugh. Anyway, I guess I'd like to
know what the options really are there...

> I assume that Apacha Infra will have to activate GitHub bots like
dependabot for us as that requires ownership of the GitHub organization. I
hope that they can afterwards give us permissions to change the settings,
but I'm not sure about that. Maybe these permissions also only work on an
organization basis (which would include all ASF repos). Worst case would be
that ASF Infra has to configure such a bot for us.

all good questions to answer. no one seem to really be objecting to this
direction (i only want to know more about the details of how it will work)
so I guess we're at a point where you could try to move things forward with
infra and the bot folks to see how these final issues we're discussing
would work out. please let us know what you find out. thanks.

On Wed, Apr 10, 2019 at 9:52 AM Florian Hockmann <fh...@florian-hockmann.de>
wrote:

> > always using the most recent has been disastrous in python. our build
> breaks all the time with no changes from us because of that style where we
> don't pin to specific dependencies. i don't understand that model at all. i
> know you're not saying that we blindly upgrade, i was just making a point
> about python that is semi-related.
>
> Understood and I agree completely that we should pin versions.
>
> > we're not always sure of what a change in dependency will bring in terms
> of change to the API but i agree that upgrades can take place (and as you
> pointed out at the start, are already taking place in a more manual
> fashion).
>
> IF our dependencies use semantic versioning, then this shouldn't be a
> problem as long as we don't do major version updates in our patch versions.
>
> > Why no PR for spark, hadoop, etc? not that they would pass compilation -
> i'd expect failures, but I'm just wondering if you knew the reason it
> didn't catch those?
>
> I also wondered about that as I had exactly the same expectation of
> immediately getting a failed build for these dependencies. Maybe they have
> a list of dependencies where they don't attempt an update? We can of course
> ask them about this if we decide to use this bot.
>
> > it looks like it submits all PRs against master. what about tp33? will
> we have to cherry-pick out of the PR to tp33, test and then merge forward
> to master?
>
> We can define the target branch for the PRs. So, we could set that to tp33
> and then merge though to master like we usually do it for other PRs. (Maybe
> it's also possible to create different configurations for the different
> branches if we want to get one PR per branch.)
>
> > i assume that you setup an account for dependabot that gets it
> configured at our repo? I assume that the account is bound to your github
> account?
>
> Yes, I enabled dependabot for my account and could then give it access to
> my own repos which is a fork of our TinkerPop repo in this case.
>
> > can that dependabot dashboard for "tinkerpop" be accessed by others or
> just your account? how will that work?
>
> I assume that Apacha Infra will have to activate GitHub bots like
> dependabot for us as that requires ownership of the GitHub organization. I
> hope that they can afterwards give us permissions to change the settings,
> but I'm not sure about that. Maybe these permissions also only work on an
> organization basis (which would include all ASF repos). Worst case would be
> that ASF Infra has to configure such a bot for us.
>
> > i don't see much information about what the dashboard allows users to do
> - could you summarize some of the options available and what we see there?
>
> It allows to add different configurations for different languages or
> directories in a repo. We can then configure the following aspects per
> directory / language:
>
> Update schedule: daily / weekly / monthly
> Directory (important for the GLVs as that directory should contain the
> project file, like the .sln file for .NET)
> Target branch
> Filters: They currently only have a filter to only get security updates.
> Automatic PR merging: Can be configured for only some versions (like patch
> versions, only security fixes, ...) and only for specific dependencies. (I
> don't think that we want automatic merging.)
> For the GitHub PRs, we can also add default reviewers, assignees, and
> labels.
>
> On top of that, the UI also allows to manually trigger an update run,
> irrespective of the configured update schedule. I used that on the first
> day as dependabot by default only creates 5 PRs per directory/language and
> per update run, but I wanted to get more PRs without having to wait several
> days.
>
> -----Ursprüngliche Nachricht-----
> Von: Stephen Mallette <sp...@gmail.com>
> Gesendet: Mittwoch, 10. April 2019 14:41
> An: dev@tinkerpop.apache.org
> Betreff: Re: Using a bot to keep dependencies up to date
>
> >
> > I'm still not that experienced with Java/Maven, but the reasoning is
> > that we want to avoid dependency version conflicts for our users, right?
> >
>
> well, we just try to keep things stable as possible without introducing a
> breaking change by way of a dependency. it's not to say that we never do
> it, but it's always with done with careful consideration.
>
>
> > In that case, it's specific to Java I'd say. At least for .NET, I
> > would expect a library to use recent versions of its dependencies.
> >
>
> always using the most recent has been disastrous in python. our build
> breaks all the time with no changes from us because of that style where we
> don't pin to specific dependencies. i don't understand that model at all. i
> know you're not saying that we blindly upgrade, i was just making a point
> about python that is semi-related.
>
>
> > Since we add new features in our "patch versions" (we aren't doing
> > semantic versioning so those aren't actually patch versions), I think
> > it's also OK in general to update dependencies in those versions, at
> > least to a higher patch version, but I think a higher minor version
> > should also be OK if we also add features ourselves in those versions.
> >
>
> We allow new features into patch versions because we control the new
> features and make choices as to whether they are "safe" or not. we're not
> always sure of what a change in dependency will bring in terms of change to
> the API but i agree that upgrades can take place (and as you pointed out at
> the start, are already taking place in a more manual fashion). Again, I
> just feel like we need to take each upgrade on a case by case basis.
>
>
> > > We also typically have JIRAs for version bumps of dependencies - not
> > always plugins - so if we continued with that those would be need to
> > be created and CHANGELOG entries added manually after a merge i guess.
> >
> > Do we need a JIRA + CHANGELOG entry for all dependency updates or only
> > for important ones?
> > If we need them, then the bot at least notifies us of an available
> > update and provides a PR for the update itself. We can then still
> > create a JIRA issue + CHANGELOG entry manually if necessary.
> > The notification should also include fixes for security problems in
> > which case the bot sometimes also includes this information directly in
> the PR.
> >
>
> maybe we just need it for "important" ones. i guess that's how we've been
> doing it so far.......
>
>
> > > kuppitz makes the good point of the mess Travis has been in lately.
> >
> > Is that a reason against using a bot for automation or am I
> > misunderstanding your/Daniel's point here? If a Travis build fails
> > non-deterministically for a PR created by a bot, then I would simply
> > trigger a rebuild manually.
> >
>
> i guess that's just us railing about travis........not a reason to avoid
> automation.
>
>
> > We could also use such a bot only for the GLVs / some of the GLVs if
> > we don't want to update dependencies in general outside of major
> > releases (like 3.5.0). At least for .NET I would appreciate such a
> > bot, but I can't say much for Python / JavaScript as I don't know
> > enough about how they handle dependency updates.
>
>
> if we use this, i think we should use it across the project. i looked at
> the list of PRs that were generated by it in your fork more carefully and
> it was curious that the larger version bumps weren't present. Why no PR for
> spark, hadoop, etc? not that they would pass compilation - i'd expect
> failures, but I'm just wondering if you knew the reason it didn't catch
> those?
>
> some more questions:
>
> 1. I see the list of commands that dependabot has - it looks like it
> submits all PRs against master. what about tp33? will we have to
> cherry-pick out of the PR to tp33, test and then merge forward to master?
> or is there something easier?
> 2. i assume that you setup an account for dependabot that gets it
> configured at our repo? I assume that the account is bound to your github
> account? can that dependabot dashboard for "tinkerpop" be accessed by
> others or just your account? how will that work?
> 3. i don't see much information about what the dashboard allows users to do
> - could you summarize some of the options available and what we see there?
>
> thanks!
>
>
> On Wed, Apr 10, 2019 at 7:59 AM Florian Hockmann <fh...@florian-hockmann.de>
> wrote:
>
> > > we don't usually upgrade them in a release line unless
> >
> > I'm still not that experienced with Java/Maven, but the reasoning is
> > that we want to avoid dependency version conflicts for our users, right?
> >
> > In that case, it's specific to Java I'd say. At least for .NET, I
> > would expect a library to use recent versions of its dependencies.
> >
> > Since we add new features in our "patch versions" (we aren't doing
> > semantic versioning so those aren't actually patch versions), I think
> > it's also OK in general to update dependencies in those versions, at
> > least to a higher patch version, but I think a higher minor version
> > should also be OK if we also add features ourselves in those versions.
> >
> > > We also typically have JIRAs for version bumps of dependencies - not
> > always plugins - so if we continued with that those would be need to
> > be created and CHANGELOG entries added manually after a merge i guess.
> >
> > Do we need a JIRA + CHANGELOG entry for all dependency updates or only
> > for important ones?
> > If we need them, then the bot at least notifies us of an available
> > update and provides a PR for the update itself. We can then still
> > create a JIRA issue + CHANGELOG entry manually if necessary.
> > The notification should also include fixes for security problems in
> > which case the bot sometimes also includes this information directly in
> the PR.
> >
> > > kuppitz makes the good point of the mess Travis has been in lately.
> >
> > Is that a reason against using a bot for automation or am I
> > misunderstanding your/Daniel's point here? If a Travis build fails
> > non-deterministically for a PR created by a bot, then I would simply
> > trigger a rebuild manually.
> >
> > We could also use such a bot only for the GLVs / some of the GLVs if
> > we don't want to update dependencies in general outside of major
> > releases (like 3.5.0). At least for .NET I would appreciate such a
> > bot, but I can't say much for Python / JavaScript as I don't know
> > enough about how they handle dependency updates.
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Stephen Mallette <sp...@gmail.com>
> > Gesendet: Mittwoch, 3. April 2019 20:27
> > An: dev@tinkerpop.apache.org
> > Betreff: Re: Using a bot to keep dependencies up to date
> >
> > hmm - so far i've been the bot. i run maven commands to find out what
> > needs upgrading and then make decisions on when to do those upgrades
> > as we don't usually upgrade them in a release line unless:
> >
> > 1. they are maven plugins
> > 2. they are fixes for security problems 3. they introduce something
> > important for our code base - like a performance fix or other
> > enhancements
> >
> > That said, my maven commands only deal with Java and not the GLVs so
> > not much attention has been paid there unfortunately now that I think
> about it.
> > We also typically have JIRAs for version bumps of dependencies - not
> > always plugins - so if we continued with that those would be need to
> > be created and CHANGELOG entries added manually after a merge i guess.
> >
> > I'm not completely against the idea, just pointing out some points of
> > friction with what we currently do and kuppitz makes the good point of
> > the mess Travis has been in lately. Utterly useless the last few
> > weeks. If we didn't use a bot, at minimum it would be nice to document
> > the manual methods for getting reports for the GLVs that show upgrade
> > paths (I assume that those exist for other language ecosystems as they
> > do for maven). it would be nice to hear additional thoughts on the
> matter.
> >
> > Florian, thanks for taking the time to look into improving our build
> > process.
> >
> >
> >
> > On Wed, Apr 3, 2019 at 1:29 PM Daniel Kuppitz <me...@gremlin.guru> wrote:
> >
> > > Pretty cool, I like that (if only Travis would be a little more
> > reliable).
> > >
> > > Cheers,
> > > Daniel
> > >
> > >
> > > On Wed, Apr 3, 2019 at 9:43 AM Florian Hockmann
> > > <fh...@florian-hockmann.de>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > we have a lot of dependencies in TinkerPop in different projects
> > > > and even across different languages. That makes it hard to keep
> > > > them updated which sometimes has security implications.
> > > >
> > > > I recently noticed that other open source projects use a bot that
> > > > regularly checks whether any updates are available for their
> > > > dependencies and then creates one PR per dependency. Just to try
> > > > it out with TinkerPop, I activated such a bot on my fork:
> > > >
> > > > https://github.com/florianhockmann/tinkerpop/pulls
> > > >
> > > > and the overall result looks quite good in my opinion. It created
> > > > a lot of PRs* and most could probably be directly merged. The bot
> > > > can also be easily configured just by adding comments to its PR,
> > > > for example to ignore a certain (major/minor/patch) version of a
> > dependency:
> > > >
> > > >
> > > https://github.com/FlorianHockmann/tinkerpop/pull/24#issuecomment-47
> > > 39
> > > 36360
> > > >
> > > > What do you think about adding such a bot for our repo?
> > > >
> > > >
> > > > * This is limited to only 5 PRs per day at first to not overwhelm
> > > > a project with PRs.
> > > >
> > > >
> > > >
> > >
> >
> >
>
>

AW: Using a bot to keep dependencies up to date

[Florian Hockmann <fh...@florian-hockmann.de>]

> always using the most recent has been disastrous in python. our build breaks all the time with no changes from us because of that style where we don't pin to specific dependencies. i don't understand that model at all. i know you're not saying that we blindly upgrade, i was just making a point about python that is semi-related.

Understood and I agree completely that we should pin versions.

> we're not always sure of what a change in dependency will bring in terms of change to the API but i agree that upgrades can take place (and as you pointed out at the start, are already taking place in a more manual fashion).

IF our dependencies use semantic versioning, then this shouldn't be a problem as long as we don't do major version updates in our patch versions.

> Why no PR for spark, hadoop, etc? not that they would pass compilation - i'd expect failures, but I'm just wondering if you knew the reason it didn't catch those?

I also wondered about that as I had exactly the same expectation of immediately getting a failed build for these dependencies. Maybe they have a list of dependencies where they don't attempt an update? We can of course ask them about this if we decide to use this bot.

> it looks like it submits all PRs against master. what about tp33? will we have to cherry-pick out of the PR to tp33, test and then merge forward to master?

We can define the target branch for the PRs. So, we could set that to tp33 and then merge though to master like we usually do it for other PRs. (Maybe it's also possible to create different configurations for the different branches if we want to get one PR per branch.)

> i assume that you setup an account for dependabot that gets it configured at our repo? I assume that the account is bound to your github account? 

Yes, I enabled dependabot for my account and could then give it access to my own repos which is a fork of our TinkerPop repo in this case.

> can that dependabot dashboard for "tinkerpop" be accessed by others or just your account? how will that work?

I assume that Apacha Infra will have to activate GitHub bots like dependabot for us as that requires ownership of the GitHub organization. I hope that they can afterwards give us permissions to change the settings, but I'm not sure about that. Maybe these permissions also only work on an organization basis (which would include all ASF repos). Worst case would be that ASF Infra has to configure such a bot for us.

> i don't see much information about what the dashboard allows users to do
- could you summarize some of the options available and what we see there?

It allows to add different configurations for different languages or directories in a repo. We can then configure the following aspects per directory / language:

Update schedule: daily / weekly / monthly
Directory (important for the GLVs as that directory should contain the project file, like the .sln file for .NET)
Target branch
Filters: They currently only have a filter to only get security updates.
Automatic PR merging: Can be configured for only some versions (like patch versions, only security fixes, ...) and only for specific dependencies. (I don't think that we want automatic merging.)
For the GitHub PRs, we can also add default reviewers, assignees, and labels.

On top of that, the UI also allows to manually trigger an update run, irrespective of the configured update schedule. I used that on the first day as dependabot by default only creates 5 PRs per directory/language and per update run, but I wanted to get more PRs without having to wait several days.

-----Ursprüngliche Nachricht-----
Von: Stephen Mallette <sp...@gmail.com> 
Gesendet: Mittwoch, 10. April 2019 14:41
An: dev@tinkerpop.apache.org
Betreff: Re: Using a bot to keep dependencies up to date

>
> I'm still not that experienced with Java/Maven, but the reasoning is 
> that we want to avoid dependency version conflicts for our users, right?
>

well, we just try to keep things stable as possible without introducing a breaking change by way of a dependency. it's not to say that we never do it, but it's always with done with careful consideration.


> In that case, it's specific to Java I'd say. At least for .NET, I 
> would expect a library to use recent versions of its dependencies.
>

always using the most recent has been disastrous in python. our build breaks all the time with no changes from us because of that style where we don't pin to specific dependencies. i don't understand that model at all. i know you're not saying that we blindly upgrade, i was just making a point about python that is semi-related.


> Since we add new features in our "patch versions" (we aren't doing 
> semantic versioning so those aren't actually patch versions), I think 
> it's also OK in general to update dependencies in those versions, at 
> least to a higher patch version, but I think a higher minor version 
> should also be OK if we also add features ourselves in those versions.
>

We allow new features into patch versions because we control the new features and make choices as to whether they are "safe" or not. we're not always sure of what a change in dependency will bring in terms of change to the API but i agree that upgrades can take place (and as you pointed out at the start, are already taking place in a more manual fashion). Again, I just feel like we need to take each upgrade on a case by case basis.


> > We also typically have JIRAs for version bumps of dependencies - not
> always plugins - so if we continued with that those would be need to 
> be created and CHANGELOG entries added manually after a merge i guess.
>
> Do we need a JIRA + CHANGELOG entry for all dependency updates or only 
> for important ones?
> If we need them, then the bot at least notifies us of an available 
> update and provides a PR for the update itself. We can then still 
> create a JIRA issue + CHANGELOG entry manually if necessary.
> The notification should also include fixes for security problems in 
> which case the bot sometimes also includes this information directly in the PR.
>

maybe we just need it for "important" ones. i guess that's how we've been doing it so far.......


> > kuppitz makes the good point of the mess Travis has been in lately.
>
> Is that a reason against using a bot for automation or am I 
> misunderstanding your/Daniel's point here? If a Travis build fails 
> non-deterministically for a PR created by a bot, then I would simply 
> trigger a rebuild manually.
>

i guess that's just us railing about travis........not a reason to avoid automation.


> We could also use such a bot only for the GLVs / some of the GLVs if 
> we don't want to update dependencies in general outside of major 
> releases (like 3.5.0). At least for .NET I would appreciate such a 
> bot, but I can't say much for Python / JavaScript as I don't know 
> enough about how they handle dependency updates.


if we use this, i think we should use it across the project. i looked at the list of PRs that were generated by it in your fork more carefully and it was curious that the larger version bumps weren't present. Why no PR for spark, hadoop, etc? not that they would pass compilation - i'd expect failures, but I'm just wondering if you knew the reason it didn't catch those?

some more questions:

1. I see the list of commands that dependabot has - it looks like it submits all PRs against master. what about tp33? will we have to cherry-pick out of the PR to tp33, test and then merge forward to master?
or is there something easier?
2. i assume that you setup an account for dependabot that gets it configured at our repo? I assume that the account is bound to your github account? can that dependabot dashboard for "tinkerpop" be accessed by others or just your account? how will that work?
3. i don't see much information about what the dashboard allows users to do
- could you summarize some of the options available and what we see there?

thanks!


On Wed, Apr 10, 2019 at 7:59 AM Florian Hockmann <fh...@florian-hockmann.de>
wrote:

> > we don't usually upgrade them in a release line unless
>
> I'm still not that experienced with Java/Maven, but the reasoning is 
> that we want to avoid dependency version conflicts for our users, right?
>
> In that case, it's specific to Java I'd say. At least for .NET, I 
> would expect a library to use recent versions of its dependencies.
>
> Since we add new features in our "patch versions" (we aren't doing 
> semantic versioning so those aren't actually patch versions), I think 
> it's also OK in general to update dependencies in those versions, at 
> least to a higher patch version, but I think a higher minor version 
> should also be OK if we also add features ourselves in those versions.
>
> > We also typically have JIRAs for version bumps of dependencies - not
> always plugins - so if we continued with that those would be need to 
> be created and CHANGELOG entries added manually after a merge i guess.
>
> Do we need a JIRA + CHANGELOG entry for all dependency updates or only 
> for important ones?
> If we need them, then the bot at least notifies us of an available 
> update and provides a PR for the update itself. We can then still 
> create a JIRA issue + CHANGELOG entry manually if necessary.
> The notification should also include fixes for security problems in 
> which case the bot sometimes also includes this information directly in the PR.
>
> > kuppitz makes the good point of the mess Travis has been in lately.
>
> Is that a reason against using a bot for automation or am I 
> misunderstanding your/Daniel's point here? If a Travis build fails 
> non-deterministically for a PR created by a bot, then I would simply 
> trigger a rebuild manually.
>
> We could also use such a bot only for the GLVs / some of the GLVs if 
> we don't want to update dependencies in general outside of major 
> releases (like 3.5.0). At least for .NET I would appreciate such a 
> bot, but I can't say much for Python / JavaScript as I don't know 
> enough about how they handle dependency updates.
>
> -----Ursprüngliche Nachricht-----
> Von: Stephen Mallette <sp...@gmail.com>
> Gesendet: Mittwoch, 3. April 2019 20:27
> An: dev@tinkerpop.apache.org
> Betreff: Re: Using a bot to keep dependencies up to date
>
> hmm - so far i've been the bot. i run maven commands to find out what 
> needs upgrading and then make decisions on when to do those upgrades 
> as we don't usually upgrade them in a release line unless:
>
> 1. they are maven plugins
> 2. they are fixes for security problems 3. they introduce something 
> important for our code base - like a performance fix or other 
> enhancements
>
> That said, my maven commands only deal with Java and not the GLVs so 
> not much attention has been paid there unfortunately now that I think about it.
> We also typically have JIRAs for version bumps of dependencies - not 
> always plugins - so if we continued with that those would be need to 
> be created and CHANGELOG entries added manually after a merge i guess.
>
> I'm not completely against the idea, just pointing out some points of 
> friction with what we currently do and kuppitz makes the good point of 
> the mess Travis has been in lately. Utterly useless the last few 
> weeks. If we didn't use a bot, at minimum it would be nice to document 
> the manual methods for getting reports for the GLVs that show upgrade 
> paths (I assume that those exist for other language ecosystems as they 
> do for maven). it would be nice to hear additional thoughts on the matter.
>
> Florian, thanks for taking the time to look into improving our build 
> process.
>
>
>
> On Wed, Apr 3, 2019 at 1:29 PM Daniel Kuppitz <me...@gremlin.guru> wrote:
>
> > Pretty cool, I like that (if only Travis would be a little more
> reliable).
> >
> > Cheers,
> > Daniel
> >
> >
> > On Wed, Apr 3, 2019 at 9:43 AM Florian Hockmann 
> > <fh...@florian-hockmann.de>
> > wrote:
> >
> > > Hi,
> > >
> > > we have a lot of dependencies in TinkerPop in different projects 
> > > and even across different languages. That makes it hard to keep 
> > > them updated which sometimes has security implications.
> > >
> > > I recently noticed that other open source projects use a bot that 
> > > regularly checks whether any updates are available for their 
> > > dependencies and then creates one PR per dependency. Just to try 
> > > it out with TinkerPop, I activated such a bot on my fork:
> > >
> > > https://github.com/florianhockmann/tinkerpop/pulls
> > >
> > > and the overall result looks quite good in my opinion. It created 
> > > a lot of PRs* and most could probably be directly merged. The bot 
> > > can also be easily configured just by adding comments to its PR, 
> > > for example to ignore a certain (major/minor/patch) version of a
> dependency:
> > >
> > >
> > https://github.com/FlorianHockmann/tinkerpop/pull/24#issuecomment-47
> > 39
> > 36360
> > >
> > > What do you think about adding such a bot for our repo?
> > >
> > >
> > > * This is limited to only 5 PRs per day at first to not overwhelm 
> > > a project with PRs.
> > >
> > >
> > >
> >
>
>

Re: Using a bot to keep dependencies up to date

[Stephen Mallette <sp...@gmail.com>]

>
> I'm still not that experienced with Java/Maven, but the reasoning is that
> we want to avoid dependency version conflicts for our users, right?
>

well, we just try to keep things stable as possible without introducing a
breaking change by way of a dependency. it's not to say that we never do
it, but it's always with done with careful consideration.


> In that case, it's specific to Java I'd say. At least for .NET, I would
> expect a library to use recent versions of its dependencies.
>

always using the most recent has been disastrous in python. our build
breaks all the time with no changes from us because of that style where we
don't pin to specific dependencies. i don't understand that model at all. i
know you're not saying that we blindly upgrade, i was just making a point
about python that is semi-related.


> Since we add new features in our "patch versions" (we aren't doing
> semantic versioning so those aren't actually patch versions), I think it's
> also OK in general to update dependencies in those versions, at least to a
> higher patch version, but I think a higher minor version should also be OK
> if we also add features ourselves in those versions.
>

We allow new features into patch versions because we control the new
features and make choices as to whether they are "safe" or not. we're not
always sure of what a change in dependency will bring in terms of change to
the API but i agree that upgrades can take place (and as you pointed out at
the start, are already taking place in a more manual fashion). Again, I
just feel like we need to take each upgrade on a case by case basis.


> > We also typically have JIRAs for version bumps of dependencies - not
> always plugins - so if we continued with that those would be need to be
> created and CHANGELOG entries added manually after a merge i guess.
>
> Do we need a JIRA + CHANGELOG entry for all dependency updates or only for
> important ones?
> If we need them, then the bot at least notifies us of an available update
> and provides a PR for the update itself. We can then still create a JIRA
> issue + CHANGELOG entry manually if necessary.
> The notification should also include fixes for security problems in which
> case the bot sometimes also includes this information directly in the PR.
>

maybe we just need it for "important" ones. i guess that's how we've been
doing it so far.......


> > kuppitz makes the good point of the mess Travis has been in lately.
>
> Is that a reason against using a bot for automation or am I
> misunderstanding your/Daniel's point here? If a Travis build fails
> non-deterministically for a PR created by a bot, then I would simply
> trigger a rebuild manually.
>

i guess that's just us railing about travis........not a reason to avoid
automation.


> We could also use such a bot only for the GLVs / some of the GLVs if we
> don't want to update dependencies in general outside of major releases
> (like 3.5.0). At least for .NET I would appreciate such a bot, but I can't
> say much for Python / JavaScript as I don't know enough about how they
> handle dependency updates.


if we use this, i think we should use it across the project. i looked at
the list of PRs that were generated by it in your fork more carefully and
it was curious that the larger version bumps weren't present. Why no PR for
spark, hadoop, etc? not that they would pass compilation - i'd expect
failures, but I'm just wondering if you knew the reason it didn't catch
those?

some more questions:

1. I see the list of commands that dependabot has - it looks like it
submits all PRs against master. what about tp33? will we have to
cherry-pick out of the PR to tp33, test and then merge forward to master?
or is there something easier?
2. i assume that you setup an account for dependabot that gets it
configured at our repo? I assume that the account is bound to your github
account? can that dependabot dashboard for "tinkerpop" be accessed by
others or just your account? how will that work?
3. i don't see much information about what the dashboard allows users to do
- could you summarize some of the options available and what we see there?

thanks!


On Wed, Apr 10, 2019 at 7:59 AM Florian Hockmann <fh...@florian-hockmann.de>
wrote:

> > we don't usually upgrade them in a release line unless
>
> I'm still not that experienced with Java/Maven, but the reasoning is that
> we want to avoid dependency version conflicts for our users, right?
>
> In that case, it's specific to Java I'd say. At least for .NET, I would
> expect a library to use recent versions of its dependencies.
>
> Since we add new features in our "patch versions" (we aren't doing
> semantic versioning so those aren't actually patch versions), I think it's
> also OK in general to update dependencies in those versions, at least to a
> higher patch version, but I think a higher minor version should also be OK
> if we also add features ourselves in those versions.
>
> > We also typically have JIRAs for version bumps of dependencies - not
> always plugins - so if we continued with that those would be need to be
> created and CHANGELOG entries added manually after a merge i guess.
>
> Do we need a JIRA + CHANGELOG entry for all dependency updates or only for
> important ones?
> If we need them, then the bot at least notifies us of an available update
> and provides a PR for the update itself. We can then still create a JIRA
> issue + CHANGELOG entry manually if necessary.
> The notification should also include fixes for security problems in which
> case the bot sometimes also includes this information directly in the PR.
>
> > kuppitz makes the good point of the mess Travis has been in lately.
>
> Is that a reason against using a bot for automation or am I
> misunderstanding your/Daniel's point here? If a Travis build fails
> non-deterministically for a PR created by a bot, then I would simply
> trigger a rebuild manually.
>
> We could also use such a bot only for the GLVs / some of the GLVs if we
> don't want to update dependencies in general outside of major releases
> (like 3.5.0). At least for .NET I would appreciate such a bot, but I can't
> say much for Python / JavaScript as I don't know enough about how they
> handle dependency updates.
>
> -----Ursprüngliche Nachricht-----
> Von: Stephen Mallette <sp...@gmail.com>
> Gesendet: Mittwoch, 3. April 2019 20:27
> An: dev@tinkerpop.apache.org
> Betreff: Re: Using a bot to keep dependencies up to date
>
> hmm - so far i've been the bot. i run maven commands to find out what
> needs upgrading and then make decisions on when to do those upgrades as we
> don't usually upgrade them in a release line unless:
>
> 1. they are maven plugins
> 2. they are fixes for security problems
> 3. they introduce something important for our code base - like a
> performance fix or other enhancements
>
> That said, my maven commands only deal with Java and not the GLVs so not
> much attention has been paid there unfortunately now that I think about it.
> We also typically have JIRAs for version bumps of dependencies - not
> always plugins - so if we continued with that those would be need to be
> created and CHANGELOG entries added manually after a merge i guess.
>
> I'm not completely against the idea, just pointing out some points of
> friction with what we currently do and kuppitz makes the good point of the
> mess Travis has been in lately. Utterly useless the last few weeks. If we
> didn't use a bot, at minimum it would be nice to document the manual
> methods for getting reports for the GLVs that show upgrade paths (I assume
> that those exist for other language ecosystems as they do for maven). it
> would be nice to hear additional thoughts on the matter.
>
> Florian, thanks for taking the time to look into improving our build
> process.
>
>
>
> On Wed, Apr 3, 2019 at 1:29 PM Daniel Kuppitz <me...@gremlin.guru> wrote:
>
> > Pretty cool, I like that (if only Travis would be a little more
> reliable).
> >
> > Cheers,
> > Daniel
> >
> >
> > On Wed, Apr 3, 2019 at 9:43 AM Florian Hockmann
> > <fh...@florian-hockmann.de>
> > wrote:
> >
> > > Hi,
> > >
> > > we have a lot of dependencies in TinkerPop in different projects and
> > > even across different languages. That makes it hard to keep them
> > > updated which sometimes has security implications.
> > >
> > > I recently noticed that other open source projects use a bot that
> > > regularly checks whether any updates are available for their
> > > dependencies and then creates one PR per dependency. Just to try it
> > > out with TinkerPop, I activated such a bot on my fork:
> > >
> > > https://github.com/florianhockmann/tinkerpop/pulls
> > >
> > > and the overall result looks quite good in my opinion. It created a
> > > lot of PRs* and most could probably be directly merged. The bot can
> > > also be easily configured just by adding comments to its PR, for
> > > example to ignore a certain (major/minor/patch) version of a
> dependency:
> > >
> > >
> > https://github.com/FlorianHockmann/tinkerpop/pull/24#issuecomment-4739
> > 36360
> > >
> > > What do you think about adding such a bot for our repo?
> > >
> > >
> > > * This is limited to only 5 PRs per day at first to not overwhelm a
> > > project with PRs.
> > >
> > >
> > >
> >
>
>

AW: Using a bot to keep dependencies up to date

[Florian Hockmann <fh...@florian-hockmann.de>]

> we don't usually upgrade them in a release line unless

I'm still not that experienced with Java/Maven, but the reasoning is that we want to avoid dependency version conflicts for our users, right?

In that case, it's specific to Java I'd say. At least for .NET, I would expect a library to use recent versions of its dependencies.

Since we add new features in our "patch versions" (we aren't doing semantic versioning so those aren't actually patch versions), I think it's also OK in general to update dependencies in those versions, at least to a higher patch version, but I think a higher minor version should also be OK if we also add features ourselves in those versions.

> We also typically have JIRAs for version bumps of dependencies - not always plugins - so if we continued with that those would be need to be created and CHANGELOG entries added manually after a merge i guess.

Do we need a JIRA + CHANGELOG entry for all dependency updates or only for important ones?
If we need them, then the bot at least notifies us of an available update and provides a PR for the update itself. We can then still create a JIRA issue + CHANGELOG entry manually if necessary.
The notification should also include fixes for security problems in which case the bot sometimes also includes this information directly in the PR.

> kuppitz makes the good point of the mess Travis has been in lately. 

Is that a reason against using a bot for automation or am I misunderstanding your/Daniel's point here? If a Travis build fails non-deterministically for a PR created by a bot, then I would simply trigger a rebuild manually.

We could also use such a bot only for the GLVs / some of the GLVs if we don't want to update dependencies in general outside of major releases (like 3.5.0). At least for .NET I would appreciate such a bot, but I can't say much for Python / JavaScript as I don't know enough about how they handle dependency updates.

-----Ursprüngliche Nachricht-----
Von: Stephen Mallette <sp...@gmail.com> 
Gesendet: Mittwoch, 3. April 2019 20:27
An: dev@tinkerpop.apache.org
Betreff: Re: Using a bot to keep dependencies up to date

hmm - so far i've been the bot. i run maven commands to find out what needs upgrading and then make decisions on when to do those upgrades as we don't usually upgrade them in a release line unless:

1. they are maven plugins
2. they are fixes for security problems
3. they introduce something important for our code base - like a performance fix or other enhancements

That said, my maven commands only deal with Java and not the GLVs so not much attention has been paid there unfortunately now that I think about it.
We also typically have JIRAs for version bumps of dependencies - not always plugins - so if we continued with that those would be need to be created and CHANGELOG entries added manually after a merge i guess.

I'm not completely against the idea, just pointing out some points of friction with what we currently do and kuppitz makes the good point of the mess Travis has been in lately. Utterly useless the last few weeks. If we didn't use a bot, at minimum it would be nice to document the manual methods for getting reports for the GLVs that show upgrade paths (I assume that those exist for other language ecosystems as they do for maven). it would be nice to hear additional thoughts on the matter.

Florian, thanks for taking the time to look into improving our build process.



On Wed, Apr 3, 2019 at 1:29 PM Daniel Kuppitz <me...@gremlin.guru> wrote:

> Pretty cool, I like that (if only Travis would be a little more reliable).
>
> Cheers,
> Daniel
>
>
> On Wed, Apr 3, 2019 at 9:43 AM Florian Hockmann 
> <fh...@florian-hockmann.de>
> wrote:
>
> > Hi,
> >
> > we have a lot of dependencies in TinkerPop in different projects and 
> > even across different languages. That makes it hard to keep them 
> > updated which sometimes has security implications.
> >
> > I recently noticed that other open source projects use a bot that 
> > regularly checks whether any updates are available for their 
> > dependencies and then creates one PR per dependency. Just to try it 
> > out with TinkerPop, I activated such a bot on my fork:
> >
> > https://github.com/florianhockmann/tinkerpop/pulls
> >
> > and the overall result looks quite good in my opinion. It created a 
> > lot of PRs* and most could probably be directly merged. The bot can 
> > also be easily configured just by adding comments to its PR, for 
> > example to ignore a certain (major/minor/patch) version of a dependency:
> >
> >
> https://github.com/FlorianHockmann/tinkerpop/pull/24#issuecomment-4739
> 36360
> >
> > What do you think about adding such a bot for our repo?
> >
> >
> > * This is limited to only 5 PRs per day at first to not overwhelm a 
> > project with PRs.
> >
> >
> >
>

Re: Using a bot to keep dependencies up to date

[Stephen Mallette <sp...@gmail.com>]

hmm - so far i've been the bot. i run maven commands to find out what needs
upgrading and then make decisions on when to do those upgrades as we don't
usually upgrade them in a release line unless:

1. they are maven plugins
2. they are fixes for security problems
3. they introduce something important for our code base - like a
performance fix or other enhancements

That said, my maven commands only deal with Java and not the GLVs so not
much attention has been paid there unfortunately now that I think about it.
We also typically have JIRAs for version bumps of dependencies - not always
plugins - so if we continued with that those would be need to be created
and CHANGELOG entries added manually after a merge i guess.

I'm not completely against the idea, just pointing out some points of
friction with what we currently do and kuppitz makes the good point of the
mess Travis has been in lately. Utterly useless the last few weeks. If we
didn't use a bot, at minimum it would be nice to document the manual
methods for getting reports for the GLVs that show upgrade paths (I assume
that those exist for other language ecosystems as they do for maven). it
would be nice to hear additional thoughts on the matter.

Florian, thanks for taking the time to look into improving our build
process.



On Wed, Apr 3, 2019 at 1:29 PM Daniel Kuppitz <me...@gremlin.guru> wrote:

> Pretty cool, I like that (if only Travis would be a little more reliable).
>
> Cheers,
> Daniel
>
>
> On Wed, Apr 3, 2019 at 9:43 AM Florian Hockmann <fh...@florian-hockmann.de>
> wrote:
>
> > Hi,
> >
> > we have a lot of dependencies in TinkerPop in different projects and
> > even across different languages. That makes it hard to keep them updated
> > which sometimes has security implications.
> >
> > I recently noticed that other open source projects use a bot that
> > regularly checks whether any updates are available for their
> > dependencies and then creates one PR per dependency. Just to try it out
> > with TinkerPop, I activated such a bot on my fork:
> >
> > https://github.com/florianhockmann/tinkerpop/pulls
> >
> > and the overall result looks quite good in my opinion. It created a lot
> > of PRs* and most could probably be directly merged. The bot can also be
> > easily configured just by adding comments to its PR, for example to
> > ignore a certain (major/minor/patch) version of a dependency:
> >
> >
> https://github.com/FlorianHockmann/tinkerpop/pull/24#issuecomment-473936360
> >
> > What do you think about adding such a bot for our repo?
> >
> >
> > * This is limited to only 5 PRs per day at first to not overwhelm a
> > project with PRs.
> >
> >
> >
>

Re: Using a bot to keep dependencies up to date

[Daniel Kuppitz <me...@gremlin.guru>]

Pretty cool, I like that (if only Travis would be a little more reliable).

Cheers,
Daniel


On Wed, Apr 3, 2019 at 9:43 AM Florian Hockmann <fh...@florian-hockmann.de>
wrote:

> Hi,
>
> we have a lot of dependencies in TinkerPop in different projects and
> even across different languages. That makes it hard to keep them updated
> which sometimes has security implications.
>
> I recently noticed that other open source projects use a bot that
> regularly checks whether any updates are available for their
> dependencies and then creates one PR per dependency. Just to try it out
> with TinkerPop, I activated such a bot on my fork:
>
> https://github.com/florianhockmann/tinkerpop/pulls
>
> and the overall result looks quite good in my opinion. It created a lot
> of PRs* and most could probably be directly merged. The bot can also be
> easily configured just by adding comments to its PR, for example to
> ignore a certain (major/minor/patch) version of a dependency:
>
> https://github.com/FlorianHockmann/tinkerpop/pull/24#issuecomment-473936360
>
> What do you think about adding such a bot for our repo?
>
>
> * This is limited to only 5 PRs per day at first to not overwhelm a
> project with PRs.
>
>
>