You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@druid.apache.org by Andreas Maechler <am...@apache.org> on 2026/10/23 02:22:36 UTC

[DISCUSS] Weekly, grouped Dependabot updates to cut PR volume and CI load

Hi all,

I'd like to propose changing how Dependabot updates our Maven dependencies:
run weekly instead of daily, group minor and patch updates into one PR, and
keep major updates as separate PRs. I'd also like to discuss how we handle
licenses.yaml for these PRs, since that is the most common reason they fail
and need some manual intervention.

This should hopefully help with both the CI load (and mental PR load), and
also streamline the dependency updates more. My apologies if this has been
discussed before. I also know that Frank's Codex automation helped
tremendously with getting through many of the trivial and not-so-trivial
dependency updates, which is great.

Why

Apart from the obvious answer of "it's a lot!", I had Claude pull some
stats (2026-07-22 to 2026-10-22):

- 381 of the 696 commits on master were Dependabot PRs.
- Dependabot triggered 422 of the 2,389 PR runs of the "Unit & Integration
tests CI" workflow (about 18%), plus a run of Static Checks and CodeQL each
time.
- One CI pass for a Maven bump PR costs about 450 runner-minutes (median of
a dozen PRs from September and October), or roughly 14,600 runner-minutes a
week. That is well within the ASF caps [1], but the runner queue is shared
across the foundation, and each PR also needs a committer to review, often
fix, and merge it (so additional commits and additional runs).
- A single upstream release often arrives as several PRs. gRPC 1.84.1
opened four today (#20676 grpc-api, #20679 grpc-bom, #20681 grpc-context,
#20677 shade.grpc.version), because those artifacts are pinned in four
places.

Also, over a third of Dependabot's "Static Checks" runs fail (58 of 159
since 2026-09-08, against 6% on master): because licenses.yaml doesn't
match, because maven-enforcer throws RequireUpperBoundDeps when an artifact
moves without its siblings, or because of compile errors (kinda normal, at
least for major bumps). Separately, about 30% of their unit-test runs fail,
the same rate as on master, so that's general flaky tests and another
problem.

Proposal

At various projects I work on, we update dependencies weekly, so I was
thinking, why don't we do that for apache/druid as well? Idea:

1. Schedule: weekly (Monday) instead of daily. The existing 14-day cooldown
stays. (Neither applies to security updates: Dependabot opens those when
GitHub raises an alert for a vulnerable dependency, and bumps to the lowest
fixed version [2]. Security updates are already enabled for our repo.)
2. One group for all minor and patch updates, to keep the configuration
simple. Not every library follows semantic versioning, so the reviewer of
the weekly PR can drop an update that looks risky or breaking (maybe best
by adding an ignore rule to the yml file so we can easily track it) and
handle it in its own PR. This works well in other codebases I work on.
Major updates stay ungrouped, one PR each, since they are the ones most
likely to need code changes. The existing maven-resolver group and all
ignore rules stay as they are.
3. One group per ecosystem for security updates (Maven, website,
web-console, GitHub Actions), so a burst of advisories opens one PR instead
of one per advisory.

Again, I asked Claude to replay this proposal against the last 13 weeks of
merged bumps: we get about 68 PRs instead of 381, 43 of them majors from
the August catch-up. The last seven weeks go from 117 PRs to 17 (rough
estimate).

ASF Infra's Dependabot guidance [3] suggests weekly updates, grouping, and
grouped security updates.

licenses.yaml

licenses.yaml records an exact version for every bundled dependency, and
check-licenses matches on group, artifact and version. Every bump of a
shipped dependency therefore fails Static Checks until someone edits
licenses.yaml by hand: 166 of the 381 Dependabot PRs merged in the last 13
weeks needed that edit. Grouping cuts it to one edit per week, but here's
another idea.

Options, in the order I'd prefer them:

a) Check versions at release time, not on every PR. In PR CI,
check-licenses would keep failing a PR that adds a dependency or changes a
license, but only warn when a version differs. The release build keeps the
strict check, so a release still cannot ship a stale LICENSE. Before
cutting a release branch, the release manager runs a new script that
refreshes the versions in licenses.yaml from the Maven license reports, and
commits the result to master. Dependabot PRs then pass as they are. Pinot
runs Dependabot the same way and refreshes its LICENSE-binary with a script
at release time [4].
b) Keep the strict check, and have a workflow push the licenses.yaml fix to
the Dependabot branch. This is the most automated option, but I'm not sure
how easy it is to do. I think we'd have to mint a GH App token instead of
the GITHUB_TOKEN so CI would run, and store the app's key as a Dependabot
secret, since workflows triggered by Dependabot only see those. We do that
in private repos, but I'm not sure whether that works on Apache Infra.
c) Keep the strict check, and have a workflow post the needed licenses.yaml
change on each Dependabot PR as a comment. But that would still need manual
work on each PR, and we might as well not do the additional workflow stuff
and just run the script locally on that branch quickly.

Thoughts?

Cheers,
Andy

[1] https://infra.apache.org/github-actions-policy.html
[2]
https://docs.github.com/en/code-security/concepts/supply-chain-security/dependabot-security-updates
[3] https://infra.apache.org/dependabot.html
[4] https://github.com/apache/pinot/pull/17818

Re: [DISCUSS] Weekly, grouped Dependabot updates to cut PR volume and CI load

Posted by Andreas Maechler <am...@gmail.com>.
Ok, thank you Frank.

Regarding the manual updates, that's correct. Basically, the workflow would
change from closing the PR (or adding it to the dependency exclude list) to
changing the PR (dropping a version). I think the net overall work for a
human will still be small though since we don't have to look at dozens of
PRs but only one.

If there isn't any opposition, I will try to break up the above proposal
and a few items and then create a tracking issue (similar to Frank's
flaky test issue, or the JUnit migration one) to keep track of individual
work / PRs to achieve the above.

Cheers,
Andy

On Thu, Oct 22, 2026 at 9:52 PM Frank Chen <fr...@apache.org> wrote:

> I agree with the proposal.
>
> Reducing the number of these PRs would also save the tokens/time for the
> triage automation.
>
> One small problem is that, even for minor upgrades, some libs would have
> incompatibility problems like the recent zstd-jni upgrades.
> In this case, we have to update the PR manually to exclude these
> dependencies.
> But I think the odds that we encounter such a problem are small.
>
>
>
>
>
> On Fri, Oct 23, 2026 at 10:22 AM Andreas Maechler <am...@apache.org>
> wrote:
>
> > Hi all,
> >
> > I'd like to propose changing how Dependabot updates our Maven
> dependencies:
> > run weekly instead of daily, group minor and patch updates into one PR,
> and
> > keep major updates as separate PRs. I'd also like to discuss how we
> handle
> > licenses.yaml for these PRs, since that is the most common reason they
> fail
> > and need some manual intervention.
> >
> > This should hopefully help with both the CI load (and mental PR load),
> and
> > also streamline the dependency updates more. My apologies if this has
> been
> > discussed before. I also know that Frank's Codex automation helped
> > tremendously with getting through many of the trivial and not-so-trivial
> > dependency updates, which is great.
> >
> > Why
> >
> > Apart from the obvious answer of "it's a lot!", I had Claude pull some
> > stats (2026-07-22 to 2026-10-22):
> >
> > - 381 of the 696 commits on master were Dependabot PRs.
> > - Dependabot triggered 422 of the 2,389 PR runs of the "Unit &
> Integration
> > tests CI" workflow (about 18%), plus a run of Static Checks and CodeQL
> each
> > time.
> > - One CI pass for a Maven bump PR costs about 450 runner-minutes (median
> of
> > a dozen PRs from September and October), or roughly 14,600
> runner-minutes a
> > week. That is well within the ASF caps [1], but the runner queue is
> shared
> > across the foundation, and each PR also needs a committer to review,
> often
> > fix, and merge it (so additional commits and additional runs).
> > - A single upstream release often arrives as several PRs. gRPC 1.84.1
> > opened four today (#20676 grpc-api, #20679 grpc-bom, #20681 grpc-context,
> > #20677 shade.grpc.version), because those artifacts are pinned in four
> > places.
> >
> > Also, over a third of Dependabot's "Static Checks" runs fail (58 of 159
> > since 2026-09-08, against 6% on master): because licenses.yaml doesn't
> > match, because maven-enforcer throws RequireUpperBoundDeps when an
> artifact
> > moves without its siblings, or because of compile errors (kinda normal,
> at
> > least for major bumps). Separately, about 30% of their unit-test runs
> fail,
> > the same rate as on master, so that's general flaky tests and another
> > problem.
> >
> > Proposal
> >
> > At various projects I work on, we update dependencies weekly, so I was
> > thinking, why don't we do that for apache/druid as well? Idea:
> >
> > 1. Schedule: weekly (Monday) instead of daily. The existing 14-day
> cooldown
> > stays. (Neither applies to security updates: Dependabot opens those when
> > GitHub raises an alert for a vulnerable dependency, and bumps to the
> lowest
> > fixed version [2]. Security updates are already enabled for our repo.)
> > 2. One group for all minor and patch updates, to keep the configuration
> > simple. Not every library follows semantic versioning, so the reviewer of
> > the weekly PR can drop an update that looks risky or breaking (maybe best
> > by adding an ignore rule to the yml file so we can easily track it) and
> > handle it in its own PR. This works well in other codebases I work on.
> > Major updates stay ungrouped, one PR each, since they are the ones most
> > likely to need code changes. The existing maven-resolver group and all
> > ignore rules stay as they are.
> > 3. One group per ecosystem for security updates (Maven, website,
> > web-console, GitHub Actions), so a burst of advisories opens one PR
> instead
> > of one per advisory.
> >
> > Again, I asked Claude to replay this proposal against the last 13 weeks
> of
> > merged bumps: we get about 68 PRs instead of 381, 43 of them majors from
> > the August catch-up. The last seven weeks go from 117 PRs to 17 (rough
> > estimate).
> >
> > ASF Infra's Dependabot guidance [3] suggests weekly updates, grouping,
> and
> > grouped security updates.
> >
> > licenses.yaml
> >
> > licenses.yaml records an exact version for every bundled dependency, and
> > check-licenses matches on group, artifact and version. Every bump of a
> > shipped dependency therefore fails Static Checks until someone edits
> > licenses.yaml by hand: 166 of the 381 Dependabot PRs merged in the last
> 13
> > weeks needed that edit. Grouping cuts it to one edit per week, but here's
> > another idea.
> >
> > Options, in the order I'd prefer them:
> >
> > a) Check versions at release time, not on every PR. In PR CI,
> > check-licenses would keep failing a PR that adds a dependency or changes
> a
> > license, but only warn when a version differs. The release build keeps
> the
> > strict check, so a release still cannot ship a stale LICENSE. Before
> > cutting a release branch, the release manager runs a new script that
> > refreshes the versions in licenses.yaml from the Maven license reports,
> and
> > commits the result to master. Dependabot PRs then pass as they are. Pinot
> > runs Dependabot the same way and refreshes its LICENSE-binary with a
> script
> > at release time [4].
> > b) Keep the strict check, and have a workflow push the licenses.yaml fix
> to
> > the Dependabot branch. This is the most automated option, but I'm not
> sure
> > how easy it is to do. I think we'd have to mint a GH App token instead of
> > the GITHUB_TOKEN so CI would run, and store the app's key as a Dependabot
> > secret, since workflows triggered by Dependabot only see those. We do
> that
> > in private repos, but I'm not sure whether that works on Apache Infra.
> > c) Keep the strict check, and have a workflow post the needed
> licenses.yaml
> > change on each Dependabot PR as a comment. But that would still need
> manual
> > work on each PR, and we might as well not do the additional workflow
> stuff
> > and just run the script locally on that branch quickly.
> >
> > Thoughts?
> >
> > Cheers,
> > Andy
> >
> > [1] https://infra.apache.org/github-actions-policy.html
> > [2]
> >
> >
> https://docs.github.com/en/code-security/concepts/supply-chain-security/dependabot-security-updates
> > [3] https://infra.apache.org/dependabot.html
> > [4] https://github.com/apache/pinot/pull/17818
> >
>

Re: [DISCUSS] Weekly, grouped Dependabot updates to cut PR volume and CI load

Posted by Frank Chen <fr...@apache.org>.
I agree with the proposal.

Reducing the number of these PRs would also save the tokens/time for the
triage automation.

One small problem is that, even for minor upgrades, some libs would have
incompatibility problems like the recent zstd-jni upgrades.
In this case, we have to update the PR manually to exclude these
dependencies.
But I think the odds that we encounter such a problem are small.





On Fri, Oct 23, 2026 at 10:22 AM Andreas Maechler <am...@apache.org>
wrote:

> Hi all,
>
> I'd like to propose changing how Dependabot updates our Maven dependencies:
> run weekly instead of daily, group minor and patch updates into one PR, and
> keep major updates as separate PRs. I'd also like to discuss how we handle
> licenses.yaml for these PRs, since that is the most common reason they fail
> and need some manual intervention.
>
> This should hopefully help with both the CI load (and mental PR load), and
> also streamline the dependency updates more. My apologies if this has been
> discussed before. I also know that Frank's Codex automation helped
> tremendously with getting through many of the trivial and not-so-trivial
> dependency updates, which is great.
>
> Why
>
> Apart from the obvious answer of "it's a lot!", I had Claude pull some
> stats (2026-07-22 to 2026-10-22):
>
> - 381 of the 696 commits on master were Dependabot PRs.
> - Dependabot triggered 422 of the 2,389 PR runs of the "Unit & Integration
> tests CI" workflow (about 18%), plus a run of Static Checks and CodeQL each
> time.
> - One CI pass for a Maven bump PR costs about 450 runner-minutes (median of
> a dozen PRs from September and October), or roughly 14,600 runner-minutes a
> week. That is well within the ASF caps [1], but the runner queue is shared
> across the foundation, and each PR also needs a committer to review, often
> fix, and merge it (so additional commits and additional runs).
> - A single upstream release often arrives as several PRs. gRPC 1.84.1
> opened four today (#20676 grpc-api, #20679 grpc-bom, #20681 grpc-context,
> #20677 shade.grpc.version), because those artifacts are pinned in four
> places.
>
> Also, over a third of Dependabot's "Static Checks" runs fail (58 of 159
> since 2026-09-08, against 6% on master): because licenses.yaml doesn't
> match, because maven-enforcer throws RequireUpperBoundDeps when an artifact
> moves without its siblings, or because of compile errors (kinda normal, at
> least for major bumps). Separately, about 30% of their unit-test runs fail,
> the same rate as on master, so that's general flaky tests and another
> problem.
>
> Proposal
>
> At various projects I work on, we update dependencies weekly, so I was
> thinking, why don't we do that for apache/druid as well? Idea:
>
> 1. Schedule: weekly (Monday) instead of daily. The existing 14-day cooldown
> stays. (Neither applies to security updates: Dependabot opens those when
> GitHub raises an alert for a vulnerable dependency, and bumps to the lowest
> fixed version [2]. Security updates are already enabled for our repo.)
> 2. One group for all minor and patch updates, to keep the configuration
> simple. Not every library follows semantic versioning, so the reviewer of
> the weekly PR can drop an update that looks risky or breaking (maybe best
> by adding an ignore rule to the yml file so we can easily track it) and
> handle it in its own PR. This works well in other codebases I work on.
> Major updates stay ungrouped, one PR each, since they are the ones most
> likely to need code changes. The existing maven-resolver group and all
> ignore rules stay as they are.
> 3. One group per ecosystem for security updates (Maven, website,
> web-console, GitHub Actions), so a burst of advisories opens one PR instead
> of one per advisory.
>
> Again, I asked Claude to replay this proposal against the last 13 weeks of
> merged bumps: we get about 68 PRs instead of 381, 43 of them majors from
> the August catch-up. The last seven weeks go from 117 PRs to 17 (rough
> estimate).
>
> ASF Infra's Dependabot guidance [3] suggests weekly updates, grouping, and
> grouped security updates.
>
> licenses.yaml
>
> licenses.yaml records an exact version for every bundled dependency, and
> check-licenses matches on group, artifact and version. Every bump of a
> shipped dependency therefore fails Static Checks until someone edits
> licenses.yaml by hand: 166 of the 381 Dependabot PRs merged in the last 13
> weeks needed that edit. Grouping cuts it to one edit per week, but here's
> another idea.
>
> Options, in the order I'd prefer them:
>
> a) Check versions at release time, not on every PR. In PR CI,
> check-licenses would keep failing a PR that adds a dependency or changes a
> license, but only warn when a version differs. The release build keeps the
> strict check, so a release still cannot ship a stale LICENSE. Before
> cutting a release branch, the release manager runs a new script that
> refreshes the versions in licenses.yaml from the Maven license reports, and
> commits the result to master. Dependabot PRs then pass as they are. Pinot
> runs Dependabot the same way and refreshes its LICENSE-binary with a script
> at release time [4].
> b) Keep the strict check, and have a workflow push the licenses.yaml fix to
> the Dependabot branch. This is the most automated option, but I'm not sure
> how easy it is to do. I think we'd have to mint a GH App token instead of
> the GITHUB_TOKEN so CI would run, and store the app's key as a Dependabot
> secret, since workflows triggered by Dependabot only see those. We do that
> in private repos, but I'm not sure whether that works on Apache Infra.
> c) Keep the strict check, and have a workflow post the needed licenses.yaml
> change on each Dependabot PR as a comment. But that would still need manual
> work on each PR, and we might as well not do the additional workflow stuff
> and just run the script locally on that branch quickly.
>
> Thoughts?
>
> Cheers,
> Andy
>
> [1] https://infra.apache.org/github-actions-policy.html
> [2]
>
> https://docs.github.com/en/code-security/concepts/supply-chain-security/dependabot-security-updates
> [3] https://infra.apache.org/dependabot.html
> [4] https://github.com/apache/pinot/pull/17818
>