You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by "Piotr P. Karwasz" <pi...@mailing.copernik.eu> on 2025/11/05 11:24:21 UTC
Introduce reusable workflows
Hi all,
Maintaining CI workflows across more than 40 Commons repositories
creates significant maintenance overhead and Dependabot noise.
I’d like to propose refactoring our four common workflows, CodeQL
Analysis, Dependency Review, Java CI, and Scorecards Analysis,
to use reusable workflows defined in `commons-parent` instead.
As an example, I’ve opened commons-parent#681 [1], which refactors
maven.yml, and triggered a demo run in commons-lang [2]. If there’s
agreement, I can refactor the remaining three workflows as well.
Adopting shared workflows raises the question of how they should be
updated across projects. Some existing approaches in Apache projects
include:
1. Pinning to a commit (SHA-1): reliable but reintroduces Dependabot
churn.
2. Pinning to release tags: used by the Logging Services PMC. Updates
happen with parent releases, useful if workflow and POM changes must
align.
3. Branch-based sharing: used by the Maven PMC in
`maven-gh-actions-shared` [3], where branches correspond to workflow
major versions.
To start simple, I suggest we reference the master branch of
commons-parent. This provides automatic propagation of workflow
improvements with zero maintenance effort in downstream repositories.
What do you think?
Piotr
[1] https://github.com/apache/commons-parent/pull/681
[2] https://github.com/apache/commons-lang/actions/runs/19098960862
[3] https://github.com/apache/maven-gh-actions-shared
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: Introduce reusable workflows
Posted by Mike Drob <md...@mdrob.com>.
Conceptually, I am in favor of this. The reusable workflows feel similar to
having a maven parent pom, you do things in one place and it works for the
majority of our projects and then where it doesn't we make a tweak or
deviate. I haven't looked at the specific implementation here, but the
argument that you have to look in a different repo doesn't really sway me
much - you already have to do this for maven poms anyway.
On Wed, Nov 5, 2025 at 1:30 PM Piotr P. Karwasz <pi...@mailing.copernik.eu>
wrote:
> Hi Gary,
>
> On 5.11.2025 16:40, Gary Gregory wrote:
> > Another level of delegation just to build on GH CI is a complication I
> have
> > no need for, YMMV. I don't see how this makes building better or easier.
> >
> > Seeing the GH CI build in one small file is straight forward and
> > _easy to understand_.
>
> It should make maintenance easier: you don't get 40 Dependabot PRs for
> each action upgrade in 40 repositories, but just a single PR.
>
> The workflows would still be in a single file, just not in the same
> repository. Personally I don't look at workflow files too often, so it
> should not be a problem to look into another repo from time to time.
>
>
> > From what I see, all components should ignore this maven configuration
> > because it doesn't serve the need of any component. All components build
> on
> > Java 26-ea as an experimental build. This configuration explicitly turns
> > this off because... PMD doesn't support Java 26 yet? Not only is PMD not
> > used by all default Maven goals but this is how we find out what's
> working
> > in our stuff and in the FOSS ecosystem at large. We can then give
> feedback.
> > Which I do.
>
>
> I copied the part about the PMD from commons-compress#724 [1] and I
> think the details of the PoC can be discussed further in
> commons-parent#681 [2].
>
> My main point here is to highlight how reusable workflows make these
> kinds of decisions much easier. Instead of adding a new Java version to
> 40 separate workflow files every six months, we can update it once in a
> single shared workflow.
>
> > We have many components that have a special tweak to their GH CI
> > configuration like setting this or that command line argument. We have
> > others that are a complex world of their own. Trying to create reuse in
> GH
> > CI just makes it even _more_ complex _and_ harder to debug builds than
> they
> > already are. Debugging a CI build on GH doesn't happen often but it's
> never
> > simple. Adding another layer doesn't make it easier.
>
>
> Sure, there are differences between the GitHub workflows, but they don’t
> always reflect real requirements of each repository. In many cases,
> those workflows simply weren’t updated in sync with others.
>
> Workflows like CodeQL (for GitHub Actions and Java), Dependency Review,
> and Scorecards *should* be universal. I agree that the Maven workflow
> may need some project-specific tweaks, but by using a reusable workflow
> we can ensure that build parameters come from a consistent, well-defined
> set of sources. For example, if special parameters are needed, we can
> agree, and document, that they should be provided:
>
> - either in `.mvn/maven.config` and `.mvn/jvm.config`, which also
> benefits local builds,
> - or via `MAVEN_ARGS` and `MAVEN_OPTS`,
>
> but not both.
>
>
> > After much delay, we finally have a dependabot email list, so the "noise"
> > argument is moot IMO.
> >
> > We should only make the GH CI builds more complex if we must IMO.
>
>
> The `dependabot@` mailing list won’t eliminate all the “noise” from
> Dependabot activity: it only receives messages caused by Dependabot, not
> the human interactions that follow.
>
> Even setting that aside, the time you (and others) spend handling
> Dependabot PRs manually could be used much more effectively. I’d rather
> see your expertise applied to improving our code and infrastructure than
> to clicking “Merge” on routine updates.
>
> Piotr
>
> [1] https://github.com/apache/commons-compress/pull/724
> [2] https://github.com/apache/commons-parent/pull/681
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
>
Re: Introduce reusable workflows
Posted by Alex Herbert <al...@gmail.com>.
The maven build for each repository can differ. For example the
configuration used for the multi-module repositories (Statistics; RNG;
Numbers) is more intricate due to the way the build occurs across the
module dependencies. I also do not think that building on 5 JDKs (6 with
EA) and 3 platforms is required for all repositories. Code that interacts
with the platform would benefit; for others the compute resource is better
shared with other users and repositories.
I do see the benefit of a single common file for codeql and scorecard
analysis where configuration is more standard.
Alex
On Wed, 5 Nov 2025 at 22:32, Gary Gregory <ga...@gmail.com> wrote:
> On Wed, Nov 5, 2025, 14:30 Piotr P. Karwasz <pi...@mailing.copernik.eu>
> wrote:
>
> > Hi Gary,
> >
> > On 5.11.2025 16:40, Gary Gregory wrote:
> > > Another level of delegation just to build on GH CI is a complication I
> > have
> > > no need for, YMMV. I don't see how this makes building better or
> easier.
> > >
> > > Seeing the GH CI build in one small file is straight forward and
> > > _easy to understand_.
> >
> > It should make maintenance easier: you don't get 40 Dependabot PRs for
> > each action upgrade in 40 repositories, but just a single PR.
> >
>
>
> You're confusing build maintenance and email box maintenance.
>
> TLDR.
>
> Gary
>
> >
> > The workflows would still be in a single file, just not in the same
> > repository. Personally I don't look at workflow files too often, so it
> > should not be a problem to look into another repo from time to time.
> >
> >
> > > From what I see, all components should ignore this maven configuration
> > > because it doesn't serve the need of any component. All components
> build
> > on
> > > Java 26-ea as an experimental build. This configuration explicitly
> turns
> > > this off because... PMD doesn't support Java 26 yet? Not only is PMD
> not
> > > used by all default Maven goals but this is how we find out what's
> > working
> > > in our stuff and in the FOSS ecosystem at large. We can then give
> > feedback.
> > > Which I do.
> >
> >
> > I copied the part about the PMD from commons-compress#724 [1] and I
> > think the details of the PoC can be discussed further in
> > commons-parent#681 [2].
> >
> > My main point here is to highlight how reusable workflows make these
> > kinds of decisions much easier. Instead of adding a new Java version to
> > 40 separate workflow files every six months, we can update it once in a
> > single shared workflow.
> >
> > > We have many components that have a special tweak to their GH CI
> > > configuration like setting this or that command line argument. We have
> > > others that are a complex world of their own. Trying to create reuse in
> > GH
> > > CI just makes it even _more_ complex _and_ harder to debug builds than
> > they
> > > already are. Debugging a CI build on GH doesn't happen often but it's
> > never
> > > simple. Adding another layer doesn't make it easier.
> >
> >
> > Sure, there are differences between the GitHub workflows, but they don’t
> > always reflect real requirements of each repository. In many cases,
> > those workflows simply weren’t updated in sync with others.
> >
> > Workflows like CodeQL (for GitHub Actions and Java), Dependency Review,
> > and Scorecards *should* be universal. I agree that the Maven workflow
> > may need some project-specific tweaks, but by using a reusable workflow
> > we can ensure that build parameters come from a consistent, well-defined
> > set of sources. For example, if special parameters are needed, we can
> > agree, and document, that they should be provided:
> >
> > - either in `.mvn/maven.config` and `.mvn/jvm.config`, which also
> > benefits local builds,
> > - or via `MAVEN_ARGS` and `MAVEN_OPTS`,
> >
> > but not both.
> >
> >
> > > After much delay, we finally have a dependabot email list, so the
> "noise"
> > > argument is moot IMO.
> > >
> > > We should only make the GH CI builds more complex if we must IMO.
> >
> >
> > The `dependabot@` mailing list won’t eliminate all the “noise” from
> > Dependabot activity: it only receives messages caused by Dependabot, not
> > the human interactions that follow.
> >
> > Even setting that aside, the time you (and others) spend handling
> > Dependabot PRs manually could be used much more effectively. I’d rather
> > see your expertise applied to improving our code and infrastructure than
> > to clicking “Merge” on routine updates.
> >
> > Piotr
> >
> > [1] https://github.com/apache/commons-compress/pull/724
> > [2] https://github.com/apache/commons-parent/pull/681
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> > For additional commands, e-mail: dev-help@commons.apache.org
> >
> >
>
Re: Introduce reusable workflows
Posted by Gary Gregory <ga...@gmail.com>.
On Wed, Nov 5, 2025, 14:30 Piotr P. Karwasz <pi...@mailing.copernik.eu>
wrote:
> Hi Gary,
>
> On 5.11.2025 16:40, Gary Gregory wrote:
> > Another level of delegation just to build on GH CI is a complication I
> have
> > no need for, YMMV. I don't see how this makes building better or easier.
> >
> > Seeing the GH CI build in one small file is straight forward and
> > _easy to understand_.
>
> It should make maintenance easier: you don't get 40 Dependabot PRs for
> each action upgrade in 40 repositories, but just a single PR.
>
You're confusing build maintenance and email box maintenance.
TLDR.
Gary
>
> The workflows would still be in a single file, just not in the same
> repository. Personally I don't look at workflow files too often, so it
> should not be a problem to look into another repo from time to time.
>
>
> > From what I see, all components should ignore this maven configuration
> > because it doesn't serve the need of any component. All components build
> on
> > Java 26-ea as an experimental build. This configuration explicitly turns
> > this off because... PMD doesn't support Java 26 yet? Not only is PMD not
> > used by all default Maven goals but this is how we find out what's
> working
> > in our stuff and in the FOSS ecosystem at large. We can then give
> feedback.
> > Which I do.
>
>
> I copied the part about the PMD from commons-compress#724 [1] and I
> think the details of the PoC can be discussed further in
> commons-parent#681 [2].
>
> My main point here is to highlight how reusable workflows make these
> kinds of decisions much easier. Instead of adding a new Java version to
> 40 separate workflow files every six months, we can update it once in a
> single shared workflow.
>
> > We have many components that have a special tweak to their GH CI
> > configuration like setting this or that command line argument. We have
> > others that are a complex world of their own. Trying to create reuse in
> GH
> > CI just makes it even _more_ complex _and_ harder to debug builds than
> they
> > already are. Debugging a CI build on GH doesn't happen often but it's
> never
> > simple. Adding another layer doesn't make it easier.
>
>
> Sure, there are differences between the GitHub workflows, but they don’t
> always reflect real requirements of each repository. In many cases,
> those workflows simply weren’t updated in sync with others.
>
> Workflows like CodeQL (for GitHub Actions and Java), Dependency Review,
> and Scorecards *should* be universal. I agree that the Maven workflow
> may need some project-specific tweaks, but by using a reusable workflow
> we can ensure that build parameters come from a consistent, well-defined
> set of sources. For example, if special parameters are needed, we can
> agree, and document, that they should be provided:
>
> - either in `.mvn/maven.config` and `.mvn/jvm.config`, which also
> benefits local builds,
> - or via `MAVEN_ARGS` and `MAVEN_OPTS`,
>
> but not both.
>
>
> > After much delay, we finally have a dependabot email list, so the "noise"
> > argument is moot IMO.
> >
> > We should only make the GH CI builds more complex if we must IMO.
>
>
> The `dependabot@` mailing list won’t eliminate all the “noise” from
> Dependabot activity: it only receives messages caused by Dependabot, not
> the human interactions that follow.
>
> Even setting that aside, the time you (and others) spend handling
> Dependabot PRs manually could be used much more effectively. I’d rather
> see your expertise applied to improving our code and infrastructure than
> to clicking “Merge” on routine updates.
>
> Piotr
>
> [1] https://github.com/apache/commons-compress/pull/724
> [2] https://github.com/apache/commons-parent/pull/681
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
>
Re: Introduce reusable workflows
Posted by "Piotr P. Karwasz" <pi...@mailing.copernik.eu>.
Hi Gary,
On 5.11.2025 16:40, Gary Gregory wrote:
> Another level of delegation just to build on GH CI is a complication I have
> no need for, YMMV. I don't see how this makes building better or easier.
>
> Seeing the GH CI build in one small file is straight forward and
> _easy to understand_.
It should make maintenance easier: you don't get 40 Dependabot PRs for
each action upgrade in 40 repositories, but just a single PR.
The workflows would still be in a single file, just not in the same
repository. Personally I don't look at workflow files too often, so it
should not be a problem to look into another repo from time to time.
> From what I see, all components should ignore this maven configuration
> because it doesn't serve the need of any component. All components build on
> Java 26-ea as an experimental build. This configuration explicitly turns
> this off because... PMD doesn't support Java 26 yet? Not only is PMD not
> used by all default Maven goals but this is how we find out what's working
> in our stuff and in the FOSS ecosystem at large. We can then give feedback.
> Which I do.
I copied the part about the PMD from commons-compress#724 [1] and I
think the details of the PoC can be discussed further in
commons-parent#681 [2].
My main point here is to highlight how reusable workflows make these
kinds of decisions much easier. Instead of adding a new Java version to
40 separate workflow files every six months, we can update it once in a
single shared workflow.
> We have many components that have a special tweak to their GH CI
> configuration like setting this or that command line argument. We have
> others that are a complex world of their own. Trying to create reuse in GH
> CI just makes it even _more_ complex _and_ harder to debug builds than they
> already are. Debugging a CI build on GH doesn't happen often but it's never
> simple. Adding another layer doesn't make it easier.
Sure, there are differences between the GitHub workflows, but they don’t
always reflect real requirements of each repository. In many cases,
those workflows simply weren’t updated in sync with others.
Workflows like CodeQL (for GitHub Actions and Java), Dependency Review,
and Scorecards *should* be universal. I agree that the Maven workflow
may need some project-specific tweaks, but by using a reusable workflow
we can ensure that build parameters come from a consistent, well-defined
set of sources. For example, if special parameters are needed, we can
agree, and document, that they should be provided:
- either in `.mvn/maven.config` and `.mvn/jvm.config`, which also
benefits local builds,
- or via `MAVEN_ARGS` and `MAVEN_OPTS`,
but not both.
> After much delay, we finally have a dependabot email list, so the "noise"
> argument is moot IMO.
>
> We should only make the GH CI builds more complex if we must IMO.
The `dependabot@` mailing list won’t eliminate all the “noise” from
Dependabot activity: it only receives messages caused by Dependabot, not
the human interactions that follow.
Even setting that aside, the time you (and others) spend handling
Dependabot PRs manually could be used much more effectively. I’d rather
see your expertise applied to improving our code and infrastructure than
to clicking “Merge” on routine updates.
Piotr
[1] https://github.com/apache/commons-compress/pull/724
[2] https://github.com/apache/commons-parent/pull/681
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: Introduce reusable workflows
Posted by sebb <se...@gmail.com>.
On Wed, 5 Nov 2025 at 15:40, Gary Gregory <ga...@gmail.com> wrote:
>
> Another level of delegation just to build on GH CI is a complication I have
> no need for, YMMV. I don't see how this makes building better or easier.
>
> Seeing the GH CI build in one small file is straight forward and _easy to
> understand_.
>
> From what I see, all components should ignore this maven configuration
> because it doesn't serve the need of any component. All components build on
> Java 26-ea as an experimental build. This configuration explicitly turns
> this off because... PMD doesn't support Java 26 yet? Not only is PMD not
> used by all default Maven goals but this is how we find out what's working
> in our stuff and in the FOSS ecosystem at large. We can then give feedback.
> Which I do.
>
> We have many components that have a special tweak to their GH CI
> configuration like setting this or that command line argument. We have
> others that are a complex world of their own. Trying to create reuse in GH
> CI just makes it even _more_ complex _and_ harder to debug builds than they
> already are. Debugging a CI build on GH doesn't happen often but it's never
> simple. Adding another layer doesn't make it easier.
Agreed.
> After much delay, we finally have a dependabot email list, so the "noise"
> argument is moot IMO.
That remains to be seen; it's not currently possible to direct
Dependabot commit emails to the new list.
> We should only make the GH CI builds more complex if we must IMO.
>
> Huh,
> Gary
>
>
>
> On Wed, Nov 5, 2025, 06:24 Piotr P. Karwasz <pi...@mailing.copernik.eu>
> wrote:
>
> >
> > Hi all,
> >
> > Maintaining CI workflows across more than 40 Commons repositories
> > creates significant maintenance overhead and Dependabot noise.
> > I’d like to propose refactoring our four common workflows, CodeQL
> > Analysis, Dependency Review, Java CI, and Scorecards Analysis,
> > to use reusable workflows defined in `commons-parent` instead.
> >
> > As an example, I’ve opened commons-parent#681 [1], which refactors
> > maven.yml, and triggered a demo run in commons-lang [2]. If there’s
> > agreement, I can refactor the remaining three workflows as well.
> >
> > Adopting shared workflows raises the question of how they should be
> > updated across projects. Some existing approaches in Apache projects
> > include:
> >
> > 1. Pinning to a commit (SHA-1): reliable but reintroduces Dependabot
> > churn.
> > 2. Pinning to release tags: used by the Logging Services PMC. Updates
> > happen with parent releases, useful if workflow and POM changes must
> > align.
> > 3. Branch-based sharing: used by the Maven PMC in
> > `maven-gh-actions-shared` [3], where branches correspond to workflow
> > major versions.
> >
> > To start simple, I suggest we reference the master branch of
> > commons-parent. This provides automatic propagation of workflow
> > improvements with zero maintenance effort in downstream repositories.
> >
> > What do you think?
> >
> > Piotr
> >
> > [1] https://github.com/apache/commons-parent/pull/681
> > [2] https://github.com/apache/commons-lang/actions/runs/19098960862
> > [3] https://github.com/apache/maven-gh-actions-shared
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> > For additional commands, e-mail: dev-help@commons.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: Introduce reusable workflows
Posted by Gary Gregory <ga...@gmail.com>.
Another level of delegation just to build on GH CI is a complication I have
no need for, YMMV. I don't see how this makes building better or easier.
Seeing the GH CI build in one small file is straight forward and _easy to
understand_.
From what I see, all components should ignore this maven configuration
because it doesn't serve the need of any component. All components build on
Java 26-ea as an experimental build. This configuration explicitly turns
this off because... PMD doesn't support Java 26 yet? Not only is PMD not
used by all default Maven goals but this is how we find out what's working
in our stuff and in the FOSS ecosystem at large. We can then give feedback.
Which I do.
We have many components that have a special tweak to their GH CI
configuration like setting this or that command line argument. We have
others that are a complex world of their own. Trying to create reuse in GH
CI just makes it even _more_ complex _and_ harder to debug builds than they
already are. Debugging a CI build on GH doesn't happen often but it's never
simple. Adding another layer doesn't make it easier.
After much delay, we finally have a dependabot email list, so the "noise"
argument is moot IMO.
We should only make the GH CI builds more complex if we must IMO.
Huh,
Gary
On Wed, Nov 5, 2025, 06:24 Piotr P. Karwasz <pi...@mailing.copernik.eu>
wrote:
>
> Hi all,
>
> Maintaining CI workflows across more than 40 Commons repositories
> creates significant maintenance overhead and Dependabot noise.
> I’d like to propose refactoring our four common workflows, CodeQL
> Analysis, Dependency Review, Java CI, and Scorecards Analysis,
> to use reusable workflows defined in `commons-parent` instead.
>
> As an example, I’ve opened commons-parent#681 [1], which refactors
> maven.yml, and triggered a demo run in commons-lang [2]. If there’s
> agreement, I can refactor the remaining three workflows as well.
>
> Adopting shared workflows raises the question of how they should be
> updated across projects. Some existing approaches in Apache projects
> include:
>
> 1. Pinning to a commit (SHA-1): reliable but reintroduces Dependabot
> churn.
> 2. Pinning to release tags: used by the Logging Services PMC. Updates
> happen with parent releases, useful if workflow and POM changes must
> align.
> 3. Branch-based sharing: used by the Maven PMC in
> `maven-gh-actions-shared` [3], where branches correspond to workflow
> major versions.
>
> To start simple, I suggest we reference the master branch of
> commons-parent. This provides automatic propagation of workflow
> improvements with zero maintenance effort in downstream repositories.
>
> What do you think?
>
> Piotr
>
> [1] https://github.com/apache/commons-parent/pull/681
> [2] https://github.com/apache/commons-lang/actions/runs/19098960862
> [3] https://github.com/apache/maven-gh-actions-shared
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
>