You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by Arnout Engelen <en...@apache.org> on 2025/05/09 11:44:00 UTC
'experimental' GitHub Actions builds
Hello Commons Dev,
I noticed that, for many commons components, our GitHub Actions build
matrix includes building against 'experimental' JVM versions - e.g.
https://github.com/apache/commons-logging/blob/master/.github/workflows/maven.yml#L33-L35
.
These builds commonly fail all the time. While I guess this is fine as
they're 'experimental', I do think it makes the new/casual-contributor
workflow worse, as they also show up as 'build failures' in PRs and it
wouldn't be entirely obvious to newcomers that they're harmless.
I'd like to understand what these jobs are for, and see if we can find an
approach that avoids those distracting build failures?
Kind regards,
--
Arnout Engelen
ASF Security Response
Apache Pekko PMC member, ASF Member
NixOS Committer
Independent Open Source consultant
Re: 'experimental' GitHub Actions builds
Posted by Arnout Engelen <en...@apache.org>.
On Fri, May 9, 2025 at 7:05 PM Gary Gregory <ga...@gmail.com> wrote:
> I personally feel this will just clutter the PR queue for some components
> and force one to go digging for status information which is now readily
> available. The PR will eventually be "lost" in the queue.
While I agree the 'boolean' status information of 'the build fails'
will be less in-your-face, I think the actually useful status
information (what seems to be wrong? are we waiting on some upstream
component? If so, which one? If not, what changes do we need to make
ourselves? is this a regression or did it never work?) will be easier
to collect on a PR.
> A major problem with the PR idea is that it is out of sync with master as
> soon as it is created, unless you recreate the PR or update the branch on
> each commit, which feels like busy work for the CI or a person.
I'm indeed not suggesting 'randomly' updating that PR all the time.
I'd expect it to be pretty rare that an experimental build
'spontaneously' goes green (but maybe I'm wrong there? you definitely
have more experience with this). Given that, updating the PR when
you're actually working on adding the support for a new JDK version
doesn't seem like a big burden?
The current practice of re-running some of these builds that we
already know just fail all the time on all PRs and merges seems "busy
work for the CI" (and distracting for contributors) as well...
> If you want to pick up on Gilles' offer for mathy components, then go for
> it I suppose but I really prefer the current setup for the others.
Happy to start there, but if you're "a priori" rejecting the whole
idea regardless of how that experiment would work out then I won't
pursue it :).
Kind regards,
Arnout
> On Fri, May 9, 2025, 10:27 Arnout Engelen <en...@apache.org> wrote:
>
> > On Fri, May 9, 2025 at 3:33 PM Gary Gregory <ga...@gmail.com>
> > wrote:
> > > I think we need to build with the next EA Java version especially when
> > the
> > > next Java version is an LTS version like 25. This serves as an early
> > > warning signal for work to do but also as playing our part in the FOSS at
> > > large ecosystem where we should report failures to the 3rd party tooling
> > we
> > > use.
> >
> > I agree.
> >
> > > Some other builds are experimental for released versions of Java like 24
> > > because they are known to fail and serve as either a reminder of work to
> > do
> > > or a warning to users.
> >
> > OK
> >
> > > All these GH CI builds show the world what we test and what to expect.
> > >
> > > If a user sees a failing experimental build we can offer it as a to do
> > for
> > > contributions which I've done in the past.
> > >
> > > In general, I think we should build with all LTS versions plus the next
> > EA
> > > version as experimental. Even if an EA build is green it should stay
> > > experimental since the EA behavior could be a moving target.
> > >
> > > IOW, please don't remove experimental builds.
> >
> > Oh I didn't mean to suggest to 'simply' remove the experimental
> > builds. But I think it might be worth considering changing/replacing
> > them.
> >
> > One option would be to, instead of having a failing build in the
> > matrix, have an open draft pull request that adds that (still failing)
> > build.
> >
> > I think a PR does a better job of playing our part in the ecosystem:
> > it allows us to add comments to the pull request highlighting the
> > saillant part of the failure, sharing analysis of the root cause,
> > possible fixes, and links to other projects that may have the same
> > problem (or even a solution).
> >
> > I think it also does a better job of showing the world what we test
> > and what to expect: to the world, a failing job means "this project
> > intended to support this, but it broke, and it seems poorly maintained
> > because nobody fixed the breakage". An open PR conveys "we're still
> > working on this" without us having to explain 'experimental builds'.
> >
> > I think this approach also does a better job of tracking regressions
> > once an experimental build has been green: right now, I bet we
> > wouldn't notice if a once-green experimental build started failing -
> > experimental builds fail all the time anyway. I'd propose we merge the
> > experimental build to the main branch as soon as it's green, and have
> > it fail the build normally. That way, if that happens, we notice and
> > can either fix it directly or remove it from CI (and add another PR to
> > track fixing it). That is a little more work, but valuable in
> > detecting regressions (which we can then report upstream if they turn
> > out to be unintentional).
> >
> > WDYT? Perhaps at least worth trying out on a pilot component?
> >
> >
> > Kind regards,
> >
> > Arnout
> >
> > > On Fri, May 9, 2025, 07:44 Arnout Engelen <en...@apache.org> wrote:
> > >
> > > > Hello Commons Dev,
> > > >
> > > > I noticed that, for many commons components, our GitHub Actions build
> > > > matrix includes building against 'experimental' JVM versions - e.g.
> > > >
> > > >
> > https://github.com/apache/commons-logging/blob/master/.github/workflows/maven.yml#L33-L35
> > > > .
> > > >
> > > > These builds commonly fail all the time. While I guess this is fine as
> > > > they're 'experimental', I do think it makes the new/casual-contributor
> > > > workflow worse, as they also show up as 'build failures' in PRs and it
> > > > wouldn't be entirely obvious to newcomers that they're harmless.
> > > >
> > > > I'd like to understand what these jobs are for, and see if we can find
> > an
> > > > approach that avoids those distracting build failures?
> > > >
> > > >
> > > > Kind regards,
> > > >
> > > > --
> > > > Arnout Engelen
> > > > ASF Security Response
> > > > Apache Pekko PMC member, ASF Member
> > > > NixOS Committer
> > > > Independent Open Source consultant
> > > >
> >
> >
> >
> > --
> > Arnout Engelen
> > ASF Security Response
> > Apache Pekko PMC member, ASF Member
> > NixOS Committer
> > Independent Open Source consultant
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> > For additional commands, e-mail: dev-help@commons.apache.org
> >
> >
--
Arnout Engelen
ASF Security Response
Apache Pekko PMC member, ASF Member
NixOS Committer
Independent Open Source consultant
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: 'experimental' GitHub Actions builds
Posted by Gary Gregory <ga...@gmail.com>.
Hi,
I personally feel this will just clutter the PR queue for some components
and force one to go digging for status information which is now readily
available. The PR will eventually be "lost" in the queue.
I prefer to see all builds in one place which GitHub shows nicely.
A major problem with the PR idea is that it is out of sync with master as
soon as it is created, unless you recreate the PR or update the branch on
each commit, which feels like busy work for the CI or a person.
I think it is simpler to leave a build experimental until a Java version is
LTS.
If you want to pick up on Gilles' offer for mathy components, then go for
it I suppose but I really prefer the current setup for the others.
HTH,
Gary
On Fri, May 9, 2025, 10:27 Arnout Engelen <en...@apache.org> wrote:
> On Fri, May 9, 2025 at 3:33 PM Gary Gregory <ga...@gmail.com>
> wrote:
> > I think we need to build with the next EA Java version especially when
> the
> > next Java version is an LTS version like 25. This serves as an early
> > warning signal for work to do but also as playing our part in the FOSS at
> > large ecosystem where we should report failures to the 3rd party tooling
> we
> > use.
>
> I agree.
>
> > Some other builds are experimental for released versions of Java like 24
> > because they are known to fail and serve as either a reminder of work to
> do
> > or a warning to users.
>
> OK
>
> > All these GH CI builds show the world what we test and what to expect.
> >
> > If a user sees a failing experimental build we can offer it as a to do
> for
> > contributions which I've done in the past.
> >
> > In general, I think we should build with all LTS versions plus the next
> EA
> > version as experimental. Even if an EA build is green it should stay
> > experimental since the EA behavior could be a moving target.
> >
> > IOW, please don't remove experimental builds.
>
> Oh I didn't mean to suggest to 'simply' remove the experimental
> builds. But I think it might be worth considering changing/replacing
> them.
>
> One option would be to, instead of having a failing build in the
> matrix, have an open draft pull request that adds that (still failing)
> build.
>
> I think a PR does a better job of playing our part in the ecosystem:
> it allows us to add comments to the pull request highlighting the
> saillant part of the failure, sharing analysis of the root cause,
> possible fixes, and links to other projects that may have the same
> problem (or even a solution).
>
> I think it also does a better job of showing the world what we test
> and what to expect: to the world, a failing job means "this project
> intended to support this, but it broke, and it seems poorly maintained
> because nobody fixed the breakage". An open PR conveys "we're still
> working on this" without us having to explain 'experimental builds'.
>
> I think this approach also does a better job of tracking regressions
> once an experimental build has been green: right now, I bet we
> wouldn't notice if a once-green experimental build started failing -
> experimental builds fail all the time anyway. I'd propose we merge the
> experimental build to the main branch as soon as it's green, and have
> it fail the build normally. That way, if that happens, we notice and
> can either fix it directly or remove it from CI (and add another PR to
> track fixing it). That is a little more work, but valuable in
> detecting regressions (which we can then report upstream if they turn
> out to be unintentional).
>
> WDYT? Perhaps at least worth trying out on a pilot component?
>
>
> Kind regards,
>
> Arnout
>
> > On Fri, May 9, 2025, 07:44 Arnout Engelen <en...@apache.org> wrote:
> >
> > > Hello Commons Dev,
> > >
> > > I noticed that, for many commons components, our GitHub Actions build
> > > matrix includes building against 'experimental' JVM versions - e.g.
> > >
> > >
> https://github.com/apache/commons-logging/blob/master/.github/workflows/maven.yml#L33-L35
> > > .
> > >
> > > These builds commonly fail all the time. While I guess this is fine as
> > > they're 'experimental', I do think it makes the new/casual-contributor
> > > workflow worse, as they also show up as 'build failures' in PRs and it
> > > wouldn't be entirely obvious to newcomers that they're harmless.
> > >
> > > I'd like to understand what these jobs are for, and see if we can find
> an
> > > approach that avoids those distracting build failures?
> > >
> > >
> > > Kind regards,
> > >
> > > --
> > > Arnout Engelen
> > > ASF Security Response
> > > Apache Pekko PMC member, ASF Member
> > > NixOS Committer
> > > Independent Open Source consultant
> > >
>
>
>
> --
> Arnout Engelen
> ASF Security Response
> Apache Pekko PMC member, ASF Member
> NixOS Committer
> Independent Open Source consultant
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
>
Re: 'experimental' GitHub Actions builds
Posted by Gilles Sadowski <gi...@gmail.com>.
Le ven. 9 mai 2025 à 16:27, Arnout Engelen <en...@apache.org> a écrit :
>
> On Fri, May 9, 2025 at 3:33 PM Gary Gregory <ga...@gmail.com> wrote:
> > I think we need to build with the next EA Java version especially when the
> > next Java version is an LTS version like 25. This serves as an early
> > warning signal for work to do but also as playing our part in the FOSS at
> > large ecosystem where we should report failures to the 3rd party tooling we
> > use.
>
> I agree.
>
> > Some other builds are experimental for released versions of Java like 24
> > because they are known to fail and serve as either a reminder of work to do
> > or a warning to users.
>
> OK
>
> > All these GH CI builds show the world what we test and what to expect.
> >
> > If a user sees a failing experimental build we can offer it as a to do for
> > contributions which I've done in the past.
> >
> > In general, I think we should build with all LTS versions plus the next EA
> > version as experimental. Even if an EA build is green it should stay
> > experimental since the EA behavior could be a moving target.
> >
> > IOW, please don't remove experimental builds.
>
> Oh I didn't mean to suggest to 'simply' remove the experimental
> builds. But I think it might be worth considering changing/replacing
> them.
>
> One option would be to, instead of having a failing build in the
> matrix, have an open draft pull request that adds that (still failing)
> build.
>
> I think a PR does a better job of playing our part in the ecosystem:
> it allows us to add comments to the pull request highlighting the
> saillant part of the failure, sharing analysis of the root cause,
> possible fixes, and links to other projects that may have the same
> problem (or even a solution).
>
> I think it also does a better job of showing the world what we test
> and what to expect: to the world, a failing job means "this project
> intended to support this, but it broke, and it seems poorly maintained
> because nobody fixed the breakage". An open PR conveys "we're still
> working on this" without us having to explain 'experimental builds'.
>
> I think this approach also does a better job of tracking regressions
> once an experimental build has been green: right now, I bet we
> wouldn't notice if a once-green experimental build started failing -
> experimental builds fail all the time anyway. I'd propose we merge the
> experimental build to the main branch as soon as it's green, and have
> it fail the build normally. That way, if that happens, we notice and
> can either fix it directly or remove it from CI (and add another PR to
> track fixing it). That is a little more work, but valuable in
> detecting regressions (which we can then report upstream if they turn
> out to be unintentional).
>
> WDYT? Perhaps at least worth trying out on a pilot component?
I did not understand everything, but it sounds like a well ironed out plan.
;-)
For experimenting, do you need components that change often, or that
are fairly stable? If the latter, you are welcome to use any (or all of) the
math-related ones:
* RNG
* Numbers
* Geometry
* Statistics
* Math
Somewhat unrelated (or maybe not), I wonder whether benchmarks
(designed by Alex) in these components could be performed automatically,
and a slow-down would count as regression.
Regards,
Gilles
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: 'experimental' GitHub Actions builds
Posted by Arnout Engelen <en...@apache.org>.
On Fri, May 9, 2025 at 3:33 PM Gary Gregory <ga...@gmail.com> wrote:
> I think we need to build with the next EA Java version especially when the
> next Java version is an LTS version like 25. This serves as an early
> warning signal for work to do but also as playing our part in the FOSS at
> large ecosystem where we should report failures to the 3rd party tooling we
> use.
I agree.
> Some other builds are experimental for released versions of Java like 24
> because they are known to fail and serve as either a reminder of work to do
> or a warning to users.
OK
> All these GH CI builds show the world what we test and what to expect.
>
> If a user sees a failing experimental build we can offer it as a to do for
> contributions which I've done in the past.
>
> In general, I think we should build with all LTS versions plus the next EA
> version as experimental. Even if an EA build is green it should stay
> experimental since the EA behavior could be a moving target.
>
> IOW, please don't remove experimental builds.
Oh I didn't mean to suggest to 'simply' remove the experimental
builds. But I think it might be worth considering changing/replacing
them.
One option would be to, instead of having a failing build in the
matrix, have an open draft pull request that adds that (still failing)
build.
I think a PR does a better job of playing our part in the ecosystem:
it allows us to add comments to the pull request highlighting the
saillant part of the failure, sharing analysis of the root cause,
possible fixes, and links to other projects that may have the same
problem (or even a solution).
I think it also does a better job of showing the world what we test
and what to expect: to the world, a failing job means "this project
intended to support this, but it broke, and it seems poorly maintained
because nobody fixed the breakage". An open PR conveys "we're still
working on this" without us having to explain 'experimental builds'.
I think this approach also does a better job of tracking regressions
once an experimental build has been green: right now, I bet we
wouldn't notice if a once-green experimental build started failing -
experimental builds fail all the time anyway. I'd propose we merge the
experimental build to the main branch as soon as it's green, and have
it fail the build normally. That way, if that happens, we notice and
can either fix it directly or remove it from CI (and add another PR to
track fixing it). That is a little more work, but valuable in
detecting regressions (which we can then report upstream if they turn
out to be unintentional).
WDYT? Perhaps at least worth trying out on a pilot component?
Kind regards,
Arnout
> On Fri, May 9, 2025, 07:44 Arnout Engelen <en...@apache.org> wrote:
>
> > Hello Commons Dev,
> >
> > I noticed that, for many commons components, our GitHub Actions build
> > matrix includes building against 'experimental' JVM versions - e.g.
> >
> > https://github.com/apache/commons-logging/blob/master/.github/workflows/maven.yml#L33-L35
> > .
> >
> > These builds commonly fail all the time. While I guess this is fine as
> > they're 'experimental', I do think it makes the new/casual-contributor
> > workflow worse, as they also show up as 'build failures' in PRs and it
> > wouldn't be entirely obvious to newcomers that they're harmless.
> >
> > I'd like to understand what these jobs are for, and see if we can find an
> > approach that avoids those distracting build failures?
> >
> >
> > Kind regards,
> >
> > --
> > Arnout Engelen
> > ASF Security Response
> > Apache Pekko PMC member, ASF Member
> > NixOS Committer
> > Independent Open Source consultant
> >
--
Arnout Engelen
ASF Security Response
Apache Pekko PMC member, ASF Member
NixOS Committer
Independent Open Source consultant
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: 'experimental' GitHub Actions builds
Posted by Gary Gregory <ga...@gmail.com>.
Hi all,
I think we need to build with the next EA Java version especially when the
next Java version is an LTS version like 25. This serves as an early
warning signal for work to do but also as playing out part in the FOSS at
large ecosystem where we should report failures to the 3rd party tooling we
use.
Some other builds are experimental for released versions of Java like 24
because they are known to fail and serve as either a reminder of work to do
or a warning to users.
An failing experimental build is OK IMO. GH knows the difference and
doesn't send a failure email if all production builds are green and an
experimental build is red.
All these GH CI builds show the world what we test and what to expect.
If a user sees a failing experimental build we can offer it as a to do for
contributions which I've done in the past.
In general, I think we should build with all LTS versions plus the next EA
version as experimental. Even if an EA build is green it should stay
experimental since the EA behavior could be a moving target.
IOW, please don't remove experimental builds.
HTH,
Gary
On Fri, May 9, 2025, 07:44 Arnout Engelen <en...@apache.org> wrote:
> Hello Commons Dev,
>
> I noticed that, for many commons components, our GitHub Actions build
> matrix includes building against 'experimental' JVM versions - e.g.
>
> https://github.com/apache/commons-logging/blob/master/.github/workflows/maven.yml#L33-L35
> .
>
> These builds commonly fail all the time. While I guess this is fine as
> they're 'experimental', I do think it makes the new/casual-contributor
> workflow worse, as they also show up as 'build failures' in PRs and it
> wouldn't be entirely obvious to newcomers that they're harmless.
>
> I'd like to understand what these jobs are for, and see if we can find an
> approach that avoids those distracting build failures?
>
>
> Kind regards,
>
> --
> Arnout Engelen
> ASF Security Response
> Apache Pekko PMC member, ASF Member
> NixOS Committer
> Independent Open Source consultant
>