Broken CI

[Gary Gregory <ga...@gmail.com>]

After getting https://github.com/apache/logging-log4j2/pull/646 in what I
think is a good state, I have no idea if it is safe or not to merge because
the 1st build GitHub shows is red:
https://github.com/apache/logging-log4j2/runs/4589771553?check_suite_focus=true

I don't use GH actions the way we do here and I'm not sure why we need the
publish test result set when anyone can just look at the build logs.

Gary

Re: Broken CI

[Tim Perry <ti...@gmail.com>]

My issues were all on master. I've always been able to work on release-2.x
in IntelliJ or Eclipse without any issue.

I'll have to check out master and see if it works now. I have it on the
back burner to try to figure out an improved status logger life cycle.

On Thu, Dec 23, 2021 at 9:34 AM Carter Kozak <ck...@ckozak.net> wrote:

> I haven't had a test flake locally in the last 6 months (at least), if we
> see flaky tests I'm in favor of fixing them rather than working around them.
>
> fwiw the non GitHub-actions tests work great on the release-2.x branch in
> my experience, when they aren't overwhelmed accessing build nodes anyhow.
> That said, I would prefer to get everything workin on GitHub actions as it
> seems to be the gold standard these days.
>
> > Last time I tried I couldn't get the mainline code to load in IntelliJ
> or Eclipse because of the packing changes that were in progress.
>
> I find release-2.x works nicely with IntelliJ idea when I set the
> project-level sdk to jdk8. the master branch may be another story. I have
> an INFRA ticket open to switch the default branch to reduce confusion:
> https://issues.apache.org/jira/browse/INFRA-22641
>
> -ck
>
> On Thu, Dec 23, 2021, at 12:25, Tim Perry wrote:
> > Is it worth marching those tests with @Ignore and filing a Jira for each
> > one? That does seem to set a bad precedent though.
> >
> > Last time I tried I couldn't get the mainline code to load in IntelliJ or
> > Eclipse because of the packing changes that were in progress. Did that
> get
> > fixed? If so, I might be able to carve out some time to look at a couple
> > tests if you point me in the right direction.
> >
> > Tim
> >
> > On Thu, Dec 23, 2021 at 7:24 AM Volkan Yazıcı <vo...@yazi.ci> wrote:
> >
> > > Since, there are some tests which occasionally constantly fail, we use
> > > `-Dmaven.test.failure.ignore=true` in `verify` goal. This causes a
> build
> > > with test failures to be marked green. The 3rd party component,
> > > `scacap/action-surefire-report@v1`, is used to feed these failures
> back
> > > into GitHub Actions status with some pretty printing. But since the PR
> is
> > > opened by a user who doesn't have commit rights, the 3rd party
> component
> > > fails to mark the failures due to insufficient privileges. In
> > > `.github/workflows/main.yml`, I had introduced the `if:
> github.repository
> > > == 'apache/logging-log4j2'` line to work around this, but apparently
> it is
> > > broken again.
> > >
> > > I totally share your frustration, same here. Though sparing time to fix
> > > this is pretty difficult nowadays.
> > >
> > > I also need to confess, in those brief moments of insanity, I
> contemplate
> > > nuking all those flaky tests. This will simplify the CI magic a lot and
> > > enhance our confidence in the tests.
> > >
> > > On Tue, Dec 21, 2021 at 3:10 AM Gary Gregory <ga...@gmail.com>
> > > wrote:
> > >
> > > > After getting https://github.com/apache/logging-log4j2/pull/646 in
> what
> > > I
> > > > think is a good state, I have no idea if it is safe or not to merge
> > > because
> > > > the 1st build GitHub shows is red:
> > > >
> > > >
> > >
> https://github.com/apache/logging-log4j2/runs/4589771553?check_suite_focus=true
> > > >
> > > > I don't use GH actions the way we do here and I'm not sure why we
> need
> > > the
> > > > publish test result set when anyone can just look at the build logs.
> > > >
> > > > Gary
> > > >
> > >
> >
>

Re: Broken CI

[Gary Gregory <ga...@gmail.com>]

I rarely have issues running tests from the mvn command line. Whatever we
have set up in GitHub seems quite complicated compared to most projects
I've seen. From Eclipse, sometimes tests fail, but I never researched it,
the mvm cmd line is what matters most anyway.

G

Gary

On Thu, Dec 23, 2021, 12:34 Carter Kozak <ck...@ckozak.net> wrote:

> I haven't had a test flake locally in the last 6 months (at least), if we
> see flaky tests I'm in favor of fixing them rather than working around them.
>
> fwiw the non GitHub-actions tests work great on the release-2.x branch in
> my experience, when they aren't overwhelmed accessing build nodes anyhow.
> That said, I would prefer to get everything workin on GitHub actions as it
> seems to be the gold standard these days.
>
> > Last time I tried I couldn't get the mainline code to load in IntelliJ
> or Eclipse because of the packing changes that were in progress.
>
> I find release-2.x works nicely with IntelliJ idea when I set the
> project-level sdk to jdk8. the master branch may be another story. I have
> an INFRA ticket open to switch the default branch to reduce confusion:
> https://issues.apache.org/jira/browse/INFRA-22641
>
> -ck
>
> On Thu, Dec 23, 2021, at 12:25, Tim Perry wrote:
> > Is it worth marching those tests with @Ignore and filing a Jira for each
> > one? That does seem to set a bad precedent though.
> >
> > Last time I tried I couldn't get the mainline code to load in IntelliJ or
> > Eclipse because of the packing changes that were in progress. Did that
> get
> > fixed? If so, I might be able to carve out some time to look at a couple
> > tests if you point me in the right direction.
> >
> > Tim
> >
> > On Thu, Dec 23, 2021 at 7:24 AM Volkan Yazıcı <vo...@yazi.ci> wrote:
> >
> > > Since, there are some tests which occasionally constantly fail, we use
> > > `-Dmaven.test.failure.ignore=true` in `verify` goal. This causes a
> build
> > > with test failures to be marked green. The 3rd party component,
> > > `scacap/action-surefire-report@v1`, is used to feed these failures
> back
> > > into GitHub Actions status with some pretty printing. But since the PR
> is
> > > opened by a user who doesn't have commit rights, the 3rd party
> component
> > > fails to mark the failures due to insufficient privileges. In
> > > `.github/workflows/main.yml`, I had introduced the `if:
> github.repository
> > > == 'apache/logging-log4j2'` line to work around this, but apparently
> it is
> > > broken again.
> > >
> > > I totally share your frustration, same here. Though sparing time to fix
> > > this is pretty difficult nowadays.
> > >
> > > I also need to confess, in those brief moments of insanity, I
> contemplate
> > > nuking all those flaky tests. This will simplify the CI magic a lot and
> > > enhance our confidence in the tests.
> > >
> > > On Tue, Dec 21, 2021 at 3:10 AM Gary Gregory <ga...@gmail.com>
> > > wrote:
> > >
> > > > After getting https://github.com/apache/logging-log4j2/pull/646 in
> what
> > > I
> > > > think is a good state, I have no idea if it is safe or not to merge
> > > because
> > > > the 1st build GitHub shows is red:
> > > >
> > > >
> > >
> https://github.com/apache/logging-log4j2/runs/4589771553?check_suite_focus=true
> > > >
> > > > I don't use GH actions the way we do here and I'm not sure why we
> need
> > > the
> > > > publish test result set when anyone can just look at the build logs.
> > > >
> > > > Gary
> > > >
> > >
> >
>

Re: Broken CI

[Carter Kozak <ck...@ckozak.net>]

I haven't had a test flake locally in the last 6 months (at least), if we see flaky tests I'm in favor of fixing them rather than working around them.

fwiw the non GitHub-actions tests work great on the release-2.x branch in my experience, when they aren't overwhelmed accessing build nodes anyhow. That said, I would prefer to get everything workin on GitHub actions as it seems to be the gold standard these days.

> Last time I tried I couldn't get the mainline code to load in IntelliJ or Eclipse because of the packing changes that were in progress.

I find release-2.x works nicely with IntelliJ idea when I set the project-level sdk to jdk8. the master branch may be another story. I have an INFRA ticket open to switch the default branch to reduce confusion: https://issues.apache.org/jira/browse/INFRA-22641

-ck

On Thu, Dec 23, 2021, at 12:25, Tim Perry wrote:
> Is it worth marching those tests with @Ignore and filing a Jira for each
> one? That does seem to set a bad precedent though.
> 
> Last time I tried I couldn't get the mainline code to load in IntelliJ or
> Eclipse because of the packing changes that were in progress. Did that get
> fixed? If so, I might be able to carve out some time to look at a couple
> tests if you point me in the right direction.
> 
> Tim
> 
> On Thu, Dec 23, 2021 at 7:24 AM Volkan Yazıcı <vo...@yazi.ci> wrote:
> 
> > Since, there are some tests which occasionally constantly fail, we use
> > `-Dmaven.test.failure.ignore=true` in `verify` goal. This causes a build
> > with test failures to be marked green. The 3rd party component,
> > `scacap/action-surefire-report@v1`, is used to feed these failures back
> > into GitHub Actions status with some pretty printing. But since the PR is
> > opened by a user who doesn't have commit rights, the 3rd party component
> > fails to mark the failures due to insufficient privileges. In
> > `.github/workflows/main.yml`, I had introduced the `if: github.repository
> > == 'apache/logging-log4j2'` line to work around this, but apparently it is
> > broken again.
> >
> > I totally share your frustration, same here. Though sparing time to fix
> > this is pretty difficult nowadays.
> >
> > I also need to confess, in those brief moments of insanity, I contemplate
> > nuking all those flaky tests. This will simplify the CI magic a lot and
> > enhance our confidence in the tests.
> >
> > On Tue, Dec 21, 2021 at 3:10 AM Gary Gregory <ga...@gmail.com>
> > wrote:
> >
> > > After getting https://github.com/apache/logging-log4j2/pull/646 in what
> > I
> > > think is a good state, I have no idea if it is safe or not to merge
> > because
> > > the 1st build GitHub shows is red:
> > >
> > >
> > https://github.com/apache/logging-log4j2/runs/4589771553?check_suite_focus=true
> > >
> > > I don't use GH actions the way we do here and I'm not sure why we need
> > the
> > > publish test result set when anyone can just look at the build logs.
> > >
> > > Gary
> > >
> >
> 

Re: Broken CI

[Tim Perry <ti...@gmail.com>]

Is it worth marching those tests with @Ignore and filing a Jira for each
one? That does seem to set a bad precedent though.

Last time I tried I couldn't get the mainline code to load in IntelliJ or
Eclipse because of the packing changes that were in progress. Did that get
fixed? If so, I might be able to carve out some time to look at a couple
tests if you point me in the right direction.

Tim

On Thu, Dec 23, 2021 at 7:24 AM Volkan Yazıcı <vo...@yazi.ci> wrote:

> Since, there are some tests which occasionally constantly fail, we use
> `-Dmaven.test.failure.ignore=true` in `verify` goal. This causes a build
> with test failures to be marked green. The 3rd party component,
> `scacap/action-surefire-report@v1`, is used to feed these failures back
> into GitHub Actions status with some pretty printing. But since the PR is
> opened by a user who doesn't have commit rights, the 3rd party component
> fails to mark the failures due to insufficient privileges. In
> `.github/workflows/main.yml`, I had introduced the `if: github.repository
> == 'apache/logging-log4j2'` line to work around this, but apparently it is
> broken again.
>
> I totally share your frustration, same here. Though sparing time to fix
> this is pretty difficult nowadays.
>
> I also need to confess, in those brief moments of insanity, I contemplate
> nuking all those flaky tests. This will simplify the CI magic a lot and
> enhance our confidence in the tests.
>
> On Tue, Dec 21, 2021 at 3:10 AM Gary Gregory <ga...@gmail.com>
> wrote:
>
> > After getting https://github.com/apache/logging-log4j2/pull/646 in what
> I
> > think is a good state, I have no idea if it is safe or not to merge
> because
> > the 1st build GitHub shows is red:
> >
> >
> https://github.com/apache/logging-log4j2/runs/4589771553?check_suite_focus=true
> >
> > I don't use GH actions the way we do here and I'm not sure why we need
> the
> > publish test result set when anyone can just look at the build logs.
> >
> > Gary
> >
>

Re: Broken CI

[Volkan Yazıcı <vo...@yazi.ci>]

`Error: Resource not accessible by integration` failure caused by
`action-surefire-report` trying to access the job triggered by a user who
doesn't have commit rights to the repo. I have actually tried to explain
this in detail in my earlier email. Let me know if that wasn't clear enough.

On Thu, Dec 23, 2021 at 6:34 PM Gary Gregory <ga...@gmail.com> wrote:

> I'm not talking about failing tests but about some other build silliness
> that does not feel necessary, for example:
>
> https://github.com/apache/logging-log4j2/runs/4620228443?check_suite_focus=true
>
> Shows:
>
>
> Run scacap/action-surefire-report@v1
> Going to parse results form **/*-reports/TEST-*.xml
> Result: 4015 tests run, 15 skipped, 0 failed.
> Posting status 'completed' with conclusion 'success' to
> https://github.com/apache/logging-log4j2/pull/651 (sha:
> 06c61ddc60b24610fda694aa54f4c1dd2e29ff07)
> Error: Resource not accessible by integration
>
> Gary
>
>
> On Thu, Dec 23, 2021, 10:24 Volkan Yazıcı <vo...@yazi.ci> wrote:
>
> > Since, there are some tests which occasionally constantly fail, we use
> > `-Dmaven.test.failure.ignore=true` in `verify` goal. This causes a build
> > with test failures to be marked green. The 3rd party component,
> > `scacap/action-surefire-report@v1`, is used to feed these failures back
> > into GitHub Actions status with some pretty printing. But since the PR is
> > opened by a user who doesn't have commit rights, the 3rd party component
> > fails to mark the failures due to insufficient privileges. In
> > `.github/workflows/main.yml`, I had introduced the `if: github.repository
> > == 'apache/logging-log4j2'` line to work around this, but apparently it
> is
> > broken again.
> >
> > I totally share your frustration, same here. Though sparing time to fix
> > this is pretty difficult nowadays.
> >
> > I also need to confess, in those brief moments of insanity, I contemplate
> > nuking all those flaky tests. This will simplify the CI magic a lot and
> > enhance our confidence in the tests.
> >
> > On Tue, Dec 21, 2021 at 3:10 AM Gary Gregory <ga...@gmail.com>
> > wrote:
> >
> > > After getting https://github.com/apache/logging-log4j2/pull/646 in
> what
> > I
> > > think is a good state, I have no idea if it is safe or not to merge
> > because
> > > the 1st build GitHub shows is red:
> > >
> > >
> >
> https://github.com/apache/logging-log4j2/runs/4589771553?check_suite_focus=true
> > >
> > > I don't use GH actions the way we do here and I'm not sure why we need
> > the
> > > publish test result set when anyone can just look at the build logs.
> > >
> > > Gary
> > >
> >
>

Re: Broken CI

[Gary Gregory <ga...@gmail.com>]

I'm not talking about failing tests but about some other build silliness
that does not feel necessary, for example:
https://github.com/apache/logging-log4j2/runs/4620228443?check_suite_focus=true

Shows:


Run scacap/action-surefire-report@v1
Going to parse results form **/*-reports/TEST-*.xml
Result: 4015 tests run, 15 skipped, 0 failed.
Posting status 'completed' with conclusion 'success' to
https://github.com/apache/logging-log4j2/pull/651 (sha:
06c61ddc60b24610fda694aa54f4c1dd2e29ff07)
Error: Resource not accessible by integration

Gary


On Thu, Dec 23, 2021, 10:24 Volkan Yazıcı <vo...@yazi.ci> wrote:

> Since, there are some tests which occasionally constantly fail, we use
> `-Dmaven.test.failure.ignore=true` in `verify` goal. This causes a build
> with test failures to be marked green. The 3rd party component,
> `scacap/action-surefire-report@v1`, is used to feed these failures back
> into GitHub Actions status with some pretty printing. But since the PR is
> opened by a user who doesn't have commit rights, the 3rd party component
> fails to mark the failures due to insufficient privileges. In
> `.github/workflows/main.yml`, I had introduced the `if: github.repository
> == 'apache/logging-log4j2'` line to work around this, but apparently it is
> broken again.
>
> I totally share your frustration, same here. Though sparing time to fix
> this is pretty difficult nowadays.
>
> I also need to confess, in those brief moments of insanity, I contemplate
> nuking all those flaky tests. This will simplify the CI magic a lot and
> enhance our confidence in the tests.
>
> On Tue, Dec 21, 2021 at 3:10 AM Gary Gregory <ga...@gmail.com>
> wrote:
>
> > After getting https://github.com/apache/logging-log4j2/pull/646 in what
> I
> > think is a good state, I have no idea if it is safe or not to merge
> because
> > the 1st build GitHub shows is red:
> >
> >
> https://github.com/apache/logging-log4j2/runs/4589771553?check_suite_focus=true
> >
> > I don't use GH actions the way we do here and I'm not sure why we need
> the
> > publish test result set when anyone can just look at the build logs.
> >
> > Gary
> >
>

Re: Broken CI

[Volkan Yazıcı <vo...@yazi.ci>]

Since, there are some tests which occasionally constantly fail, we use
`-Dmaven.test.failure.ignore=true` in `verify` goal. This causes a build
with test failures to be marked green. The 3rd party component,
`scacap/action-surefire-report@v1`, is used to feed these failures back
into GitHub Actions status with some pretty printing. But since the PR is
opened by a user who doesn't have commit rights, the 3rd party component
fails to mark the failures due to insufficient privileges. In
`.github/workflows/main.yml`, I had introduced the `if: github.repository
== 'apache/logging-log4j2'` line to work around this, but apparently it is
broken again.

I totally share your frustration, same here. Though sparing time to fix
this is pretty difficult nowadays.

I also need to confess, in those brief moments of insanity, I contemplate
nuking all those flaky tests. This will simplify the CI magic a lot and
enhance our confidence in the tests.

On Tue, Dec 21, 2021 at 3:10 AM Gary Gregory <ga...@gmail.com> wrote:

> After getting https://github.com/apache/logging-log4j2/pull/646 in what I
> think is a good state, I have no idea if it is safe or not to merge because
> the 1st build GitHub shows is red:
>
> https://github.com/apache/logging-log4j2/runs/4589771553?check_suite_focus=true
>
> I don't use GH actions the way we do here and I'm not sure why we need the
> publish test result set when anyone can just look at the build logs.
>
> Gary
>

Re: Broken CI

[Volkan Yazıcı <vo...@yazi.ci>]

I have simplified(?) the GitHub Actions `build` workflow in `release-2.x`.

Removed `action-surefire-report` usage, hence no extra privileges should be
necessary anymore.
Removed the `-Dmaven.test.failure.ignore=true` flag, hence all tests need
to pass.

On Tue, Dec 21, 2021 at 3:10 AM Gary Gregory <ga...@gmail.com> wrote:

> After getting https://github.com/apache/logging-log4j2/pull/646 in what I
> think is a good state, I have no idea if it is safe or not to merge because
> the 1st build GitHub shows is red:
>
> https://github.com/apache/logging-log4j2/runs/4589771553?check_suite_focus=true
>
> I don't use GH actions the way we do here and I'm not sure why we need the
> publish test result set when anyone can just look at the build logs.
>
> Gary
>