You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@logging.apache.org by Ralph Goers <ra...@dslextreme.com> on 2022/05/28 08:17:46 UTC
Disabled tests
Matt, it seems you have disabled a bunch of tests. I’ve been working on one that I recently committed but for the life of me cannot figure out why it is failing on Windows.
What I don’t understand is why several of the Jira issues seemingly have 100 commits on various branches and flooded my inbox with email.
I am also not comfortable performing a release with tests that seemingly used to work now failing. Disabling the tests is not the answer.
So although I had intended to start work on the 2.18.0 release this weekend it seems I now have to figure out what tests you disabled and figure out how to fix them. I would have preferred that you would have discussed this before doing that.
Ralph
Re: Disabled tests
Posted by Ralph Goers <ra...@dslextreme.com>.
It also just failed for me on my Mac. It got the same error on line 73.
Ralph
> On May 28, 2022, at 2:32 PM, Ralph Goers <ra...@dslextreme.com> wrote:
>
> You need to mark a test you recently added as flaky as it has failed for me twice on Windows.
>
> [INFO]
> [ERROR] Failures:
> [ERROR] OnPropertyConditionTest.whenPropertyMatches:80 expected: <truth> but was: <goodbye>
> [ERROR] OnPropertyConditionTest.whenPropertyPresent:73 expected: <hello> but was: <goodbye>
>
> Ralph
>
>> On May 28, 2022, at 12:52 PM, Matt Sicker <bo...@gmail.com> wrote:
>>
>> Well, let’s take a look at the commits that have failed CI builds lately. Of course CI was green before, MutableThreadContextMapFilterTest was only introduced recently. Since then, the following completely unrelated commits have CI build failures:
>>
>> * https://github.com/apache/logging-log4j2/commit/22382a42bb888cdfb371553ec74c1da1f343ea02 <https://github.com/apache/logging-log4j2/commit/22382a42bb888cdfb371553ec74c1da1f343ea02> (unrelated dependency change)
>> * https://github.com/apache/logging-log4j2/commit/a4d562b0363d7f4ceef22aa7ab3d0ff11c2d6376 <https://github.com/apache/logging-log4j2/commit/a4d562b0363d7f4ceef22aa7ab3d0ff11c2d6376> (change in manual)
>> * https://github.com/apache/logging-log4j2/commit/26efbc801b8e87088c0924abde17001cff34a88e <https://github.com/apache/logging-log4j2/commit/26efbc801b8e87088c0924abde17001cff34a88e> (unrelated test updates for arbiters)
>> * https://github.com/apache/logging-log4j2/commit/ecd2d7783090683d3b7ba0ceb26e5c953f5c5a44 <https://github.com/apache/logging-log4j2/commit/ecd2d7783090683d3b7ba0ceb26e5c953f5c5a44> (modifying the Dependabot config)
>> * several Dependabot PRs because fuck it at that point
>>
>> So a test was added that flakes on Windows CI which causes churn in trying to merge PRs. The fact that this test was left enabled in production like this for as long as it was is surprising.
>>
>> The GitHub action build failure spam sent directly to committers doesn’t help the case, either. Once I start getting those after I manually verify my commits before pushing them, unless the test failures are related to changes I just made (which does happen once in a while due to platform-specific differences or forgetting to commit a file), these emails are simply just spam. Thus, I go and fix the source of spam. If you prefer the spam, then change the action config to email yourself instead for all build failures and re-enable all the flaky tests.
>> —
>> Matt Sicker
>>
>>> On May 28, 2022, at 14:26, Ralph Goers <ra...@dslextreme.com> wrote:
>>>
>>> There is only one flaw with that argument. When I did release 2.17.2 we weren’t having random test failures like this. So something changed.
>>>
>>> Ralph
>>>
>>>> On May 28, 2022, at 11:34 AM, Matt Sicker <bo...@gmail.com> wrote:
>>>>
>>>> Oh, and if these tests used to work, they wouldn’t be flaking from unrelated changes such as when someone updates a README file or other non-code area that somehow results in a failed CI run.
>>>> —
>>>> Matt Sicker
>>>>
>>>>> On May 28, 2022, at 10:29, Matt Sicker <bo...@gmail.com> wrote:
>>>>>
>>>>> I sent a separate email complaining about how Dependabot does this. Anyways, the disabled tests are flakes. As I’ve said before, I run the full build and suite of tests locally before pushing commits, but I’ve been getting tons of build failure emails regardless. So instead of ignoring CI failures as seems to be standard right now, I disabled the flaky tests where applicable until someone cares enough to fix them. I filed Jira issues so we don’t forget, either. It was also the only real feasible way to get through dependency upgrade PRs without them randomly failing due to unrelated flaky tests.
>>>>>
>>>>> —
>>>>> Matt Sicker
>>>>>
>>>>>> On May 28, 2022, at 03:46, Piotr P. Karwasz <pi...@gmail.com> wrote:
>>>>>>
>>>>>> Hi Ralph,
>>>>>>
>>>>>> On Sat, 28 May 2022 at 10:17, Ralph Goers <ra...@dslextreme.com>
>>>>>> wrote:
>>>>>>
>>>>>>> What I don’t understand is why several of the Jira issues seemingly have
>>>>>>> 100 commits on various branches and flooded my inbox with email.
>>>>>>>
>>>>>>> This is Dependabot-related: every time a commit with "LOG4J2" appears in
>>>>>> *any* branch, JIRA sends an e-mail. Rebasing many Dependabot branches
>>>>>> caused a storm of e-mails (I had some 150 e-mails this morning).
>>>>>>
>>>>>> Piotr
>>>>
>>>
>>
>
Re: Disabled tests
Posted by Ralph Goers <ra...@dslextreme.com>.
You need to mark a test you recently added as flaky as it has failed for me twice on Windows.
[INFO]
[ERROR] Failures:
[ERROR] OnPropertyConditionTest.whenPropertyMatches:80 expected: <truth> but was: <goodbye>
[ERROR] OnPropertyConditionTest.whenPropertyPresent:73 expected: <hello> but was: <goodbye>
Ralph
> On May 28, 2022, at 12:52 PM, Matt Sicker <bo...@gmail.com> wrote:
>
> Well, let’s take a look at the commits that have failed CI builds lately. Of course CI was green before, MutableThreadContextMapFilterTest was only introduced recently. Since then, the following completely unrelated commits have CI build failures:
>
> * https://github.com/apache/logging-log4j2/commit/22382a42bb888cdfb371553ec74c1da1f343ea02 <https://github.com/apache/logging-log4j2/commit/22382a42bb888cdfb371553ec74c1da1f343ea02> (unrelated dependency change)
> * https://github.com/apache/logging-log4j2/commit/a4d562b0363d7f4ceef22aa7ab3d0ff11c2d6376 <https://github.com/apache/logging-log4j2/commit/a4d562b0363d7f4ceef22aa7ab3d0ff11c2d6376> (change in manual)
> * https://github.com/apache/logging-log4j2/commit/26efbc801b8e87088c0924abde17001cff34a88e <https://github.com/apache/logging-log4j2/commit/26efbc801b8e87088c0924abde17001cff34a88e> (unrelated test updates for arbiters)
> * https://github.com/apache/logging-log4j2/commit/ecd2d7783090683d3b7ba0ceb26e5c953f5c5a44 <https://github.com/apache/logging-log4j2/commit/ecd2d7783090683d3b7ba0ceb26e5c953f5c5a44> (modifying the Dependabot config)
> * several Dependabot PRs because fuck it at that point
>
> So a test was added that flakes on Windows CI which causes churn in trying to merge PRs. The fact that this test was left enabled in production like this for as long as it was is surprising.
>
> The GitHub action build failure spam sent directly to committers doesn’t help the case, either. Once I start getting those after I manually verify my commits before pushing them, unless the test failures are related to changes I just made (which does happen once in a while due to platform-specific differences or forgetting to commit a file), these emails are simply just spam. Thus, I go and fix the source of spam. If you prefer the spam, then change the action config to email yourself instead for all build failures and re-enable all the flaky tests.
> —
> Matt Sicker
>
>> On May 28, 2022, at 14:26, Ralph Goers <ra...@dslextreme.com> wrote:
>>
>> There is only one flaw with that argument. When I did release 2.17.2 we weren’t having random test failures like this. So something changed.
>>
>> Ralph
>>
>>> On May 28, 2022, at 11:34 AM, Matt Sicker <bo...@gmail.com> wrote:
>>>
>>> Oh, and if these tests used to work, they wouldn’t be flaking from unrelated changes such as when someone updates a README file or other non-code area that somehow results in a failed CI run.
>>> —
>>> Matt Sicker
>>>
>>>> On May 28, 2022, at 10:29, Matt Sicker <bo...@gmail.com> wrote:
>>>>
>>>> I sent a separate email complaining about how Dependabot does this. Anyways, the disabled tests are flakes. As I’ve said before, I run the full build and suite of tests locally before pushing commits, but I’ve been getting tons of build failure emails regardless. So instead of ignoring CI failures as seems to be standard right now, I disabled the flaky tests where applicable until someone cares enough to fix them. I filed Jira issues so we don’t forget, either. It was also the only real feasible way to get through dependency upgrade PRs without them randomly failing due to unrelated flaky tests.
>>>>
>>>> —
>>>> Matt Sicker
>>>>
>>>>> On May 28, 2022, at 03:46, Piotr P. Karwasz <pi...@gmail.com> wrote:
>>>>>
>>>>> Hi Ralph,
>>>>>
>>>>> On Sat, 28 May 2022 at 10:17, Ralph Goers <ra...@dslextreme.com>
>>>>> wrote:
>>>>>
>>>>>> What I don’t understand is why several of the Jira issues seemingly have
>>>>>> 100 commits on various branches and flooded my inbox with email.
>>>>>>
>>>>>> This is Dependabot-related: every time a commit with "LOG4J2" appears in
>>>>> *any* branch, JIRA sends an e-mail. Rebasing many Dependabot branches
>>>>> caused a storm of e-mails (I had some 150 e-mails this morning).
>>>>>
>>>>> Piotr
>>>
>>
>
Re: Disabled tests
Posted by Matt Sicker <bo...@gmail.com>.
Well, let’s take a look at the commits that have failed CI builds lately. Of course CI was green before, MutableThreadContextMapFilterTest was only introduced recently. Since then, the following completely unrelated commits have CI build failures:
* https://github.com/apache/logging-log4j2/commit/22382a42bb888cdfb371553ec74c1da1f343ea02 <https://github.com/apache/logging-log4j2/commit/22382a42bb888cdfb371553ec74c1da1f343ea02> (unrelated dependency change)
* https://github.com/apache/logging-log4j2/commit/a4d562b0363d7f4ceef22aa7ab3d0ff11c2d6376 <https://github.com/apache/logging-log4j2/commit/a4d562b0363d7f4ceef22aa7ab3d0ff11c2d6376> (change in manual)
* https://github.com/apache/logging-log4j2/commit/26efbc801b8e87088c0924abde17001cff34a88e <https://github.com/apache/logging-log4j2/commit/26efbc801b8e87088c0924abde17001cff34a88e> (unrelated test updates for arbiters)
* https://github.com/apache/logging-log4j2/commit/ecd2d7783090683d3b7ba0ceb26e5c953f5c5a44 <https://github.com/apache/logging-log4j2/commit/ecd2d7783090683d3b7ba0ceb26e5c953f5c5a44> (modifying the Dependabot config)
* several Dependabot PRs because fuck it at that point
So a test was added that flakes on Windows CI which causes churn in trying to merge PRs. The fact that this test was left enabled in production like this for as long as it was is surprising.
The GitHub action build failure spam sent directly to committers doesn’t help the case, either. Once I start getting those after I manually verify my commits before pushing them, unless the test failures are related to changes I just made (which does happen once in a while due to platform-specific differences or forgetting to commit a file), these emails are simply just spam. Thus, I go and fix the source of spam. If you prefer the spam, then change the action config to email yourself instead for all build failures and re-enable all the flaky tests.
—
Matt Sicker
> On May 28, 2022, at 14:26, Ralph Goers <ra...@dslextreme.com> wrote:
>
> There is only one flaw with that argument. When I did release 2.17.2 we weren’t having random test failures like this. So something changed.
>
> Ralph
>
>> On May 28, 2022, at 11:34 AM, Matt Sicker <bo...@gmail.com> wrote:
>>
>> Oh, and if these tests used to work, they wouldn’t be flaking from unrelated changes such as when someone updates a README file or other non-code area that somehow results in a failed CI run.
>> —
>> Matt Sicker
>>
>>> On May 28, 2022, at 10:29, Matt Sicker <bo...@gmail.com> wrote:
>>>
>>> I sent a separate email complaining about how Dependabot does this. Anyways, the disabled tests are flakes. As I’ve said before, I run the full build and suite of tests locally before pushing commits, but I’ve been getting tons of build failure emails regardless. So instead of ignoring CI failures as seems to be standard right now, I disabled the flaky tests where applicable until someone cares enough to fix them. I filed Jira issues so we don’t forget, either. It was also the only real feasible way to get through dependency upgrade PRs without them randomly failing due to unrelated flaky tests.
>>>
>>> —
>>> Matt Sicker
>>>
>>>> On May 28, 2022, at 03:46, Piotr P. Karwasz <pi...@gmail.com> wrote:
>>>>
>>>> Hi Ralph,
>>>>
>>>> On Sat, 28 May 2022 at 10:17, Ralph Goers <ra...@dslextreme.com>
>>>> wrote:
>>>>
>>>>> What I don’t understand is why several of the Jira issues seemingly have
>>>>> 100 commits on various branches and flooded my inbox with email.
>>>>>
>>>>> This is Dependabot-related: every time a commit with "LOG4J2" appears in
>>>> *any* branch, JIRA sends an e-mail. Rebasing many Dependabot branches
>>>> caused a storm of e-mails (I had some 150 e-mails this morning).
>>>>
>>>> Piotr
>>
>
Re: Disabled tests
Posted by Ralph Goers <ra...@dslextreme.com>.
There is only one flaw with that argument. When I did release 2.17.2 we weren’t having random test failures like this. So something changed.
Ralph
> On May 28, 2022, at 11:34 AM, Matt Sicker <bo...@gmail.com> wrote:
>
> Oh, and if these tests used to work, they wouldn’t be flaking from unrelated changes such as when someone updates a README file or other non-code area that somehow results in a failed CI run.
> —
> Matt Sicker
>
>> On May 28, 2022, at 10:29, Matt Sicker <bo...@gmail.com> wrote:
>>
>> I sent a separate email complaining about how Dependabot does this. Anyways, the disabled tests are flakes. As I’ve said before, I run the full build and suite of tests locally before pushing commits, but I’ve been getting tons of build failure emails regardless. So instead of ignoring CI failures as seems to be standard right now, I disabled the flaky tests where applicable until someone cares enough to fix them. I filed Jira issues so we don’t forget, either. It was also the only real feasible way to get through dependency upgrade PRs without them randomly failing due to unrelated flaky tests.
>>
>> —
>> Matt Sicker
>>
>>> On May 28, 2022, at 03:46, Piotr P. Karwasz <pi...@gmail.com> wrote:
>>>
>>> Hi Ralph,
>>>
>>> On Sat, 28 May 2022 at 10:17, Ralph Goers <ra...@dslextreme.com>
>>> wrote:
>>>
>>>> What I don’t understand is why several of the Jira issues seemingly have
>>>> 100 commits on various branches and flooded my inbox with email.
>>>>
>>>> This is Dependabot-related: every time a commit with "LOG4J2" appears in
>>> *any* branch, JIRA sends an e-mail. Rebasing many Dependabot branches
>>> caused a storm of e-mails (I had some 150 e-mails this morning).
>>>
>>> Piotr
>
Re: Disabled tests
Posted by Matt Sicker <bo...@gmail.com>.
Oh, and if these tests used to work, they wouldn’t be flaking from unrelated changes such as when someone updates a README file or other non-code area that somehow results in a failed CI run.
—
Matt Sicker
> On May 28, 2022, at 10:29, Matt Sicker <bo...@gmail.com> wrote:
>
> I sent a separate email complaining about how Dependabot does this. Anyways, the disabled tests are flakes. As I’ve said before, I run the full build and suite of tests locally before pushing commits, but I’ve been getting tons of build failure emails regardless. So instead of ignoring CI failures as seems to be standard right now, I disabled the flaky tests where applicable until someone cares enough to fix them. I filed Jira issues so we don’t forget, either. It was also the only real feasible way to get through dependency upgrade PRs without them randomly failing due to unrelated flaky tests.
>
> —
> Matt Sicker
>
>> On May 28, 2022, at 03:46, Piotr P. Karwasz <pi...@gmail.com> wrote:
>>
>> Hi Ralph,
>>
>> On Sat, 28 May 2022 at 10:17, Ralph Goers <ra...@dslextreme.com>
>> wrote:
>>
>>> What I don’t understand is why several of the Jira issues seemingly have
>>> 100 commits on various branches and flooded my inbox with email.
>>>
>>> This is Dependabot-related: every time a commit with "LOG4J2" appears in
>> *any* branch, JIRA sends an e-mail. Rebasing many Dependabot branches
>> caused a storm of e-mails (I had some 150 e-mails this morning).
>>
>> Piotr
Re: Disabled tests
Posted by Matt Sicker <bo...@gmail.com>.
I sent a separate email complaining about how Dependabot does this. Anyways, the disabled tests are flakes. As I’ve said before, I run the full build and suite of tests locally before pushing commits, but I’ve been getting tons of build failure emails regardless. So instead of ignoring CI failures as seems to be standard right now, I disabled the flaky tests where applicable until someone cares enough to fix them. I filed Jira issues so we don’t forget, either. It was also the only real feasible way to get through dependency upgrade PRs without them randomly failing due to unrelated flaky tests.
—
Matt Sicker
> On May 28, 2022, at 03:46, Piotr P. Karwasz <pi...@gmail.com> wrote:
>
> Hi Ralph,
>
>> On Sat, 28 May 2022 at 10:17, Ralph Goers <ra...@dslextreme.com>
>> wrote:
>>
>> What I don’t understand is why several of the Jira issues seemingly have
>> 100 commits on various branches and flooded my inbox with email.
>>
>> This is Dependabot-related: every time a commit with "LOG4J2" appears in
> *any* branch, JIRA sends an e-mail. Rebasing many Dependabot branches
> caused a storm of e-mails (I had some 150 e-mails this morning).
>
> Piotr
Re: Disabled tests
Posted by "Piotr P. Karwasz" <pi...@gmail.com>.
Hi Ralph,
On Sat, 28 May 2022 at 10:17, Ralph Goers <ra...@dslextreme.com>
wrote:
> What I don’t understand is why several of the Jira issues seemingly have
> 100 commits on various branches and flooded my inbox with email.
>
> This is Dependabot-related: every time a commit with "LOG4J2" appears in
*any* branch, JIRA sends an e-mail. Rebasing many Dependabot branches
caused a storm of e-mails (I had some 150 e-mails this morning).
Piotr