You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Robert Muir (JIRA)" <ji...@apache.org> on 2018/02/25 14:34:00 UTC
[jira] [Deleted] (SOLR-12016) Reduce noise from flakey tests
[ https://issues.apache.org/jira/browse/SOLR-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir deleted SOLR-12016:
-------------------------------
> Reduce noise from flakey tests
> ------------------------------
>
> Key: SOLR-12016
> URL: https://issues.apache.org/jira/browse/SOLR-12016
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Erick Erickson
> Assignee: Erick Erickson
> Priority: Major
>
> We had a discussion of this topic on the dev list, look for a thread titled: "Test failures are out of control.....". I'll try to summarize that discussion here and we can move this JIRA forward. This may become an umbrella issue.
> Current situation concerns:
> > There is so much noise from flakey tests (particularly Solr tests) that they are difficult to use.
> > The number of tests that regularly fail is increasing
> > Failures are being ignored
> > The number of failing tests makes releasing more difficult.
> > The number of failing tests make it harder to determine whether recent changes actually caused problems. Running the tests again until they succeed is used commonly at present, which is not robust.
> > e-mail notifications of failing tests are largely being ignored.
> Propsal:
> > Mark all currently "flakey" tests as BadApple or AwaitsFix
> > Run Jenkins jobs with BadApple (and/or AwaitsFix) enabled and disabled. Frequency TBD, depends partly on whether we can label emails from these runs for easy filtering of the two flavors.
> >> Label these runs with something suitable in the subject line (wish list)
> > Weekly reports on the tests labeled BadApple or AwaitsFix
> >> Perhaps this could be incorporated in the reports linked below (wish list)
> > Committers should enable BadApple (or AwaitsFix) regularly as a sanity check. Leave these as defaults.
> > We start getting _much_ more aggressive about not allowing _new_ flakey tests.
> NOTE: It's perfectly acceptable to have failing flakey tests as long as someone is activey working on _fixing_ them.
> Concerns with solution
> > Decreases test coverage
> > Decreases visibility of flakey tests, making fixing them less likely.
> > Some tools (see below) that report on bad tests will not see tests that are annotated with BadApple or AwaitsFix.
> > Running unit tests and reporting errors are being conflated
> To be decided:
> > Can we label e-mails with failing tests with something in the subject line identifying whether they were run with BadApple/Awaits fix enabled or disabled? Can someone volunteer?
> > Is there any difference between BadApple and AwaitsFix? If not should we deprecate one? I propose we just use AwaitsFix and deprecate BadApple.
> > Can the automated reports (see below) be enhanced to also report tests labeled BadApple or AwaitsFix?
> Useful tools:
> > Steve Rowe's work on a Jenkins job to reproduce test failures (LUCENE-8106)
> > Hoss has worked on aggregating all test failures from the 3 Jenkins systems (ASF, Policeman, and Steve's), downloading the test results & logs, and running some reports/stats on failures.
> >> http://fucit.org/solr-jenkins-reports/
> >> https://github.com/hossman/jenkins-reports/
> >> http://fucit.org/solr-jenkins-reports/failure-report.html
> I've assigned this JIRA to myslef, but all volunteers welcome, especially anything that changes the build system.....
> I've decided to make this a SOLR jira on the theory that most of the offending tests are in the Solr hive, any sub-tasks for touching the build system can go under LUCENE if wanted.
> Also, I expect to add the annotation to some more tests for a few days as infrequent failures occur. Once we have stability (defined by there being little noise) that'll stop.
> 3 BadApple 23 AwaitsFix annotations are currently in the code, linked to these issues:
> HADOOP-14044
> HADOOP-9893
> LUCENE-3869
> LUCENE-5575")
> LUCENE-5595
> LUCENE-5737
> LUCENE-6709
> LUCENE-7161
> SOLR-2715
> SOLR-6213
> SOLR-6443
> SOLR-6944
> SOLR-7736
> SOLR-9036
> SOLR-10071
> SOLR-10107
> SOLR-10136
> SOLR-10734
> SOLR-10191
> SOLR-11134
> SOLR-11458
> SOLR-11714
> SOLR-11974
> Solr JIRAS about bad tests
> SOLR-2175
> SOLR-4147
> SOLR-5880
> SOLR-6423
> SOLR-6944
> SOLR-6961
> SOLR-6974
> SOLR-8122
> SOLR-8182
> SOLR-9869
> SOLR-10053
> SOLR-10070
> SOLR-10071
> SOLR-10139
> SOLR-10287
> SOLR-10815
> SOLR-11911
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [jira] [Deleted] (SOLR-12016) Reduce noise from flakey tests
Posted by Uwe Schindler <uw...@thetaphi.de>.
Was this intended?
Uwe
Am February 25, 2018 2:34:00 PM UTC schrieb "Robert Muir (JIRA)" <ji...@apache.org>:
>
>[
>https://issues.apache.org/jira/browse/SOLR-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
>]
>
>Robert Muir deleted SOLR-12016:
>-------------------------------
>
>
>> Reduce noise from flakey tests
>> ------------------------------
>>
>> Key: SOLR-12016
>> URL: https://issues.apache.org/jira/browse/SOLR-12016
>> Project: Solr
>> Issue Type: Improvement
>> Security Level: Public(Default Security Level. Issues are
>Public)
>> Reporter: Erick Erickson
>> Assignee: Erick Erickson
>> Priority: Major
>>
>> We had a discussion of this topic on the dev list, look for a thread
>titled: "Test failures are out of control.....". I'll try to summarize
>that discussion here and we can move this JIRA forward. This may become
>an umbrella issue.
>> Current situation concerns:
>> > There is so much noise from flakey tests (particularly Solr tests)
>that they are difficult to use.
>> > The number of tests that regularly fail is increasing
>> > Failures are being ignored
>> > The number of failing tests makes releasing more difficult.
>> > The number of failing tests make it harder to determine whether
>recent changes actually caused problems. Running the tests again until
>they succeed is used commonly at present, which is not robust.
>> > e-mail notifications of failing tests are largely being ignored.
>> Propsal:
>> > Mark all currently "flakey" tests as BadApple or AwaitsFix
>> > Run Jenkins jobs with BadApple (and/or AwaitsFix) enabled and
>disabled. Frequency TBD, depends partly on whether we can label emails
>from these runs for easy filtering of the two flavors.
>> >> Label these runs with something suitable in the subject line (wish
>list)
>> > Weekly reports on the tests labeled BadApple or AwaitsFix
>> >> Perhaps this could be incorporated in the reports linked below
>(wish list)
>> > Committers should enable BadApple (or AwaitsFix) regularly as a
>sanity check. Leave these as defaults.
>> > We start getting _much_ more aggressive about not allowing _new_
>flakey tests.
>> NOTE: It's perfectly acceptable to have failing flakey tests as long
>as someone is activey working on _fixing_ them.
>> Concerns with solution
>> > Decreases test coverage
>> > Decreases visibility of flakey tests, making fixing them less
>likely.
>> > Some tools (see below) that report on bad tests will not see tests
>that are annotated with BadApple or AwaitsFix.
>> > Running unit tests and reporting errors are being conflated
>> To be decided:
>> > Can we label e-mails with failing tests with something in the
>subject line identifying whether they were run with BadApple/Awaits fix
>enabled or disabled? Can someone volunteer?
>> > Is there any difference between BadApple and AwaitsFix? If not
>should we deprecate one? I propose we just use AwaitsFix and deprecate
>BadApple.
>> > Can the automated reports (see below) be enhanced to also report
>tests labeled BadApple or AwaitsFix?
>> Useful tools:
>> > Steve Rowe's work on a Jenkins job to reproduce test failures
>(LUCENE-8106)
>> > Hoss has worked on aggregating all test failures from the 3 Jenkins
>systems (ASF, Policeman, and Steve's), downloading the test results &
>logs, and running some reports/stats on failures.
>> >> http://fucit.org/solr-jenkins-reports/
>> >> https://github.com/hossman/jenkins-reports/
>> >> http://fucit.org/solr-jenkins-reports/failure-report.html
>> I've assigned this JIRA to myslef, but all volunteers welcome,
>especially anything that changes the build system.....
>> I've decided to make this a SOLR jira on the theory that most of the
>offending tests are in the Solr hive, any sub-tasks for touching the
>build system can go under LUCENE if wanted.
>> Also, I expect to add the annotation to some more tests for a few
>days as infrequent failures occur. Once we have stability (defined by
>there being little noise) that'll stop.
>> 3 BadApple 23 AwaitsFix annotations are currently in the code, linked
>to these issues:
>> HADOOP-14044
>> HADOOP-9893
>> LUCENE-3869
>> LUCENE-5575")
>> LUCENE-5595
>> LUCENE-5737
>> LUCENE-6709
>> LUCENE-7161
>> SOLR-2715
>> SOLR-6213
>> SOLR-6443
>> SOLR-6944
>> SOLR-7736
>> SOLR-9036
>> SOLR-10071
>> SOLR-10107
>> SOLR-10136
>> SOLR-10734
>> SOLR-10191
>> SOLR-11134
>> SOLR-11458
>> SOLR-11714
>> SOLR-11974
>> Solr JIRAS about bad tests
>> SOLR-2175
>> SOLR-4147
>> SOLR-5880
>> SOLR-6423
>> SOLR-6944
>> SOLR-6961
>> SOLR-6974
>> SOLR-8122
>> SOLR-8182
>> SOLR-9869
>> SOLR-10053
>> SOLR-10070
>> SOLR-10071
>> SOLR-10139
>> SOLR-10287
>> SOLR-10815
>> SOLR-11911
>
>
>
>--
>This message was sent by Atlassian JIRA
>(v7.6.3#76005)
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>For additional commands, e-mail: dev-help@lucene.apache.org
--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de