You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geode.apache.org by Anthony Baker <ab...@apache.org> on 2016/05/10 04:49:41 UTC
Review Request 47157: GEODE-1374: Add a flakyTest build task
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47157/
-----------------------------------------------------------
Review request for geode, Dick Cavender, Kirk Lund, Mark Bretl, and Dan Smith.
Repository: geode
Description
-------
Add a new test task specifically for running tests annotated with
@FlakyTest. This allows us to isolate tests that exhibit
intermittent failures. Other changes include:
- Exclude flaky tests from test, integrationTest, uiTest, and
distributedTest tasks
- Cleanup how tests are found using the minimum set of
include/exclude categories
Diffs
-----
gradle/test.gradle 2dd75210b38fe37120778e35a4f96dbd08de97fe
Diff: https://reviews.apache.org/r/47157/diff/
Testing
-------
Tested by running test, flakyTest, integrationTest. Will run further tests before pushing.
Thanks,
Anthony Baker
Re: Review Request 47157: GEODE-1374: Add a flakyTest build task
Posted by Dan Smith <ds...@pivotal.io>.
> On May 10, 2016, 4:09 p.m., Dan Smith wrote:
> > I thought we were going to continue to make precheckin run these tests. You should make precheckin depend on flakyTest.
> >
> > Weren't we going to run these with forkEvery 1? The ingregration tests run with forkEvery 1, and the dunits with forkEvery 30. These don't fork at all, so they may be even less stable running as part of this target.
>
> Kirk Lund wrote:
> I think some of us discussed the possibility of adding "flakyTest" to the end of "precheckin" and that the addition of "forkEvery 1" on the flakTest task might be a good start. If the flakies are still flickering during most precheckin runs then we might want to remove from precheckin. We would need a separate task (from precheckin) for the primary consourse pipeline so that flakyTest can be parallel and not block later build tasks.
If you want to run a build without the flaky tests, you can run ./gradlew precheckin -xflakyTest
- Dan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47157/#review132460
-----------------------------------------------------------
On June 14, 2016, 4:58 p.m., Anthony Baker wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47157/
> -----------------------------------------------------------
>
> (Updated June 14, 2016, 4:58 p.m.)
>
>
> Review request for geode, Dick Cavender, Jens Deppe, Kirk Lund, Mark Bretl, and Dan Smith.
>
>
> Repository: geode
>
>
> Description
> -------
>
> Add a new test task specifically for running tests annotated with
> @FlakyTest. This allows us to isolate tests that exhibit
> intermittent failures. Other changes include:
>
> - Exclude flaky tests from test, integrationTest, uiTest, and
> distributedTest tasks
> - Cleanup how tests are found using the minimum set of
> include/exclude categories
>
>
> Diffs
> -----
>
> gradle/test.gradle eb10cad78272a54f84e17ca6b984da224dccfbba
>
> Diff: https://reviews.apache.org/r/47157/diff/
>
>
> Testing
> -------
>
> Tested by running test, flakyTest, integrationTest. Will run further tests before pushing.
>
>
> Thanks,
>
> Anthony Baker
>
>
Re: Review Request 47157: GEODE-1374: Add a flakyTest build task
Posted by Kirk Lund <ki...@gmail.com>.
> On May 10, 2016, 4:09 p.m., Dan Smith wrote:
> > I thought we were going to continue to make precheckin run these tests. You should make precheckin depend on flakyTest.
> >
> > Weren't we going to run these with forkEvery 1? The ingregration tests run with forkEvery 1, and the dunits with forkEvery 30. These don't fork at all, so they may be even less stable running as part of this target.
I think some of us discussed the possibility of adding "flakyTest" to the end of "precheckin" and that the addition of "forkEvery 1" on the flakTest task might be a good start. If the flakies are still flickering during most precheckin runs then we might want to remove from precheckin. We would need a separate task (from precheckin) for the primary consourse pipeline so that flakyTest can be parallel and not block later build tasks.
- Kirk
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47157/#review132460
-----------------------------------------------------------
On May 10, 2016, 4:49 a.m., Anthony Baker wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47157/
> -----------------------------------------------------------
>
> (Updated May 10, 2016, 4:49 a.m.)
>
>
> Review request for geode, Dick Cavender, Kirk Lund, Mark Bretl, and Dan Smith.
>
>
> Repository: geode
>
>
> Description
> -------
>
> Add a new test task specifically for running tests annotated with
> @FlakyTest. This allows us to isolate tests that exhibit
> intermittent failures. Other changes include:
>
> - Exclude flaky tests from test, integrationTest, uiTest, and
> distributedTest tasks
> - Cleanup how tests are found using the minimum set of
> include/exclude categories
>
>
> Diffs
> -----
>
> gradle/test.gradle 2dd75210b38fe37120778e35a4f96dbd08de97fe
>
> Diff: https://reviews.apache.org/r/47157/diff/
>
>
> Testing
> -------
>
> Tested by running test, flakyTest, integrationTest. Will run further tests before pushing.
>
>
> Thanks,
>
> Anthony Baker
>
>
Re: Review Request 47157: GEODE-1374: Add a flakyTest build task
Posted by Dan Smith <ds...@pivotal.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47157/#review132460
-----------------------------------------------------------
I thought we were going to continue to make precheckin run these tests. You should make precheckin depend on flakyTest.
Weren't we going to run these with forkEvery 1? The ingregration tests run with forkEvery 1, and the dunits with forkEvery 30. These don't fork at all, so they may be even less stable running as part of this target.
- Dan Smith
On May 10, 2016, 4:49 a.m., Anthony Baker wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47157/
> -----------------------------------------------------------
>
> (Updated May 10, 2016, 4:49 a.m.)
>
>
> Review request for geode, Dick Cavender, Kirk Lund, Mark Bretl, and Dan Smith.
>
>
> Repository: geode
>
>
> Description
> -------
>
> Add a new test task specifically for running tests annotated with
> @FlakyTest. This allows us to isolate tests that exhibit
> intermittent failures. Other changes include:
>
> - Exclude flaky tests from test, integrationTest, uiTest, and
> distributedTest tasks
> - Cleanup how tests are found using the minimum set of
> include/exclude categories
>
>
> Diffs
> -----
>
> gradle/test.gradle 2dd75210b38fe37120778e35a4f96dbd08de97fe
>
> Diff: https://reviews.apache.org/r/47157/diff/
>
>
> Testing
> -------
>
> Tested by running test, flakyTest, integrationTest. Will run further tests before pushing.
>
>
> Thanks,
>
> Anthony Baker
>
>
Re: Review Request 47157: GEODE-1374: Add a flakyTest build task
Posted by Anthony Baker <ab...@apache.org>.
> On May 11, 2016, 11:01 p.m., Kirk Lund wrote:
> > gradle/test.gradle, line 112
> > <https://reviews.apache.org/r/47157/diff/1/?file=1377352#file1377352line112>
> >
> > Dan added exclude *DUnitTest here because of GEODE-837 -- I don't really understand the problem well enough to know if it's ok to remove the exclusion
Unit we figure out GEODE-837 we will have a problem trying to run a separate test target for flaky tests. The tests run and results are reported in the html output but not in the TEST-* xml files used by Jenkins and other post-processors.
- Anthony
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47157/#review132779
-----------------------------------------------------------
On May 10, 2016, 4:49 a.m., Anthony Baker wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47157/
> -----------------------------------------------------------
>
> (Updated May 10, 2016, 4:49 a.m.)
>
>
> Review request for geode, Dick Cavender, Kirk Lund, Mark Bretl, and Dan Smith.
>
>
> Repository: geode
>
>
> Description
> -------
>
> Add a new test task specifically for running tests annotated with
> @FlakyTest. This allows us to isolate tests that exhibit
> intermittent failures. Other changes include:
>
> - Exclude flaky tests from test, integrationTest, uiTest, and
> distributedTest tasks
> - Cleanup how tests are found using the minimum set of
> include/exclude categories
>
>
> Diffs
> -----
>
> gradle/test.gradle 2dd75210b38fe37120778e35a4f96dbd08de97fe
>
> Diff: https://reviews.apache.org/r/47157/diff/
>
>
> Testing
> -------
>
> Tested by running test, flakyTest, integrationTest. Will run further tests before pushing.
>
>
> Thanks,
>
> Anthony Baker
>
>
Re: Review Request 47157: GEODE-1374: Add a flakyTest build task
Posted by Kirk Lund <ki...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47157/#review132779
-----------------------------------------------------------
Fix it, then Ship it!
Add "forkEvery 1" and review GEODE-837. Need to address policy about precheckin and flaky tests. And then Ship It!
gradle/test.gradle
<https://reviews.apache.org/r/47157/#comment197026>
Dan added exclude *DUnitTest here because of GEODE-837 -- I don't really understand the problem well enough to know if it's ok to remove the exclusion
gradle/test.gradle (line 134)
<https://reviews.apache.org/r/47157/#comment197028>
Please add "forkEvery 1" to this task
- Kirk Lund
On May 10, 2016, 4:49 a.m., Anthony Baker wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47157/
> -----------------------------------------------------------
>
> (Updated May 10, 2016, 4:49 a.m.)
>
>
> Review request for geode, Dick Cavender, Kirk Lund, Mark Bretl, and Dan Smith.
>
>
> Repository: geode
>
>
> Description
> -------
>
> Add a new test task specifically for running tests annotated with
> @FlakyTest. This allows us to isolate tests that exhibit
> intermittent failures. Other changes include:
>
> - Exclude flaky tests from test, integrationTest, uiTest, and
> distributedTest tasks
> - Cleanup how tests are found using the minimum set of
> include/exclude categories
>
>
> Diffs
> -----
>
> gradle/test.gradle 2dd75210b38fe37120778e35a4f96dbd08de97fe
>
> Diff: https://reviews.apache.org/r/47157/diff/
>
>
> Testing
> -------
>
> Tested by running test, flakyTest, integrationTest. Will run further tests before pushing.
>
>
> Thanks,
>
> Anthony Baker
>
>
Re: Review Request 47157: GEODE-1374: Add a flakyTest build task
Posted by Kirk Lund <ki...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47157/#review138052
-----------------------------------------------------------
Ship it!
Ship It!
- Kirk Lund
On June 14, 2016, 4:58 p.m., Anthony Baker wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47157/
> -----------------------------------------------------------
>
> (Updated June 14, 2016, 4:58 p.m.)
>
>
> Review request for geode, Dick Cavender, Jens Deppe, Kirk Lund, Mark Bretl, and Dan Smith.
>
>
> Repository: geode
>
>
> Description
> -------
>
> Add a new test task specifically for running tests annotated with
> @FlakyTest. This allows us to isolate tests that exhibit
> intermittent failures. Other changes include:
>
> - Exclude flaky tests from test, integrationTest, uiTest, and
> distributedTest tasks
> - Cleanup how tests are found using the minimum set of
> include/exclude categories
>
>
> Diffs
> -----
>
> gradle/test.gradle eb10cad78272a54f84e17ca6b984da224dccfbba
>
> Diff: https://reviews.apache.org/r/47157/diff/
>
>
> Testing
> -------
>
> Tested by running test, flakyTest, integrationTest. Will run further tests before pushing.
>
>
> Thanks,
>
> Anthony Baker
>
>
Re: Review Request 47157: GEODE-1374: Add a flakyTest build task
Posted by Mark Bretl <mb...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47157/#review137547
-----------------------------------------------------------
Ship it!
Ship It!
- Mark Bretl
On June 14, 2016, 9:58 a.m., Anthony Baker wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47157/
> -----------------------------------------------------------
>
> (Updated June 14, 2016, 9:58 a.m.)
>
>
> Review request for geode, Dick Cavender, Jens Deppe, Kirk Lund, Mark Bretl, and Dan Smith.
>
>
> Repository: geode
>
>
> Description
> -------
>
> Add a new test task specifically for running tests annotated with
> @FlakyTest. This allows us to isolate tests that exhibit
> intermittent failures. Other changes include:
>
> - Exclude flaky tests from test, integrationTest, uiTest, and
> distributedTest tasks
> - Cleanup how tests are found using the minimum set of
> include/exclude categories
>
>
> Diffs
> -----
>
> gradle/test.gradle eb10cad78272a54f84e17ca6b984da224dccfbba
>
> Diff: https://reviews.apache.org/r/47157/diff/
>
>
> Testing
> -------
>
> Tested by running test, flakyTest, integrationTest. Will run further tests before pushing.
>
>
> Thanks,
>
> Anthony Baker
>
>
Re: Review Request 47157: GEODE-1374: Add a flakyTest build task
Posted by Dan Smith <ds...@pivotal.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47157/#review137558
-----------------------------------------------------------
Ship it!
Ship It!
- Dan Smith
On June 14, 2016, 4:58 p.m., Anthony Baker wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47157/
> -----------------------------------------------------------
>
> (Updated June 14, 2016, 4:58 p.m.)
>
>
> Review request for geode, Dick Cavender, Jens Deppe, Kirk Lund, Mark Bretl, and Dan Smith.
>
>
> Repository: geode
>
>
> Description
> -------
>
> Add a new test task specifically for running tests annotated with
> @FlakyTest. This allows us to isolate tests that exhibit
> intermittent failures. Other changes include:
>
> - Exclude flaky tests from test, integrationTest, uiTest, and
> distributedTest tasks
> - Cleanup how tests are found using the minimum set of
> include/exclude categories
>
>
> Diffs
> -----
>
> gradle/test.gradle eb10cad78272a54f84e17ca6b984da224dccfbba
>
> Diff: https://reviews.apache.org/r/47157/diff/
>
>
> Testing
> -------
>
> Tested by running test, flakyTest, integrationTest. Will run further tests before pushing.
>
>
> Thanks,
>
> Anthony Baker
>
>
Re: Review Request 47157: GEODE-1374: Add a flakyTest build task
Posted by Dick Cavender <dc...@pivotal.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47157/#review137567
-----------------------------------------------------------
Ship it!
Ship It!
- Dick Cavender
On June 14, 2016, 4:58 p.m., Anthony Baker wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47157/
> -----------------------------------------------------------
>
> (Updated June 14, 2016, 4:58 p.m.)
>
>
> Review request for geode, Dick Cavender, Jens Deppe, Kirk Lund, Mark Bretl, and Dan Smith.
>
>
> Repository: geode
>
>
> Description
> -------
>
> Add a new test task specifically for running tests annotated with
> @FlakyTest. This allows us to isolate tests that exhibit
> intermittent failures. Other changes include:
>
> - Exclude flaky tests from test, integrationTest, uiTest, and
> distributedTest tasks
> - Cleanup how tests are found using the minimum set of
> include/exclude categories
>
>
> Diffs
> -----
>
> gradle/test.gradle eb10cad78272a54f84e17ca6b984da224dccfbba
>
> Diff: https://reviews.apache.org/r/47157/diff/
>
>
> Testing
> -------
>
> Tested by running test, flakyTest, integrationTest. Will run further tests before pushing.
>
>
> Thanks,
>
> Anthony Baker
>
>
Re: Review Request 47157: GEODE-1374: Add a flakyTest build task
Posted by Anthony Baker <ab...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47157/
-----------------------------------------------------------
(Updated June 14, 2016, 4:58 p.m.)
Review request for geode, Dick Cavender, Jens Deppe, Kirk Lund, Mark Bretl, and Dan Smith.
Repository: geode
Description
-------
Add a new test task specifically for running tests annotated with
@FlakyTest. This allows us to isolate tests that exhibit
intermittent failures. Other changes include:
- Exclude flaky tests from test, integrationTest, uiTest, and
distributedTest tasks
- Cleanup how tests are found using the minimum set of
include/exclude categories
Diffs (updated)
-----
gradle/test.gradle eb10cad78272a54f84e17ca6b984da224dccfbba
Diff: https://reviews.apache.org/r/47157/diff/
Testing
-------
Tested by running test, flakyTest, integrationTest. Will run further tests before pushing.
Thanks,
Anthony Baker