You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "David Van Couvering (JIRA)" <de...@db.apache.org> on 2006/03/15 18:58:58 UTC
[jira] Created: (DERBY-1116) Define a minimal acceptance test suite
for checkins
Define a minimal acceptance test suite for checkins
---------------------------------------------------
Key: DERBY-1116
URL: http://issues.apache.org/jira/browse/DERBY-1116
Project: Derby
Type: Improvement
Components: Test
Reporter: David Van Couvering
Priority: Minor
Now that we have an excellent notification system for tinderbox/nightly regression failures, I would like to suggest that we reduce the size of the test suite being run prior to checkin. I am not sure what should be in such a minimal test, but in particular I would like to remove things such as the stress test and generally reduce the number of tests being run for each subsystem/area of code.
As an example of how derbyall currently affects my productivity, I was running derbyall on my machine starting at 2pm, and by evening it was still running. At 9pm my machine was accidentally powered down, and this morning I am restarting the test run.
I have been tempted (and acted on such temptation) in the past to run a smaller set of tests, only to find out that I have blocked others who are running derbyall prior to checkin. For this reason, we need to define a minimal acceptance test (MATS) that we all agree to run prior to checkin.
One could argue that you can run your tests on another machine and thus reduce productivity, but we can't assume everybody in the community has nice big test servers to run their tests on.
If there are no objections, I can take a first pass at defining what this test suite should look like, but I suspect many others in the community have strong opinions about this and may even wish to volunteer to do this definition themselves (for example, some of you who may be working in the QA division in some of our Big Companies :) ).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1116) Define a minimal acceptance test
suite for checkins
Posted by "David Van Couvering (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1116?page=comments#action_12370587 ]
David Van Couvering commented on DERBY-1116:
--------------------------------------------
Submitted client-side test, revision 386187. Derbyall passed except for wisconsin.sql, which looks completely unrelated
> Define a minimal acceptance test suite for checkins
> ---------------------------------------------------
>
> Key: DERBY-1116
> URL: http://issues.apache.org/jira/browse/DERBY-1116
> Project: Derby
> Type: Improvement
> Components: Test
> Reporter: David Van Couvering
> Priority: Minor
>
> Now that we have an excellent notification system for tinderbox/nightly regression failures, I would like to suggest that we reduce the size of the test suite being run prior to checkin. I am not sure what should be in such a minimal test, but in particular I would like to remove things such as the stress test and generally reduce the number of tests being run for each subsystem/area of code.
> As an example of how derbyall currently affects my productivity, I was running derbyall on my machine starting at 2pm, and by evening it was still running. At 9pm my machine was accidentally powered down, and this morning I am restarting the test run.
> I have been tempted (and acted on such temptation) in the past to run a smaller set of tests, only to find out that I have blocked others who are running derbyall prior to checkin. For this reason, we need to define a minimal acceptance test (MATS) that we all agree to run prior to checkin.
> One could argue that you can run your tests on another machine and thus reduce productivity, but we can't assume everybody in the community has nice big test servers to run their tests on.
> If there are no objections, I can take a first pass at defining what this test suite should look like, but I suspect many others in the community have strong opinions about this and may even wish to volunteer to do this definition themselves (for example, some of you who may be working in the QA division in some of our Big Companies :) ).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1116) Define a minimal acceptance test
suite for checkins
Posted by "Rick Hillegas (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1116?page=comments#action_12370770 ]
Rick Hillegas commented on DERBY-1116:
--------------------------------------
Our regression tests clearly take too long. For the record, I think that 1 hour is a reasonable amount of time for a mats suite to churn. To date, several suggestions have been made for reducing the churn. I'm listing them from (off the top of my head) least to most expensive:
1) Fix the test harness so that it doesn't create a new database and bounce the server for every test.
2) Better parameterize the tests so that you could parallelize test runs.
3) Prune back the existing suites. I'm more or less clear about how you identify victims: e.g., tests which routinely generate false negatives, including the luckless tests which always flunk the tinderbox. I'm also reasonably certain that we could identify a handful of really good tests, the super-sensitive canaries in the mine shaft. Other than that, there is a huge mass of middling tests and I'm not clear on how you sift out the winners.
4) Start converting tests to JUnit to eliminate the drag of writing test chatter to disk. Dan's experiments with JUnit seemed to show some promise here.
> Define a minimal acceptance test suite for checkins
> ---------------------------------------------------
>
> Key: DERBY-1116
> URL: http://issues.apache.org/jira/browse/DERBY-1116
> Project: Derby
> Type: Improvement
> Components: Test
> Reporter: David Van Couvering
> Priority: Minor
>
> Now that we have an excellent notification system for tinderbox/nightly regression failures, I would like to suggest that we reduce the size of the test suite being run prior to checkin. I am not sure what should be in such a minimal test, but in particular I would like to remove things such as the stress test and generally reduce the number of tests being run for each subsystem/area of code.
> As an example of how derbyall currently affects my productivity, I was running derbyall on my machine starting at 2pm, and by evening it was still running. At 9pm my machine was accidentally powered down, and this morning I am restarting the test run.
> I have been tempted (and acted on such temptation) in the past to run a smaller set of tests, only to find out that I have blocked others who are running derbyall prior to checkin. For this reason, we need to define a minimal acceptance test (MATS) that we all agree to run prior to checkin.
> One could argue that you can run your tests on another machine and thus reduce productivity, but we can't assume everybody in the community has nice big test servers to run their tests on.
> If there are no objections, I can take a first pass at defining what this test suite should look like, but I suspect many others in the community have strong opinions about this and may even wish to volunteer to do this definition themselves (for example, some of you who may be working in the QA division in some of our Big Companies :) ).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1116) Define a minimal acceptance test
suite for checkins
Posted by "Daniel John Debrunner (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1116?page=comments#action_12370599 ]
Daniel John Debrunner commented on DERBY-1116:
----------------------------------------------
+1 to everything David just said - we are in agreement.
> Define a minimal acceptance test suite for checkins
> ---------------------------------------------------
>
> Key: DERBY-1116
> URL: http://issues.apache.org/jira/browse/DERBY-1116
> Project: Derby
> Type: Improvement
> Components: Test
> Reporter: David Van Couvering
> Priority: Minor
>
> Now that we have an excellent notification system for tinderbox/nightly regression failures, I would like to suggest that we reduce the size of the test suite being run prior to checkin. I am not sure what should be in such a minimal test, but in particular I would like to remove things such as the stress test and generally reduce the number of tests being run for each subsystem/area of code.
> As an example of how derbyall currently affects my productivity, I was running derbyall on my machine starting at 2pm, and by evening it was still running. At 9pm my machine was accidentally powered down, and this morning I am restarting the test run.
> I have been tempted (and acted on such temptation) in the past to run a smaller set of tests, only to find out that I have blocked others who are running derbyall prior to checkin. For this reason, we need to define a minimal acceptance test (MATS) that we all agree to run prior to checkin.
> One could argue that you can run your tests on another machine and thus reduce productivity, but we can't assume everybody in the community has nice big test servers to run their tests on.
> If there are no objections, I can take a first pass at defining what this test suite should look like, but I suspect many others in the community have strong opinions about this and may even wish to volunteer to do this definition themselves (for example, some of you who may be working in the QA division in some of our Big Companies :) ).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1116) Define a minimal acceptance test
suite for checkins
Posted by "Rick Hillegas (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1116?page=comments#action_12370873 ]
Rick Hillegas commented on DERBY-1116:
--------------------------------------
At the end of the day, we want to end up with a basket of tests which 1) covers Derby features and 2) doesn't take too long to run. I think your heuristic makes it easy to deliver on (2) but I'm not convinced it will do a very thorough job for (1). Handling (1) involves more analysis. Maybe you could run the code coverage tool and try to end up with a basket of tests which has similar code coverage results as the full derbyall run.
Again, however you pick the initial basket of tests, it's worth revising this basket as tinderbox failures disclose which excluded tests are actually catching real problems. We can add them back in and remove lower value tests from the basket.
> Define a minimal acceptance test suite for checkins
> ---------------------------------------------------
>
> Key: DERBY-1116
> URL: http://issues.apache.org/jira/browse/DERBY-1116
> Project: Derby
> Type: Improvement
> Components: Test
> Reporter: David Van Couvering
> Priority: Minor
>
> Now that we have an excellent notification system for tinderbox/nightly regression failures, I would like to suggest that we reduce the size of the test suite being run prior to checkin. I am not sure what should be in such a minimal test, but in particular I would like to remove things such as the stress test and generally reduce the number of tests being run for each subsystem/area of code.
> As an example of how derbyall currently affects my productivity, I was running derbyall on my machine starting at 2pm, and by evening it was still running. At 9pm my machine was accidentally powered down, and this morning I am restarting the test run.
> I have been tempted (and acted on such temptation) in the past to run a smaller set of tests, only to find out that I have blocked others who are running derbyall prior to checkin. For this reason, we need to define a minimal acceptance test (MATS) that we all agree to run prior to checkin.
> One could argue that you can run your tests on another machine and thus reduce productivity, but we can't assume everybody in the community has nice big test servers to run their tests on.
> If there are no objections, I can take a first pass at defining what this test suite should look like, but I suspect many others in the community have strong opinions about this and may even wish to volunteer to do this definition themselves (for example, some of you who may be working in the QA division in some of our Big Companies :) ).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (DERBY-1116) Define a minimal acceptance test
suite for checkins
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I also do not believe running derbyall is a requirement currently.
So far I run derbyall whenever I can. Recently for some patches that
are obviously only affecting backup code I have been running storeall,
and then paying attention to the tinderbox runs to make sure I didn't
miss anything.
It is obvious from the kinds of issues we have seen over the past few
months that no matter how much you test there may be an issue on a
platform that you don't have access to. So a committer should not only
worry about how much tests he can run, but should try to help out
watching the change in test results after a commit.
I believe the minimum set of tests a commiter should run is what they
believe is the "right" set of tests. For me I would say most of my
commits I have run derbyall, some I have run just a single suite, and
some test/javadoc changes just that test or javadoc build.
Daniel John Debrunner (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-1116?page=comments#action_12370592 ]
>
> Daniel John Debrunner commented on DERBY-1116:
> ----------------------------------------------
>
> Is there any actual requirement to run derbyall at the moment?
>
> I think a goal for everyone is to have derbyall running clean at all times on all platforms, which is not quite the same.
>
> I found this thread which talks about what folks do:
> http://www.nabble.com/Running-derbyall-before-submitting-patches-t30537.html#a86475
>
> I think I've mentioned before that expecting people to run derbyall on all platforms is unrealistic and therefore not a requirement,
> and running derbyall on a single common platform/jdk is sufficient, but I'm not sure anyone has said it must be done.
>
> If we do have a mandated smaller subset then we would need some guidelines on what to do when there are multiple commits,
> each that ran the minimal set, but we end up with several derbyall failures. How do we (who?) match failures to commits, how do we decide
> which commit to revert? If one of those commits ran derbyall before contributing, then is that change blessed and the others suspect?
>
> I've always assumed a model of run tests the contributor thinks are sufficient, maybe for some changes it's derbyall, for some it's none, for modifying
> a single test it's that test, for xa code it's the xa suite and the jdbcapi suite, etc. etc. Having the great tinderbox and regression testing is a huge help here, as you say, to running reduced testing. The downside is when multiple failures exist due to reduced contributor testing then it can effect a lot of other folks, in different time-zones. If I break derbyall at 5pm pacific time then it can effect the Europeans and Tomihito for a complete day until I come back the next day and address it, unless someone else has the itch to fix my mistake.
>
> I didn't understand your comment about "being tempted to run a smaller set of tests, but being blocked by others running derbyall".
> Is this because you are sharing test machines or something else?
>
>
>
>
>
>>Define a minimal acceptance test suite for checkins
>>---------------------------------------------------
>>
>> Key: DERBY-1116
>> URL: http://issues.apache.org/jira/browse/DERBY-1116
>> Project: Derby
>> Type: Improvement
>> Components: Test
>> Reporter: David Van Couvering
>> Priority: Minor
>
>
>>Now that we have an excellent notification system for tinderbox/nightly regression failures, I would like to suggest that we reduce the size of the test suite being run prior to checkin. I am not sure what should be in such a minimal test, but in particular I would like to remove things such as the stress test and generally reduce the number of tests being run for each subsystem/area of code.
>>As an example of how derbyall currently affects my productivity, I was running derbyall on my machine starting at 2pm, and by evening it was still running. At 9pm my machine was accidentally powered down, and this morning I am restarting the test run.
>>I have been tempted (and acted on such temptation) in the past to run a smaller set of tests, only to find out that I have blocked others who are running derbyall prior to checkin. For this reason, we need to define a minimal acceptance test (MATS) that we all agree to run prior to checkin.
>>One could argue that you can run your tests on another machine and thus reduce productivity, but we can't assume everybody in the community has nice big test servers to run their tests on.
>>If there are no objections, I can take a first pass at defining what this test suite should look like, but I suspect many others in the community have strong opinions about this and may even wish to volunteer to do this definition themselves (for example, some of you who may be working in the QA division in some of our Big Companies :) ).
>
>
[jira] Commented: (DERBY-1116) Define a minimal acceptance test
suite for checkins
Posted by "Daniel John Debrunner (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1116?page=comments#action_12370592 ]
Daniel John Debrunner commented on DERBY-1116:
----------------------------------------------
Is there any actual requirement to run derbyall at the moment?
I think a goal for everyone is to have derbyall running clean at all times on all platforms, which is not quite the same.
I found this thread which talks about what folks do:
http://www.nabble.com/Running-derbyall-before-submitting-patches-t30537.html#a86475
I think I've mentioned before that expecting people to run derbyall on all platforms is unrealistic and therefore not a requirement,
and running derbyall on a single common platform/jdk is sufficient, but I'm not sure anyone has said it must be done.
If we do have a mandated smaller subset then we would need some guidelines on what to do when there are multiple commits,
each that ran the minimal set, but we end up with several derbyall failures. How do we (who?) match failures to commits, how do we decide
which commit to revert? If one of those commits ran derbyall before contributing, then is that change blessed and the others suspect?
I've always assumed a model of run tests the contributor thinks are sufficient, maybe for some changes it's derbyall, for some it's none, for modifying
a single test it's that test, for xa code it's the xa suite and the jdbcapi suite, etc. etc. Having the great tinderbox and regression testing is a huge help here, as you say, to running reduced testing. The downside is when multiple failures exist due to reduced contributor testing then it can effect a lot of other folks, in different time-zones. If I break derbyall at 5pm pacific time then it can effect the Europeans and Tomihito for a complete day until I come back the next day and address it, unless someone else has the itch to fix my mistake.
I didn't understand your comment about "being tempted to run a smaller set of tests, but being blocked by others running derbyall".
Is this because you are sharing test machines or something else?
> Define a minimal acceptance test suite for checkins
> ---------------------------------------------------
>
> Key: DERBY-1116
> URL: http://issues.apache.org/jira/browse/DERBY-1116
> Project: Derby
> Type: Improvement
> Components: Test
> Reporter: David Van Couvering
> Priority: Minor
>
> Now that we have an excellent notification system for tinderbox/nightly regression failures, I would like to suggest that we reduce the size of the test suite being run prior to checkin. I am not sure what should be in such a minimal test, but in particular I would like to remove things such as the stress test and generally reduce the number of tests being run for each subsystem/area of code.
> As an example of how derbyall currently affects my productivity, I was running derbyall on my machine starting at 2pm, and by evening it was still running. At 9pm my machine was accidentally powered down, and this morning I am restarting the test run.
> I have been tempted (and acted on such temptation) in the past to run a smaller set of tests, only to find out that I have blocked others who are running derbyall prior to checkin. For this reason, we need to define a minimal acceptance test (MATS) that we all agree to run prior to checkin.
> One could argue that you can run your tests on another machine and thus reduce productivity, but we can't assume everybody in the community has nice big test servers to run their tests on.
> If there are no objections, I can take a first pass at defining what this test suite should look like, but I suspect many others in the community have strong opinions about this and may even wish to volunteer to do this definition themselves (for example, some of you who may be working in the QA division in some of our Big Companies :) ).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1116) Define a minimal acceptance test
suite for checkins
Posted by "David Van Couvering (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1116?page=comments#action_12370866 ]
David Van Couvering commented on DERBY-1116:
--------------------------------------------
The approach I was thinking of employing here is to modify RunTest to report the time to run each test (if it doesn't already do this), do a run of derbyall, and build a table of tests grouped by subsystem and then ordered by execution time. That would give us a sense of the low-hanging fruit we could pick out. The goal would be to reduce the number of tests until we get to an hour running on a "standard" development machine (I am thinking of a fairly modern desktop with 1GB of memory and a fairly fast disk).
Thoughts?
> Define a minimal acceptance test suite for checkins
> ---------------------------------------------------
>
> Key: DERBY-1116
> URL: http://issues.apache.org/jira/browse/DERBY-1116
> Project: Derby
> Type: Improvement
> Components: Test
> Reporter: David Van Couvering
> Priority: Minor
>
> Now that we have an excellent notification system for tinderbox/nightly regression failures, I would like to suggest that we reduce the size of the test suite being run prior to checkin. I am not sure what should be in such a minimal test, but in particular I would like to remove things such as the stress test and generally reduce the number of tests being run for each subsystem/area of code.
> As an example of how derbyall currently affects my productivity, I was running derbyall on my machine starting at 2pm, and by evening it was still running. At 9pm my machine was accidentally powered down, and this morning I am restarting the test run.
> I have been tempted (and acted on such temptation) in the past to run a smaller set of tests, only to find out that I have blocked others who are running derbyall prior to checkin. For this reason, we need to define a minimal acceptance test (MATS) that we all agree to run prior to checkin.
> One could argue that you can run your tests on another machine and thus reduce productivity, but we can't assume everybody in the community has nice big test servers to run their tests on.
> If there are no objections, I can take a first pass at defining what this test suite should look like, but I suspect many others in the community have strong opinions about this and may even wish to volunteer to do this definition themselves (for example, some of you who may be working in the QA division in some of our Big Companies :) ).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1116) Define a minimal acceptance test
suite for checkins
Posted by "David Van Couvering (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1116?page=comments#action_12370596 ]
David Van Couvering commented on DERBY-1116:
--------------------------------------------
Sorry, I should have made clear this was more a guideline than a requirement.
The issue is, if we all generally agree to follow a guideline in terms of what tests to run, then when someone *doesn't* run those tests, they potentially cause failures for other developers. That's what I meant about "blocking others" -- their derbyall starts failing because of my choice to run fewer tests and missing something -- same as what you mentioned about breaking derbyall and impacting others across multiple timezones. If Tomohito doesn't run derbyall but derbymats (a proposed smaller set of tests) then I can feel more comfortable running derbymats myself, rather than derbyall.
I agree that it is up to the contributor to decide what is sufficient, but if we had a general agreement in terms of what the Big Suite is for running prior to checkin, and that is a smaller set of tests than derbyall, then I think we can be more productive overall.
I do agree that if there are five submissions between tinderbox runs, it's hard to tell which one it is, but it's a lot easier than when we *didn't* have tinderbox tests and we might not find out for a day or two that there was a problem. I think that's why we all very seriously ran derbyall, it was as if all of us were taking responsibility for running full regression tests. Now I think we have Ole's automated suites doing that heavy lifting for us, and we can pull back a bit in terms of what we *recommend* developers do prior to committing a patch.
> Define a minimal acceptance test suite for checkins
> ---------------------------------------------------
>
> Key: DERBY-1116
> URL: http://issues.apache.org/jira/browse/DERBY-1116
> Project: Derby
> Type: Improvement
> Components: Test
> Reporter: David Van Couvering
> Priority: Minor
>
> Now that we have an excellent notification system for tinderbox/nightly regression failures, I would like to suggest that we reduce the size of the test suite being run prior to checkin. I am not sure what should be in such a minimal test, but in particular I would like to remove things such as the stress test and generally reduce the number of tests being run for each subsystem/area of code.
> As an example of how derbyall currently affects my productivity, I was running derbyall on my machine starting at 2pm, and by evening it was still running. At 9pm my machine was accidentally powered down, and this morning I am restarting the test run.
> I have been tempted (and acted on such temptation) in the past to run a smaller set of tests, only to find out that I have blocked others who are running derbyall prior to checkin. For this reason, we need to define a minimal acceptance test (MATS) that we all agree to run prior to checkin.
> One could argue that you can run your tests on another machine and thus reduce productivity, but we can't assume everybody in the community has nice big test servers to run their tests on.
> If there are no objections, I can take a first pass at defining what this test suite should look like, but I suspect many others in the community have strong opinions about this and may even wish to volunteer to do this definition themselves (for example, some of you who may be working in the QA division in some of our Big Companies :) ).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1116) Define a minimal acceptance test
suite for checkins
Posted by "David Van Couvering (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1116?page=comments#action_12370538 ]
David Van Couvering commented on DERBY-1116:
--------------------------------------------
"One could argue that you can run your tests on another machine and thus reduce productivity" -- what I *meant* to say was "and thus *improve* productivity" :)
> Define a minimal acceptance test suite for checkins
> ---------------------------------------------------
>
> Key: DERBY-1116
> URL: http://issues.apache.org/jira/browse/DERBY-1116
> Project: Derby
> Type: Improvement
> Components: Test
> Reporter: David Van Couvering
> Priority: Minor
>
> Now that we have an excellent notification system for tinderbox/nightly regression failures, I would like to suggest that we reduce the size of the test suite being run prior to checkin. I am not sure what should be in such a minimal test, but in particular I would like to remove things such as the stress test and generally reduce the number of tests being run for each subsystem/area of code.
> As an example of how derbyall currently affects my productivity, I was running derbyall on my machine starting at 2pm, and by evening it was still running. At 9pm my machine was accidentally powered down, and this morning I am restarting the test run.
> I have been tempted (and acted on such temptation) in the past to run a smaller set of tests, only to find out that I have blocked others who are running derbyall prior to checkin. For this reason, we need to define a minimal acceptance test (MATS) that we all agree to run prior to checkin.
> One could argue that you can run your tests on another machine and thus reduce productivity, but we can't assume everybody in the community has nice big test servers to run their tests on.
> If there are no objections, I can take a first pass at defining what this test suite should look like, but I suspect many others in the community have strong opinions about this and may even wish to volunteer to do this definition themselves (for example, some of you who may be working in the QA division in some of our Big Companies :) ).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1116) Define a minimal acceptance test
suite for checkins
Posted by "David Van Couvering (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1116?page=comments#action_12370542 ]
David Van Couvering commented on DERBY-1116:
--------------------------------------------
One more point: currently if you try to run derbyall more than once on the same machine, it fails due to network port conflicts. So you can only be testing one patch at a time, further impacting productivity.
> Define a minimal acceptance test suite for checkins
> ---------------------------------------------------
>
> Key: DERBY-1116
> URL: http://issues.apache.org/jira/browse/DERBY-1116
> Project: Derby
> Type: Improvement
> Components: Test
> Reporter: David Van Couvering
> Priority: Minor
>
> Now that we have an excellent notification system for tinderbox/nightly regression failures, I would like to suggest that we reduce the size of the test suite being run prior to checkin. I am not sure what should be in such a minimal test, but in particular I would like to remove things such as the stress test and generally reduce the number of tests being run for each subsystem/area of code.
> As an example of how derbyall currently affects my productivity, I was running derbyall on my machine starting at 2pm, and by evening it was still running. At 9pm my machine was accidentally powered down, and this morning I am restarting the test run.
> I have been tempted (and acted on such temptation) in the past to run a smaller set of tests, only to find out that I have blocked others who are running derbyall prior to checkin. For this reason, we need to define a minimal acceptance test (MATS) that we all agree to run prior to checkin.
> One could argue that you can run your tests on another machine and thus reduce productivity, but we can't assume everybody in the community has nice big test servers to run their tests on.
> If there are no objections, I can take a first pass at defining what this test suite should look like, but I suspect many others in the community have strong opinions about this and may even wish to volunteer to do this definition themselves (for example, some of you who may be working in the QA division in some of our Big Companies :) ).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1116) Define a minimal acceptance test
suite for checkins
Posted by "Rick Hillegas (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1116?page=comments#action_12370771 ]
Rick Hillegas commented on DERBY-1116:
--------------------------------------
If we prune out a slimmer mats, we can keep an eye on the full tinderbox run to spot the canaries in the mine shaft. We can then add them back into the mats.
> Define a minimal acceptance test suite for checkins
> ---------------------------------------------------
>
> Key: DERBY-1116
> URL: http://issues.apache.org/jira/browse/DERBY-1116
> Project: Derby
> Type: Improvement
> Components: Test
> Reporter: David Van Couvering
> Priority: Minor
>
> Now that we have an excellent notification system for tinderbox/nightly regression failures, I would like to suggest that we reduce the size of the test suite being run prior to checkin. I am not sure what should be in such a minimal test, but in particular I would like to remove things such as the stress test and generally reduce the number of tests being run for each subsystem/area of code.
> As an example of how derbyall currently affects my productivity, I was running derbyall on my machine starting at 2pm, and by evening it was still running. At 9pm my machine was accidentally powered down, and this morning I am restarting the test run.
> I have been tempted (and acted on such temptation) in the past to run a smaller set of tests, only to find out that I have blocked others who are running derbyall prior to checkin. For this reason, we need to define a minimal acceptance test (MATS) that we all agree to run prior to checkin.
> One could argue that you can run your tests on another machine and thus reduce productivity, but we can't assume everybody in the community has nice big test servers to run their tests on.
> If there are no objections, I can take a first pass at defining what this test suite should look like, but I suspect many others in the community have strong opinions about this and may even wish to volunteer to do this definition themselves (for example, some of you who may be working in the QA division in some of our Big Companies :) ).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1116) Define a minimal acceptance test
suite for checkins
Posted by "Satheesh Bandaram (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1116?page=comments#action_12370875 ]
Satheesh Bandaram commented on DERBY-1116:
------------------------------------------
I am not sure picking fastest tests is the way to go for derbymats. May be we could look for tests that actually cought issues in the past as good candidates and also based on perceived value of a test. This may be harder to do than based on automatic process, but if derbymats are not going to catch issues, it may "block" others who do run derbyAll.
With so many new contributors and so many new changes going in these days, I am little nervous that we are trying to prune down minimum tests needed to checkin. I wonder if effort could be placed else where, like making tests run faster by removing database creation etc. Even with everyone running derbyAll, we have so many failures these days..
> Define a minimal acceptance test suite for checkins
> ---------------------------------------------------
>
> Key: DERBY-1116
> URL: http://issues.apache.org/jira/browse/DERBY-1116
> Project: Derby
> Type: Improvement
> Components: Test
> Reporter: David Van Couvering
> Priority: Minor
>
> Now that we have an excellent notification system for tinderbox/nightly regression failures, I would like to suggest that we reduce the size of the test suite being run prior to checkin. I am not sure what should be in such a minimal test, but in particular I would like to remove things such as the stress test and generally reduce the number of tests being run for each subsystem/area of code.
> As an example of how derbyall currently affects my productivity, I was running derbyall on my machine starting at 2pm, and by evening it was still running. At 9pm my machine was accidentally powered down, and this morning I am restarting the test run.
> I have been tempted (and acted on such temptation) in the past to run a smaller set of tests, only to find out that I have blocked others who are running derbyall prior to checkin. For this reason, we need to define a minimal acceptance test (MATS) that we all agree to run prior to checkin.
> One could argue that you can run your tests on another machine and thus reduce productivity, but we can't assume everybody in the community has nice big test servers to run their tests on.
> If there are no objections, I can take a first pass at defining what this test suite should look like, but I suspect many others in the community have strong opinions about this and may even wish to volunteer to do this definition themselves (for example, some of you who may be working in the QA division in some of our Big Companies :) ).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1116) Define a minimal acceptance test
suite for checkins
Posted by "David Van Couvering (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1116?page=comments#action_12370588 ]
David Van Couvering commented on DERBY-1116:
--------------------------------------------
Sorry, that last comment was for DERBY-1115, off-by-one error.
> Define a minimal acceptance test suite for checkins
> ---------------------------------------------------
>
> Key: DERBY-1116
> URL: http://issues.apache.org/jira/browse/DERBY-1116
> Project: Derby
> Type: Improvement
> Components: Test
> Reporter: David Van Couvering
> Priority: Minor
>
> Now that we have an excellent notification system for tinderbox/nightly regression failures, I would like to suggest that we reduce the size of the test suite being run prior to checkin. I am not sure what should be in such a minimal test, but in particular I would like to remove things such as the stress test and generally reduce the number of tests being run for each subsystem/area of code.
> As an example of how derbyall currently affects my productivity, I was running derbyall on my machine starting at 2pm, and by evening it was still running. At 9pm my machine was accidentally powered down, and this morning I am restarting the test run.
> I have been tempted (and acted on such temptation) in the past to run a smaller set of tests, only to find out that I have blocked others who are running derbyall prior to checkin. For this reason, we need to define a minimal acceptance test (MATS) that we all agree to run prior to checkin.
> One could argue that you can run your tests on another machine and thus reduce productivity, but we can't assume everybody in the community has nice big test servers to run their tests on.
> If there are no objections, I can take a first pass at defining what this test suite should look like, but I suspect many others in the community have strong opinions about this and may even wish to volunteer to do this definition themselves (for example, some of you who may be working in the QA division in some of our Big Companies :) ).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira