You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geronimo.apache.org by David Jencks <da...@yahoo.com> on 2007/12/31 22:41:49 UTC
Running integration tests by default?
First of all I appear to have broken the build last night with some
changes to get the roller plugin building again. I think I've
managed to fix all the problems -- the it tests all pass for me. Let
me know if there are still problems.
I think its still too hard to run the integration tests. I've made a
possibly annoying change so that the default build includes IT. If
you don't want them run
mvn clean install -P no-it
If this is too annoying we could reverse the profiles and have the
default leave out the it as before and a with-it profile that
includes them.
Comments?
This might have bad effects on Prasad's automation but I'm not sure
how that is run.
thanks && Happy New Year!
david jencks
Re: Running integration tests by default?
Posted by Jacek Laskowski <ja...@laskowski.net.pl>.
On Dec 31, 2007 10:41 PM, David Jencks <da...@yahoo.com> wrote:
> I think its still too hard to run the integration tests. I've made a
> possibly annoying change so that the default build includes IT. If
> you don't want them run
>
> mvn clean install -P no-it
+1 (the developers should run the tests every small change and an hour
away from a computer will make our health better - no worries the
build's broken and more moves ;-)).
Jacek
--
Jacek Laskowski
http://www.JacekLaskowski.pl
Re: Running integration tests by default?
Posted by Donald Woods <dw...@apache.org>.
I finally got the testsuite to run by setting up a new OpenSUSE 10.3 VM
image with Maven 2.0.7 and Sun 1.5.0_13, but I'm still seeing the same
mvn unit test failures when building some of the server plugins with
car-maven-plugin.....
Guess you guys with the Macs have such a controlled OS/Java setup, that
you don't run into the problems that we Linux and Windows users often
have... :-)
-Donald
Donald Woods wrote:
> I have yet to get the testsuite to complete on my laptop running
> RHEL5u1. Guess I need to find some time to debug each failure until I
> get down to the same number of failures Prasad gets on his automated
> runs....
>
> -Donald
>
>
> David Jencks wrote:
>>
>> On Jan 5, 2008, at 11:34 AM, Donald Woods wrote:
>>
>>> Taking another hour to run the IT tests on Tomcat and Jetty (on top
>>> of the 30 mins. to run a full build) is just too much wasted time in
>>> my busy schedule... :-) Besides, we have Prasad's scheduled builds
>>> running with IT on several times a day, which is close enough to
>>> continuous builds to catch build breaks.
>>>
>>> I'd rather see more automated builds with every build running the IT
>>> instead of forcing it as the default on all our developers. If
>>> someone is working on an isolated piece of code (like Console
>>> portlets or non-core plugin like remote mgmt or clustering) then I
>>> don't see how always running IT would outweigh the decreased
>>> productivity of increased build times...
>>
>> I don't think I understand your point of view yet. IIUC we are
>> discussing whether we want
>>
>> A. mvn clean install runs the integration tests, and if you don't want
>> to wait for them you can type ctrl-c at any point or run mvn clean
>> install -P no-it
>>
>> B. mvn clean install does not run the integration tests and if you do
>> want them you type mvn clean install -P with-it
>>
>> Obviously its not a lot of fun to wait around while the it tests run
>> and I don't expect everyone to run them before most commits. To me
>> however its a good principle to run all the available tests before
>> committing ( #1 below) so we should express that principle by making
>> that happen in the default build ( # 2 below). If in your judgement
>> your changes are not likely to affect the integration tests you are
>> free to not run them (#3 below), or indeed to not do a complete build
>> of the server: I for one frequently don't do a full server build after
>> changes I think are minor. (Sometimes this results in me staying up
>> very late trying to fix the build after I later discover I broke it)
>>
>> I understand you don't want to run the integration tests before every
>> commit, and I don't either and don't expect to. I'm thinking more of
>> the case when I've worked until 3 AM and think I have everything
>> working so I do a full build overnight as a last check before
>> committing in the morning..... no way will I remember to type -P
>> with-it but in this situation there's plenty of time to run all the
>> integration tests and a big advantage to doing so. If you are sitting
>> around waiting for a build to complete and you forgot to type -P no-it
>> you can still interrupt the build anytime you want -- after your
>> module builds, after one assembly completes so you can try starting
>> it, after some itests finish.... whenever you want.
>>
>>
>>>
>>> Also, there have been 4 console tests broken for well over a week
>>> now, with no apparent rush to fix them, so everyone is now used to
>>> ignoring the Build Failure notices....
>>>
>>
>> I don't think you've been following closely, or my mail isn't
>> working. For quite a while there was an incompatible xbean version
>> problem with openejb which Jarek identified and fixed on Jan 2. We
>> then had 8 successful 2.1 builds from Jan 2 to yesterday. We've had 3
>> failed builds since, one possibly anomolous in geronimo-system, and 2
>> with the broken console tests you mention. I ran the itests before my
>> last commit and these console tests passed, and I am not a console
>> expert, so I haven't looked into them yet.
>>
>> However, whatever the specific facts about the current failures, I
>> think we generally don't pay enough attention to itest failures.
>> Personally running them myself a couple times a week is the best way
>> I've thought of to keep them on my personal radar. Do you have any
>> ideas how to generally raise the level of attention paid to them?
>>
>> thanks
>> david jencks
>>
>>>
>>>
>>> -Donald
>>>
>>> David Jencks wrote:
>>>> On Jan 4, 2008, at 1:07 PM, Donald Woods wrote:
>>>>> How about not running the integration tests by default and letting
>>>>> those who want them either 1) use a "-P testsuite" option from
>>>>> trunk or 2) run mvn inside the testsuite subdir?
>>>> (2) is what we had before my change and didn't work for me -- I
>>>> never ran the testsuite.
>>>> My thinking on my change is:
>>>> 1. Its better to run more tests. For instance if you see a IT
>>>> failure on your machine I think you are much more likely to fix it
>>>> than if you have to wait for the notification from Prasad's
>>>> automation. I think this is why we have unit tests as well :-)
>>>> 2. People are lazy and don't like to type, so the default option
>>>> should include the maximum level of testing.
>>>> 3. In the case of the testsuite, these are run after the rest of the
>>>> build is complete, so you are free to interrupt the build if you get
>>>> too bored/impatient.
>>>> Could you explain why you disagree with this or what factors you
>>>> consider more important?
>>>> thanks
>>>> david jencks
>>>>>
>>>>> -Donald
>>>>>
>>>>>
>>>>> David Jencks wrote:
>>>>>> On Jan 2, 2008, at 7:58 AM, Prasad Kashyap wrote:
>>>>>>> On Dec 31, 2007 4:41 PM, David Jencks <da...@yahoo.com>
>>>>>>> wrote:
>>>>>>>> First of all I appear to have broken the build last night with some
>>>>>>>> changes to get the roller plugin building again. I think I've
>>>>>>>> managed to fix all the problems -- the it tests all pass for
>>>>>>>> me. Let
>>>>>>>> me know if there are still problems.
>>>>>>>>
>>>>>>>> I think its still too hard to run the integration tests.
>>>>>>>
>>>>>>> I would like to know what exactly you think is hard about it. It
>>>>>>> would
>>>>>>> be great if you could please share your thoughts and ideas on making
>>>>>>> it simpler.
>>>>>> I didn't see a way to run the integration tests with the main
>>>>>> build without a separate command. Unless I can run everything in
>>>>>> one command, I'm pretty sure laziness will take over and I won't
>>>>>> run the integration tests.
>>>>>>>
>>>>>>>> I've made a
>>>>>>>> possibly annoying change so that the default build includes IT. If
>>>>>>>> you don't want them run
>>>>>>>>
>>>>>>>> mvn clean install -P no-it
>>>>>>>>
>>>>>>>> If this is too annoying we could reverse the profiles and have the
>>>>>>>> default leave out the it as before and a with-it profile that
>>>>>>>> includes them.
>>>>>>>
>>>>>>> Yeah. I think the default profile should not run the IT. IMHO, I
>>>>>>> think
>>>>>>> it should not even run the unit tests by default. Developers
>>>>>>> (should)
>>>>>>> run unit & IT tests before committing their code. And we have
>>>>>>> automation builds with all tests that run 4 times a day anyways. So
>>>>>>> the default profile can well do away with tests. But that may
>>>>>>> just be
>>>>>>> my opinion.
>>>>>> After working through some hard problem and getting ready to
>>>>>> commit the last thing anyone wants to do is remember a second
>>>>>> command or even a command line option to run some tests: they are
>>>>>> apt to type the simplest command that will check the build. I
>>>>>> think that command should run all the tests, including the
>>>>>> integration tests. Even though there's enough time to go eat
>>>>>> dinner while they run, our most comprehensive checks will be run.
>>>>>> If you get too bored you can always stop the build after you think
>>>>>> enough stuff has been checked, and if you remember you can run the
>>>>>> build with options to turn off whatever tests you want to skip.
>>>>>> ApacheDS had their integration tests run using an option and most
>>>>>> people did not run them due to the extra effort of trying to
>>>>>> remember to type more on the command line. I think they've
>>>>>> changed so the integration tests are run by default.
>>>>>> I'm happy to keep talking about this.... and I'll be happy enough
>>>>>> with a "with-it" option, but I think running everything by default
>>>>>> is the best strategy.
>>>>>>>
>>>>>>>>
>>>>>>>> Comments?
>>>>>>>>
>>>>>>>> This might have bad effects on Prasad's automation but I'm not sure
>>>>>>>> how that is run.
>>>>>>>
>>>>>>> For now, the automation builds have been modified to use the
>>>>>>> no-it profile.
>>>>>>>
>>>>>> thanks!
>>>>>> david jencks
>>>>>>>>
>>>>>>>> thanks && Happy New Year!
>>>>>>>> david jencks
>>>>>>>
>>>>>>> Happy New Year to ya'll !
>>>>>>> Prasad
>>
>>
Re: Running integration tests by default?
Posted by Donald Woods <dw...@apache.org>.
I have yet to get the testsuite to complete on my laptop running
RHEL5u1. Guess I need to find some time to debug each failure until I
get down to the same number of failures Prasad gets on his automated
runs....
-Donald
David Jencks wrote:
>
> On Jan 5, 2008, at 11:34 AM, Donald Woods wrote:
>
>> Taking another hour to run the IT tests on Tomcat and Jetty (on top of
>> the 30 mins. to run a full build) is just too much wasted time in my
>> busy schedule... :-) Besides, we have Prasad's scheduled builds
>> running with IT on several times a day, which is close enough to
>> continuous builds to catch build breaks.
>>
>> I'd rather see more automated builds with every build running the IT
>> instead of forcing it as the default on all our developers. If
>> someone is working on an isolated piece of code (like Console portlets
>> or non-core plugin like remote mgmt or clustering) then I don't see
>> how always running IT would outweigh the decreased productivity of
>> increased build times...
>
> I don't think I understand your point of view yet. IIUC we are
> discussing whether we want
>
> A. mvn clean install runs the integration tests, and if you don't want
> to wait for them you can type ctrl-c at any point or run mvn clean
> install -P no-it
>
> B. mvn clean install does not run the integration tests and if you do
> want them you type mvn clean install -P with-it
>
> Obviously its not a lot of fun to wait around while the it tests run and
> I don't expect everyone to run them before most commits. To me however
> its a good principle to run all the available tests before committing (
> #1 below) so we should express that principle by making that happen in
> the default build ( # 2 below). If in your judgement your changes are
> not likely to affect the integration tests you are free to not run them
> (#3 below), or indeed to not do a complete build of the server: I for
> one frequently don't do a full server build after changes I think are
> minor. (Sometimes this results in me staying up very late trying to fix
> the build after I later discover I broke it)
>
> I understand you don't want to run the integration tests before every
> commit, and I don't either and don't expect to. I'm thinking more of
> the case when I've worked until 3 AM and think I have everything working
> so I do a full build overnight as a last check before committing in the
> morning..... no way will I remember to type -P with-it but in this
> situation there's plenty of time to run all the integration tests and a
> big advantage to doing so. If you are sitting around waiting for a
> build to complete and you forgot to type -P no-it you can still
> interrupt the build anytime you want -- after your module builds, after
> one assembly completes so you can try starting it, after some itests
> finish.... whenever you want.
>
>
>>
>> Also, there have been 4 console tests broken for well over a week now,
>> with no apparent rush to fix them, so everyone is now used to ignoring
>> the Build Failure notices....
>>
>
> I don't think you've been following closely, or my mail isn't working.
> For quite a while there was an incompatible xbean version problem with
> openejb which Jarek identified and fixed on Jan 2. We then had 8
> successful 2.1 builds from Jan 2 to yesterday. We've had 3 failed
> builds since, one possibly anomolous in geronimo-system, and 2 with the
> broken console tests you mention. I ran the itests before my last
> commit and these console tests passed, and I am not a console expert, so
> I haven't looked into them yet.
>
> However, whatever the specific facts about the current failures, I think
> we generally don't pay enough attention to itest failures. Personally
> running them myself a couple times a week is the best way I've thought
> of to keep them on my personal radar. Do you have any ideas how to
> generally raise the level of attention paid to them?
>
> thanks
> david jencks
>
>>
>>
>> -Donald
>>
>> David Jencks wrote:
>>> On Jan 4, 2008, at 1:07 PM, Donald Woods wrote:
>>>> How about not running the integration tests by default and letting
>>>> those who want them either 1) use a "-P testsuite" option from trunk
>>>> or 2) run mvn inside the testsuite subdir?
>>> (2) is what we had before my change and didn't work for me -- I never
>>> ran the testsuite.
>>> My thinking on my change is:
>>> 1. Its better to run more tests. For instance if you see a IT
>>> failure on your machine I think you are much more likely to fix it
>>> than if you have to wait for the notification from Prasad's
>>> automation. I think this is why we have unit tests as well :-)
>>> 2. People are lazy and don't like to type, so the default option
>>> should include the maximum level of testing.
>>> 3. In the case of the testsuite, these are run after the rest of the
>>> build is complete, so you are free to interrupt the build if you get
>>> too bored/impatient.
>>> Could you explain why you disagree with this or what factors you
>>> consider more important?
>>> thanks
>>> david jencks
>>>>
>>>> -Donald
>>>>
>>>>
>>>> David Jencks wrote:
>>>>> On Jan 2, 2008, at 7:58 AM, Prasad Kashyap wrote:
>>>>>> On Dec 31, 2007 4:41 PM, David Jencks <da...@yahoo.com> wrote:
>>>>>>> First of all I appear to have broken the build last night with some
>>>>>>> changes to get the roller plugin building again. I think I've
>>>>>>> managed to fix all the problems -- the it tests all pass for me.
>>>>>>> Let
>>>>>>> me know if there are still problems.
>>>>>>>
>>>>>>> I think its still too hard to run the integration tests.
>>>>>>
>>>>>> I would like to know what exactly you think is hard about it. It
>>>>>> would
>>>>>> be great if you could please share your thoughts and ideas on making
>>>>>> it simpler.
>>>>> I didn't see a way to run the integration tests with the main build
>>>>> without a separate command. Unless I can run everything in one
>>>>> command, I'm pretty sure laziness will take over and I won't run
>>>>> the integration tests.
>>>>>>
>>>>>>> I've made a
>>>>>>> possibly annoying change so that the default build includes IT. If
>>>>>>> you don't want them run
>>>>>>>
>>>>>>> mvn clean install -P no-it
>>>>>>>
>>>>>>> If this is too annoying we could reverse the profiles and have the
>>>>>>> default leave out the it as before and a with-it profile that
>>>>>>> includes them.
>>>>>>
>>>>>> Yeah. I think the default profile should not run the IT. IMHO, I
>>>>>> think
>>>>>> it should not even run the unit tests by default. Developers (should)
>>>>>> run unit & IT tests before committing their code. And we have
>>>>>> automation builds with all tests that run 4 times a day anyways. So
>>>>>> the default profile can well do away with tests. But that may just be
>>>>>> my opinion.
>>>>> After working through some hard problem and getting ready to commit
>>>>> the last thing anyone wants to do is remember a second command or
>>>>> even a command line option to run some tests: they are apt to type
>>>>> the simplest command that will check the build. I think that
>>>>> command should run all the tests, including the integration tests.
>>>>> Even though there's enough time to go eat dinner while they run,
>>>>> our most comprehensive checks will be run. If you get too bored
>>>>> you can always stop the build after you think enough stuff has been
>>>>> checked, and if you remember you can run the build with options to
>>>>> turn off whatever tests you want to skip.
>>>>> ApacheDS had their integration tests run using an option and most
>>>>> people did not run them due to the extra effort of trying to
>>>>> remember to type more on the command line. I think they've changed
>>>>> so the integration tests are run by default.
>>>>> I'm happy to keep talking about this.... and I'll be happy enough
>>>>> with a "with-it" option, but I think running everything by default
>>>>> is the best strategy.
>>>>>>
>>>>>>>
>>>>>>> Comments?
>>>>>>>
>>>>>>> This might have bad effects on Prasad's automation but I'm not sure
>>>>>>> how that is run.
>>>>>>
>>>>>> For now, the automation builds have been modified to use the no-it
>>>>>> profile.
>>>>>>
>>>>> thanks!
>>>>> david jencks
>>>>>>>
>>>>>>> thanks && Happy New Year!
>>>>>>> david jencks
>>>>>>
>>>>>> Happy New Year to ya'll !
>>>>>> Prasad
>
>
Re: Running integration tests by default?
Posted by David Jencks <da...@yahoo.com>.
On Jan 5, 2008, at 11:34 AM, Donald Woods wrote:
> Taking another hour to run the IT tests on Tomcat and Jetty (on top
> of the 30 mins. to run a full build) is just too much wasted time
> in my busy schedule... :-) Besides, we have Prasad's scheduled
> builds running with IT on several times a day, which is close
> enough to continuous builds to catch build breaks.
>
> I'd rather see more automated builds with every build running the
> IT instead of forcing it as the default on all our developers. If
> someone is working on an isolated piece of code (like Console
> portlets or non-core plugin like remote mgmt or clustering) then I
> don't see how always running IT would outweigh the decreased
> productivity of increased build times...
I don't think I understand your point of view yet. IIUC we are
discussing whether we want
A. mvn clean install runs the integration tests, and if you don't
want to wait for them you can type ctrl-c at any point or run mvn
clean install -P no-it
B. mvn clean install does not run the integration tests and if you do
want them you type mvn clean install -P with-it
Obviously its not a lot of fun to wait around while the it tests run
and I don't expect everyone to run them before most commits. To me
however its a good principle to run all the available tests before
committing ( #1 below) so we should express that principle by making
that happen in the default build ( # 2 below). If in your judgement
your changes are not likely to affect the integration tests you are
free to not run them (#3 below), or indeed to not do a complete build
of the server: I for one frequently don't do a full server build
after changes I think are minor. (Sometimes this results in me
staying up very late trying to fix the build after I later discover I
broke it)
I understand you don't want to run the integration tests before every
commit, and I don't either and don't expect to. I'm thinking more of
the case when I've worked until 3 AM and think I have everything
working so I do a full build overnight as a last check before
committing in the morning..... no way will I remember to type -P with-
it but in this situation there's plenty of time to run all the
integration tests and a big advantage to doing so. If you are
sitting around waiting for a build to complete and you forgot to type
-P no-it you can still interrupt the build anytime you want -- after
your module builds, after one assembly completes so you can try
starting it, after some itests finish.... whenever you want.
>
> Also, there have been 4 console tests broken for well over a week
> now, with no apparent rush to fix them, so everyone is now used to
> ignoring the Build Failure notices....
>
I don't think you've been following closely, or my mail isn't
working. For quite a while there was an incompatible xbean version
problem with openejb which Jarek identified and fixed on Jan 2. We
then had 8 successful 2.1 builds from Jan 2 to yesterday. We've had
3 failed builds since, one possibly anomolous in geronimo-system, and
2 with the broken console tests you mention. I ran the itests before
my last commit and these console tests passed, and I am not a console
expert, so I haven't looked into them yet.
However, whatever the specific facts about the current failures, I
think we generally don't pay enough attention to itest failures.
Personally running them myself a couple times a week is the best way
I've thought of to keep them on my personal radar. Do you have any
ideas how to generally raise the level of attention paid to them?
thanks
david jencks
>
>
> -Donald
>
> David Jencks wrote:
>> On Jan 4, 2008, at 1:07 PM, Donald Woods wrote:
>>> How about not running the integration tests by default and
>>> letting those who want them either 1) use a "-P testsuite" option
>>> from trunk or 2) run mvn inside the testsuite subdir?
>> (2) is what we had before my change and didn't work for me -- I
>> never ran the testsuite.
>> My thinking on my change is:
>> 1. Its better to run more tests. For instance if you see a IT
>> failure on your machine I think you are much more likely to fix it
>> than if you have to wait for the notification from Prasad's
>> automation. I think this is why we have unit tests as well :-)
>> 2. People are lazy and don't like to type, so the default option
>> should include the maximum level of testing.
>> 3. In the case of the testsuite, these are run after the rest of
>> the build is complete, so you are free to interrupt the build if
>> you get too bored/impatient.
>> Could you explain why you disagree with this or what factors you
>> consider more important?
>> thanks
>> david jencks
>>>
>>> -Donald
>>>
>>>
>>> David Jencks wrote:
>>>> On Jan 2, 2008, at 7:58 AM, Prasad Kashyap wrote:
>>>>> On Dec 31, 2007 4:41 PM, David Jencks <da...@yahoo.com>
>>>>> wrote:
>>>>>> First of all I appear to have broken the build last night with
>>>>>> some
>>>>>> changes to get the roller plugin building again. I think I've
>>>>>> managed to fix all the problems -- the it tests all pass for
>>>>>> me. Let
>>>>>> me know if there are still problems.
>>>>>>
>>>>>> I think its still too hard to run the integration tests.
>>>>>
>>>>> I would like to know what exactly you think is hard about it.
>>>>> It would
>>>>> be great if you could please share your thoughts and ideas on
>>>>> making
>>>>> it simpler.
>>>> I didn't see a way to run the integration tests with the main
>>>> build without a separate command. Unless I can run everything
>>>> in one command, I'm pretty sure laziness will take over and I
>>>> won't run the integration tests.
>>>>>
>>>>>> I've made a
>>>>>> possibly annoying change so that the default build includes
>>>>>> IT. If
>>>>>> you don't want them run
>>>>>>
>>>>>> mvn clean install -P no-it
>>>>>>
>>>>>> If this is too annoying we could reverse the profiles and have
>>>>>> the
>>>>>> default leave out the it as before and a with-it profile that
>>>>>> includes them.
>>>>>
>>>>> Yeah. I think the default profile should not run the IT. IMHO,
>>>>> I think
>>>>> it should not even run the unit tests by default. Developers
>>>>> (should)
>>>>> run unit & IT tests before committing their code. And we have
>>>>> automation builds with all tests that run 4 times a day
>>>>> anyways. So
>>>>> the default profile can well do away with tests. But that may
>>>>> just be
>>>>> my opinion.
>>>> After working through some hard problem and getting ready to
>>>> commit the last thing anyone wants to do is remember a second
>>>> command or even a command line option to run some tests: they
>>>> are apt to type the simplest command that will check the build.
>>>> I think that command should run all the tests, including the
>>>> integration tests. Even though there's enough time to go eat
>>>> dinner while they run, our most comprehensive checks will be
>>>> run. If you get too bored you can always stop the build after
>>>> you think enough stuff has been checked, and if you remember you
>>>> can run the build with options to turn off whatever tests you
>>>> want to skip.
>>>> ApacheDS had their integration tests run using an option and
>>>> most people did not run them due to the extra effort of trying
>>>> to remember to type more on the command line. I think they've
>>>> changed so the integration tests are run by default.
>>>> I'm happy to keep talking about this.... and I'll be happy
>>>> enough with a "with-it" option, but I think running everything
>>>> by default is the best strategy.
>>>>>
>>>>>>
>>>>>> Comments?
>>>>>>
>>>>>> This might have bad effects on Prasad's automation but I'm not
>>>>>> sure
>>>>>> how that is run.
>>>>>
>>>>> For now, the automation builds have been modified to use the no-
>>>>> it profile.
>>>>>
>>>> thanks!
>>>> david jencks
>>>>>>
>>>>>> thanks && Happy New Year!
>>>>>> david jencks
>>>>>
>>>>> Happy New Year to ya'll !
>>>>> Prasad
Re: Running integration tests by default?
Posted by Donald Woods <dw...@apache.org>.
Taking another hour to run the IT tests on Tomcat and Jetty (on top of
the 30 mins. to run a full build) is just too much wasted time in my
busy schedule... :-) Besides, we have Prasad's scheduled builds running
with IT on several times a day, which is close enough to continuous
builds to catch build breaks.
I'd rather see more automated builds with every build running the IT
instead of forcing it as the default on all our developers. If someone
is working on an isolated piece of code (like Console portlets or
non-core plugin like remote mgmt or clustering) then I don't see how
always running IT would outweigh the decreased productivity of increased
build times...
Also, there have been 4 console tests broken for well over a week now,
with no apparent rush to fix them, so everyone is now used to ignoring
the Build Failure notices....
-Donald
David Jencks wrote:
>
> On Jan 4, 2008, at 1:07 PM, Donald Woods wrote:
>
>> How about not running the integration tests by default and letting
>> those who want them either 1) use a "-P testsuite" option from trunk
>> or 2) run mvn inside the testsuite subdir?
> (2) is what we had before my change and didn't work for me -- I never
> ran the testsuite.
>
> My thinking on my change is:
>
> 1. Its better to run more tests. For instance if you see a IT failure
> on your machine I think you are much more likely to fix it than if you
> have to wait for the notification from Prasad's automation. I think
> this is why we have unit tests as well :-)
> 2. People are lazy and don't like to type, so the default option should
> include the maximum level of testing.
> 3. In the case of the testsuite, these are run after the rest of the
> build is complete, so you are free to interrupt the build if you get too
> bored/impatient.
>
> Could you explain why you disagree with this or what factors you
> consider more important?
>
> thanks
> david jencks
>
>>
>> -Donald
>>
>>
>> David Jencks wrote:
>>> On Jan 2, 2008, at 7:58 AM, Prasad Kashyap wrote:
>>>> On Dec 31, 2007 4:41 PM, David Jencks <da...@yahoo.com> wrote:
>>>>> First of all I appear to have broken the build last night with some
>>>>> changes to get the roller plugin building again. I think I've
>>>>> managed to fix all the problems -- the it tests all pass for me. Let
>>>>> me know if there are still problems.
>>>>>
>>>>> I think its still too hard to run the integration tests.
>>>>
>>>> I would like to know what exactly you think is hard about it. It would
>>>> be great if you could please share your thoughts and ideas on making
>>>> it simpler.
>>> I didn't see a way to run the integration tests with the main build
>>> without a separate command. Unless I can run everything in one
>>> command, I'm pretty sure laziness will take over and I won't run the
>>> integration tests.
>>>>
>>>>> I've made a
>>>>> possibly annoying change so that the default build includes IT. If
>>>>> you don't want them run
>>>>>
>>>>> mvn clean install -P no-it
>>>>>
>>>>> If this is too annoying we could reverse the profiles and have the
>>>>> default leave out the it as before and a with-it profile that
>>>>> includes them.
>>>>
>>>> Yeah. I think the default profile should not run the IT. IMHO, I think
>>>> it should not even run the unit tests by default. Developers (should)
>>>> run unit & IT tests before committing their code. And we have
>>>> automation builds with all tests that run 4 times a day anyways. So
>>>> the default profile can well do away with tests. But that may just be
>>>> my opinion.
>>> After working through some hard problem and getting ready to commit
>>> the last thing anyone wants to do is remember a second command or
>>> even a command line option to run some tests: they are apt to type
>>> the simplest command that will check the build. I think that command
>>> should run all the tests, including the integration tests. Even
>>> though there's enough time to go eat dinner while they run, our most
>>> comprehensive checks will be run. If you get too bored you can
>>> always stop the build after you think enough stuff has been checked,
>>> and if you remember you can run the build with options to turn off
>>> whatever tests you want to skip.
>>> ApacheDS had their integration tests run using an option and most
>>> people did not run them due to the extra effort of trying to remember
>>> to type more on the command line. I think they've changed so the
>>> integration tests are run by default.
>>> I'm happy to keep talking about this.... and I'll be happy enough
>>> with a "with-it" option, but I think running everything by default is
>>> the best strategy.
>>>>
>>>>>
>>>>> Comments?
>>>>>
>>>>> This might have bad effects on Prasad's automation but I'm not sure
>>>>> how that is run.
>>>>
>>>> For now, the automation builds have been modified to use the no-it
>>>> profile.
>>>>
>>> thanks!
>>> david jencks
>>>>>
>>>>> thanks && Happy New Year!
>>>>> david jencks
>>>>
>>>> Happy New Year to ya'll !
>>>> Prasad
>
>
Re: Running integration tests by default?
Posted by Shiva Kumar H R <sh...@gmail.com>.
On Jan 5, 2008 3:23 AM, David Jencks <da...@yahoo.com> wrote:
>
> On Jan 4, 2008, at 1:07 PM, Donald Woods wrote:
>
> > How about not running the integration tests by default and letting
> > those who want them either 1) use a "-P testsuite" option from
> > trunk or 2) run mvn inside the testsuite subdir?
> (2) is what we had before my change and didn't work for me -- I never
> ran the testsuite.
>
> My thinking on my change is:
>
> 1. Its better to run more tests. For instance if you see a IT
> failure on your machine I think you are much more likely to fix it
> than if you have to wait for the notification from Prasad's
> automation. I think this is why we have unit tests as well :-)
> 2. People are lazy and don't like to type, so the default option
> should include the maximum level of testing.
+1. I never remembered and ran those IT tests by the way!
>
> 3. In the case of the testsuite, these are run after the rest of the
> build is complete, so you are free to interrupt the build if you get
> too bored/impatient.
>
> Could you explain why you disagree with this or what factors you
> consider more important?
>
> thanks
> david jencks
>
> >
> > -Donald
> >
> >
> > David Jencks wrote:
> >> On Jan 2, 2008, at 7:58 AM, Prasad Kashyap wrote:
> >>> On Dec 31, 2007 4:41 PM, David Jencks <da...@yahoo.com>
> >>> wrote:
> >>>> First of all I appear to have broken the build last night with some
> >>>> changes to get the roller plugin building again. I think I've
> >>>> managed to fix all the problems -- the it tests all pass for
> >>>> me. Let
> >>>> me know if there are still problems.
> >>>>
> >>>> I think its still too hard to run the integration tests.
> >>>
> >>> I would like to know what exactly you think is hard about it. It
> >>> would
> >>> be great if you could please share your thoughts and ideas on making
> >>> it simpler.
> >> I didn't see a way to run the integration tests with the main
> >> build without a separate command. Unless I can run everything in
> >> one command, I'm pretty sure laziness will take over and I won't
> >> run the integration tests.
> >>>
> >>>> I've made a
> >>>> possibly annoying change so that the default build includes IT. If
> >>>> you don't want them run
> >>>>
> >>>> mvn clean install -P no-it
> >>>>
> >>>> If this is too annoying we could reverse the profiles and have the
> >>>> default leave out the it as before and a with-it profile that
> >>>> includes them.
> >>>
> >>> Yeah. I think the default profile should not run the IT. IMHO, I
> >>> think
> >>> it should not even run the unit tests by default. Developers
> >>> (should)
> >>> run unit & IT tests before committing their code. And we have
> >>> automation builds with all tests that run 4 times a day anyways. So
> >>> the default profile can well do away with tests. But that may
> >>> just be
> >>> my opinion.
> >> After working through some hard problem and getting ready to
> >> commit the last thing anyone wants to do is remember a second
> >> command or even a command line option to run some tests: they are
> >> apt to type the simplest command that will check the build. I
> >> think that command should run all the tests, including the
> >> integration tests. Even though there's enough time to go eat
> >> dinner while they run, our most comprehensive checks will be run.
> >> If you get too bored you can always stop the build after you think
> >> enough stuff has been checked, and if you remember you can run the
> >> build with options to turn off whatever tests you want to skip.
> >> ApacheDS had their integration tests run using an option and most
> >> people did not run them due to the extra effort of trying to
> >> remember to type more on the command line. I think they've
> >> changed so the integration tests are run by default.
> >> I'm happy to keep talking about this.... and I'll be happy enough
> >> with a "with-it" option, but I think running everything by default
> >> is the best strategy.
> >>>
> >>>>
> >>>> Comments?
> >>>>
> >>>> This might have bad effects on Prasad's automation but I'm not sure
> >>>> how that is run.
> >>>
> >>> For now, the automation builds have been modified to use the no-
> >>> it profile.
> >>>
> >> thanks!
> >> david jencks
> >>>>
> >>>> thanks && Happy New Year!
> >>>> david jencks
> >>>
> >>> Happy New Year to ya'll !
> >>> Prasad
>
>
--
Thanks,
Shiva
Re: Running integration tests by default?
Posted by David Jencks <da...@yahoo.com>.
On Jan 4, 2008, at 1:07 PM, Donald Woods wrote:
> How about not running the integration tests by default and letting
> those who want them either 1) use a "-P testsuite" option from
> trunk or 2) run mvn inside the testsuite subdir?
(2) is what we had before my change and didn't work for me -- I never
ran the testsuite.
My thinking on my change is:
1. Its better to run more tests. For instance if you see a IT
failure on your machine I think you are much more likely to fix it
than if you have to wait for the notification from Prasad's
automation. I think this is why we have unit tests as well :-)
2. People are lazy and don't like to type, so the default option
should include the maximum level of testing.
3. In the case of the testsuite, these are run after the rest of the
build is complete, so you are free to interrupt the build if you get
too bored/impatient.
Could you explain why you disagree with this or what factors you
consider more important?
thanks
david jencks
>
> -Donald
>
>
> David Jencks wrote:
>> On Jan 2, 2008, at 7:58 AM, Prasad Kashyap wrote:
>>> On Dec 31, 2007 4:41 PM, David Jencks <da...@yahoo.com>
>>> wrote:
>>>> First of all I appear to have broken the build last night with some
>>>> changes to get the roller plugin building again. I think I've
>>>> managed to fix all the problems -- the it tests all pass for
>>>> me. Let
>>>> me know if there are still problems.
>>>>
>>>> I think its still too hard to run the integration tests.
>>>
>>> I would like to know what exactly you think is hard about it. It
>>> would
>>> be great if you could please share your thoughts and ideas on making
>>> it simpler.
>> I didn't see a way to run the integration tests with the main
>> build without a separate command. Unless I can run everything in
>> one command, I'm pretty sure laziness will take over and I won't
>> run the integration tests.
>>>
>>>> I've made a
>>>> possibly annoying change so that the default build includes IT. If
>>>> you don't want them run
>>>>
>>>> mvn clean install -P no-it
>>>>
>>>> If this is too annoying we could reverse the profiles and have the
>>>> default leave out the it as before and a with-it profile that
>>>> includes them.
>>>
>>> Yeah. I think the default profile should not run the IT. IMHO, I
>>> think
>>> it should not even run the unit tests by default. Developers
>>> (should)
>>> run unit & IT tests before committing their code. And we have
>>> automation builds with all tests that run 4 times a day anyways. So
>>> the default profile can well do away with tests. But that may
>>> just be
>>> my opinion.
>> After working through some hard problem and getting ready to
>> commit the last thing anyone wants to do is remember a second
>> command or even a command line option to run some tests: they are
>> apt to type the simplest command that will check the build. I
>> think that command should run all the tests, including the
>> integration tests. Even though there's enough time to go eat
>> dinner while they run, our most comprehensive checks will be run.
>> If you get too bored you can always stop the build after you think
>> enough stuff has been checked, and if you remember you can run the
>> build with options to turn off whatever tests you want to skip.
>> ApacheDS had their integration tests run using an option and most
>> people did not run them due to the extra effort of trying to
>> remember to type more on the command line. I think they've
>> changed so the integration tests are run by default.
>> I'm happy to keep talking about this.... and I'll be happy enough
>> with a "with-it" option, but I think running everything by default
>> is the best strategy.
>>>
>>>>
>>>> Comments?
>>>>
>>>> This might have bad effects on Prasad's automation but I'm not sure
>>>> how that is run.
>>>
>>> For now, the automation builds have been modified to use the no-
>>> it profile.
>>>
>> thanks!
>> david jencks
>>>>
>>>> thanks && Happy New Year!
>>>> david jencks
>>>
>>> Happy New Year to ya'll !
>>> Prasad
Re: Running integration tests by default?
Posted by Donald Woods <dw...@apache.org>.
How about not running the integration tests by default and letting those
who want them either 1) use a "-P testsuite" option from trunk or 2) run
mvn inside the testsuite subdir?
-Donald
David Jencks wrote:
>
> On Jan 2, 2008, at 7:58 AM, Prasad Kashyap wrote:
>
>> On Dec 31, 2007 4:41 PM, David Jencks <da...@yahoo.com> wrote:
>>> First of all I appear to have broken the build last night with some
>>> changes to get the roller plugin building again. I think I've
>>> managed to fix all the problems -- the it tests all pass for me. Let
>>> me know if there are still problems.
>>>
>>> I think its still too hard to run the integration tests.
>>
>> I would like to know what exactly you think is hard about it. It would
>> be great if you could please share your thoughts and ideas on making
>> it simpler.
>
> I didn't see a way to run the integration tests with the main build
> without a separate command. Unless I can run everything in one command,
> I'm pretty sure laziness will take over and I won't run the integration
> tests.
>
>>
>>> I've made a
>>> possibly annoying change so that the default build includes IT. If
>>> you don't want them run
>>>
>>> mvn clean install -P no-it
>>>
>>> If this is too annoying we could reverse the profiles and have the
>>> default leave out the it as before and a with-it profile that
>>> includes them.
>>
>> Yeah. I think the default profile should not run the IT. IMHO, I think
>> it should not even run the unit tests by default. Developers (should)
>> run unit & IT tests before committing their code. And we have
>> automation builds with all tests that run 4 times a day anyways. So
>> the default profile can well do away with tests. But that may just be
>> my opinion.
>
> After working through some hard problem and getting ready to commit the
> last thing anyone wants to do is remember a second command or even a
> command line option to run some tests: they are apt to type the simplest
> command that will check the build. I think that command should run all
> the tests, including the integration tests. Even though there's enough
> time to go eat dinner while they run, our most comprehensive checks will
> be run. If you get too bored you can always stop the build after you
> think enough stuff has been checked, and if you remember you can run the
> build with options to turn off whatever tests you want to skip.
>
> ApacheDS had their integration tests run using an option and most people
> did not run them due to the extra effort of trying to remember to type
> more on the command line. I think they've changed so the integration
> tests are run by default.
>
> I'm happy to keep talking about this.... and I'll be happy enough with a
> "with-it" option, but I think running everything by default is the best
> strategy.
>
>>
>>>
>>> Comments?
>>>
>>> This might have bad effects on Prasad's automation but I'm not sure
>>> how that is run.
>>
>> For now, the automation builds have been modified to use the no-it
>> profile.
>>
>
> thanks!
> david jencks
>
>>>
>>> thanks && Happy New Year!
>>> david jencks
>>
>> Happy New Year to ya'll !
>> Prasad
>
>
Re: Running integration tests by default?
Posted by David Jencks <da...@yahoo.com>.
On Jan 2, 2008, at 7:58 AM, Prasad Kashyap wrote:
> On Dec 31, 2007 4:41 PM, David Jencks <da...@yahoo.com> wrote:
>> First of all I appear to have broken the build last night with some
>> changes to get the roller plugin building again. I think I've
>> managed to fix all the problems -- the it tests all pass for me. Let
>> me know if there are still problems.
>>
>> I think its still too hard to run the integration tests.
>
> I would like to know what exactly you think is hard about it. It would
> be great if you could please share your thoughts and ideas on making
> it simpler.
I didn't see a way to run the integration tests with the main build
without a separate command. Unless I can run everything in one
command, I'm pretty sure laziness will take over and I won't run the
integration tests.
>
>> I've made a
>> possibly annoying change so that the default build includes IT. If
>> you don't want them run
>>
>> mvn clean install -P no-it
>>
>> If this is too annoying we could reverse the profiles and have the
>> default leave out the it as before and a with-it profile that
>> includes them.
>
> Yeah. I think the default profile should not run the IT. IMHO, I think
> it should not even run the unit tests by default. Developers (should)
> run unit & IT tests before committing their code. And we have
> automation builds with all tests that run 4 times a day anyways. So
> the default profile can well do away with tests. But that may just be
> my opinion.
After working through some hard problem and getting ready to commit
the last thing anyone wants to do is remember a second command or
even a command line option to run some tests: they are apt to type
the simplest command that will check the build. I think that command
should run all the tests, including the integration tests. Even
though there's enough time to go eat dinner while they run, our most
comprehensive checks will be run. If you get too bored you can
always stop the build after you think enough stuff has been checked,
and if you remember you can run the build with options to turn off
whatever tests you want to skip.
ApacheDS had their integration tests run using an option and most
people did not run them due to the extra effort of trying to remember
to type more on the command line. I think they've changed so the
integration tests are run by default.
I'm happy to keep talking about this.... and I'll be happy enough
with a "with-it" option, but I think running everything by default is
the best strategy.
>
>>
>> Comments?
>>
>> This might have bad effects on Prasad's automation but I'm not sure
>> how that is run.
>
> For now, the automation builds have been modified to use the no-it
> profile.
>
thanks!
david jencks
>>
>> thanks && Happy New Year!
>> david jencks
>
> Happy New Year to ya'll !
> Prasad
Re: Running integration tests by default?
Posted by Prasad Kashyap <go...@gmail.com>.
On Dec 31, 2007 4:41 PM, David Jencks <da...@yahoo.com> wrote:
> First of all I appear to have broken the build last night with some
> changes to get the roller plugin building again. I think I've
> managed to fix all the problems -- the it tests all pass for me. Let
> me know if there are still problems.
>
> I think its still too hard to run the integration tests.
I would like to know what exactly you think is hard about it. It would
be great if you could please share your thoughts and ideas on making
it simpler.
> I've made a
> possibly annoying change so that the default build includes IT. If
> you don't want them run
>
> mvn clean install -P no-it
>
> If this is too annoying we could reverse the profiles and have the
> default leave out the it as before and a with-it profile that
> includes them.
Yeah. I think the default profile should not run the IT. IMHO, I think
it should not even run the unit tests by default. Developers (should)
run unit & IT tests before committing their code. And we have
automation builds with all tests that run 4 times a day anyways. So
the default profile can well do away with tests. But that may just be
my opinion.
>
> Comments?
>
> This might have bad effects on Prasad's automation but I'm not sure
> how that is run.
For now, the automation builds have been modified to use the no-it profile.
>
> thanks && Happy New Year!
> david jencks
Happy New Year to ya'll !
Prasad