You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beehive.apache.org by Eddie O'Neil <ek...@bea.com> on 2005/04/15 16:29:56 UTC

fixing TCH to fail on error in a dev environment

Joe and Zach--

   As we get closer to shipping a Beehive release, it would be great if 
we could get this controls test framework issue fixed.  Still seems 
right to fail the Ant build when the TCH tests fail.  This will still 
work with Cruisecontrol if you set a flag "cc.mode" when running in that 
environment.  Would it be possible to address this in the next few weeks?

Thoughts?

Eddie



Eddie O'Neil wrote:
> 
>   Definitely understand the cmd.exe issues.  :)
> 
>   But, isn't this just a matter of setting a flag for whether to fail a 
> test run?  It would be an addition to what we are doing now.
> 
>   I'd propose that this work as follows:
> 
> - default TCH behavior is to fail the build in the case of a test 
> failure / abort / etc.  TCH does this by setting an Ant property, for 
> example "test.failure" that can be used in Ant like:
> 
> <target name="drt.errors" if="test.failure">
>     <echo>!!!!! Errors or failures occurred running tests !!!!!</echo>
>     <fail message="Controls TCH Tests Failed"/>
> </target>
> 
> - this behavior can be over-ridden in CruiseControl mode.  The override 
> will still print some failure message (or even set a property) but 
> doesn't set the "test.failure" property so that the build fails.
> 
> The latter is the behavior that we have today; the former is just an 
> addition to the TCH behavior / API.  This would make TCH more like JUnit 
> in the way that failures can be controlled through properties.
> 
>   Thoughts?
> 
> Eddie
> 
> 
> 
> 
> James Song wrote:
> 
>> First, this is a windows only issue. Ant does not have this problem on
>> other platform. There is no need to do String searching for failure
>> detection on Linux/Unix.
>>
>> Second, on Windows, someone mentioned python and perl works better than
>> cmd.exe. Maybe cmd.exe could be switched to python or perl?
>>
>> -James
>>
>> -----Original Message-----
>> From: Eddie O'Neil Sent: Wednesday, March 16, 2005 1:27 PM
>> To: Beehive Developers
>> Cc: Joe Pemberton; Zachary Smith
>> Subject: Re: Beehive Check-in Test (Linux) Failed
>>
>>
>>    Hm.  I've seen CC machine configs that can handle this sort of thing,
>>
>> so we should dig into how to fix it.
>>
>>    IMHO, it's pretty important to be very obvious about test failures 
>> when running in a shell, so we should look at addressing this.
>>
>>    Any other thoughts?
>>
>> Eddie
>>
>>
>>
>> James Song wrote:
>>
>>> I come across this issue when I set up a cc machine.
>>>
>>> http://issues.apache.org/bugzilla/show_bug.cgi?id=30216
>>>
>>> CC code boils down to the following:
>>>
>>> <exec dir="." executable="cmd" failonerror="false"
>>>            resultproperty="build.result">
>>>
>>> The resultproperty does not return the correct value on Windows
>>> sometimes.
>>>
>>> That is why I switched the failure detecting code to look for "Build
>>> Failed" from the log file.
>>>
>>>
>>> -James
>>>
>>>
>>> -----Original Message-----
>>> From: Joe Pemberton Sent: Wednesday, March 16, 2005 1:03 PM
>>> To: 'Beehive Developers'; Zachary Smith; James Song
>>> Subject: RE: Beehive Check-in Test (Linux) Failed
>>>
>>>> From what I understand, the reason the test run does not fail early
>>>
>>> there is that for a CruiseControl run, it is useful to do the full
>>
>>
>> test
>>
>>> run (and not fail early if the controls tests have errors).  After the
>>> full controls test build/run, CC greps for "BUILD FAILED", and fails
>>
>>
>> the
>>
>>> build accordingly.
>>>
>>> James may know more about this..
>>>
>>> -Joe
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Eddie O'Neil Sent: Wednesday, March 16, 2005 12:58 PM
>>> To: Beehive Developers
>>> Subject: Re: Beehive Check-in Test (Linux) Failed
>>>
>>>
>>>   My take is that those three test aborts should have failed the
>>
>>
>> entire
>>
>>> Ant process at the point where the controls tests failed.
>>>
>>>   Without that, the tests fail and the build continues, and the Ant 
>>> process can still end in BUILD SUCCESSFUL even if there were aborts.
>>>
>>>   Probably good to follow the JUnit / Ant model here -- fail early.
>>>
>>>   :)
>>>
>>> Eddie
>>>
>>>
>>>
>>> Joe Pemberton wrote:
>>>
>>>
>>>> Eddie,
>>>> I'm not sure I follow you.  I see the three aborts in the TCH run,
>>>
>>>
>>> and
>>>
>>>
>>>> I see a BUILD FAILED at the end of the run.  Also, the CC run itself
>>>> failed.
>>>>
>>>> -Joe
>>>> -----Original Message-----
>>>> From: Eddie O'Neil Sent: Wednesday, March 16, 2005 10:45 AM
>>>> To: Beehive Developers
>>>> Subject: Re: Beehive Check-in Test (Linux) Failed
>>>>
>>>>
>>>>  A second-order problem is that there are three tests in the TCH
>>>
>>>
>>> test
>>>
>>>> suite that were aborted (and even stack traced), but these didn't fail
>>>
>>>
>>>
>>>> the controls test suite.  Look at this link and scroll down to the TCH
>>>
>>>
>>>
>>>> output:
>>>>
>>>>  http://beehive01.bea.com/downloads/20050316052618/general/build.out
>>>>
>>>>  Zach, can you take a look at this?  We saw the same thing with some
>>>
>>>
>>>
>>>> controls test failures last week.
>>>>
>>>> Eddie
>>>>
>>>>
>>>>
>>>> Daryl Olander wrote:
>>>>
>>>>
>>>>
>>>>> So it appears that there is a break in the controls checkin tests
>>
>>
>> that
>>
>>>>> was caused by the same checkin from this morning.
>>>>>
>>>>>
>>>>> http://svn.apache.org/viewcvs.cgi?rev=157742&view=rev
>>>>>
>>>>> CC was broken at 5:26 this morning.  There were two sets of failures,
>>>>> the WSM Drts failed in addition the  controls check in test failed
>>>>>
>>>>> http://beehive01.bea.com/downloads/20050316052618/controls/tch.cout
>>>>>
>>>>> There is only a single checkin here.
>>>>>
>>>>>
>>>>> On Wed, 16 Mar 2005 10:33:09 -0800 (PST), jpembert@bea.com
>>>>> <jp...@bea.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> View results here ->
>>>>
>>>>
>>>>
>> http://beehive01.bea.com/cruisecontrol/buildresults?log=log2005031610210
>>
>>>> 0
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>