how to write a test case

[Adam Heath <do...@brainfood.com>]

I've been doing a bit of test case work the past day.  While doing
this, I came across a very bad pattern.  There were test cases
committed to ofbiz, that *never* worked.  If you are going to write a
test case, or commit a patch that includes a test case, make 100%
absolutely certain without a doubt this is an ultimatum that the test
case *WORKS*.

All of the accounting test cases failed when they were initially
checked in.  They modified entities that *DID NOT EXIST* in any
seed/demo data.  This is just so very bad.

I'm having to go back thru and *reopen* the bugs that contained these
test cases.

ps: I'm very frustrated by this.  If you don't understand something,
then do *not* commit it.  Please.  With sugar on top.

Re: how to write a test case

[Vikas Mayur <vi...@hotwaxmedia.com>]

On Mar 7, 2009, at 2:01 AM, Adam Heath wrote:

> Vikas Mayur wrote:
>
>>> How did it work?  I reverted back to 660193, the last patch for
>>> OFBIZ-1790, and the accounting tests failed.
>>>
>>> If they worked in the past, I'd like to know when.  If so, then that
>>> means something since then has caused them to break, and I will more
>>> than gladly track that down.
>>>
>>> However, if they have never worked(which is what I'm strongly
>>> suspecting), then I stand by my original assessment.
>>>
>> Do not know why it is not working for you and I have no idea/solution
>> for this.
>
> If you run the test individually, and follow the instructions in the
> file, it'll probably work.

Yeah, I think so.

>
>
> However, that's not how things are done.
>
> All tests are run together.  Every testdef/*.xml file that is in any
> ofbiz-component.xml is run one after the other, with no chance for any
> manual setup between each test.
>
> In this circumstance, they do not work, and never did work.  It is
> this circumstance that an *automated* test case must work.

I do not know what is the point here to discuss same thing again and  
again. I agree to your point of making test automated and lot of  
people have complaint about
this in past but no one really come forward for the contribution.

Its really useless point to discuss on that these things in the trunk  
are making you frustrated because they are not written properly so why  
not complain early in the process and not after a YEAR or so. Sorry  
man, no time to look back and why not fix them by yourself if you see  
issues.

  

Re: how to write a test case

[Adam Heath <do...@brainfood.com>]

Vikas Mayur wrote:

>> How did it work?  I reverted back to 660193, the last patch for
>> OFBIZ-1790, and the accounting tests failed.
>>
>> If they worked in the past, I'd like to know when.  If so, then that
>> means something since then has caused them to break, and I will more
>> than gladly track that down.
>>
>> However, if they have never worked(which is what I'm strongly
>> suspecting), then I stand by my original assessment.
>>
> Do not know why it is not working for you and I have no idea/solution
> for this.

If you run the test individually, and follow the instructions in the
file, it'll probably work.

However, that's not how things are done.

All tests are run together.  Every testdef/*.xml file that is in any
ofbiz-component.xml is run one after the other, with no chance for any
manual setup between each test.

In this circumstance, they do not work, and never did work.  It is
this circumstance that an *automated* test case must work.

Re: how to write a test case

[Vikas Mayur <vi...@hotwaxmedia.com>]

On Mar 5, 2009, at 9:14 PM, Adam Heath wrote:

> Vikas Mayur wrote:
>
>> I appreciate your efforts in the recent commits to clean up test case
>> and its relative code. I am happy now people are getting off sleep  
>> and
>> want to contribute to this important part of the OFBiz.
>>
>> So just you know,  almost every test case in accounting require  
>> certain
>> pre-condition that need to fulfilled before a test case actually run.
>> They can be in any various form like either running the pre-condition
>> manually, step by step from the OFBiz UI or through defining a demo  
>> data
>> (this is the best way to go with).
>
> If it requires a pre-condition, then it's not a test case, and is just
> some code that does something.
>
> Test-cases are supposed to be *automated*.

Agreed. That is why I refer to demo data is the best way to deal with.  
Just rephrasing again.

>
>
>> To make it clear, every test case was committed after it WORKS.
>
> How did it work?  I reverted back to 660193, the last patch for
> OFBIZ-1790, and the accounting tests failed.
>
> If they worked in the past, I'd like to know when.  If so, then that
> means something since then has caused them to break, and I will more
> than gladly track that down.
>
> However, if they have never worked(which is what I'm strongly
> suspecting), then I stand by my original assessment.
>
Do not know why it is not working for you and I have no idea/solution  
for this.

Re: how to write a test case

[Adam Heath <do...@brainfood.com>]

Vikas Mayur wrote:

> I appreciate your efforts in the recent commits to clean up test case
> and its relative code. I am happy now people are getting off sleep and
> want to contribute to this important part of the OFBiz.
> 
> So just you know,  almost every test case in accounting require certain
> pre-condition that need to fulfilled before a test case actually run.
> They can be in any various form like either running the pre-condition
> manually, step by step from the OFBiz UI or through defining a demo data
> (this is the best way to go with).

If it requires a pre-condition, then it's not a test case, and is just
some code that does something.

Test-cases are supposed to be *automated*.

> To make it clear, every test case was committed after it WORKS.

How did it work?  I reverted back to 660193, the last patch for
OFBIZ-1790, and the accounting tests failed.

If they worked in the past, I'd like to know when.  If so, then that
means something since then has caused them to break, and I will more
than gladly track that down.

However, if they have never worked(which is what I'm strongly
suspecting), then I stand by my original assessment.

Re: how to write a test case

[Vikas Mayur <vi...@hotwaxmedia.com>]

Adam,

I appreciate your efforts in the recent commits to clean up test case  
and its relative code. I am happy now people are getting off sleep and  
want to contribute to this important part of the OFBiz.

So just you know,  almost every test case in accounting require  
certain pre-condition that need to fulfilled before a test case  
actually run. They can be in any various form like either running the  
pre-condition manually, step by step from the OFBiz UI or through  
defining a demo data (this is the best way to go with).

To make it clear, every test case was committed after it WORKS.

AFAIK, Most of the issues relative to JUnit test case are now closed.

We were working on in improving JUnit test case some time back so  
that  they run successfully and independent of each other, for that we  
need to define big pile of demo data.
In that direction, there are certain patches uploaded to improve them  
and the recent patches contain the demo data.

But unfortunately these are uploaded on closed issue. I wasn't have  
time to look into them and fix them (priorities).

I hope now you can take on some issues.


Vikas




On Mar 5, 2009, at 8:36 AM, Adam Heath wrote:

> I've been doing a bit of test case work the past day.  While doing
> this, I came across a very bad pattern.  There were test cases
> committed to ofbiz, that *never* worked.  If you are going to write a
> test case, or commit a patch that includes a test case, make 100%
> absolutely certain without a doubt this is an ultimatum that the test
> case *WORKS*.
>
> All of the accounting test cases failed when they were initially
> checked in.  They modified entities that *DID NOT EXIST* in any
> seed/demo data.  This is just so very bad.
>
> I'm having to go back thru and *reopen* the bugs that contained these
> test cases.
>
> ps: I'm very frustrated by this.  If you don't understand something,
> then do *not* commit it.  Please.  With sugar on top.