Back to OOTB testing - Re: OFBiz self-hosting

[Adam Heath <do...@brainfood.com>]

David E Jones wrote:

> [snip stuff about tests]

On that note, it has become my primary non-paid-work concern, to make
the automated tests work out of the box.

One idea that has been mentioned, is running each test in a transaction,
then rolling it back at the end.  This won't work, for several reasons.
 One, is that tests themselves may do multiple transaction things, even
going so far as calling services that spawn separate transactions.  Two,
if the test calls commit, it actually won't, as it's running inside
another transaction started by the test container.

Another idea that has been mentioned, is using the EntityAuditLog
feature.  Since that only deals with single fields, it would need to be
extended.

One I had this morning, is to use Entity Data xml.  Whenever the new
entity is stored, save the old entity into a per-transaction based list.
 When the transaction status changes, either throw it away, or commit
it.  This would  need to be extended to support deletes.

And probably the simplest of all, is to save a copy of
runtime/data/derby, then after each test, shutdown stuff, and copy the
files back in.  Preliminary testing shows that tarring up that dir, not
compressing, then deleting and untarring is the fastest.

Re: Back to OOTB testing - Re: OFBiz self-hosting

[Scott Gray <le...@gmail.com>]

Makes sense, thanks Adam

Regards
Scott

2008/11/14 Adam Heath <do...@brainfood.com>:
> Scott Gray wrote:
>> Hi Adam,
>>
>> I'm no transaction expert but how sure are we that starting a new
>> transaction just before each test and rolling it back at the end won't
>> work?  From what I can gather Derby does actually alter the database
>> prior to committing the transaction but it also creates log records
>> that allow the changes to undone in case of a rollback.
>>
>> I just tried it using the JUnitListener class to start and end a
>> transaction and I can't see any obvious negative effects (aside from
>> tests failing that relied on data created in previous tests).
>
> The reason that won't work, is that some services are configured to run
> in a *separate* transaction, completely separate from the one that is in
> the current thread.  In those cases, the current transaction is
> suspended, then resumed.
>
> To do the transaction rollback stuff in those cases, becomes rather more
> complex.  It's just simpler to save the disk files, and revert them all,
> between tests.  Just need to make certain any background threads are
> restarted/shutdown.
>

Re: Back to OOTB testing - Re: OFBiz self-hosting

[David E Jones <da...@hotwaxmedia.com>]

On Nov 13, 2008, at 6:57 PM, Adam Heath wrote:

> Scott Gray wrote:
>> Hi Adam,
>>
>> I'm no transaction expert but how sure are we that starting a new
>> transaction just before each test and rolling it back at the end  
>> won't
>> work?  From what I can gather Derby does actually alter the database
>> prior to committing the transaction but it also creates log records
>> that allow the changes to undone in case of a rollback.
>>
>> I just tried it using the JUnitListener class to start and end a
>> transaction and I can't see any obvious negative effects (aside from
>> tests failing that relied on data created in previous tests).
>
> The reason that won't work, is that some services are configured to  
> run
> in a *separate* transaction, completely separate from the one that  
> is in
> the current thread.  In those cases, the current transaction is
> suspended, then resumed.
>
> To do the transaction rollback stuff in those cases, becomes rather  
> more
> complex.  It's just simpler to save the disk files, and revert them  
> all,
> between tests.  Just need to make certain any background threads are
> restarted/shutdown.

Yes, this is a good point. It should only be the ones that do a  
suspend/resume that cause a problem, but there could be other things  
too.

I just did a quick run on my machine and found that loading the seed  
and demo data takes about 41 seconds. To do this while running we'd  
need to truncate all tables and then reload the seed data, which would  
take a bit more time (probably have to drop all foreign keys, truncate  
tables, restore fks).

The tar method on my machine is quite a bit faster, and uses a file  
that is nearly 100MB. I guess the trick is... can we do that while  
OFBiz is running? In other words, can we stop derby and replace its  
files right under it and then restart it and get it to play nice?

There is a delay restarting OFBiz, but it's not too bad (and probably  
something we could profile and speed up if we wanted to). It would be  
pretty easy to create an ant target that does a fresh run-install,  
saves the runtime/data directory, runs the tests in one component at a  
time, and between them stops ofbiz then resets runtime/data then  
starts ofbiz again.

It will be slower, but maybe not be too bad...

Then we'd just have to introduce a basic rule that the tests in a  
component run together, and can only depend on other tests run  
previously in that component. It would be nice to have more  
independent tests, but we'd require at least that level of independence.

-David

Re: Back to OOTB testing - Re: OFBiz self-hosting

[Adam Heath <do...@brainfood.com>]

Scott Gray wrote:
> Hi Adam,
> 
> I'm no transaction expert but how sure are we that starting a new
> transaction just before each test and rolling it back at the end won't
> work?  From what I can gather Derby does actually alter the database
> prior to committing the transaction but it also creates log records
> that allow the changes to undone in case of a rollback.
> 
> I just tried it using the JUnitListener class to start and end a
> transaction and I can't see any obvious negative effects (aside from
> tests failing that relied on data created in previous tests).

The reason that won't work, is that some services are configured to run
in a *separate* transaction, completely separate from the one that is in
the current thread.  In those cases, the current transaction is
suspended, then resumed.

To do the transaction rollback stuff in those cases, becomes rather more
complex.  It's just simpler to save the disk files, and revert them all,
between tests.  Just need to make certain any background threads are
restarted/shutdown.

Re: Back to OOTB testing - Re: OFBiz self-hosting

[Scott Gray <le...@gmail.com>]

Hi Adam,

I'm no transaction expert but how sure are we that starting a new
transaction just before each test and rolling it back at the end won't
work?  From what I can gather Derby does actually alter the database
prior to committing the transaction but it also creates log records
that allow the changes to undone in case of a rollback.

I just tried it using the JUnitListener class to start and end a
transaction and I can't see any obvious negative effects (aside from
tests failing that relied on data created in previous tests).

Regards
Scott

2008/11/14 Adam Heath <do...@brainfood.com>:
> David E Jones wrote:
>
>> [snip stuff about tests]
>
> On that note, it has become my primary non-paid-work concern, to make
> the automated tests work out of the box.
>
> One idea that has been mentioned, is running each test in a transaction,
> then rolling it back at the end.  This won't work, for several reasons.
>  One, is that tests themselves may do multiple transaction things, even
> going so far as calling services that spawn separate transactions.  Two,
> if the test calls commit, it actually won't, as it's running inside
> another transaction started by the test container.
>
> Another idea that has been mentioned, is using the EntityAuditLog
> feature.  Since that only deals with single fields, it would need to be
> extended.
>
> One I had this morning, is to use Entity Data xml.  Whenever the new
> entity is stored, save the old entity into a per-transaction based list.
>  When the transaction status changes, either throw it away, or commit
> it.  This would  need to be extended to support deletes.
>
> And probably the simplest of all, is to save a copy of
> runtime/data/derby, then after each test, shutdown stuff, and copy the
> files back in.  Preliminary testing shows that tarring up that dir, not
> compressing, then deleting and untarring is the fastest.
>