Speeding up tests

[Emmanuel Lecharny <el...@gmail.com>]

Hi guys,

we currently have insanely long tests, especially when it comes to
integration tests. This is not only due to the fact that our server is
slow ;), but mainly because we are using an old version of JUnit,
which run setup() and teardown() for _each_ test. We have around 3000
tests currently, with around 420 core-unit tests and 180 server-unit
tests.

Those last tests are really time consuming, as the server is started
and stopped for each single test, and that cost around 2 seconds for
each server start. It cost more than 15 minutes on my laptop ...

Junit 4, the latest version we are currently using, offers a new
annotation system where you can add a @BeforeClass and @AfterClass,
applied on a startup and teardown methods only once for all the tests
in a simple class. If we tune the tests correctly, we can then reduce
the number of server startup.shutdown to 100 startup instead if 600.
The gain will be quite important, as we may save 70% of the total
cost.

I have done some test with JUnit 3.8, using a very ugly hack, and the
result is that for the SearchTest, I went down from 46 seconds to 6
seconds.

There are a few things to do now :
- I have not tested JUnit 4 right now, but there are everything we
need to speed up the test, as far as I can see
- if we are to switch to JUnit 4, we have to figure out which impact
it has on the existing code, on Eclipse integration, and more
important, on Maven (does Surefire support Junit 4 natively ?)
- another alternative would be to use the more evolved TestNG
framework ; the very same steps should be followed (see previous
point)
- If we want to start the server only once, we have to be very careful
when writing a new test : it should not impact the existing data
loaded. That mean that a test should left the data in the same state
when it has been completed (successfully or not) than when it started.

There may be some other options, like scripting tests (I was initially
thinking about using JMeter, but it's far from being perfect), or
adding a very simple framework to compare expected entries with those
get from the server.

At this point, I would like to get your opinion.

Thanks for any suggestion !

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Fwd: Speeding up tests

[Emmanuel Lecharny <el...@gmail.com>]

Oops, forget to send tis mail _with the content_ to the ML ;)

---------- Forwarded message ----------
From: Emmanuel Lecharny <el...@gmail.com>
Date: Sep 19, 2007 2:13 AM
Subject: Re: Speeding up tests
To: Alex Karasulu <ak...@apache.org>


Hi Alex,

some comment in the body :
<SNIP/>


> Been thinking about this for some time as you know.  At first I just made me
> a ram drive on linux to deal with this :D - it saved like 50% of the time
> but yeah that's laughable (both the technique and the time saved).  But this
> was just a quick remedy.  Still 7 minutes is too long to run integration
> tests.  The bulk of the cost of integration tests comes from bootstrapping
> specifically off the top of my head I'd list the following top time suckers:
>
>  o partition creation (with various db files)
>      o schema partition
>      o system partition
>  o schema loading and checks
>
> I suspect you can do thousands of adds, deletes and modifies for the cost of
> bootstrapping.

Exactly. The schema loading is damn costly (more than 600 entries to read)

So my recommendation is to find a way to snapshot the state
> of the server then roll it back with anti-requests.  This really is not as
> complex as it sounds.  We have the capability to do this easily using an
> interceptor much like the changelog interceptor in my embedding workshop
> which Ersin knocked out a while back.  Perhaps for each log operation
> tracked by this interceptor we can have a configuration switch to track the
> anti-ldif.  This would not be too hard to do.
>
> Add => Delete
> Delete => Add
> Modify (Replace) => Modify (Replace)
> Modify (Remove) => Modify (Add)
> Modify (Add) => Modify (Remove)
> ModifyDn => ModifyDN
>
> The order would be reversed.  So if you have this train of requests { R1,
> R2, R3, R4, R5 } that correspond to the following LDIF sequence { L1, L2,
> L3, L4, L5 } then you just need the following anti operations { A5, A4, A3.
> A2, A1 }.  Another way to have done this nicely is if we had transactions
> enabled properly.  We could just rolllback :).  But this is a quick
> workaround to the problem.

There is more than that. This bring up some very interesting features :
(1) test speed boost (minimal value)
(2) recording and playback for complex test cases
(3) for snapshoting and rolling back the server to previous states
(4) for creating journals to be used if the server crash

Let's discuss that in a separate thread.

Thanks Alex


>
> Alex
>
>
>


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Fwd: Speeding up tests

[Emmanuel Lecharny <el...@gmail.com>]

Oops, forget to send this mail to the ML

Re: Speeding up tests

[Alex Karasulu <ak...@apache.org>]

Hi Emmanuel,

On 9/18/07, Emmanuel Lecharny <el...@gmail.com> wrote:
>
> Hi guys,
>
> we currently have insanely long tests, especially when it comes to
> integration tests. This is not only due to the fact that our server is
> slow ;), but mainly because we are using an old version of JUnit,
> which run setup() and teardown() for _each_ test. We have around 3000
> tests currently, with around 420 core-unit tests and 180 server-unit
> tests.


Yep this is crazy!

Those last tests are really time consuming, as the server is started
> and stopped for each single test, and that cost around 2 seconds for
> each server start. It cost more than 15 minutes on my laptop ...


Exactly.

Junit 4, the latest version we are currently using, offers a new
> annotation system where you can add a @BeforeClass and @AfterClass,
> applied on a startup and teardown methods only once for all the tests
> in a simple class. If we tune the tests correctly, we can then reduce
> the number of server startup.shutdown to 100 startup instead if 600.
> The gain will be quite important, as we may save 70% of the total
> cost.


How did you get the 100 instead of 600 figure?

I have done some test with JUnit 3.8, using a very ugly hack, and the
> result is that for the SearchTest, I went down from 46 seconds to 6
> seconds.


This is excellent.

There are a few things to do now :
> - I have not tested JUnit 4 right now, but there are everything we
> need to speed up the test, as far as I can see
> - if we are to switch to JUnit 4, we have to figure out which impact
> it has on the existing code, on Eclipse integration, and more
> important, on Maven (does Surefire support Junit 4 natively ?)


Oh yeah.  I'm scared to find out.

- another alternative would be to use the more evolved TestNG
> framework ; the very same steps should be followed (see previous
> point)


TestNG - I checked this out a while back.  I love it but Maven can't deal
well with it although technically it supports TestNG in surefire.  It's a
work in progress.  JUnit is more mainstream tho - I'd stick to that.

- If we want to start the server only once, we have to be very careful
> when writing a new test : it should not impact the existing data
> loaded. That mean that a test should left the data in the same state
> when it has been completed (successfully or not) than when it started.


This is the tricky part.  Perhaps a reverse application of changes can be
done using a Test interceptor.  We can btw do many things with a test
interceptor.

There may be some other options, like scripting tests (I was initially
> thinking about using JMeter, but it's far from being perfect), or
> adding a very simple framework to compare expected entries with those
> get from the server.
>
> At this point, I would like to get your opinion.
>

Been thinking about this for some time as you know.  At first I just made me
a ram drive on linux to deal with this :D - it saved like 50% of the time
but yeah that's laughable (both the technique and the time saved).  But this
was just a quick remedy.  Still 7 minutes is too long to run integration
tests.  The bulk of the cost of integration tests comes from bootstrapping
specifically off the top of my head I'd list the following top time suckers:

 o partition creation (with various db files)
     o schema partition
     o system partition
 o schema loading and checks

I suspect you can do thousands of adds, deletes and modifies for the cost of
bootstrapping.  So my recommendation is to find a way to snapshot the state
of the server then roll it back with anti-requests.  This really is not as
complex as it sounds.  We have the capability to do this easily using an
interceptor much like the changelog interceptor in my embedding workshop
which Ersin knocked out a while back.  Perhaps for each log operation
tracked by this interceptor we can have a configuration switch to track the
anti-ldif.  This would not be too hard to do.

Add => Delete
Delete => Add
Modify (Replace) => Modify (Replace)
Modify (Remove) => Modify (Add)
Modify (Add) => Modify (Remove)
ModifyDn => ModifyDN

The order would be reversed.  So if you have this train of requests { R1,
R2, R3, R4, R5 } that correspond to the following LDIF sequence { L1, L2,
L3, L4, L5 } then you just need the following anti operations { A5, A4, A3.
A2, A1 }.  Another way to have done this nicely is if we had transactions
enabled properly.  We could just rolllback :).  But this is a quick
workaround to the problem.

Alex