Re: [Fwd: Re: [Testing] Coping with so many integration tests]

[Emmanuel Lecharny <el...@gmail.com>]

Don't know why, the original mail from Alex never hit my mail box ...

On 2/5/07, Alex Karasulu <ak...@apache.org> wrote:
>
>
>
>
> ---------- Forwarded message ----------
> Emmanuel Lecharny wrote:
> > Yeah, running integration tests is time consumming. For 1.0.1, it's
> > acceptable, as it takes only 3min (depending on your computer), but in
> > 1.5, it's around 5 times slower.
>
> We have many more tests about 40% more.


Yeah. In 1.0 we have  218 tests in core-unit, and 365 in 1.5

> We may consider finding out why it takes so long in 1.5 (server
> > initialization takes around 10 to 15 seconds, which is not really
> normal)
>
> Please Emmanuel it does not take this long for startup. You're going to
> freak people out by saying stuff like this.  Let's watch these kinds of
> exaggerations.


Sorry, I was totally wrong about those numbers. I guess I pulled them out of
my head just when I woke up this morning, after a very bad sleep.

Let me correct them :
- each unit test is around 1 sec, startup and shutdown included in 1.5.0,
not 10 to 15 seconds.
- in 1.0.1, each test is around 0,2 seconds, startup and shutdown included.

Be aware that those number has been taken on *my* computer, and by no means
they are the official numbers of Apache Directory Server.

It does take a little longer in 1.5 but really we have about 40% more
> tests and initialization time is slower but no where near 10-15 seconds.
>
> A single integration test case (with one test method) takes about 3.5
> seconds to start, stop and cleanup on my old hardware (AMD 2800+).


It's faster on my computer. 3.5 seconds is really above what I really get. I
got around 1.8 to 2.5 seconds for single tests, but you have to ake into
account the classloading of the unit test and stuff like that, which is not
free...

Here's a run of just one empty test case. Committed here:
>
>     mvn -Dtest=OneEmptyITest -Dintegration test
> Running org.apache.directory.server.core.schema.OneEmptyITest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.538 sec


I get better results :)

Running org.apache.directory.server.core.schema.OneEmptyITest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.929 sec

Man, my imac is twice faster than your old AMD ;)

And as you can see 3.538 seconds is not anywhere near 10-15 seconds
> exaggeration.


Very true. My bad ... Never ever drink on sunday evening ;(


> Now you said that this is 5x slower in 1.5 branch than it is in the 1.0
> branch.
> So we're only 2x slower in the 1.5 branch than in the 1.0 branch.  Again
> that's way less than the 5 times figure you just threw out there.


You are doing micro benchmark here. What I was talking about was the whole
Integration test. It takes 2 mins and a half on 1.0.1, and  12 mins 06
seconds on 1.5.0
If you eliminate all the artefacts, it's around 4 times slower, considering
we have 40% more tests...

Anyway, beside the wrong numbers I throwed to the world in my mail (trully
sorry about that, my bad), here are some elements we must consider :
- all the modifications done in 1.5 are so important that we can live for a
while with a slower integration phase
- we have 40% more tests, so it make sense that the integration test phase
is slower, but, much more important, that do *not* means that 1.0.1 is not
tested correctly : the 40% new tests are related to the new functions we
have introduced in 1.5
- Important point too is that we have not tried to improve the performances
in 1.5, because it's an intermediate version, the stable one will be 1.6. We
can live with this lower performances (we don't launch integration tests
every 5 mins, are we ???)

ok, just wanted to clarify my first mail, which was not correct. Thanks Alex
for the heads up !
-- 
Cordialement,
Emmanuel Lécharny
www.iktek.com