About tests and Ldif injection annotations (completed)

[Emmanuel Lecharny <el...@apache.org>]

Hi guys,

the cool annotations we have, combined with the rollback system,
allows us to run tests without any problem. Each test can play with
data without interacting with the other tests. However, as the LDIF
injection is done globally for each test, the rollback is also done
globally. ie, if you inject a lot of data, you will have to revert a
lot of data too, for each single test.

We are using it today for one of our client, and as we have around 400
entries to inject for our tests, it takes more or less a couple of
seconds to run each test. At the end of the day, when we run our 150
tests, we have to wait 10 minutes to run all of them.

The data we are injecting are not likely to change often, and we would
rather prefer to inject them only once per class, and be able to
revert them only when all the methods are run. Even better, if we
define a suite, we should be able to inject all the data when the
suite starts, and reverts when it's done.

The current system does not allow such a possibility.

I have tested something on the core tests : I inject the LDIFs at each
level (suite, class and method), and revert accordingly. (when I run a
test, I just inject the associated LDIFs, and I revert them, but don't
revert the class ldif or the suite ldif).

No need to tell that it's way faster ! I was able to run the
SearchITest in less than 5 seconds compared to the usual 15 seconds
for the very same tests (and i'm just talking of 4 entries injected
and rollbacked for every tests).

So I think this is the way to go. The only difficult point is that we
have to deal with 5 different modes for tests :
- PRISTINE
- RESTART
- ROLLBACK (the default)
- CUMULATIVE
- NOSERVICE.

To me, the RESTART, CUMULATIVE and NOSERVICE mode seems a bit useless.
The PRISTINE mode is used for one single test in which we declare two
different factories, hence the need to clean up completely the server.

I would suggest we get rid of the RESTART, CUMULATIVE and NOSERVICE
mode in order to ease the implementation of my small improvement
(otehriwse, there are too many cases to deal with, and i don't realy
have time for that).

wdyt?

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: About tests and Ldif injection annotations (completed)

[Emmanuel Lecharny <el...@apache.org>]

On Wed, Feb 11, 2009 at 1:11 AM, Alex Karasulu <ak...@gmail.com> wrote:
>
>
> On Mon, Feb 9, 2009 at 11:52 AM, Emmanuel Lecharny <el...@apache.org>
> wrote:
>>
>> Hi guys,
>>
>> the cool annotations we have, combined with the rollback system,
>> allows us to run tests without any problem. Each test can play with
>> data without interacting with the other tests. However, as the LDIF
>> injection is done globally for each test, the rollback is also done
>> globally. ie, if you inject a lot of data, you will have to revert a
>> lot of data too, for each single test.
>>
>> We are using it today for one of our client, and as we have around 400
>> entries to inject for our tests, it takes more or less a couple of
>> seconds to run each test. At the end of the day, when we run our 150
>> tests, we have to wait 10 minutes to run all of them.
>
> Use suites and have a full cleanup take place between the suites.

It does not work. The LDIF you inject at the suite level are injected
for any method, as the settings are considered globally (here, the
code is broken)

You have
> various cleanup levels (test level, class level, suite level etc) and you
> can set it up so that a suite class has an annotation which make it cleanup
> when the suite is started with a PRISTINE server.  That's why all this was
> put in there.  We use this today for the Apache DS tests.

This was what it was intended for. But we can have a BIG improvement
in tests if we correctly inject LDIF only when needed.

>
>>
>> No need to tell that it's way faster ! I was able to run the
>> SearchITest in less than 5 seconds compared to the usual 15 seconds
>> for the very same tests (and i'm just talking of 4 entries injected
>> and rollbacked for every tests).
>>
>> So I think this is the way to go. The only difficult point is that we
>> have to deal with 5 different modes for tests :
>> - PRISTINE
>> - RESTART
>> - ROLLBACK (the default)
>> - CUMULATIVE
>> - NOSERVICE.
>>
>> To me, the RESTART, CUMULATIVE and NOSERVICE mode seems a bit useless.
>
> There are reasons for these - I can explain this tomorrow.  Please don't nix
> this stuff until we discuss it somemore.

Not a big deal to keep them if they are usefull.

>>
>> The PRISTINE mode is used for one single test in which we declare two
>> different factories, hence the need to clean up completely the server.
>>
>> I would suggest we get rid of the RESTART, CUMULATIVE and NOSERVICE
>> mode in order to ease the implementation of my small improvement
>> (otehriwse, there are too many cases to deal with, and i don't realy
>> have time for that).
>>
>> wdyt?
>
> I think it is a bad idea but I may be wrong.  Shit cannot remember that far
> back when we did all this stuff.

I think we tried to cover all of your bases with all the possible
cases. Right now, only the TOOLBACK and PRISTINE are used, and
PRISTINE is only used when we define different factories inside a
test.

CUMULATIVE was defined for the case we want to run tests without doing
any cleanup between each test, but I don't see how we can use it
asuuming we can't guarantee the test order.

RESTART is a bit stronger than PRISTINE, as it stop the directory
service, and start it again, cleaning the whole server.

NOSERVICE? I have no idea about why it was defined... I have to check the code.

>
> Alex
>



-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: About tests and Ldif injection annotations (completed)

[Emmanuel Lecharny <el...@apache.org>]

>> So I think this is the way to go. The only difficult point is that we
>> have to deal with 5 different modes for tests :
>> - PRISTINE
>> - RESTART
>> - ROLLBACK (the default)
>> - CUMULATIVE
>> - NOSERVICE.
>>
>> To me, the RESTART, CUMULATIVE and NOSERVICE mode seems a bit useless.
>
> There are reasons for these - I can explain this tomorrow.  Please don't nix
> this stuff until we discuss it somemore.
>
>>
>> The PRISTINE mode is used for one single test in which we declare two
>> different factories, hence the need to clean up completely the server.
>>
>> I would suggest we get rid of the RESTART, CUMULATIVE and NOSERVICE
>> mode in order to ease the implementation of my small improvement
>> (otehriwse, there are too many cases to deal with, and i don't realy
>> have time for that).
>>
>> wdyt?
>
> I think it is a bad idea but I may be wrong.  Shit cannot remember that far
> back when we did all this stuff.

regarding CUMULATIVE, have a loo, at what you wrote months ago on the wiki :
http://cwiki.apache.org/confluence/display/DIRxSRVx11/Core+Integration+Testing+Framework

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: About tests and Ldif injection annotations (completed)

[Alex Karasulu <ak...@gmail.com>]

On Mon, Feb 9, 2009 at 11:52 AM, Emmanuel Lecharny <el...@apache.org>wrote:

> Hi guys,
>
> the cool annotations we have, combined with the rollback system,
> allows us to run tests without any problem. Each test can play with
> data without interacting with the other tests. However, as the LDIF
> injection is done globally for each test, the rollback is also done
> globally. ie, if you inject a lot of data, you will have to revert a
> lot of data too, for each single test.
>
> We are using it today for one of our client, and as we have around 400
> entries to inject for our tests, it takes more or less a couple of
> seconds to run each test. At the end of the day, when we run our 150
> tests, we have to wait 10 minutes to run all of them.
>

Use suites and have a full cleanup take place between the suites. You have
various cleanup levels (test level, class level, suite level etc) and you
can set it up so that a suite class has an annotation which make it cleanup
when the suite is started with a PRISTINE server.  That's why all this was
put in there.  We use this today for the Apache DS tests.


>
> No need to tell that it's way faster ! I was able to run the
> SearchITest in less than 5 seconds compared to the usual 15 seconds
> for the very same tests (and i'm just talking of 4 entries injected
> and rollbacked for every tests).
>
> So I think this is the way to go. The only difficult point is that we
> have to deal with 5 different modes for tests :
> - PRISTINE
> - RESTART
> - ROLLBACK (the default)
> - CUMULATIVE
> - NOSERVICE.
>
> To me, the RESTART, CUMULATIVE and NOSERVICE mode seems a bit useless.


There are reasons for these - I can explain this tomorrow.  Please don't nix
this stuff until we discuss it somemore.


>
> The PRISTINE mode is used for one single test in which we declare two
> different factories, hence the need to clean up completely the server.
>
> I would suggest we get rid of the RESTART, CUMULATIVE and NOSERVICE
> mode in order to ease the implementation of my small improvement
> (otehriwse, there are too many cases to deal with, and i don't realy
> have time for that).
>
> wdyt?
>

I think it is a bad idea but I may be wrong.  Shit cannot remember that far
back when we did all this stuff.

Alex