[ApacheDS] Separating base unittest classes (Was: Re: svn commit: r307398)

[Alex Karasulu <ao...@bellsouth.net>]

Emmanuel Lecharny wrote:

>>>Ooops. May be a little convo on IRC could help a lot avoiding the code
>>>breaking I induced...
>>> 
>>>
>>>      
>>>
>>Let's keep this on the list.  
>>    
>>
>
>+1
>
>  
>
>>Also we *MUST* always make sure we summarize our conclusions on IRC.
>>    
>>
>
>+1 to this kind of wish
>
>  
>
>>Please don't take this Emmanuel this is more a reminder to myself :-) as 
>>well.  We neeed to watch how much info is lost on IRC now that our IRC 
>>channel's are not being logged anymore.
>>    
>>
>
>We must find a way to log those infos, and how to publish them.
>
>
>Back to the pb, what about a test-setup subproject in apacheds project?
>  
>
What do we have today?
========================

Right now apacheds has the following subprojects:

shared
core
main
plugin

Let's try to understand some of the dependencies:

main deps core
main deps shared
core deps plugin
core deps shared
plugin deps shared

The the unit testing "framework" (if you could call it that) for 
apacheds' *core* is just composed of three classes.  We're talking about 
AbstractTestCase, AbstractAdminTestCase and AbstractNonAdminTestCase.  
These test cases do not fire up the protocol providers for apacheds ... 
just the core.  This means you cannot do over the wire LDAP tests using 
the SUN JNDI Provider or with JLDAP with these TestCase base classes.  
This code just depends on the core and on shared.

In the *main* project resides another testcase base class called 
AbstractServerTest which turns on the protocols in addition to firing up 
the core.  This code depends on the main, core and shared.


Why should we separate unit test base classes?
==============================================

Now let's ask ourselves why we need to separate this stuff out from core 
and main.  I think its because we don't want these necessarily shipping 
with the core or the main since users may not be interested in unit 
testing and hence do not require those classes or the jars they depend 
on.  It thus makes sense, why, you Emmanuel, did not want these files in 
the src/main/java directory but in the src/test directory.  So 
separating them out of the core and main is a good idea.  How this is 
done or if it's worth doing is another story.

The How?
==========

Let's start first by just considering the core.  It has the 3 abstract 
test cases which must be removed however these test cases depend on core 
classes.  Core in turn depends on the test cases to drive its tests.  
This imposes a cyclic dependency.  So we cannot just separate the test 
base classes from the core.  We need to create some shared project that 
both the core and the core-unit tests depend on to break the cycle.  We 
have the same situation for the main.  In the end if we were to do this 
right then we need to create 4 new maven child projects for apacheds:

core-shared
core-unit
main-shared
main-unit

Is it worth it?
==============

IMHO it is not worth creating 4 new maven projects to get 3 classes out 
of the way.  This is the conclusion I came to 6 months ago when I had 
the idea of making this separation. 

What do I think now? Well let's not mess with this:

(1) the entropy incurred for absolute correctness is too much
(2) the unit tests are nice to have in the jar: they're just there if 
you need em
(3) the junit jar is not required as a dependency at runtime unless you 
use the base unit tests

So in the end, I actually think it becomes more convenient to leave the 
base test cases as-is in their respective src/main/java directories in 
core and main respectively.  Let's not bother with 4 new maven 
projects.  WDYT?

Alex

Re: [ApacheDS] Separating base unittest classes (Was: Re: svn commit: r307398)

[Emmanuel Lecharny <el...@apache.org>]

I will start from the bottom of your message :

<snip/>
> Let's not bother with 4 new maven projects.  WDYT?

YEAH ! Let's not bother with 4 more maven projects ;)

Man, you now how to talk to men : KIS, S

Ok, I rollback the change I did.

-- Emmanuel, the last S in KISS.

Re: [ApacheDS] Separating base unittest classes (Was: Re: svn commit: r307398)

[Trustin Lee <tr...@gmail.com>]

2005/10/13, Alex Karasulu <ao...@bellsouth.net>:
>
> Trustin Lee wrote:
>
> > 2005/10/12, Alex Karasulu <aok123@bellsouth.net
> > <ma...@bellsouth.net>>:
> >
> > Trustin Lee wrote:
> >
> > > The essential problem in our project is that our tests are not unit
> > > tests. If we're doing unit tests, we don't need to start up the
> > whole
> > > ApacheDS everytime we test each classes. But we're doing so by
> > some
> > > reason and the tests takes too much time. Ideally we should change
> > > all of them to unit tests strictly speaking.
> >
> > You're right these are more *integration* tests and not simple unit
> > tests. Again correctness is not always the most sensible approach in
> > our present not so perfect situation :).
> >
> > It's very hard to unit test these interceptors properly though because
> > many presume access to the core. I see how you performed unit
> > tests on
> > the ACI code but I think this was a special case. Without a
> > harness you
> > cannot really test an interceptor.
> >
> >
> > Once we implement in-memory DirectoryPartition (formerly
> > ContextPArtition) implementation, then we can implement mock
> > NextInterceptor very easily which is the hardest part. Nothing is
> > different in this case.
> >
> > So what you're suggesting is we move these tests to an apacheds test
> > project? Hmmm ... That would mean we don't have to create the extra
> > maven projects since there would be no dep in core and in main for
> > these
> > abstract testcases. I like the sound of this, but it raises some
> > concerns for me. I kind of like the fact that nothing deploys unless
> > those tests pass even though it takes a long time for them to run.
> >
> >
> > Yes. But this doesn't necessarily mean that we have to run integration
> > test for *every* test case just like now. We can create a single
> > integration test that tries to launch ApacheDS with full set of
> > interceptors and to test basic LDAP operations. If the core still has
> > some issues, then it's a bug of our test code primarily so I cannot
> > believe the integration test helps us to test individual interceptors.
>
> I don't know what you're talking about now. Sorry I'm lost.


What I'm pointing out are two:

1) We can write unit test for most interceptors and if we can't, it means
we've got some design problem which means that our code is tightly coupled
to each other that leads to the integration which hinders unit testing.

2) For now all interceptor tests are integration tests, so it takes more
than a minute to run whole tests. We don't have to do this only if we have
proper unit tests.

Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/

Re: [ApacheDS] Separating base unittest classes (Was: Re: svn commit: r307398)

[Alex Karasulu <ao...@bellsouth.net>]

Trustin Lee wrote:

> 2005/10/12, Alex Karasulu <aok123@bellsouth.net 
> <ma...@bellsouth.net>>:
>
>     Trustin Lee wrote:
>
>     > The essential problem in our project is that our tests are not unit
>     > tests.  If we're doing unit tests, we don't need to start up the
>     whole
>     > ApacheDS everytime we test each classes.  But we're doing so by
>     some
>     > reason and the tests takes too much time.  Ideally we should change
>     > all of them to unit tests strictly speaking.
>
>     You're right these are more *integration* tests and not simple unit
>     tests.  Again correctness is not always the most sensible approach in
>     our present not so perfect situation :).
>
>     It's very hard to unit test these interceptors properly though because
>     many presume access to the core.  I see how you performed unit
>     tests on
>     the ACI code but I think this was a special case.  Without a
>     harness you
>     cannot really test an interceptor.
>
>
> Once we implement in-memory DirectoryPartition (formerly 
> ContextPArtition) implementation, then we can implement mock 
> NextInterceptor very easily which is the hardest part.  Nothing is 
> different in this case.
>
>     So what you're suggesting is we move these tests to an apacheds test
>     project?  Hmmm ... That would mean we don't have to create the extra
>     maven projects since there would be no dep in core and in main for
>     these
>     abstract testcases.  I like the sound of this, but it raises some
>     concerns for me. I kind of like the fact that nothing deploys unless
>     those tests pass even though it takes a long time for them to run.
>
>
> Yes. But this doesn't necessarily mean that we have to run integration 
> test for *every* test case just like now.  We can create a single 
> integration test that tries to launch ApacheDS with full set of 
> interceptors and to test basic LDAP operations.  If the core still has 
> some issues, then it's a bug of our test code primarily so I cannot 
> believe the integration test helps us to test individual interceptors.

I don't know what you're talking about now.   Sorry I'm lost.

Alex

Re: [ApacheDS] Separating base unittest classes (Was: Re: svn commit: r307398)

[Trustin Lee <tr...@gmail.com>]

2005/10/12, Alex Karasulu <ao...@bellsouth.net>:
>
> Trustin Lee wrote:
>
> > The essential problem in our project is that our tests are not unit
> > tests. If we're doing unit tests, we don't need to start up the whole
> > ApacheDS everytime we test each classes. But we're doing so by some
> > reason and the tests takes too much time. Ideally we should change
> > all of them to unit tests strictly speaking.
>
> You're right these are more *integration* tests and not simple unit
> tests. Again correctness is not always the most sensible approach in
> our present not so perfect situation :).
>
> It's very hard to unit test these interceptors properly though because
> many presume access to the core. I see how you performed unit tests on
> the ACI code but I think this was a special case. Without a harness you
> cannot really test an interceptor.


Once we implement in-memory DirectoryPartition (formerly ContextPArtition)
implementation, then we can implement mock NextInterceptor very easily which
is the hardest part. Nothing is different in this case.

So what you're suggesting is we move these tests to an apacheds test
> project? Hmmm ... That would mean we don't have to create the extra
> maven projects since there would be no dep in core and in main for these
> abstract testcases. I like the sound of this, but it raises some
> concerns for me. I kind of like the fact that nothing deploys unless
> those tests pass even though it takes a long time for them to run.


Yes. But this doesn't necessarily mean that we have to run integration test
for *every* test case just like now. We can create a single integration test
that tries to launch ApacheDS with full set of interceptors and to test
basic LDAP operations. If the core still has some issues, then it's a bug of
our test code primarily so I cannot believe the integration test helps us to
test individual interceptors.

Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/

Re: [ApacheDS] Separating base unittest classes (Was: Re: svn commit: r307398)

[Alex Karasulu <ao...@bellsouth.net>]

Trustin Lee wrote:

> The essential problem in our project is that our tests are not unit 
> tests.  If we're doing unit tests, we don't need to start up the whole 
> ApacheDS everytime we test each classes.  But we're doing so by some 
> reason and the tests takes too much time.  Ideally we should change 
> all of them to unit tests strictly speaking.

You're right these are more *integration* tests and not simple unit 
tests.  Again correctness is not always the most sensible approach in 
our present not so perfect situation :).

It's very hard to unit test these interceptors properly though because 
many presume access to the core.  I see how you performed unit tests on 
the ACI code but I think this was a special case.  Without a harness you 
cannot really test an interceptor.

So what you're suggesting is we move these tests to an apacheds test 
project?  Hmmm ... That would mean we don't have to create the extra 
maven projects since there would be no dep in core and in main for these 
abstract testcases.  I like the sound of this, but it raises some 
concerns for me. I kind of like the fact that nothing deploys unless 
those tests pass even though it takes a long time for them to run.

What else do you think we can do to make sure we protect ourselves?  
Because if we move these "integration" tests into a separate subproject 
it opens the door to the deployment of code in core and main that might 
not work.  The fact that it compiles and simple unit tests run does not 
mean the server will run.  Furthermore if we move the tests, we're going 
to have a lag before the proper unit tests are written.  Who's going to 
retrofit unit tests to all the methods.  No one is my guess.  Right now 
with these integration tests we at least have some kind of coverage to 
keep us from shooting ourselves in the foot.

WDYT?

I should not be advertising this hehe but if test execution times are an 
issue you could always bypass them on builds instead of deploys with 
this option to maven:

-Dmaven.tests.skip=true

Question is: does the fact that someone can do this to deploy artifacts 
to the remote repository make keeping these integration tests in core 
and main moot?  Probably so.  If we do remove these tests then we're 
going to need some serious protocol for managing uploads just to make 
sure these tests run and pass.

Alex

Re: [ApacheDS] Separating base unittest classes (Was: Re: svn commit: r307398)

[Trustin Lee <tr...@gmail.com>]

The essential problem in our project is that our tests are not unit tests.
If we're doing unit tests, we don't need to start up the whole ApacheDS
everytime we test each classes. But we're doing so by some reason and the
tests takes too much time. Ideally we should change all of them to unit
tests strictly speaking.

I think most interceptors can be converted into unit tests just like I did
for classes in org.apache.ldap.server.authz.support package.

If we're going to create a separate testing framework project, it could be a
JndiUnit project which works for all JNDI implementations IMHO.

Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/