Testing, Test Coverage, etc. was Re: Adding Clover to hudson

[Grant Ingersoll <gs...@apache.org>]

OK, Clover should be hooked in and hopefully doesn't have to go  
through many hoops.  You can see reports on the latest at: http://hudson.zones.apache.org/hudson/job/Solr-trunk/clover/

Too bad we didn't put in a Google Summer of Code option to write test  
cases like the Derby folks did.  :-)

I think we may want to consider, similarly to what we do in Lucene- 
Java, is introducing a code freeze before official release, and one of  
the things we do is tackle the fact that a pretty sizable chunk of  
Solr code is not ever tested.  Typically, in Lucene land we freeze for  
7 to 10 days and the only allowed commits in that time are blockers  
and documentation.  We probably could add to that test cases.

I also find the BasicFunctionalityTest to be a bit annoying from a  
purist Unit Testing standpoint.  For instance, today I was looking at  
the Date Faceting patch as applied to SimpleFacets and instinctively  
went to look for TestSimpleFacets (or, SimpleFacetsTest) which doesn't  
exist.  Then, later, discovered that the Faceting test stuff is in the  
BFT class, which is a bit harder to remember and I think causes people  
to wonder where tests are located for a given class.

Anyway, just my two cents on how we can improve testing in Solr.

-Grant


On Apr 16, 2008, at 12:21 PM, Chris Hostetter wrote:

>
> : I'm testing adding clover to hudson builds, so please disregard  
> any failure
> : notices.
>
> Oh yeah ... i've been meaning to get to that.
>
> FWIW: I'm not sure how far you've gotten, but i don't think we need to
> jump through as many hoops as Lucene does to build with clover, then  
> build
> again without -- we don't advertise the artifacts from Hudson  
> because we
> also still have nightly.sh.   Having both is a nice little bit of  
> sanity
> check, and I think it would be fine to make the hudson build have no
> artifacts, just verify the tests and run the metrics.
>
>
> -Hoss
>

Re: Testing, Test Coverage, etc. was Re: Adding Clover to hudson

[Grant Ingersoll <gs...@apache.org>]

+1

On Apr 17, 2008, at 1:15 AM, Chris Hostetter wrote:

>
> : I think we may want to consider, similarly to what we do in Lucene- 
> Java, is
> : introducing a code freeze before official release, and one of the  
> things we do
> : is tackle the fact that a pretty sizable chunk of Solr code is not  
> ever
> : tested.  Typically, in Lucene land we freeze for 7 to 10 days and  
> the only
> : allowed commits in that time are blockers and documentation.  We  
> probably
> : could add to that test cases.
>
> Agreed ... we have some TODOs on the "How To Release" page about  
> trying to
> include the process *arround* the actual release creation as well
> (preperation, and post release tasks) that we can beef up with stuff  
> like
> this, but like most documentation: it's fallen through the cracks...
> 	http://wiki.apache.org/solr/HowToRelease
>
> : I also find the BasicFunctionalityTest to be a bit annoying from a  
> purist Unit
> : Testing standpoint.  For instance, today I was looking at the Date  
> Faceting
> : patch as applied to SimpleFacets and instinctively went to look for
> : TestSimpleFacets (or, SimpleFacetsTest) which doesn't exist.   
> Then, later,
> : discovered that the Faceting test stuff is in the BFT class, which  
> is a bit
> : harder to remember and I think causes people to wonder where tests  
> are located
> : for a given class.
>
> I take full responsibility for the mess that is  
> BasicFunctionalityTest ...
> it started out as exactly what the name implies, but little by  
> little it
> was easier to add more stuff to that one class then to create new  
> classes.
> I agree it should be refactored, but writing new tests is probably a
> bigger win ... the general strategy that would probably make the most
> sense is: when writing some new test cases, if part of the  
> functionality
> is already covered in BFT, refactor the existing stuff inot a new  
> class,
> and add your new tests.  we keep refactoring until BFT is empty, and  
> then
> add some redundent tests for truly basic stuff back in and leave the  
> test
> as what it should have been: an example of the way to write simple
> "functionality" tests without needing to know a lot about the internal
> APIs.
>
>
>
> -Hoss
>

--------------------------
Grant Ingersoll

Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

Re: Testing, Test Coverage, etc. was Re: Adding Clover to hudson

[Chris Hostetter <ho...@fucit.org>]

: I think we may want to consider, similarly to what we do in Lucene-Java, is
: introducing a code freeze before official release, and one of the things we do
: is tackle the fact that a pretty sizable chunk of Solr code is not ever
: tested.  Typically, in Lucene land we freeze for 7 to 10 days and the only
: allowed commits in that time are blockers and documentation.  We probably
: could add to that test cases.

Agreed ... we have some TODOs on the "How To Release" page about trying to 
include the process *arround* the actual release creation as well 
(preperation, and post release tasks) that we can beef up with stuff like 
this, but like most documentation: it's fallen through the cracks...
	http://wiki.apache.org/solr/HowToRelease

: I also find the BasicFunctionalityTest to be a bit annoying from a purist Unit
: Testing standpoint.  For instance, today I was looking at the Date Faceting
: patch as applied to SimpleFacets and instinctively went to look for
: TestSimpleFacets (or, SimpleFacetsTest) which doesn't exist.  Then, later,
: discovered that the Faceting test stuff is in the BFT class, which is a bit
: harder to remember and I think causes people to wonder where tests are located
: for a given class.

I take full responsibility for the mess that is BasicFunctionalityTest ... 
it started out as exactly what the name implies, but little by little it 
was easier to add more stuff to that one class then to create new classes.  
I agree it should be refactored, but writing new tests is probably a 
bigger win ... the general strategy that would probably make the most 
sense is: when writing some new test cases, if part of the functionality 
is already covered in BFT, refactor the existing stuff inot a new class, 
and add your new tests.  we keep refactoring until BFT is empty, and then 
add some redundent tests for truly basic stuff back in and leave the test 
as what it should have been: an example of the way to write simple 
"functionality" tests without needing to know a lot about the internal 
APIs.



-Hoss