You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jena.apache.org by Claude Warren <cl...@xenei.com> on 2012/12/19 22:38:53 UTC
core test cases
Quick question:
When I run the maven test (mvn test) on the jena-core (2.7.4) It runs
8951 tests and finishes with success.
When I tell Eclipse to run all the test cases in the jena-core source
tree it runs 29888 tests of which 118 have errors and 62 fail.
Does anyone know why there is such a huge discrepancy?
Also, should I make all 29888 tests pass or only worry about the 8951?
-- Claude
--
I like: Like Like - The likeliest place on the web
Identity: https://www.identify.nu/user.php?claude@xenei.com
LinkedIn: http://www.linkedin.com/in/claudewarren
Re: core test cases
Posted by Andy Seaborne <an...@apache.org>.
On 20/12/12 12:54, Claude Warren wrote:
> Thanks for the info.
>
> I will concentrate on getting the TestPackage to work correctly and
> include the new tests.
>
> I am thinking of volunteering to go through the test cases and put
> them in some normal form (if we can decide on a normal form). but I
> am hoping someone stops me. ;) In anycase I am not going to do it for
> the defect I am working on now as it will be too big a rewrite.
>
> I'll spend a bit more time looking at the various test cases and
> structures for test cases. Perhaps a pass through to convert to
> JUnit4 annotations is in order. But then I think we need a bit more
> discussion before anyone heads down that road.
Makes sense to clean up when the test suite isn't itself changing for
other reasons. The good news is compatibility across versions isn't
necessary for the tests.
I wish I hadn't looked (:-)) - some tidying up would be a very good thing.
Any package called X.test can be pulled up - the use of X.test packages
is an ancient decision before the days of maven directory layout and
current tool support for multiple source hierarchies (IIRC).
What do we need to discuss-and-decide before some cleaning?
I'll help out cleaning up. I'm currently integration RIOT reader into
the system properly, and migrating org.openjena to org.apache.jena (does
not affect core) and that's nearly done. Expect a large commit soon - I
tried to build a patch but Eclipse couldn't.
Andy
>
> -- Claude
>
> On Thu, Dec 20, 2012 at 12:03 PM, Andy Seaborne <an...@apache.org> wrote:
>> On 19/12/12 21:38, Claude Warren wrote:
>>>
>>> Quick question:
>>>
>>> When I run the maven test (mvn test) on the jena-core (2.7.4) It runs
>>> 8951 tests and finishes with success.
>>>
>>> When I tell Eclipse to run all the test cases in the jena-core source
>>> tree it runs 29888 tests of which 118 have errors and 62 fail.
>>>
>>> Does anyone know why there is such a huge discrepancy?
>>>
>>> Also, should I make all 29888 tests pass or only worry about the 8951?
>>>
>>> -- Claude
>>>
>>
>> Short answer
>> + run com.hp.hpl.jena.test.TestPackage which is the 8K one.
>> + Looks like the "find tests" ends up double and triple accounting
>>
>>
>> It looks to me as if you get 20K tests because it is looking for all classes
>> beginning "Test".
>>
>> But the tests are grouped by classes always called "TestPackage" which
>> passes the filter! This is a really old decision.
>>
>> Worse, there is a top level one that calls other lower level, "TestPackage",
>> so double and triple accounting is going on.
>>
>> I don't know why some fail - either there are order dependencies lurking, or
>> some are really not tests but pass the search filter or we have some bugs.
>> I don't recall anyone removing tests even from Jena1 (IIRC that's the
>> "regression" set) just updating them.
>>
>> This could be better.
>>
>> I'm not sure it is ideal but the pattern I've fallen into in ARQ is that the
>> tests are grouped as:
>>
>> * Classes of tests are called Test*
>> * Tests are grouped together as TS_ ("Test Suite") using
>> @RunWith(Suite.class)
>> * Useful groups to run TC_ ("Test Collection")
>>
>> Maven for ARQ runs all TS_* which is different from core.
>>
>> I like my tests run in some sort of consistent order - I like low level test
>> run first then higher level ones. So test tokenization before testing the
>> parser grammar constructs.
>>
>> Downside is that this is fragile - if tests are not linked in they don't get
>> run. This is bad. Running all TS_ is a partial, limited way to address
>> this.
>>
>> There is a bit of a catch by the way - some tests are scripted (generated
>> from external files - RDF WG tests, SPARQL tests and others) and that is
>> hard/impossible to do in JUnit4 as far as I know. I gave up.
>>
>> At best, parametrized tests might work but they get meaning less names. So
>> Junit3 built-into-JUnit4 is used because in Junit3 you can create and name a
>> test case from java code. Maybe this has got better in JUnit4 - I haven't
>> looked for a few versions. If anyoBut there is a lot of JUnit3 that could
>> be JUnit4.
>> ne knows of a better way to do this, please do speak.
>>
>> But there is a lot of JUnit3 that could be JUnit4.
>>
>>
>> Also - I'm looking at jena-core 2.10-SNAPSHOT and see 23871/23875 tests. I
>> think it's less because the changes for simplification removed tests.
>>
>> For ARQ I get: 18388/18390 (it finds TS_* classes)
>> and for Jena core 23871/23875
>>
>> Some of the difference is Eclispe - there are tests marked unrun but they
>> clearly have. I think this is due to same-name clashes. It's not new.
>>
>> Andy
>>
>
>
>
Re: core test cases
Posted by Claude Warren <cl...@xenei.com>.
Thanks for the info.
I will concentrate on getting the TestPackage to work correctly and
include the new tests.
I am thinking of volunteering to go through the test cases and put
them in some normal form (if we can decide on a normal form). but I
am hoping someone stops me. ;) In anycase I am not going to do it for
the defect I am working on now as it will be too big a rewrite.
I'll spend a bit more time looking at the various test cases and
structures for test cases. Perhaps a pass through to convert to
JUnit4 annotations is in order. But then I think we need a bit more
discussion before anyone heads down that road.
-- Claude
On Thu, Dec 20, 2012 at 12:03 PM, Andy Seaborne <an...@apache.org> wrote:
> On 19/12/12 21:38, Claude Warren wrote:
>>
>> Quick question:
>>
>> When I run the maven test (mvn test) on the jena-core (2.7.4) It runs
>> 8951 tests and finishes with success.
>>
>> When I tell Eclipse to run all the test cases in the jena-core source
>> tree it runs 29888 tests of which 118 have errors and 62 fail.
>>
>> Does anyone know why there is such a huge discrepancy?
>>
>> Also, should I make all 29888 tests pass or only worry about the 8951?
>>
>> -- Claude
>>
>
> Short answer
> + run com.hp.hpl.jena.test.TestPackage which is the 8K one.
> + Looks like the "find tests" ends up double and triple accounting
>
>
> It looks to me as if you get 20K tests because it is looking for all classes
> beginning "Test".
>
> But the tests are grouped by classes always called "TestPackage" which
> passes the filter! This is a really old decision.
>
> Worse, there is a top level one that calls other lower level, "TestPackage",
> so double and triple accounting is going on.
>
> I don't know why some fail - either there are order dependencies lurking, or
> some are really not tests but pass the search filter or we have some bugs.
> I don't recall anyone removing tests even from Jena1 (IIRC that's the
> "regression" set) just updating them.
>
> This could be better.
>
> I'm not sure it is ideal but the pattern I've fallen into in ARQ is that the
> tests are grouped as:
>
> * Classes of tests are called Test*
> * Tests are grouped together as TS_ ("Test Suite") using
> @RunWith(Suite.class)
> * Useful groups to run TC_ ("Test Collection")
>
> Maven for ARQ runs all TS_* which is different from core.
>
> I like my tests run in some sort of consistent order - I like low level test
> run first then higher level ones. So test tokenization before testing the
> parser grammar constructs.
>
> Downside is that this is fragile - if tests are not linked in they don't get
> run. This is bad. Running all TS_ is a partial, limited way to address
> this.
>
> There is a bit of a catch by the way - some tests are scripted (generated
> from external files - RDF WG tests, SPARQL tests and others) and that is
> hard/impossible to do in JUnit4 as far as I know. I gave up.
>
> At best, parametrized tests might work but they get meaning less names. So
> Junit3 built-into-JUnit4 is used because in Junit3 you can create and name a
> test case from java code. Maybe this has got better in JUnit4 - I haven't
> looked for a few versions. If anyoBut there is a lot of JUnit3 that could
> be JUnit4.
> ne knows of a better way to do this, please do speak.
>
> But there is a lot of JUnit3 that could be JUnit4.
>
>
> Also - I'm looking at jena-core 2.10-SNAPSHOT and see 23871/23875 tests. I
> think it's less because the changes for simplification removed tests.
>
> For ARQ I get: 18388/18390 (it finds TS_* classes)
> and for Jena core 23871/23875
>
> Some of the difference is Eclispe - there are tests marked unrun but they
> clearly have. I think this is due to same-name clashes. It's not new.
>
> Andy
>
--
I like: Like Like - The likeliest place on the web
Identity: https://www.identify.nu/user.php?claude@xenei.com
LinkedIn: http://www.linkedin.com/in/claudewarren
Re: core test cases
Posted by Andy Seaborne <an...@apache.org>.
On 19/12/12 21:38, Claude Warren wrote:
> Quick question:
>
> When I run the maven test (mvn test) on the jena-core (2.7.4) It runs
> 8951 tests and finishes with success.
>
> When I tell Eclipse to run all the test cases in the jena-core source
> tree it runs 29888 tests of which 118 have errors and 62 fail.
>
> Does anyone know why there is such a huge discrepancy?
>
> Also, should I make all 29888 tests pass or only worry about the 8951?
>
> -- Claude
>
Short answer
+ run com.hp.hpl.jena.test.TestPackage which is the 8K one.
+ Looks like the "find tests" ends up double and triple accounting
It looks to me as if you get 20K tests because it is looking for all
classes beginning "Test".
But the tests are grouped by classes always called "TestPackage" which
passes the filter! This is a really old decision.
Worse, there is a top level one that calls other lower level,
"TestPackage", so double and triple accounting is going on.
I don't know why some fail - either there are order dependencies
lurking, or some are really not tests but pass the search filter or we
have some bugs. I don't recall anyone removing tests even from Jena1
(IIRC that's the "regression" set) just updating them.
This could be better.
I'm not sure it is ideal but the pattern I've fallen into in ARQ is that
the tests are grouped as:
* Classes of tests are called Test*
* Tests are grouped together as TS_ ("Test Suite") using
@RunWith(Suite.class)
* Useful groups to run TC_ ("Test Collection")
Maven for ARQ runs all TS_* which is different from core.
I like my tests run in some sort of consistent order - I like low level
test run first then higher level ones. So test tokenization before
testing the parser grammar constructs.
Downside is that this is fragile - if tests are not linked in they don't
get run. This is bad. Running all TS_ is a partial, limited way to
address this.
There is a bit of a catch by the way - some tests are scripted
(generated from external files - RDF WG tests, SPARQL tests and others)
and that is hard/impossible to do in JUnit4 as far as I know. I gave up.
At best, parametrized tests might work but they get meaning less names.
So Junit3 built-into-JUnit4 is used because in Junit3 you can create
and name a test case from java code. Maybe this has got better in
JUnit4 - I haven't looked for a few versions. If anyoBut there is a lot
of JUnit3 that could be JUnit4.
ne knows of a better way to do this, please do speak.
But there is a lot of JUnit3 that could be JUnit4.
Also - I'm looking at jena-core 2.10-SNAPSHOT and see 23871/23875 tests.
I think it's less because the changes for simplification removed tests.
For ARQ I get: 18388/18390 (it finds TS_* classes)
and for Jena core 23871/23875
Some of the difference is Eclispe - there are tests marked unrun but
they clearly have. I think this is due to same-name clashes. It's not new.
Andy