You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stanbol.apache.org by Rupert Westenthaler <ru...@gmail.com> on 2012/11/15 15:36:21 UTC
Unit tests that depend on the execution order
Hi all,
Short Version:
* Developers: should check their test and make sure they do not assume
an certain execution order
* Users: Please help us by reporting failed Unit Test. Your issue is
most likely specific to the JVM you are using so we might not be aware
of the problem. Thanks!
- - -
Long Version with more details (for Stanbol Developers):
Since about half a year there where occasionally problems with failed
unit tests caused by nondeterministic method ordering in some JVM
versions (see STANBOL-620).
Ideally Unit Tests need to be independent and therefore the execution
order should not matter at all. However the fact was that a lot of
Unit Tests in Stanbol are not independent but assume an certain
execution oder (e.g. a CRUD test class that first creates, than
retrieves, updated and deletes items in multiple test methods).
Initial the assumption was that with the release of junit-4.11
execution orders will be re-assured, but as I discovered today this
was not the case (at least not out of the box). Because of that I
decided to fix affected Unit Tests [1-4].
However this does not mean that all such issues are solved. e.g. the
RuleManager test [4] was only failing on
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.5)
(6b24-1.11.5-0ubuntu1~12.04.1)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
but was fine on the Mac
java version "1.6.0_37"
Java(TM) SE Runtime Environment (build 1.6.0_37-b06-434-11M3909)
Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01-434, mixed mode)
best
Rupert
[1] http://svn.apache.org/viewvc?rev=1409744&view=rev
[2] http://svn.apache.org/viewvc?rev=1409745&view=rev
[3] http://svn.apache.org/viewvc?rev=1409749&view=rev
[4] http://svn.apache.org/viewvc?rev=1409755&view=rev
--
| Rupert Westenthaler rupert.westenthaler@gmail.com
| Bodenlehenstraße 11 ++43-699-11108907
| A-5500 Bischofshofen
Re: Unit tests that depend on the execution order
Posted by Peter Ansell <an...@gmail.com>.
On 16 November 2012 00:36, Rupert Westenthaler
<ru...@gmail.com> wrote:
> Hi all,
>
> Short Version:
>
> * Developers: should check their test and make sure they do not assume
> an certain execution order
> * Users: Please help us by reporting failed Unit Test. Your issue is
> most likely specific to the JVM you are using so we might not be aware
> of the problem. Thanks!
>
> - - -
>
> Long Version with more details (for Stanbol Developers):
>
> Since about half a year there where occasionally problems with failed
> unit tests caused by nondeterministic method ordering in some JVM
> versions (see STANBOL-620).
>
> Ideally Unit Tests need to be independent and therefore the execution
> order should not matter at all. However the fact was that a lot of
> Unit Tests in Stanbol are not independent but assume an certain
> execution oder (e.g. a CRUD test class that first creates, than
> retrieves, updated and deletes items in multiple test methods).
>
> Initial the assumption was that with the release of junit-4.11
> execution orders will be re-assured, but as I discovered today this
> was not the case (at least not out of the box). Because of that I
> decided to fix affected Unit Tests [1-4].
Although it is a bad idea in general, if you need to fix the order for
a test, you can do so using the new @FixMethodOrder annotation [1].
MethodSorters.JVM will not be consistent across runs or JVMs for the
latest java6 and java7 runtimes, but some of the other MethodSorters
are deterministic.
Cheers,
Peter
[1] https://github.com/KentBeck/junit/blob/master/doc/ReleaseNotes4.11.md
Re: Unit tests that depend on the execution order
Posted by Andriy Nikolov <an...@fluidops.com>.
Hi Rupert, all,
After the update I still have a failing test: Topic Classification.
Tests in error:
testCrossValidation(org.apache.stanbol.enhancer.engine.topic.TopicEngineTest):
org.apache.solr.common.SolrException
Before the RuleStoreTest threw an error as well, but it disappeared with
the latest update.
Best,
Andriy
On Thu, Nov 15, 2012 at 3:36 PM, Rupert Westenthaler <
rupert.westenthaler@gmail.com> wrote:
> Hi all,
>
> Short Version:
>
> * Developers: should check their test and make sure they do not assume
> an certain execution order
> * Users: Please help us by reporting failed Unit Test. Your issue is
> most likely specific to the JVM you are using so we might not be aware
> of the problem. Thanks!
>
> - - -
>
> Long Version with more details (for Stanbol Developers):
>
> Since about half a year there where occasionally problems with failed
> unit tests caused by nondeterministic method ordering in some JVM
> versions (see STANBOL-620).
>
> Ideally Unit Tests need to be independent and therefore the execution
> order should not matter at all. However the fact was that a lot of
> Unit Tests in Stanbol are not independent but assume an certain
> execution oder (e.g. a CRUD test class that first creates, than
> retrieves, updated and deletes items in multiple test methods).
>
> Initial the assumption was that with the release of junit-4.11
> execution orders will be re-assured, but as I discovered today this
> was not the case (at least not out of the box). Because of that I
> decided to fix affected Unit Tests [1-4].
>
> However this does not mean that all such issues are solved. e.g. the
> RuleManager test [4] was only failing on
>
> java version "1.6.0_24"
> OpenJDK Runtime Environment (IcedTea6 1.11.5)
> (6b24-1.11.5-0ubuntu1~12.04.1)
> OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
>
> but was fine on the Mac
>
> java version "1.6.0_37"
> Java(TM) SE Runtime Environment (build 1.6.0_37-b06-434-11M3909)
> Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01-434, mixed mode)
>
>
> best
> Rupert
>
> [1] http://svn.apache.org/viewvc?rev=1409744&view=rev
> [2] http://svn.apache.org/viewvc?rev=1409745&view=rev
> [3] http://svn.apache.org/viewvc?rev=1409749&view=rev
> [4] http://svn.apache.org/viewvc?rev=1409755&view=rev
>
>
>
> --
> | Rupert Westenthaler rupert.westenthaler@gmail.com
> | Bodenlehenstraße 11 ++43-699-11108907
> | A-5500 Bischofshofen
>
--
*Dr Andriy Nikolov
*
*R&D Engineer*
F +49 6227 3849-565
andriy.nikolov@fluidops.com<ht...@fluidops.com>
http://www.fluidops.com
*fluid Operations AG*
Altrottstr. 31
69190 Walldorf, Germany
Geschäftsführer/Managing Directors: Vasu Chandrasekhara, Dr. Andreas
Eberhart, Dr. Stefan Kraus, Dr. Ulrich Walther
Beirat/Advisory Board: Prof. Dr. Andreas Reuter, Thomas Reinhart
Registergericht/Commercial Register: Mannheim, HRB 704027
USt-Id Nr./VAT-No.: DE258759786
This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorised copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.
Re: Unit tests that depend on the execution order
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Thu, Nov 15, 2012 at 3:36 PM, Rupert Westenthaler
<ru...@gmail.com> wrote:
> ...the fact was that a lot of
> Unit Tests in Stanbol are not independent but assume an certain
> execution oder (e.g. a CRUD test class that first creates, than
> retrieves, updated and deletes items in multiple test methods)....
I hope we all agree that this is a bad idea ;-)
-Bertrand