junit 4.11 / JDK 1.7 and test methods execution order.

[Nicolas Liochon <nk...@gmail.com>]

Hi all,

In JDK 1.7, the order of the methods returned by reflection can vary. While
the ordering was not guaranteed previously, it happens that it was always
the same (declaration order). It's not the case anymore in JDK 1.7

Consequence for the tests: the order in which the tests are executed within
a test class may vary.

Because of this, JUnit 4.11 decided to execute the tests in a different
order than the declaration order.
That's explained here:
http://randomallsorts.blogspot.fr/2012/12/junit-411-whats-new-test-execution-order.html

So depending on the execution order used to be bad, but can now randomly
breaks the tests, especially with jdk 1.7 and JUnit < 4.11.
With JUnit 4.11, the order is determined, but different from what it used
to be. So it breaks some tests. I'm currently fixing this (low priority).
But it's worth knowing, especially if you port the tests from one HBase
version to another.

Cheers,

Nicolas

Re: junit 4.11 / JDK 1.7 and test methods execution order.

[Andrew Purtell <ap...@apache.org>]

This makes sense.

On Mon, Jan 14, 2013 at 8:33 AM, Nicolas Liochon <nk...@gmail.com> wrote:

> JUnit team seems to think a totally random order is not a perfect solution,
> because it makes tests non repeatable.
>


-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: junit 4.11 / JDK 1.7 and test methods execution order.

[Nicolas Liochon <nk...@gmail.com>]

>From the code I've seen so far, I don't think having a dependency on the
order execution was a voluntary decision, more an accident.
JUnit team seems to think a totally random order is not a perfect solution,
because it makes tests non repeatable. So the new 4.11 is not random, just
different than before, and similar on 1.7 and 1.6.

We can keep the same order for the 1.6 with the new 4.11 option
'MethodSorters.JVM'. I've not tested it. However, it won't work on 1.7.
That's why I've chosen to fix the root cause...

Nicolas





On Mon, Jan 14, 2013 at 5:21 PM, Jonathan Hsieh <jo...@cloudera.com> wrote:

> Interesting.  It's probably good to randomize test ordering.  Are the
> classes that failed supposed to be executed in order? (non independent
> test cases?)
>
> I found this when I was looking if there was an option to keep the older
> style.
>

Re: junit 4.11 / JDK 1.7 and test methods execution order.

[Jonathan Hsieh <jo...@cloudera.com>]

Interesting.  It's probably good to randomize test ordering.  Are the
classes that failed supposed to be executed in order? (non independent
test cases?)

I found this when I was looking if there was an option to keep the older style.
http://stackoverflow.com/questions/3089151/specifying-an-order-to-junit-4-tests-at-the-method-level-not-class-level

Jon.

On Mon, Jan 14, 2013 at 5:57 AM, Nicolas Liochon <nk...@gmail.com> wrote:
> Hi all,
>
> In JDK 1.7, the order of the methods returned by reflection can vary. While
> the ordering was not guaranteed previously, it happens that it was always
> the same (declaration order). It's not the case anymore in JDK 1.7
>
> Consequence for the tests: the order in which the tests are executed within
> a test class may vary.
>
> Because of this, JUnit 4.11 decided to execute the tests in a different
> order than the declaration order.
> That's explained here:
> http://randomallsorts.blogspot.fr/2012/12/junit-411-whats-new-test-execution-order.html
>
> So depending on the execution order used to be bad, but can now randomly
> breaks the tests, especially with jdk 1.7 and JUnit < 4.11.
> With JUnit 4.11, the order is determined, but different from what it used
> to be. So it breaks some tests. I'm currently fixing this (low priority).
> But it's worth knowing, especially if you port the tests from one HBase
> version to another.
>
> Cheers,
>
> Nicolas



-- 
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com