Surefire 2.13: minor test failure summary problem

[Laird Nelson <lj...@gmail.com>]

Hi; hope this is the right place to start investigating this.

I'm using the maven-surefire-plugin version 2.13 coupled with JUnit 4.10 by
way of Maven 3.0.4.

I recently completed a first pass at refactoring some of my stuff and lots
of tests failed.  I noticed that the top line of the Surefire summary omits
the name of the test that failed.

I'll snip sections of the stack below, followed by the summary reproduced
in full.  I turned off the trimming of stack traces by setting the user
property trimStackTrace to false, in case that matters.

Tests run: 16, Failures: 6, Errors: 1, Skipped: 0, Time elapsed: 0.102 sec
<<< FAILURE!
testLotsOfOpenParens(com.foobar.parser.TestCasePostfixTokenizer)  Time
elapsed: 0.006 sec  <<< FAILURE!
java.lang.AssertionError: Expected exception:
java.lang.IllegalStateException
at
org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:32)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)


Then, the summary in full:

Results :

Failed tests:
   Expected exception: java.lang.IllegalStateException
  TestCasePostfixTokenizer.testPrime:255 expected same:<CATENATION> was
not:<FILTER>
  TestCasePostfixTokenizer.testAnchoredCatnenationOfNonQualifiedFilters:114
null

TestCasePostfixTokenizer.testAnchoredCatnenationOfNonQualifiedFiltersWithWhitespace:217
expected same:<CATENATION> was not:<FILTER>
  TestCasePostfixTokenizer.testInitialCaret:30 null
   Expected exception: java.lang.IllegalStateException

Tests in error:
  TestCasePostfixTokenizer.testSimpleFilterAnchoredAtEndWithWhitespace:197
» IllegalState

Tests run: 16, Failures: 6, Errors: 1, Skipped: 0


Note the first line under "Failed tests:" which omits any mention of which
test failed.  A similar construct, correctly rendered, occurs towards the
bottom of the Failed tests: summary:

  TestCasePostfixTokenizer.testInitialCaret:30 null
   Expected exception: java.lang.IllegalStateException

I'm assuming that that is what should have been rendered at the top.

(Also I don't really know what is going on with the error line:

TestCasePostfixTokenizer.testSimpleFilterAnchoredAtEndWithWhitespace:197 »
IllegalState


Why did it say just "IllegalState", and not "IllegalStateException"?)

Should I file a bug?

Best,
Laird


-- 
http://about.me/lairdnelson

Re: Surefire 2.13: minor test failure summary problem

[Kristian Rosenvold <kr...@gmail.com>]

You should probably read this, which is new for 2.13:

http://maven.apache.org/surefire/maven-surefire-plugin/newerrorsummary.html

I think there may be a bug in the new error summary when it fails in the
"@Before" method; I believe I've seen that in the corner of my eye. I've
been meaning to test this out and will check it out before 2.14 is
released.

Example tests should have test classes attached to jiras ;) (Usually a few
lines of test code demonstrating the problem will do)

Kristian





2013/1/27 Laird Nelson <lj...@gmail.com>

> Hi; hope this is the right place to start investigating this.
>
> I'm using the maven-surefire-plugin version 2.13 coupled with JUnit 4.10 by
> way of Maven 3.0.4.
>
> I recently completed a first pass at refactoring some of my stuff and lots
> of tests failed.  I noticed that the top line of the Surefire summary omits
> the name of the test that failed.
>
> I'll snip sections of the stack below, followed by the summary reproduced
> in full.  I turned off the trimming of stack traces by setting the user
> property trimStackTrace to false, in case that matters.
>
> Tests run: 16, Failures: 6, Errors: 1, Skipped: 0, Time elapsed: 0.102 sec
> <<< FAILURE!
> testLotsOfOpenParens(com.foobar.parser.TestCasePostfixTokenizer)  Time
> elapsed: 0.006 sec  <<< FAILURE!
> java.lang.AssertionError: Expected exception:
> java.lang.IllegalStateException
> at
>
> org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:32)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>
>
> Then, the summary in full:
>
> Results :
>
> Failed tests:
>    Expected exception: java.lang.IllegalStateException
>   TestCasePostfixTokenizer.testPrime:255 expected same:<CATENATION> was
> not:<FILTER>
>   TestCasePostfixTokenizer.testAnchoredCatnenationOfNonQualifiedFilters:114
> null
>
>
> TestCasePostfixTokenizer.testAnchoredCatnenationOfNonQualifiedFiltersWithWhitespace:217
> expected same:<CATENATION> was not:<FILTER>
>   TestCasePostfixTokenizer.testInitialCaret:30 null
>    Expected exception: java.lang.IllegalStateException
>
> Tests in error:
>   TestCasePostfixTokenizer.testSimpleFilterAnchoredAtEndWithWhitespace:197
> » IllegalState
>
> Tests run: 16, Failures: 6, Errors: 1, Skipped: 0
>
>
> Note the first line under "Failed tests:" which omits any mention of which
> test failed.  A similar construct, correctly rendered, occurs towards the
> bottom of the Failed tests: summary:
>
>   TestCasePostfixTokenizer.testInitialCaret:30 null
>    Expected exception: java.lang.IllegalStateException
>
> I'm assuming that that is what should have been rendered at the top.
>
> (Also I don't really know what is going on with the error line:
>
> TestCasePostfixTokenizer.testSimpleFilterAnchoredAtEndWithWhitespace:197 »
> IllegalState
>
>
> Why did it say just "IllegalState", and not "IllegalStateException"?)
>
> Should I file a bug?
>
> Best,
> Laird
>
>
> --
> http://about.me/lairdnelson
>