[Bug 55697] New: JUnit tests skipped by haltonerror or haltonfailure are missing from skipped test reporting

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=55697

            Bug ID: 55697
           Summary: JUnit tests skipped by haltonerror or haltonfailure
                    are missing from skipped test reporting
           Product: Ant
           Version: nightly
          Hardware: PC
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: Optional Tasks
          Assignee: notifications@ant.apache.org
          Reporter: gbriggs@amazon.com

The ability to report skipped tests was recently added to the Ant JUnit
reporting (in March 2013 / bug 54670). When the "haltonerror" or
"haltonfailure" directives are used in the junit task, and the build does halt
before running all the tests, then the resulting skipped tests should be
reported via this mechanism as well. (Whether you consider this a bug or a
feature, it would still be good.)

Software versions used to repro:
 Apache Ant(TM) from SVN, version 1.9.3alpha retrieved and compiled on October
21 2013
 Apache Ant(TM) version 1.9.2 also has this issue
 JUnit 4.11 (with Hamcrest-core 1.3)
 Java version 1.7.0

How to reproduce:
 Create the following files and then run "ant" in the directory. Note that the
test "TestFail.test3" was skipped, but there is no XML report of this getting
skipped. The build directory should contain a report stating that a test was
skipped.

== build.xml ==
<project name="Repro" default="tests" basedir=".">

  <target name="compile">
     <mkdir dir="build"/>
     <javac srcdir="src" destdir="build">
        <classpath><path
path="../hamcrest-core-1.3.jar:../junit-4.11.jar"/></classpath>
     </javac>
     <jar jarfile="Repro.jar" basedir="build"/>
  </target>

  <target name="tests" depends="compile">
     <junit 
        haltonerror="true" haltonfailure="true"
        outputtoformatters="true" showoutput="true">
        <formatter type="xml" usefile="true"/>
        <test name="com.amazon.TestFailFail" todir="build"/>
        <test name="com.amazon.TestFail" todir="build"/>
        <classpath><path
path="../hamcrest-core-1.3.jar:../junit-4.11.jar:Repro.jar"/></classpath>
     </junit>
  </target>

  <target name="clean">
     <delete dir="build"/>
     <delete file="Repro.jar"/>
  </target>

</project>

== src/com/amazon/TestFailFail.java ==
package com.amazon;
import static org.junit.Assert.fail;
import org.junit.Test;

public class TestFailFail
{
    @Test public void test1()
    { fail("I am test1 fail."); }

    @Test public void test2()
    { fail("I am test2 fail."); }
}

== src/com/amazon/TestFail.java ==
package com.amazon;
import static org.junit.Assert.fail;
import org.junit.Test;

public class TestFail
{
    @Test public void test3()
    { fail("I am test3 fail."); }
}

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 55697] JUnit tests skipped by haltonerror or haltonfailure are missing from skipped test reporting

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=55697

--- Comment #3 from Greg Briggs <gb...@amazon.com> ---
I would propose that the existing JUnit functionality be used to enumerate the
tests. We should not be duplicating logic about test case discovery in Ant. The
JUnit 4 Runner interface includes a method "testCount()" which returns the
number of tests to be run. Ant could instantiate the Runners to see how many
test cases exist, and simply not call run() as it usually would have. This
covers the Parameterized test use case as well.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 55697] JUnit tests skipped by haltonerror or haltonfailure are missing from skipped test reporting

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=55697

Greg Briggs <gb...@amazon.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |gbriggs@amazon.com

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 55697] JUnit tests skipped by haltonerror or haltonfailure are missing from skipped test reporting

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=55697

--- Comment #1 from Michael Clarke <mi...@gmail.com> ---
I'm not sure there is a suitable fix for this.

If one of the classes that hasn't been executed was to return a suite
(therefore providing more tests to be run), be invoked by a runner that
identifies methods differently (different annotations or by alternative naming
conventions), use parameterised tests, or anything else that alters the number
of tests beyond what can be counted statically with the @Test annotation then
we're going to fail to count these tests properly.

We can count methods annotated with @Test (JUnit 4) or named test* (JUnit 3) in
un-executed classes, but this may lead to an inconsistent count when compared
to a successfully run suite. The ignored count was developed for tests that the
user had specifically ignored whilst tests were being executed, not that Ant
hadn't tried to execute, and I think trying to change it is just going to cause
further issues.

I therefore propose closing this as with a "won't fix" status.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 55697] JUnit tests skipped by haltonerror or haltonfailure are missing from skipped test reporting

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=55697

--- Comment #2 from Stefan Bodewig <bo...@apache.org> ---
The alternative is to document the count is a "best guess".

-- 
You are receiving this mail because:
You are the assignee for the bug.