[jira] (SUREFIRE-985) Some Parameterized tests are not executed (or not reported)

["Nicolas Liochon (JIRA)" <ji...@codehaus.org>]

Nicolas Liochon created SUREFIRE-985:
----------------------------------------

             Summary: Some Parameterized tests are not executed (or not reported)
                 Key: SUREFIRE-985
                 URL: https://jira.codehaus.org/browse/SUREFIRE-985
             Project: Maven Surefire
          Issue Type: Bug
    Affects Versions: 2.14
         Environment: linux, 2 *4 cores, 32 Gb RAM (The sample does not work on Windows)

            Reporter: Nicolas Liochon
            Priority: Critical


The test itself does nothing. The problem occurs when you have a lot of test classes with different categories, that's why I haven't been able to extract it (I suppose generating test classes would work): the report 'forgets' some tests.

here is a branch I created to reproduce the issue:
{code}
git clone -b surefire git://github.com/nkeywal/hbase.git
mvn clean install -D skipTests
{code}
There is a lot of third parties and code. Then:
{code}
mvn test -pl hbase-server
{code}
will give back a random number of test executed (from 0 to 9).

If you specify the test name with -Dtest= the number of tests is always 9 (as expected).

Building will take time, but there is a single test to execute.

It's fragile: I tried to move the test to a difference package and the problem was not reproducable anymore. But I do reproduce it on a clean install with the branch above, and I do reproduce it all the time on our test suite, on different Linux machines.
 
test code:
{code}
package org.apache.hadoop.hbase.regionserver;


import java.util.*;

@org.junit.runner.RunWith(org.junit.runners.Parameterized.class)
@org.junit.experimental.categories.Category(org.apache.hadoop.hbase.S1Tests.class)
public class TestSurefire {

  @org.junit.runners.Parameterized.Parameters
  public static Collection<Object[]> parameters() {
    ArrayList<Object[]> configurations = new ArrayList<Object[]>();
    for (int i = 1; i < 10; i++) {
      configurations.add(new Object[] { i });
    }
    return configurations;
  }

  public TestSurefire(Object o) {
  }

  @org.junit.Test
  public void testM()  {
  }

}
{code}

settings
{code}
        <plugin>
          <artifactId>maven-surefire-plugin</artifactId>
          <version>2.14</version>
          <configuration>
            <reuseForks>true</reuseForks>
            <forkCount>2</forkCount>
	          <groups>org.apache.hadoop.hbase.S1Tests</groups>
            <redirectTestOutputToFile>true</redirectTestOutputToFile>
          </configuration>
        </plugin>
{code}

Example of output:
{noformat}
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
parallel='none', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
parallel='none', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false

Results :

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
{noformat}

or

{noformat}
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
parallel='none', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
parallel='none', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
Running org.apache.hadoop.hbase.regionserver.TestSurefire
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec

Results :

Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
{noformat}





--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira