[jira] (SUREFIRE-1043) Please add possibility to specify tests groups order for sequential run and tests groups content for parallel run.

["Tibor Digana (JIRA)" <ji...@codehaus.org>]

    [ https://jira.codehaus.org/browse/SUREFIRE-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=352049#comment-352049 ] 

Tibor Digana commented on SUREFIRE-1043:
----------------------------------------

I guess the new feature SUREFIRE-1093 (surefire-2.18) would help you because it isolates @NotThreadSafe JUnit tests from parallel tests.
It works only with listed tests. This means the inclusion section may specify something like this:
<include>**/*Suite.java</include>
and you may have two main suites:
SequentialSuite class annotated with @NotThreadSafe, and ParallelSuite.
Annotation NotThreadSafe is JCIP artifact.

If you specify run-order and groups in the same <execution/> section, then the configuration applies to both suites.

> Please add possibility to specify tests groups order for sequential run and tests groups content for parallel run.
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: SUREFIRE-1043
>                 URL: https://jira.codehaus.org/browse/SUREFIRE-1043
>             Project: Maven Surefire
>          Issue Type: Improvement
>    Affects Versions: 2.16
>            Reporter: Tomasz Halama
>
> What we need is as following:
> For sequential run:
> We would like to have some means to specify run order of whole groups of tests, during one invocation of plugin. Content of the group should be determined by ant style expressions. Inside each such group, tests should be run in order, specified by 'runOrder' parameter.
> Example:
> <plugin>
>   <groupId>org.apache.maven.plugins</groupId>
>   <artifactId>maven-failsafe-plugin</artifactId>
>   <configuration>
>     <groupsOrder>
>       <group>**/*ST</group>
>       <group>com/something/SomeTest,com/something/SomeTest2</group>
>       <group>**/*IT,**/IT*</group>
>       <group>com/something/SomeTest3</group>
>     </groupsOrder>
>     <runOrder>alphabetical</runOrder>
>   </configuration>
> </plugin>
> In above example run order of tests should be as following:
> 1. all tests, which have "ST" suffix, should be run in alphabetical order
> 2. com/something/SomeTest and com/something/SomeTest2 should be run in alphabetical order
> 3. all tests, which have "IT" suffix or prefix, should be run in alphabetical order
> 4. com/something/SomeTest3 should be run
> Instead of adding some new parameter (like above 'groupsOrder' section), I think 'includes' can be used for this purpose (each entry would determine a group), with some additional boolean value, i.e. 'groupsByIncludes' (indicating, that each 'include' entry should be considered as a separate group):
> <plugin>
>   <groupId>org.apache.maven.plugins</groupId>
>   <artifactId>maven-failsafe-plugin</artifactId>
>   <configuration>
>     <includes>
>       <include>**/*ST.java</include>
>       <include>com/something/SomeTest.java,com/something/SomeTest2.java</include>
>       <include>**/*IT.java,**/IT*.java</include>
>       <include>com/something/SomeTest3.java</include>
>     </includes>
>     <runOrder>alphabetical</runOrder>
>     <groupsByIncludes>true<groupsByIncludes>
>   </configuration>
> </plugin>
> In our case we cannot simply divide one invocation of plugin into several ones (which would solve this problem), because setuping tests context takes too much time.
> This feature will also give possibility to run tests by some specific order (when each 'group'/'include' would consist of only one entry). Possibility to set specific order could be very handy, when i.e. you detected, that some of your tests are not independent and you want to reproduce the problem.
> For parallel run:
> We would like to have some means to specify tests groups for parallel run. By this I mean: tests in each group will be run in parallel, but the groups itself will be run sequentially (so only tests from the same group can be executed at the same time)
> There should be also some annotation, which can be used to marked tests, which cannot be run in parallel at all.
> If 'includes' contains such tests - they should be run sequentially at the end.
> To achieve all of this, new (i.e. groupsOrder) or 'includes' parameter can be used here, in the same way as for sequential run, except 'parallel' attribute should be set to new value: i.e. parallelGroup.
> I think, that logging information, when each test (each test method) starts and stops, would be also a good idea (we would know exact tests, which are run at the same time).



--
This message was sent by Atlassian JIRA
(v6.1.6#6162)