You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@maven.apache.org by laredotornado-3 <la...@gmail.com> on 2014/10/08 20:44:48 UTC
Why do running modules in parallel take longer to execute than
running them in sequence?
Hi,
I’m using Maven 3.2.3, Spring 3.2.11.RELEASE, JUnit 4.11, Java 6, and the
Surefire/Failsafe 2.17 plugins. I’m trying to figure out why when running a
couple of modules in parallel, it takes much longer than when I run them
individually. I’m trying to figure out ways to speed up my build. I have
this in my parent pom …
<modules>
<module>core</module>
…
<module>module-x</module>
<module>module-y</module>
</modules>
When I run “mvn clean install -T 4”, modules “module-x” and “module-y” run
in parallel. Both have a number of integration tests, all using
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration({ "classpath:test-context.xml" })
public class MyServiceIT
{
Below are my surefire/failsafe configurations …
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.17</version>
<configuration>
<reuseForks>true</reuseForks>
<argLine>-Xmx2048m
-XX:MaxPermSize=512M </argLine>
<skipTests>${skipAllTests}</skipTests>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.17</version>
<configuration>
<reuseForks>true</reuseForks>
<argLine>-Xmx4096m
-XX:MaxPermSize=512M </argLine>
<skipTests>${skipAllTests}</skipTests>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
When I run the tests in parallel, the modules take 11 minutes and 13
minutes, respectively, to run. However, running them concurrently (without
the “-T” option), they execute in 6 minutes and 9 minutes, respectively.
I’m confused about why they take so much longer to run when they both run at
the same time and was hoping some folks could suggest some ways I can
troubleshoot this further.
--
View this message in context: http://maven.40175.n5.nabble.com/Why-do-running-modules-in-parallel-take-longer-to-execute-than-running-them-in-sequence-tp5808128.html
Sent from the Maven - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Why do running modules in parallel take longer to execute than
running them in sequence?
Posted by laredotornado-3 <la...@gmail.com>.
Kristian,
Your 'actually make the "argLine" command work' comment is killing me. What
are you talking about? Everything I have read indicates that the correct
way to set memory for surefire/failsafe plugins is to have this in the
configuration -- "<argLine>-Xmx4096m</argLine>".
Laird,
Thanks for your blog post. I'm actually trying to figure out an efficient
way to run my two modules in parallel as opposed to the tests within each
module. You touch on that with regard to "forkCount" but I haven't yet
figured out how "forkCount" relates across modules running in parallel.
--
View this message in context: http://maven.40175.n5.nabble.com/Why-do-running-modules-in-parallel-take-longer-to-execute-than-running-them-in-sequence-tp5808128p5808317.html
Sent from the Maven - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Why do running modules in parallel take longer to execute than
running them in sequence?
Posted by Laird Nelson <lj...@gmail.com>.
On Thu, Oct 9, 2014 at 10:48 AM, Kristian Rosenvold <
kristian.rosenvold@gmail.com> wrote:
> I suggest you set forkCount to 1 on surefire/failsafe, which will run
> all the tests in separate VMs and *also* actually make the "argLine"
> command work. Currently, you are not using 4GB for your tests.
>
I also wrote a blog entry on this subject some time ago because it was
baffling me for a good long while:
http://lairdnelson.wordpress.com/2013/09/25/concurrent-testing/ I hope it
helps.
Best,
Laird
--
http://about.me/lairdnelson
Re: Why do running modules in parallel take longer to execute than
running them in sequence?
Posted by Kristian Rosenvold <kr...@gmail.com>.
I suggest you set forkCount to 1 on surefire/failsafe, which will run
all the tests in separate VMs and *also* actually make the "argLine"
command work. Currently, you are not using 4GB for your tests.
Kristian
2014-10-09 16:51 GMT+02:00 laredotornado-3 <la...@gmail.com>:
>> I suspect you're simply trashing your os or the VM. Look at
> disk/memory usage indicators OS level
>
> Is there any way to get the modules I'm running in parallel to use different
> heaps or VMs so they are not stepping on each other? I allocated 4 GB of
> memory and my JUnit tests aren't doing taht much.
>
>
>
> --
> View this message in context: http://maven.40175.n5.nabble.com/Why-do-running-modules-in-parallel-take-longer-to-execute-than-running-them-in-sequence-tp5808128p5808264.html
> Sent from the Maven - Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Why do running modules in parallel take longer to execute than
running them in sequence?
Posted by laredotornado-3 <la...@gmail.com>.
> I suspect you're simply trashing your os or the VM. Look at
disk/memory usage indicators OS level
Is there any way to get the modules I'm running in parallel to use different
heaps or VMs so they are not stepping on each other? I allocated 4 GB of
memory and my JUnit tests aren't doing taht much.
--
View this message in context: http://maven.40175.n5.nabble.com/Why-do-running-modules-in-parallel-take-longer-to-execute-than-running-them-in-sequence-tp5808128p5808264.html
Sent from the Maven - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Why do running modules in parallel take longer to execute than
running them in sequence?
Posted by Kristian Rosenvold <kr...@gmail.com>.
You are right that the -T option and surefire's parallelism are
complementary. Given equal-sized modules the effect tweaking surefire
vs -T parallelism should be similar.
The fork/thread related parameters to surefire have NO relation to the
-T parameters, which means "reuseForks" is ineffective in your
settings.
I suspect you're simply trashing your os or the VM. Look at
disk/memory usage indicators OS level. In terms of resource efficiency
there's a clear difference between the different alternatives:
A) Multiple threads within same (or a single forked) VM inside
surefire. By far the most resource efficient (Probably 10x faster than
the alternatives)
B) Multiple surefire forks (Can be resource heavy too !)
C) -T n option, which can amplify other settings "n-fold".
Kristian
2014-10-08 21:11 GMT+02:00 laredotornado-3 <la...@gmail.com>:
> Hi,
>
> Karl, to answer your question, I only have one processor on my MacBook Pro
> (10.9.5), which is a 2.2 GHz Intel Core i7. I haven't set "forkCount" in
> either my surefire or failsafe plugin (configs listed in question) because I
> was under the impression that the "-T" option took care of that.
>
> Jörg, My $MAVEN_OPTS var is set to -Xmx4096m, but I thought I read somewhere
> that in the case of surefire/failsafe, MAVEN_OPTS is ignored in favor of the
> configurations listed in the plugins, which are "-Xmx2048m
> -XX:MaxPermSize=512M" for surefire and "-Xmx4096m -XX:MaxPermSize=512M" for
> failsafe.
>
> Thanks, - Dave
>
>
>
> --
> View this message in context: http://maven.40175.n5.nabble.com/Why-do-running-modules-in-parallel-take-longer-to-execute-than-running-them-in-sequence-tp5808128p5808146.html
> Sent from the Maven - Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Why do running modules in parallel take longer to execute than
running them in sequence?
Posted by laredotornado-3 <la...@gmail.com>.
Hi,
Karl, to answer your question, I only have one processor on my MacBook Pro
(10.9.5), which is a 2.2 GHz Intel Core i7. I haven't set "forkCount" in
either my surefire or failsafe plugin (configs listed in question) because I
was under the impression that the "-T" option took care of that.
Jörg, My $MAVEN_OPTS var is set to -Xmx4096m, but I thought I read somewhere
that in the case of surefire/failsafe, MAVEN_OPTS is ignored in favor of the
configurations listed in the plugins, which are "-Xmx2048m
-XX:MaxPermSize=512M" for surefire and "-Xmx4096m -XX:MaxPermSize=512M" for
failsafe.
Thanks, - Dave
--
View this message in context: http://maven.40175.n5.nabble.com/Why-do-running-modules-in-parallel-take-longer-to-execute-than-running-them-in-sequence-tp5808128p5808146.html
Sent from the Maven - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Why do running modules in parallel take longer to execute than running them in sequence?
Posted by Jörg Schaible <jo...@gmx.de>.
Hi,
laredotornado-3 wrote:
> Hi,
>
> I’m using Maven 3.2.3, Spring 3.2.11.RELEASE, JUnit 4.11, Java 6, and the
> Surefire/Failsafe 2.17 plugins. I’m trying to figure out why when running
> a couple of modules in parallel, it takes much longer than when I run them
> individually. I’m trying to figure out ways to speed up my build. I have
> this in my parent pom …
>
> <modules>
> <module>core</module>
> …
> <module>module-x</module>
> <module>module-y</module>
> </modules>
>
> When I run “mvn clean install -T 4”, modules “module-x” and “module-y” run
> in parallel. Both have a number of integration tests, all using
>
> @RunWith(SpringJUnit4ClassRunner.class)
> @ContextConfiguration({ "classpath:test-context.xml" })
> public class MyServiceIT
> {
>
> Below are my surefire/failsafe configurations …
>
> <plugin>
>
<groupId>org.apache.maven.plugins</groupId>
>
> <artifactId>maven-surefire-plugin</artifactId>
> <version>2.17</version>
> <configuration>
> <reuseForks>true</reuseForks>
> <argLine>-Xmx2048m
> -XX:MaxPermSize=512M </argLine>
>
> <skipTests>${skipAllTests}</skipTests>
> </configuration>
> </plugin>
> <plugin>
>
<groupId>org.apache.maven.plugins</groupId>
>
> <artifactId>maven-failsafe-plugin</artifactId>
> <version>2.17</version>
> <configuration>
> <reuseForks>true</reuseForks>
> <argLine>-Xmx4096m
> -XX:MaxPermSize=512M </argLine>
>
> <skipTests>${skipAllTests}</skipTests>
> </configuration>
> <executions>
> <execution>
> <goals>
>
> <goal>integration-test</goal>
>
<goal>verify</goal>
> </goals>
> </execution>
> </executions>
> </plugin>
>
> When I run the tests in parallel, the modules take 11 minutes and 13
> minutes, respectively, to run. However, running them concurrently
> (without the “-T” option), they execute in 6 minutes and 9 minutes,
> respectively. I’m confused about why they take so much longer to run when
> they both run at the same time and was hoping some folks could suggest
> some ways I can troubleshoot this further.
How much memory do you have? Your test processes take quite a lot and
executing them in parallel might simply activate disk swapping.
- Jörg
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Why do running modules in parallel take longer to execute than
running them in sequence?
Posted by Karl Heinz Marbaise <kh...@gmx.de>.
Hi,
have paid attention to forkCount ? How did you set it?
How many core's do you have ?
On 10/8/14 8:44 PM, laredotornado-3 wrote:
> Hi,
>
> I’m using Maven 3.2.3, Spring 3.2.11.RELEASE, JUnit 4.11, Java 6, and the
> Surefire/Failsafe 2.17 plugins. I’m trying to figure out why when running a
> couple of modules in parallel, it takes much longer than when I run them
> individually. I’m trying to figure out ways to speed up my build. I have
> this in my parent pom …
>
> <modules>
> <module>core</module>
> …
> <module>module-x</module>
> <module>module-y</module>
> </modules>
>
> When I run “mvn clean install -T 4”, modules “module-x” and “module-y” run
> in parallel. Both have a number of integration tests, all using
>
> @RunWith(SpringJUnit4ClassRunner.class)
> @ContextConfiguration({ "classpath:test-context.xml" })
> public class MyServiceIT
> {
>
> Below are my surefire/failsafe configurations …
>
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
>
> <artifactId>maven-surefire-plugin</artifactId>
> <version>2.17</version>
> <configuration>
> <reuseForks>true</reuseForks>
> <argLine>-Xmx2048m
> -XX:MaxPermSize=512M </argLine>
>
> <skipTests>${skipAllTests}</skipTests>
> </configuration>
> </plugin>
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
>
> <artifactId>maven-failsafe-plugin</artifactId>
> <version>2.17</version>
> <configuration>
> <reuseForks>true</reuseForks>
> <argLine>-Xmx4096m
> -XX:MaxPermSize=512M </argLine>
>
> <skipTests>${skipAllTests}</skipTests>
> </configuration>
> <executions>
> <execution>
> <goals>
>
> <goal>integration-test</goal>
> <goal>verify</goal>
> </goals>
> </execution>
> </executions>
> </plugin>
>
> When I run the tests in parallel, the modules take 11 minutes and 13
> minutes, respectively, to run. However, running them concurrently (without
> the “-T” option), they execute in 6 minutes and 9 minutes, respectively.
> I’m confused about why they take so much longer to run when they both run at
> the same time and was hoping some folks could suggest some ways I can
> troubleshoot this further.
Kind regards
Karl-Heinz Marbaise
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org