You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bookkeeper.apache.org by si...@apache.org on 2020/04/21 20:27:36 UTC
[bookkeeper] branch master updated: Add forkCount option to
parallelize build and tests
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git
The following commit(s) were added to refs/heads/master by this push:
new e20a24a Add forkCount option to parallelize build and tests
e20a24a is described below
commit e20a24a6aca44968fe6a597be31e08a552d38dec
Author: Anup Ghatage <gh...@gmail.com>
AuthorDate: Tue Apr 21 13:27:30 2020 -0700
Add forkCount option to parallelize build and tests
### Motivation
Current code forks a new JVM per module. (bookkeeper-server, bookkeeper-proto etc)
This means one fork per module for build and testing, no parallelism within the module where majority of the time goes.
We need parallelism within a module during the test execution so we can have the builds complete faster and have the artifacts shipped out quicker.
### Changes
We use the maven surefire plugin but don't define the `forkCount` and hence set it to default of 1.
This means it executes each module with one thread.
This change sets `forkCount` to 5, enabling parallelism in testing and drastically reducing total turnaround time. (by about 2/3rds!)
*Total build+test time without this change*
```
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:06 h
[INFO] Finished at: 2020-03-05T02:01:29-08:00
[INFO] ------------------------------------------------------------------------
```
*Total build+test time with this change*
```
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18:23 min
[INFO] Finished at: 2020-03-05T02:38:22-08:00
[INFO] ------------------------------------------------------------------------
```
### Things to watch
Added parallelism may cause some flappers but with much trial and error I have come to the number `5`. The flappers are usually only from conflict in obtaining the same port number.
If needed, we can increase the retryCount, but as of now I consistently don't see any flappers at a `forkCount` of 5
Reviewers: Enrico Olivelli <eo...@gmail.com>, Sijie Guo <None>
This closes #2278 from Ghatage/parallelizeBuildAndTests
---
pom.xml | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/pom.xml b/pom.xml
index 278370d..45b4c3d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -196,6 +196,7 @@
<protobuf-maven-plugin.version>0.5.0</protobuf-maven-plugin.version>
<puppycrawl.checkstyle.version>6.19</puppycrawl.checkstyle.version>
<spotbugs-maven-plugin.version>3.1.8</spotbugs-maven-plugin.version>
+ <forkCount.variable>1</forkCount.variable>
</properties>
<!-- dependency definitions -->
@@ -849,6 +850,7 @@
<configuration>
<argLine>-Xmx2G -Djava.net.preferIPv4Stack=true -Dio.netty.leakDetection.level=paranoid</argLine>
<redirectTestOutputToFile>${redirectTestOutputToFile}</redirectTestOutputToFile>
+ <forkCount>${forkCount.variable}</forkCount>
<reuseForks>false</reuseForks>
<forkedProcessTimeoutInSeconds>1800</forkedProcessTimeoutInSeconds>
<rerunFailingTestsCount>${testRetryCount}</rerunFailingTestsCount>
@@ -1010,6 +1012,7 @@
<!-- @{argLine} is a variable injected by JaCoCo-->
<argLine>@{argLine} -Xmx2G -Djava.net.preferIPv4Stack=true</argLine>
<redirectTestOutputToFile>${redirectTestOutputToFile}</redirectTestOutputToFile>
+ <forkCount>${forkCount.variable}</forkCount>
<reuseForks>false</reuseForks>
<forkedProcessTimeoutInSeconds>1800</forkedProcessTimeoutInSeconds>
<!-- we want build to complete even in case of failures -->
@@ -1044,6 +1047,7 @@
<configuration>
<argLine>-Xmx2G -Djava.net.preferIPv4Stack=true</argLine>
<redirectTestOutputToFile>false</redirectTestOutputToFile>
+ <forkCount>${forkCount.variable}</forkCount>
<reuseForks>false</reuseForks>
<forkedProcessTimeoutInSeconds>1800</forkedProcessTimeoutInSeconds>
<trimStackTrace>false</trimStackTrace>
@@ -1064,6 +1068,7 @@
<configuration>
<argLine>-Xmx2G -Djava.net.preferIPv4Stack=true -Dbookkeeper.root.logger=DEBUG,CONSOLE</argLine>
<redirectTestOutputToFile>false</redirectTestOutputToFile>
+ <forkCount>${forkCount.variable}</forkCount>
<reuseForks>false</reuseForks>
<forkedProcessTimeoutInSeconds>1800</forkedProcessTimeoutInSeconds>
<trimStackTrace>false</trimStackTrace>