You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@maven.apache.org by Sergey Saraev <Sa...@NetCracker.com> on 2015/12/21 13:53:32 UTC
How Maven solves the problem of long builds on large projects?
Hello!
I am developing a project with 67 modules.
I use Apache Maven 3.0.4.
Reassembly of the project take 1 hour and 50 minutes although usually commit change only one module.
The project is very large. It contains 5948 java classes (Basically, time spent on their compilation.).
Build command: mvn clean install pmd:pmd checkstyle:checkstyle cobertura:cobertura
Plugins versions:
maven-compiler-plugin:2.3.2
maven-antrun-plugin:1.6 (use wlappc task: http://docs.oracle.com/cd/E21764_01/web.1111/e13706/splitbuild.htm#WLPRG224)
maven-surefire-plugin:2.10
maven-jar-plugin:2.3.2
maven-install-plugin:2.3.1
maven-pmd-plugin:2.7.1
maven-checkstyle-plugin:2.6
cobertura-maven-plugin:2.7
How to speed up the assembly?
(Maybe skip modules, which sources have not changed or something else)
Regards,
Sergey Saraev | Research & Development | Office: +7 (846) 270-7800 ext. 2662 | Mobile: +7 (917) 813-5604 | --www.NetCracker.com--
Proven Partner to Communications Service Providers
________________________________
The information transmitted herein is intended only for the person or entity to which it is addressed and may contain confidential, proprietary and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.
Re: How Maven solves the problem of long builds on large projects?
Posted by Karl Heinz Marbaise <kh...@gmx.de>.
Hi,
On 12/21/15 1:53 PM, Sergey Saraev wrote:
> Hello!
>
> I am developing a project with 67 modules.
> I use Apache Maven 3.0.4.
>
> Reassembly of the project take 1 hour and 50 minutes although usually commit change only one module.
You should use only: mvn clean install ...
any other things like pmd, checkstyle etc. don't make sense in a usual
build..
Furthermore have you taken a look how long the build time of the
different modules is? What is the module with the longest time? How many
tests do you run? How long do the tests take?
I can give an impression of a large build (420 module) about 6500 tests,
can be built in ca. 35 minutes ca. 580,000 lines of code...running (mvn
clean deploy)...
BTW: Why are you using such old plugin versions?
(http://maven.apache.org/plugins/)...
And of course on what kind of machine do you do the build? dedicated
build machine?
Kind regards
Karl Heinz Marbaise
>
> The project is very large. It contains 5948 java classes (Basically, time spent on their compilation.).
> Build command: mvn clean install pmd:pmd checkstyle:checkstyle cobertura:cobertura
>
> Plugins versions:
> maven-compiler-plugin:2.3.2
> maven-antrun-plugin:1.6 (use wlappc task: http://docs.oracle.com/cd/E21764_01/web.1111/e13706/splitbuild.htm#WLPRG224)
> maven-surefire-plugin:2.10
> maven-jar-plugin:2.3.2
> maven-install-plugin:2.3.1
> maven-pmd-plugin:2.7.1
> maven-checkstyle-plugin:2.6
> cobertura-maven-plugin:2.7
>
> How to speed up the assembly?
> (Maybe skip modules, which sources have not changed or something else)
>
> Regards,
>
> Sergey Saraev | Research & Development | Office: +7 (846) 270-7800 ext. 2662 | Mobile: +7 (917) 813-5604 | --www.NetCracker.com--
> Proven Partner to Communications Service Providers
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: How Maven solves the problem of long builds on large projects?
Posted by Ron Wheeler <rw...@artifact-software.com>.
Ideas:
- Break your project up into modules that build separately. If you have
a well structured project that is loosely coupled, individual
issues/bugs will only affect a few modules.
We have a project with about 80 modules but never build them all at once.
- Build separately deployable modules so that you are building small
jars and wars that can be individually replaced in your run-time or test
environments.
- Build modules with 3rd party libraries so that you only have 1 copy of
each library in your project rather than a copy in each module. The JVM
can only use 1 copy of Apache commons-xxx; no point deploying 50 copies.
- Make sure that you have a repo "near" your workstation so that
downloads of maven components and third party libraries are fast. We use
the community version of Nexus.
- Use a RAMDrive to hold Maven and your compiler and your build
temporary files. I have dedicated 2 Gb of RAM on my 6 Gb laptop. makes a
big difference.
I also have Eclipse/STS on that drive which makes a big difference in
working with the IDE. If I had more memory,
I would put my Maven cache in RAM but it currently has 14,799 files
and takes 1.81GB. I probably should delete it and let Maven rebuild it
to get rid of old crap(9 versions of junit).
- Use the latest Maven (including plug-ins) and Java.
- Only clean when you need to. Maven does a pretty good job of figuring
out what needs to be re-compiled.
The first 3 ideas will give the biggest improvement.
If you have the RAM, the RAMDrive is easiest to do since it does not
require much thinking.
Ron
On 21/12/2015 8:17 AM, Stephen Connolly wrote:
> I doubt it is the compilation that sucks up the time. Try running with test
> execution skipped to see the time difference, eg `mvn clean install` vs
> `mvn clean install -DskipTests`
>
> If you are using `install` then likely there is no need to rebuild
> everything each time and you can use the -pl -am and -and cli options to
> tune how much of the reactor to rebuild
>
> On Monday 21 December 2015, Sergey Saraev <Sa...@netcracker.com> wrote:
>
>> Hello!
>>
>> I am developing a project with 67 modules.
>> I use Apache Maven 3.0.4.
>>
>> Reassembly of the project take 1 hour and 50 minutes although usually
>> commit change only one module.
>>
>> The project is very large. It contains 5948 java classes (Basically, time
>> spent on their compilation.).
>> Build command: mvn clean install pmd:pmd checkstyle:checkstyle
>> cobertura:cobertura
>>
>> Plugins versions:
>> maven-compiler-plugin:2.3.2
>> maven-antrun-plugin:1.6 (use wlappc task:
>> http://docs.oracle.com/cd/E21764_01/web.1111/e13706/splitbuild.htm#WLPRG224
>> )
>> maven-surefire-plugin:2.10
>> maven-jar-plugin:2.3.2
>> maven-install-plugin:2.3.1
>> maven-pmd-plugin:2.7.1
>> maven-checkstyle-plugin:2.6
>> cobertura-maven-plugin:2.7
>>
>> How to speed up the assembly?
>> (Maybe skip modules, which sources have not changed or something else)
>>
>> Regards,
>>
>> Sergey Saraev | Research & Development | Office: +7 (846) 270-7800 ext.
>> 2662 | Mobile: +7 (917) 813-5604 | --www.NetCracker.com--
>> Proven Partner to Communications Service Providers
>>
>>
>>
>>
>> ________________________________
>> The information transmitted herein is intended only for the person or
>> entity to which it is addressed and may contain confidential, proprietary
>> and/or privileged material. Any review, retransmission, dissemination or
>> other use of, or taking of any action in reliance upon, this information by
>> persons or entities other than the intended recipient is prohibited. If you
>> received this in error, please contact the sender and delete the material
>> from any computer.
>>
>
--
Ron Wheeler
President
Artifact Software Inc
email: rwheeler@artifact-software.com
skype: ronaldmwheeler
phone: 866-970-2435, ext 102
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: How Maven solves the problem of long builds on large projects?
Posted by Stephen Connolly <st...@gmail.com>.
I doubt it is the compilation that sucks up the time. Try running with test
execution skipped to see the time difference, eg `mvn clean install` vs
`mvn clean install -DskipTests`
If you are using `install` then likely there is no need to rebuild
everything each time and you can use the -pl -am and -and cli options to
tune how much of the reactor to rebuild
On Monday 21 December 2015, Sergey Saraev <Sa...@netcracker.com> wrote:
> Hello!
>
> I am developing a project with 67 modules.
> I use Apache Maven 3.0.4.
>
> Reassembly of the project take 1 hour and 50 minutes although usually
> commit change only one module.
>
> The project is very large. It contains 5948 java classes (Basically, time
> spent on their compilation.).
> Build command: mvn clean install pmd:pmd checkstyle:checkstyle
> cobertura:cobertura
>
> Plugins versions:
> maven-compiler-plugin:2.3.2
> maven-antrun-plugin:1.6 (use wlappc task:
> http://docs.oracle.com/cd/E21764_01/web.1111/e13706/splitbuild.htm#WLPRG224
> )
> maven-surefire-plugin:2.10
> maven-jar-plugin:2.3.2
> maven-install-plugin:2.3.1
> maven-pmd-plugin:2.7.1
> maven-checkstyle-plugin:2.6
> cobertura-maven-plugin:2.7
>
> How to speed up the assembly?
> (Maybe skip modules, which sources have not changed or something else)
>
> Regards,
>
> Sergey Saraev | Research & Development | Office: +7 (846) 270-7800 ext.
> 2662 | Mobile: +7 (917) 813-5604 | --www.NetCracker.com--
> Proven Partner to Communications Service Providers
>
>
>
>
> ________________________________
> The information transmitted herein is intended only for the person or
> entity to which it is addressed and may contain confidential, proprietary
> and/or privileged material. Any review, retransmission, dissemination or
> other use of, or taking of any action in reliance upon, this information by
> persons or entities other than the intended recipient is prohibited. If you
> received this in error, please contact the sender and delete the material
> from any computer.
>
--
Sent from my phone
Re: How Maven solves the problem of long builds on large projects?
Posted by Graham Leggett <mi...@sharp.fm>.
On 21 Dec 2015, at 12:53, Sergey Saraev <Sa...@NetCracker.com> wrote:
> I am developing a project with 67 modules.
> I use Apache Maven 3.0.4.
>
> Reassembly of the project take 1 hour and 50 minutes although usually commit change only one module.
>
> The project is very large. It contains 5948 java classes (Basically, time spent on their compilation.).
> Build command: mvn clean install pmd:pmd checkstyle:checkstyle cobertura:cobertura
Get rid of "clean" - that means completely delete all existing code and recompile from scratch, which you don't want.
Get rid of pmd, checkstyle and cobertura until you need it.
Regards,
Graham
--
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: How Maven solves the problem of long builds on large projects?
Posted by Bernd Eckenfels <ec...@zusammenkunft.net>.
Am Mon, 21 Dec 2015 21:23:56 +0100
schrieb Mirko Friedenhagen <mf...@gmail.com>:
> Hello Sergey,
>
> you may try to use
> https://github.com/timgifford/maven-buildtime-extension to identify
> which plugins contribute most to your long build times.
I typically enable the timestamps in
maven-home/conf/logging/simplelogger.properties
org.slf4j.simpleLogger.showDateTime=true
this can be done once for the maven home and all output has ms
precision stamps on the console.
Unfortunatelly this is not so easy to
turn on on automatically installed mavens in Jenkins (but jenkins has
its own extension to timestamp build output).
(however I think the OP is more interested in incremetal build
(plugins) or even Takari.io
Gruss
Bernd
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: How Maven solves the problem of long builds on large projects?
Posted by Mirko Friedenhagen <mf...@gmail.com>.
Hello Sergey,
you may try to use
https://github.com/timgifford/maven-buildtime-extension to identify
which plugins contribute most to your long build times.
Regards
Mirko
Regards Mirko
--
http://illegalstateexception.blogspot.com/
https://github.com/mfriedenhagen/ (http://osrc.dfm.io/mfriedenhagen)
https://bitbucket.org/mfriedenhagen/
On Mon, Dec 21, 2015 at 1:53 PM, Sergey Saraev <Sa...@netcracker.com> wrote:
> Hello!
>
> I am developing a project with 67 modules.
> I use Apache Maven 3.0.4.
>
> Reassembly of the project take 1 hour and 50 minutes although usually commit change only one module.
>
> The project is very large. It contains 5948 java classes (Basically, time spent on their compilation.).
> Build command: mvn clean install pmd:pmd checkstyle:checkstyle cobertura:cobertura
>
> Plugins versions:
> maven-compiler-plugin:2.3.2
> maven-antrun-plugin:1.6 (use wlappc task: http://docs.oracle.com/cd/E21764_01/web.1111/e13706/splitbuild.htm#WLPRG224)
> maven-surefire-plugin:2.10
> maven-jar-plugin:2.3.2
> maven-install-plugin:2.3.1
> maven-pmd-plugin:2.7.1
> maven-checkstyle-plugin:2.6
> cobertura-maven-plugin:2.7
>
> How to speed up the assembly?
> (Maybe skip modules, which sources have not changed or something else)
>
> Regards,
>
> Sergey Saraev | Research & Development | Office: +7 (846) 270-7800 ext. 2662 | Mobile: +7 (917) 813-5604 | --www.NetCracker.com--
> Proven Partner to Communications Service Providers
>
>
>
>
> ________________________________
> The information transmitted herein is intended only for the person or entity to which it is addressed and may contain confidential, proprietary and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: How Maven solves the problem of long builds on large projects?
Posted by "Oliver B. Fischer" <o....@swe-blog.net>.
Hi,
we had a similar problem with our build. First step was to optimize our
tests. By tracing the execution time of our tests we were able to reduce
the time spent on integration tests from 1h to 15m.
We use Team City as CI server. Therefore we were able to parallelize our
build. How we run in parallel static code analysis, unit test, component
test and similar stuff.
Bye,
Oliver
Am 21.12.15 um 13:53 schrieb Sergey Saraev:
> Hello!
>
> I am developing a project with 67 modules.
> I use Apache Maven 3.0.4.
>
> Reassembly of the project take 1 hour and 50 minutes although usually commit change only one module.
>
> The project is very large. It contains 5948 java classes (Basically, time spent on their compilation.).
> Build command: mvn clean install pmd:pmd checkstyle:checkstyle cobertura:cobertura
>
> Plugins versions:
> maven-compiler-plugin:2.3.2
> maven-antrun-plugin:1.6 (use wlappc task: http://docs.oracle.com/cd/E21764_01/web.1111/e13706/splitbuild.htm#WLPRG224)
> maven-surefire-plugin:2.10
> maven-jar-plugin:2.3.2
> maven-install-plugin:2.3.1
> maven-pmd-plugin:2.7.1
> maven-checkstyle-plugin:2.6
> cobertura-maven-plugin:2.7
>
> How to speed up the assembly?
> (Maybe skip modules, which sources have not changed or something else)
>
> Regards,
>
> Sergey Saraev | Research & Development | Office: +7 (846) 270-7800 ext. 2662 | Mobile: +7 (917) 813-5604 | --www.NetCracker.com--
> Proven Partner to Communications Service Providers
>
>
>
>
> ________________________________
> The information transmitted herein is intended only for the person or entity to which it is addressed and may contain confidential, proprietary and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.
>
--
N Oliver B. Fischer
A Schönhauser Allee 64, 10437 Berlin, Deutschland/Germany
P +49 30 44793251
M +49 178 7903538
E o.b.fischer@swe-blog.net
S oliver.b.fischer
J oliver.b.fischer@jabber.org
X http://xing.to/obf
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org