You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Kristian Rosenvold <kr...@gmail.com> on 2013/09/19 18:36:49 UTC
Parallel build scheduling explanation
Jason sent me some questions about the parallel build that included a
real nice figure. Since I'm a sucker for nice figures (especially
those that others make!), I quickly updated the wiki page with this,
and added some docs to describe his figure.
https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
Kristian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Parallel build scheduling explanation
Posted by Hervé BOUTEMY <he...@free.fr>.
ok, thanks you for the update, I'm starting to see
"Maven calculates this graph based on declared inter-module dependencies for a
multi-module build"
IIUC, dependency graph contains both explicit dependencies (in pom
dependencies section) and implicit dependencies got from parent
this would explain why level 0 has only 1 module
or do I miss something?
Regards,
Hervé
Le jeudi 19 septembre 2013 19:36:00 Kristian Rosenvold a écrit :
> Good question ;)
>
> I did a minor update to the wiki, adding something like:
>
>
> Each node in the graph represents a module in a multi-module build,
> the "levels" simply indicate the distance to the first module in the
> internal reactor dependency graph.
>
> Kristian
>
> 2013/9/19 Hervé BOUTEMY <he...@free.fr>:
> > I suppose each circle is a module
> > but I don't understand what the levels are, and how modules are dispatched
> > into levels
> >
> > Regards,
> >
> > Hervé
> >
> > Le jeudi 19 septembre 2013 18:36:49 Kristian Rosenvold a écrit :
> >> Jason sent me some questions about the parallel build that included a
> >> real nice figure. Since I'm a sucker for nice figures (especially
> >> those that others make!), I quickly updated the wiki page with this,
> >> and added some docs to describe his figure.
> >>
> >> https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Mave
> >> n+3
> >>
> >>
> >> Kristian
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >> For additional commands, e-mail: dev-help@maven.apache.org
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > For additional commands, e-mail: dev-help@maven.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Parallel build scheduling explanation
Posted by Kristian Rosenvold <kr...@gmail.com>.
Good question ;)
I did a minor update to the wiki, adding something like:
Each node in the graph represents a module in a multi-module build,
the "levels" simply indicate the distance to the first module in the
internal reactor dependency graph.
Kristian
2013/9/19 Hervé BOUTEMY <he...@free.fr>:
> I suppose each circle is a module
> but I don't understand what the levels are, and how modules are dispatched
> into levels
>
> Regards,
>
> Hervé
>
> Le jeudi 19 septembre 2013 18:36:49 Kristian Rosenvold a écrit :
>> Jason sent me some questions about the parallel build that included a
>> real nice figure. Since I'm a sucker for nice figures (especially
>> those that others make!), I quickly updated the wiki page with this,
>> and added some docs to describe his figure.
>>
>> https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
>>
>>
>> Kristian
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Parallel build scheduling explanation
Posted by Kristian Rosenvold <kr...@gmail.com>.
Herve; I added the dependency to the parent pom to the docs.
Jason: In the current project I'm working with, I get the following timings:
mvn -o -T 4 clean install : 53 seconds
mvn -o clean install : 50 seconds
mvn -DskipTests -o -T 4 clean install : 14.1 seconds
mvn -DskipTests -o clean install : 16.624 seconds
This is quite a small project with a very limited fan-out hence
limited effect. I exclusively run in offline mode when benchmarking
(which I know is "cheating").
I recommend running with "sufficient" threads, use -T 10. Also, the
plugin loading is heavily synchronized, which can have a very
detrimental effect on for instance level 4 in your graph, if all the 5
nodes on that level are first-time users of the same plugin and this
plugin is slow in construction. (This is a lot more common than one
might think, since those 5 modules end up at level 4 for the same
reason).
This is actually why the concurrent preemptive plugin loading that I
started prototyping has "significant" potential. It will easily make
*every* maven build a second or two faster, mitigate the effects of
artifact resolution and avoid global synchronization points on plugin
loading. Stuart kindly found a few problems for me that I have yet to
fix, which I might consider doing over a glass of something from
northern Italy tonight - I'm anxious to find out to what extent it
breaks things ;)
Kristian
2013/9/19 Jason van Zyl <ja...@tesla.io>:
> Each circle is a project in a multi-project build (I think we need to clean up our terminology, I stopped saying multi-module built so I don't have to explain the difference between a module and project).
>
> Kristian is correct it's the depth from the root project. Everything on level 1 can be built simultaneously without any side-effects.
>
> On Sep 19, 2013, at 10:27 AM, Hervé BOUTEMY <he...@free.fr> wrote:
>
>> I suppose each circle is a module
>> but I don't understand what the levels are, and how modules are dispatched
>> into levels
>>
>> Regards,
>>
>> Hervé
>>
>> Le jeudi 19 septembre 2013 18:36:49 Kristian Rosenvold a écrit :
>>> Jason sent me some questions about the parallel build that included a
>>> real nice figure. Since I'm a sucker for nice figures (especially
>>> those that others make!), I quickly updated the wiki page with this,
>>> and added some docs to describe his figure.
>>>
>>> https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
>>>
>>>
>>> Kristian
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>
> Thanks,
>
> Jason
>
> ----------------------------------------------------------
> Jason van Zyl
> Founder, Apache Maven
> http://twitter.com/jvanzyl
> ---------------------------------------------------------
>
>
>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Parallel build scheduling explanation
Posted by Jason van Zyl <ja...@tesla.io>.
Each circle is a project in a multi-project build (I think we need to clean up our terminology, I stopped saying multi-module built so I don't have to explain the difference between a module and project).
Kristian is correct it's the depth from the root project. Everything on level 1 can be built simultaneously without any side-effects.
On Sep 19, 2013, at 10:27 AM, Hervé BOUTEMY <he...@free.fr> wrote:
> I suppose each circle is a module
> but I don't understand what the levels are, and how modules are dispatched
> into levels
>
> Regards,
>
> Hervé
>
> Le jeudi 19 septembre 2013 18:36:49 Kristian Rosenvold a écrit :
>> Jason sent me some questions about the parallel build that included a
>> real nice figure. Since I'm a sucker for nice figures (especially
>> those that others make!), I quickly updated the wiki page with this,
>> and added some docs to describe his figure.
>>
>> https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
>>
>>
>> Kristian
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder, Apache Maven
http://twitter.com/jvanzyl
---------------------------------------------------------
Re: Parallel build scheduling explanation
Posted by Hervé BOUTEMY <he...@free.fr>.
I suppose each circle is a module
but I don't understand what the levels are, and how modules are dispatched
into levels
Regards,
Hervé
Le jeudi 19 septembre 2013 18:36:49 Kristian Rosenvold a écrit :
> Jason sent me some questions about the parallel build that included a
> real nice figure. Since I'm a sucker for nice figures (especially
> those that others make!), I quickly updated the wiki page with this,
> and added some docs to describe his figure.
>
> https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
>
>
> Kristian
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org