You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Kristian Rosenvold <kr...@gmail.com> on 2010/06/11 18:24:13 UTC
Parallel vs weave
I am picking up some positive tweets here and there, but I'd be really
happy if those of you who test parallel building could report back with
some numbers in response to this message. I am particularly interested
in the difference between parallel and weave (and linear too), mostly to
assess that weave is worth the effort.
So after tweaking around with threads a bit to find out what runs
fastest I'd really enjoy reports like this one (real data):
Linear: 50 seconds
Parallel: 40 seconds
Weave: 29 seconds
For a 10 module project using -T 1C on a 6 core box.
For those wondering:
* Both parallel and weave favorize projects with lots of modules
and "wide" dependency graphs.
* Weave is /probably/ quite a lot faster than parallel for projects
with lots of tests fairly evenly distributed among the modules.
* Putting all your tests in one module is a bad idea in this context.
* Some mojos (like war/ear) can "dwarf" all the others,
effectively limiting the overall potential somewhat.
Kristian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Parallel vs weave
Posted by Daniel Kulp <dk...@apache.org>.
On Friday 11 June 2010 1:40:07 pm Kristian Rosenvold wrote:
> Interesting info, and good numbers! Weave is a double edged sword;
> while it may be promising even
> faster builds, by design it has this irritating tendency to run mojos
> (of the same execution in different modules)
> *exactly* in parallel, more or less simulating how I'd write a unit
> test to test thread safety. So weave
> has proved to find almost *all* of the concurrency bugs I've located
> (where parallel doesn't have any problems).
> There are also a couple of minor details regarding the actual
> execution planning in weave mode that are still
> somewhat open - it's a much more complex execution model than
> parallel; but I'm concentrating on
> parallel and the ecosystem in general ATM and just making sure weave
> works for every project I test.
>
> Antrun itself is threadsafe, but whatever is being executed there is
> another matter. If this is committed to cxf
> trunk I can/will look at it once I get through the known issues.
Yep. To reproduce
1) checkout CXF trunk from:
http://svn.apache.org/repos/asf/cxf/trunk/
2) Build everything (non-parallel right now)
mvn -Pnochecks -Dmaven.test.skip.exec=true install
That gets everything set and all the code generation done and such. (I'm
pretty sure not all the code gen plugins are threadsafe yet, but they all "do
nothing" if they don't have to do anything)
Then just run:
mvn -Pnochecks -T 12W
or similar. For me, 100% of the time it barfs in antrun in the testutils
module.
You need the -Pnochecks right now when using -T until we get a release of
checkstyle that is threadsafe. Alternatively, update the parent/pom.xml to
use the snapshots of checkstyle and pmd.
Dan
>
> Kristian
>
> On Fri, Jun 11, 2010 at 6:57 PM, Daniel Kulp <dk...@apache.org> wrote:
> > Well, weave mode doesn't work for CXF. Ends up with a
> > NullPointerException down in antrun-plugin someplace. Thus, I cannot
> > help you there.
> >
> > For CXF on my i7 820QM (4 cores, 8 threads) turning off checkstyle/pmd
> > and having everything already code generated/compiled (so mostly just
> > running tests)
> >
> > Linear: 26 minutes
> > Parallel: 9 minutes (-T 12 is what I'm using)
> >
> > Dan
> >
> > On Friday 11 June 2010 12:24:13 pm Kristian Rosenvold wrote:
> >> I am picking up some positive tweets here and there, but I'd be really
> >> happy if those of you who test parallel building could report back with
> >> some numbers in response to this message. I am particularly interested
> >> in the difference between parallel and weave (and linear too), mostly to
> >> assess that weave is worth the effort.
> >>
> >> So after tweaking around with threads a bit to find out what runs
> >> fastest I'd really enjoy reports like this one (real data):
> >>
> >> Linear: 50 seconds
> >> Parallel: 40 seconds
> >> Weave: 29 seconds
> >> For a 10 module project using -T 1C on a 6 core box.
> >>
> >>
> >> For those wondering:
> >> * Both parallel and weave favorize projects with lots of modules
> >> and "wide" dependency graphs.
> >> * Weave is /probably/ quite a lot faster than parallel for projects
> >> with lots of tests fairly evenly distributed among the modules.
> >> * Putting all your tests in one module is a bad idea in this context.
> >> * Some mojos (like war/ear) can "dwarf" all the others,
> >> effectively limiting the overall potential somewhat.
> >>
> >> Kristian
> >>
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >> For additional commands, e-mail: dev-help@maven.apache.org
> >
> > --
> > Daniel Kulp
> > dkulp@apache.org
> > http://dankulp.com/blog
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
--
Daniel Kulp
dkulp@apache.org
http://dankulp.com/blog
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Parallel vs weave
Posted by Kristian Rosenvold <kr...@gmail.com>.
Interesting info, and good numbers! Weave is a double edged sword;
while it may be promising even
faster builds, by design it has this irritating tendency to run mojos
(of the same execution in different modules)
*exactly* in parallel, more or less simulating how I'd write a unit
test to test thread safety. So weave
has proved to find almost *all* of the concurrency bugs I've located
(where parallel doesn't have any problems).
There are also a couple of minor details regarding the actual
execution planning in weave mode that are still
somewhat open - it's a much more complex execution model than
parallel; but I'm concentrating on
parallel and the ecosystem in general ATM and just making sure weave
works for every project I test.
Antrun itself is threadsafe, but whatever is being executed there is
another matter. If this is committed to cxf
trunk I can/will look at it once I get through the known issues.
Kristian
On Fri, Jun 11, 2010 at 6:57 PM, Daniel Kulp <dk...@apache.org> wrote:
>
> Well, weave mode doesn't work for CXF. Ends up with a NullPointerException
> down in antrun-plugin someplace. Thus, I cannot help you there.
>
> For CXF on my i7 820QM (4 cores, 8 threads) turning off checkstyle/pmd and
> having everything already code generated/compiled (so mostly just running
> tests)
>
> Linear: 26 minutes
> Parallel: 9 minutes (-T 12 is what I'm using)
>
> Dan
>
>
> On Friday 11 June 2010 12:24:13 pm Kristian Rosenvold wrote:
>> I am picking up some positive tweets here and there, but I'd be really
>> happy if those of you who test parallel building could report back with
>> some numbers in response to this message. I am particularly interested
>> in the difference between parallel and weave (and linear too), mostly to
>> assess that weave is worth the effort.
>>
>> So after tweaking around with threads a bit to find out what runs
>> fastest I'd really enjoy reports like this one (real data):
>>
>> Linear: 50 seconds
>> Parallel: 40 seconds
>> Weave: 29 seconds
>> For a 10 module project using -T 1C on a 6 core box.
>>
>>
>> For those wondering:
>> * Both parallel and weave favorize projects with lots of modules
>> and "wide" dependency graphs.
>> * Weave is /probably/ quite a lot faster than parallel for projects
>> with lots of tests fairly evenly distributed among the modules.
>> * Putting all your tests in one module is a bad idea in this context.
>> * Some mojos (like war/ear) can "dwarf" all the others,
>> effectively limiting the overall potential somewhat.
>>
>> Kristian
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>
> --
> Daniel Kulp
> dkulp@apache.org
> http://dankulp.com/blog
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Parallel vs weave
Posted by Daniel Kulp <dk...@apache.org>.
Well, weave mode doesn't work for CXF. Ends up with a NullPointerException
down in antrun-plugin someplace. Thus, I cannot help you there.
For CXF on my i7 820QM (4 cores, 8 threads) turning off checkstyle/pmd and
having everything already code generated/compiled (so mostly just running
tests)
Linear: 26 minutes
Parallel: 9 minutes (-T 12 is what I'm using)
Dan
On Friday 11 June 2010 12:24:13 pm Kristian Rosenvold wrote:
> I am picking up some positive tweets here and there, but I'd be really
> happy if those of you who test parallel building could report back with
> some numbers in response to this message. I am particularly interested
> in the difference between parallel and weave (and linear too), mostly to
> assess that weave is worth the effort.
>
> So after tweaking around with threads a bit to find out what runs
> fastest I'd really enjoy reports like this one (real data):
>
> Linear: 50 seconds
> Parallel: 40 seconds
> Weave: 29 seconds
> For a 10 module project using -T 1C on a 6 core box.
>
>
> For those wondering:
> * Both parallel and weave favorize projects with lots of modules
> and "wide" dependency graphs.
> * Weave is /probably/ quite a lot faster than parallel for projects
> with lots of tests fairly evenly distributed among the modules.
> * Putting all your tests in one module is a bad idea in this context.
> * Some mojos (like war/ear) can "dwarf" all the others,
> effectively limiting the overall potential somewhat.
>
> Kristian
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
--
Daniel Kulp
dkulp@apache.org
http://dankulp.com/blog
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org