You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@maven.apache.org by Benedikt Thelen <th...@gmail.com> on 2008/04/09 13:21:39 UTC
Multiple CPUs
Hi there, i am sort of a maven newbee,
At our workplce we have a quite big Coccon project in developenet and we use
maven to build it. Building takes usually 5-6 minutes which is quite a
while. I noticed using gkrellm and htop that maven only uses one of the two
processors (Levono Thinkpad with intel core Duo) in my notebook.
Question: Is there a way to tell maven to use both CPU's while building? I
searched gooogle a lot but i didn't find anything.
Greetings Benedikt Thelen
Re: Multiple CPUs
Posted by Benedikt Thelen <th...@gmail.com>.
Hmm splitting up the modules isn't that easy in this case, I hoped there was
an option similar to -j X in GNU make.
Allthough the splittung up and compiling of modules parallel is probably the
best approach.
Thanks For the Help
Benedikt Thelen
On Wed, Apr 9, 2008 at 5:48 PM, Martin von Gagern <Ma...@gmx.net>
wrote:
> VELO wrote:
>
> > Well, may be the best way to do that is add support to maven run modules
> > on
> > parallel.... not sure how to, but if maven run two modules at same time,
> > on
> > a dual core machine, means a big gain, i believe.
> >
>
> I guess you'd have to do this in independent processes, i.e. foce new
> maven instances, as some plugins modify static resources like currentLocale
> during their execution.
>
> Greetings,
> Martin von Gagern
>
>
Re: Multiple CPUs
Posted by Martin von Gagern <Ma...@gmx.net>.
VELO wrote:
> Well, may be the best way to do that is add support to maven run modules on
> parallel.... not sure how to, but if maven run two modules at same time, on
> a dual core machine, means a big gain, i believe.
I guess you'd have to do this in independent processes, i.e. foce new
maven instances, as some plugins modify static resources like
currentLocale during their execution.
Greetings,
Martin von Gagern
Re: Multiple CPUs
Posted by Nigel Magnay <ni...@gmail.com>.
Yes - but all these dependencies are calculable from the POM files. I
think about this every time my machine sits with 7 idle cores...
The biggest initial issue is that the local repo is not threadsafe (2
mvn instances running in parallel can crash in interesting ways).
That's the first thing to fix, but it's not a small thing.
On Wed, Apr 9, 2008 at 3:08 PM, Nino Saturnino Martinez Vazquez Wael
<ni...@jayway.dk> wrote:
> Hmm, but isnt it a problem if modules depend on each other? if module a
> depends on module b...
>
> And I think that if implemented it should be scalable to xxx cpus.. I guess
> module a would have to wait for module b or something..
>
>
>
> VELO wrote:
>
> > Well, may be the best way to do that is add support to maven run modules
> on
> > parallel.... not sure how to, but if maven run two modules at same time,
> on
> > a dual core machine, means a big gain, i believe.
> >
> >
> > VELO
> >
> > On Wed, Apr 9, 2008 at 8:21 AM, Benedikt Thelen <th...@gmail.com> wrote:
> >
> >
> >
> > > Hi there, i am sort of a maven newbee,
> > > At our workplce we have a quite big Coccon project in developenet and we
> > > use
> > > maven to build it. Building takes usually 5-6 minutes which is quite a
> > > while. I noticed using gkrellm and htop that maven only uses one of the
> > > two
> > > processors (Levono Thinkpad with intel core Duo) in my notebook.
> > > Question: Is there a way to tell maven to use both CPU's while building?
> I
> > > searched gooogle a lot but i didn't find anything.
> > >
> > > Greetings Benedikt Thelen
> > >
> > >
> > >
> >
> >
> >
>
>
> --
> -Wicket for love
>
> Nino Martinez Wael
> Java Specialist @ Jayway DK
> http://www.jayway.dk
> +45 2936 7684
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Multiple CPUs
Posted by VELO <ve...@gmail.com>.
Lets imagine a 10 modules application.
I don't believe each module will always depend from the predecessor (10
depends 9, 9 depends 8, 8 depends....)
Is commons see 10 depends 1, 2 and 3. 9 depends 1, 2. 8 depends 7, and so
go on.
Probably some modules need to be compiled in sequence. Others can be on
parallel. So maven look to the machine, 4 cores, let me see, 4 cores,
compile module 1... compile module 2... now compile module 3, 4 and 5.....
I think maven dependencies mechanism is able to define who can be compiled
in parallel and who need to be compiled in series.
And can exist a parameter to force series work.
Well, just a idea.
VELO
On Wed, Apr 9, 2008 at 11:08 AM, Nino Saturnino Martinez Vazquez Wael <
nino.martinez@jayway.dk> wrote:
> Hmm, but isnt it a problem if modules depend on each other? if module a
> depends on module b...
>
> And I think that if implemented it should be scalable to xxx cpus.. I
> guess module a would have to wait for module b or something..
>
>
> VELO wrote:
>
> > Well, may be the best way to do that is add support to maven run modules
> > on
> > parallel.... not sure how to, but if maven run two modules at same time,
> > on
> > a dual core machine, means a big gain, i believe.
> >
> >
> > VELO
> >
> > On Wed, Apr 9, 2008 at 8:21 AM, Benedikt Thelen <th...@gmail.com>
> > wrote:
> >
> >
> >
> > > Hi there, i am sort of a maven newbee,
> > > At our workplce we have a quite big Coccon project in developenet and
> > > we
> > > use
> > > maven to build it. Building takes usually 5-6 minutes which is quite a
> > > while. I noticed using gkrellm and htop that maven only uses one of
> > > the
> > > two
> > > processors (Levono Thinkpad with intel core Duo) in my notebook.
> > > Question: Is there a way to tell maven to use both CPU's while
> > > building? I
> > > searched gooogle a lot but i didn't find anything.
> > >
> > > Greetings Benedikt Thelen
> > >
> > >
> > >
> >
> >
> >
>
> --
> -Wicket for love
>
> Nino Martinez Wael
> Java Specialist @ Jayway DK
> http://www.jayway.dk
> +45 2936 7684
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
Re: Multiple CPUs
Posted by Nino Saturnino Martinez Vazquez Wael <ni...@jayway.dk>.
Hmm, but isnt it a problem if modules depend on each other? if module a
depends on module b...
And I think that if implemented it should be scalable to xxx cpus.. I
guess module a would have to wait for module b or something..
VELO wrote:
> Well, may be the best way to do that is add support to maven run modules on
> parallel.... not sure how to, but if maven run two modules at same time, on
> a dual core machine, means a big gain, i believe.
>
>
> VELO
>
> On Wed, Apr 9, 2008 at 8:21 AM, Benedikt Thelen <th...@gmail.com> wrote:
>
>
>> Hi there, i am sort of a maven newbee,
>> At our workplce we have a quite big Coccon project in developenet and we
>> use
>> maven to build it. Building takes usually 5-6 minutes which is quite a
>> while. I noticed using gkrellm and htop that maven only uses one of the
>> two
>> processors (Levono Thinkpad with intel core Duo) in my notebook.
>> Question: Is there a way to tell maven to use both CPU's while building? I
>> searched gooogle a lot but i didn't find anything.
>>
>> Greetings Benedikt Thelen
>>
>>
>
>
--
-Wicket for love
Nino Martinez Wael
Java Specialist @ Jayway DK
http://www.jayway.dk
+45 2936 7684
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Multiple CPUs
Posted by VELO <ve...@gmail.com>.
Well, may be the best way to do that is add support to maven run modules on
parallel.... not sure how to, but if maven run two modules at same time, on
a dual core machine, means a big gain, i believe.
VELO
On Wed, Apr 9, 2008 at 8:21 AM, Benedikt Thelen <th...@gmail.com> wrote:
> Hi there, i am sort of a maven newbee,
> At our workplce we have a quite big Coccon project in developenet and we
> use
> maven to build it. Building takes usually 5-6 minutes which is quite a
> while. I noticed using gkrellm and htop that maven only uses one of the
> two
> processors (Levono Thinkpad with intel core Duo) in my notebook.
> Question: Is there a way to tell maven to use both CPU's while building? I
> searched gooogle a lot but i didn't find anything.
>
> Greetings Benedikt Thelen
>
RE: Multiple CPUs
Posted by Martin Gainty <mg...@hotmail.com>.
Agreed-
With Hotspot
I have been able to boost performance by implementing ConcurrentMarkSweep GC
JAVA_OPTS=XX:+UseConcMarkSweepGC
http://java.sun.com/docs/hotspot/gc1.4.2/faq.html
TakkMartin ______________________________________________Disclaimer and confidentiality noteEverything in this e-mail and any attachments relates to the official business of Sender. This transmission is of a confidential nature and Sender does not endorse distribution to any party other than intended recipient. Sender does not necessarily endorse content contained within this transmission.> Subject: RE: Multiple CPUs> Date: Wed, 9 Apr 2008 14:12:58 +0200> Wrom: BDXRQBGJSNBOHMKHJYFMYXOEAIJJPHSCRTNHGSWZIDREXCAXZOWCONEUQZAAFXISHJEXXIMQZUIVOTQNQEMSFDULHPQQWOYIYZUNNYCGPKYLEJGDGVCJVTLBXFGGMEPYOQKEDOTWFAOBUZXUWLSZLKBRNVWWCUFPEGAUTFJMVRESKPNKMBIPBARHDMNNSKVFVWRKJVZCMHVIBGDADRZFSQHYUCDDJBLVLMHAALPTCXLYRWTQTIPWIGYOKSTTZRCLBDXRQBGJSNBOHMKHJYFMYXOEAIJJPHSCRTNHGSWZIDREXCAXZOWCONEUQZAAFXISHJEXXIMQZUIVOTQNQEMSFDULHPQQWOYIYZUNNYCGPKYLEJGDGVCJVTLBXFGGMEPYOQKEDOTWFAOBUZXUWLSZLKBRNVWWCUFPEGAUTFJMVRESKPNKMBIPBARHDMNNSKVFVWRKJVZCMHVIBGDADRZFSQHYUCDDJBLVLMHAALPTCXLYRWTQTIPWIGYOKSTTZRCLBDXRQBGJSNBOHMKHJYFMYXOEAIJJPHSCRTNHGSWZIDREXCAXZOWCONEUQZAAFXISHJEXXIMQZUIVOTQNQEMSFDULHPQQWOYIYZUNNYCGPKYLEJGDGVCJVTLBXFGGMEPYOQKEDOTWFAOBUZXUWLSZLKBRNVWWCUFPEGAUTFJMVRESKPNKMBIPBARHDMNNSKVFVWRKJVZCMHVIBGDADRZFSQHYUCDDJBLVLMHAALPTCXLYRWTQTIPWIGYOKSTTZRCLBDXRQBGJSNBOHMKHJYFMYXOEAIJJPHSCRTNHGSWZIDREXCAXZOWCONEUQZAAFXISHJEXXIMQZUIVOTQNQEMSFDULHPQQWOYIYZUNNYCGPKYLEJGDGVCJVTLBXFGGMEPYOQKEDOTWFAOBUZXUWLSZLKBRNVWWCUFPEGAUTFJMVRESKPNKMBIPBARHDMNNSKVFVWRKJVZCMHVIBGDADRZFSQHYUCDDJBLVLMHAALPTCXLYRWTQTIPWIGYOKSTTZRCLBDXRQBGJSNBOHMKHJYFMYXOEAIJJPHSCRTNHGSWZIDREXCAXZOWCONEUQZAAFXISHJEXXIMQZUIVOTQNQEMSFDULHPQQWOYIYZUNNYCGPKYLEJGDGVCJVTLBXFGGMEPYOQKEDOTWFAOBUZXUWLSZLKBRNVWWCUFPEGAUTFJMVRESKPNKMBIPBARHDMNNSKVFVWRKJVZCMHVIBGDADRZFSQHYUCDDJBLVLMHAALPTCXLYRWTQTIPWIGYOKSTTZRCLBDXRQBGJSNBOHMKHJYFMYXOEAIJJPHSCRTNHGSWZIDREXCAXZOWCONEUQZAAFXISHJEXXIMQZUIVOTQNQEMSFDULHPQQWOYIYZUNNYCGPKYLEJGDGVCJVTLBXFGGMEPYOQKEDOTWFAOBUZXUWLSZLKBRNVWWCUFPEGAUTFJMVRESKPNKMBIPBARHDMNNSKVFVWRKJVZCMHVIBGDADRZFSQHYUCDDJBLVLMHAALPTCXLYRWTQTIPWIGYOKSTTZRCLBDXRQBGJSNBOHMKHJYFMYXOEAIJJPHSCRTNHGSWZIDREXCAXZOWCONEUQZAAFXISHJEXXIMQZUIVOTQNQEMSFDULHPQQWOYIYZUNNYCGPKYLEJGDGVCJVTLBXFGGMEPYOQKEDOTWFAOBUZXUWLSZLKBRNVWWCUFPEGAUTFJMVRESKPNKMBIPR>> >>> > >> two> >> > >>> processors (Levono Thinkpad with intel core Duo) in my notebook.> >>> Question: Is there a way to tell maven to use both CPU's while> >>> > >> building? I> >> > >>> searched gooogle a lot but i didn't find anything.> >>>> >>> Greetings Benedikt Thelen> >>>> >>> > >> ---------------------------------------------------------------------> >> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org> >> For additional commands, e-mail: users-help@maven.apache.org> >>> >>> >> > >> > > > -- > -Wicket for love> > Nino Martinez Wael> Java Specialist @ Jayway DK> http://www.jayway.dk> +45 2936 7684> > > ---------------------------------------------------------------------> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org> For additional commands, e-mail: users-help@maven.apache.org> > * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *> > This email with attachments is solely for the use of the individual or> entity to whom it is addressed. Please also be aware that the DnB NOR Group> cannot accept any payment orders or other legally binding correspondence with> customers as a part of an email. > > This email message has been virus checked by the anti virus programs used> in the DnB NOR Group.> > * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *> > > ---------------------------------------------------------------------> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org> For additional commands, e-mail: users-help@maven.apache.org>
_________________________________________________________________
Use video conversation to talk face-to-face with Windows Live Messenger.
http://www.windowslive.com/messenger/connect_your_way.html?ocid=TXT_TAGLM_WL_Refresh_messenger_video_042008
Re: Multiple CPUs
Posted by ben short <ja...@gmail.com>.
You could structure your project in to a multi module project. For
instance an api, implimentation and utils module. then if you only
make changes to the utils module you can just rebuild that module
which should save some time.
On Wed, Apr 9, 2008 at 1:12 PM, <he...@dnbnor.no> wrote:
> Hi
>
> The GC = Garbage Collection. It will help a bit.
>
> Hermod
>
>
>
> -----Original Message-----
> From: Nino Saturnino Martinez Vazquez Wael
> [mailto:nino.martinez@jayway.dk]
> Sent: Wednesday, April 09, 2008 2:06 PM
> To: Maven Users List
> Subject: Re: Multiple CPUs
>
>
> Is'nt this just a java setting, you could pass on to the jvm?:
>
>
> -XX:+UseParallelGC
> -XX:+ParallelGCThreads=2
>
> http://www.md.pp.ru/~eu/jdk6options.html
>
> Benedikt Thelen wrote:
> > if possible is there a wat to split up the source to make diffrent
> > jobs who could be run separately?
> >
> >
> > On Wed, Apr 9, 2008 at 1:24 PM, ben short <ja...@gmail.com>
> > wrote:
> >
> >
> >> I'd think that the build process is single threaded, with exception
> >> to the surefire plugin which can be forked.
> >>
> >> On Wed, Apr 9, 2008 at 12:21 PM, Benedikt Thelen <th...@gmail.com>
> >> wrote:
> >>
> >>> Hi there, i am sort of a maven newbee,
> >>> At our workplce we have a quite big Coccon project in developenet
> >>> and
> >>>
> >> we use
> >>
> >>> maven to build it. Building takes usually 5-6 minutes which is
> >>> quite a while. I noticed using gkrellm and htop that maven only
> >>> uses one of the
> >>>
> >> two
> >>
> >>> processors (Levono Thinkpad with intel core Duo) in my notebook.
> >>> Question: Is there a way to tell maven to use both CPU's while
> >>>
> >> building? I
> >>
> >>> searched gooogle a lot but i didn't find anything.
> >>>
> >>> Greetings Benedikt Thelen
> >>>
> >>>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> >> For additional commands, e-mail: users-help@maven.apache.org
> >>
> >>
> >>
> >
> >
>
> --
> -Wicket for love
>
> Nino Martinez Wael
> Java Specialist @ Jayway DK
> http://www.jayway.dk
> +45 2936 7684
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
> * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
>
> This email with attachments is solely for the use of the individual or
> entity to whom it is addressed. Please also be aware that the DnB NOR Group
> cannot accept any payment orders or other legally binding correspondence with
> customers as a part of an email.
>
> This email message has been virus checked by the anti virus programs used
> in the DnB NOR Group.
>
> * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
RE: Multiple CPUs
Posted by he...@dnbnor.no.
Hi
The GC = Garbage Collection. It will help a bit.
Hermod
-----Original Message-----
From: Nino Saturnino Martinez Vazquez Wael
[mailto:nino.martinez@jayway.dk]
Sent: Wednesday, April 09, 2008 2:06 PM
To: Maven Users List
Subject: Re: Multiple CPUs
Is'nt this just a java setting, you could pass on to the jvm?:
-XX:+UseParallelGC
-XX:+ParallelGCThreads=2
http://www.md.pp.ru/~eu/jdk6options.html
Benedikt Thelen wrote:
> if possible is there a wat to split up the source to make diffrent
> jobs who could be run separately?
>
>
> On Wed, Apr 9, 2008 at 1:24 PM, ben short <ja...@gmail.com>
> wrote:
>
>
>> I'd think that the build process is single threaded, with exception
>> to the surefire plugin which can be forked.
>>
>> On Wed, Apr 9, 2008 at 12:21 PM, Benedikt Thelen <th...@gmail.com>
>> wrote:
>>
>>> Hi there, i am sort of a maven newbee,
>>> At our workplce we have a quite big Coccon project in developenet
>>> and
>>>
>> we use
>>
>>> maven to build it. Building takes usually 5-6 minutes which is
>>> quite a while. I noticed using gkrellm and htop that maven only
>>> uses one of the
>>>
>> two
>>
>>> processors (Levono Thinkpad with intel core Duo) in my notebook.
>>> Question: Is there a way to tell maven to use both CPU's while
>>>
>> building? I
>>
>>> searched gooogle a lot but i didn't find anything.
>>>
>>> Greetings Benedikt Thelen
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
>> For additional commands, e-mail: users-help@maven.apache.org
>>
>>
>>
>
>
--
-Wicket for love
Nino Martinez Wael
Java Specialist @ Jayway DK
http://www.jayway.dk
+45 2936 7684
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
This email with attachments is solely for the use of the individual or
entity to whom it is addressed. Please also be aware that the DnB NOR Group
cannot accept any payment orders or other legally binding correspondence with
customers as a part of an email.
This email message has been virus checked by the anti virus programs used
in the DnB NOR Group.
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Multiple CPUs
Posted by Nino Saturnino Martinez Vazquez Wael <ni...@jayway.dk>.
Is'nt this just a java setting, you could pass on to the jvm?:
-XX:+UseParallelGC
-XX:+ParallelGCThreads=2
http://www.md.pp.ru/~eu/jdk6options.html
Benedikt Thelen wrote:
> if possible is there a wat to split up the source to make diffrent jobs who
> could be run separately?
>
>
> On Wed, Apr 9, 2008 at 1:24 PM, ben short <ja...@gmail.com> wrote:
>
>
>> I'd think that the build process is single threaded, with exception to
>> the surefire plugin which can be forked.
>>
>> On Wed, Apr 9, 2008 at 12:21 PM, Benedikt Thelen <th...@gmail.com>
>> wrote:
>>
>>> Hi there, i am sort of a maven newbee,
>>> At our workplce we have a quite big Coccon project in developenet and
>>>
>> we use
>>
>>> maven to build it. Building takes usually 5-6 minutes which is quite a
>>> while. I noticed using gkrellm and htop that maven only uses one of the
>>>
>> two
>>
>>> processors (Levono Thinkpad with intel core Duo) in my notebook.
>>> Question: Is there a way to tell maven to use both CPU's while
>>>
>> building? I
>>
>>> searched gooogle a lot but i didn't find anything.
>>>
>>> Greetings Benedikt Thelen
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
>> For additional commands, e-mail: users-help@maven.apache.org
>>
>>
>>
>
>
--
-Wicket for love
Nino Martinez Wael
Java Specialist @ Jayway DK
http://www.jayway.dk
+45 2936 7684
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Multiple CPUs
Posted by Benedikt Thelen <th...@gmail.com>.
if possible is there a wat to split up the source to make diffrent jobs who
could be run separately?
On Wed, Apr 9, 2008 at 1:24 PM, ben short <ja...@gmail.com> wrote:
> I'd think that the build process is single threaded, with exception to
> the surefire plugin which can be forked.
>
> On Wed, Apr 9, 2008 at 12:21 PM, Benedikt Thelen <th...@gmail.com>
> wrote:
> > Hi there, i am sort of a maven newbee,
> > At our workplce we have a quite big Coccon project in developenet and
> we use
> > maven to build it. Building takes usually 5-6 minutes which is quite a
> > while. I noticed using gkrellm and htop that maven only uses one of the
> two
> > processors (Levono Thinkpad with intel core Duo) in my notebook.
> > Question: Is there a way to tell maven to use both CPU's while
> building? I
> > searched gooogle a lot but i didn't find anything.
> >
> > Greetings Benedikt Thelen
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
Re: Multiple CPUs
Posted by ben short <ja...@gmail.com>.
I'd think that the build process is single threaded, with exception to
the surefire plugin which can be forked.
On Wed, Apr 9, 2008 at 12:21 PM, Benedikt Thelen <th...@gmail.com> wrote:
> Hi there, i am sort of a maven newbee,
> At our workplce we have a quite big Coccon project in developenet and we use
> maven to build it. Building takes usually 5-6 minutes which is quite a
> while. I noticed using gkrellm and htop that maven only uses one of the two
> processors (Levono Thinkpad with intel core Duo) in my notebook.
> Question: Is there a way to tell maven to use both CPU's while building? I
> searched gooogle a lot but i didn't find anything.
>
> Greetings Benedikt Thelen
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Multiple CPUs
Posted by Manos Batsis <ma...@geekologue.com>.
Benedikt Thelen wrote:
> Hi there, i am sort of a maven newbee,
> At our workplce we have a quite big Coccon project in developenet and we use
> maven to build it. Building takes usually 5-6 minutes which is quite a
> while. I noticed using gkrellm and htop that maven only uses one of the two
> processors (Levono Thinkpad with intel core Duo) in my notebook.
> Question: Is there a way to tell maven to use both CPU's while building? I
> searched gooogle a lot but i didn't find anything.
In case you are using antrun, we've used the "Parallel" container task
in certain build targets. Trying to use it doesn't always make sense though.
[1] http://ant.apache.org/manual/CoreTasks/parallel.html
Cheers,
Manos
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Multiple CPUs
Posted by VELO <ve...@gmail.com>.
So, the dream was killed?
VELO
On Thu, Apr 10, 2008 at 2:31 PM, Lacoste, Dana (TSG Software San Diego) <
dana.lacoste@hp.com> wrote:
> The situation below (deploying to multiple J2EE platforms in the build)
> is already easily supported: you're using ant to do it, and ant supports
> a "parallel" section http://ant.apache.org/manual/CoreTasks/parallel.html
>
> So, inside a maven-antrun-plugin section you can do a <parallel> with
> no hesitation.
>
> The main issue, though, gets back to one of "The Maven Way"
>
> Maven was designed with a very heavy emphasis on the "right" way to do
> things, and with most projects (breaking into modules, providing
> single-file
> artifact build results, assumed "src/java" layout, etc.) the only
> "parallel"
> part that could really work would be when compiling the classes from the
> **/*java files. This would theoretically result in "quicker" java
> compiling,
> but I question if the result would be a significant gain, unless you had
> a SERIOUSLY large number of classes in that jar! And with a 6 minute
> build,
> you really don't :)
>
> For NON-JAVA src files, you can turn on the make flags appropriately
> already
> (in other words, if maven isn't doing the build, it's just calling ant to
> call make or something, then you can get the make command to parallelize)
> (I remember an old comparison of gcc vs. kylix in this area: gcc benefits
> from the "make -j" to such a large extent because it's really horrible at
> building each file: kylix, with Pascal's simpler compile rules, was so
> much
> faster they weren't even in the same league!)
>
> But that's really just "the way it is" : there's no real way that maven
> can
> parallelize things without causing a lot of issues. Threading output of
> the build, handling build failures, ensuring dependency order: it would
> add
> a LOT of complexity to the build with not a lot of benefit.
>
> And, just for the record, my maven project has 100 (wow, 100 exactly.
> hadn't
> counted before) modules and a full build of EVERYTHING takes about 330
> minutes
> on my fastest build machine: if there was a way that I could speed this up
> I would GLADLY do so, but there really isn't : any speedup that's caused
> by
> a change in how maven works would result in serious usability and
> stability
> issues that would NOT be worth it, IMNSHO.
>
> Dana Lacoste
>
> -----Original Message (Trimmed for content)-----
> From: Jorg Heymans [mailto:jorg.heymans@gmail.com]
> Sent: Thursday, April 10, 2008 8:36 AM
> To: Maven Users List
> Subject: Re: Multiple CPUs
>
> well imagine that during your integration build you have to deploy 10
> EARs, and each of them to a number of application servers (weblogic,
> websphere etc etc). That adds up pretty quick. I don't know the effort
> involved, but if the ant runner for example could be made to run in a
> different thread for each invocation that could speed things up drastically
> only for this use case.
>
> Jorg
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
RE: Multiple CPUs
Posted by "Lacoste, Dana (TSG Software San Diego)" <da...@hp.com>.
> That's an interesting build you have there, care to tell us what the longest running tasks are (hope it's not the compiling :-P)
In short, it's "Compare new dataset vs. old dataset"
It takes a very long time because it has to take the newly built binaries and data, get them up and running (the DB load takes a while!) and then compare the new set to the old set for upgrade purposes.
It's not really the area I'm working on, so I can't comment as to why it takes so long, but it's pretty much 100% "outside Maven" so there's not much Maven could do to speed it up :)
Of the 100 Modules, only two take more than 10 minutes to "mvn install" : this one and the cross-platform build (maven triggers ant which triggers remote builds on multiple platforms.) so the time per module is very small (and 99.9% of our builds are isolated to a single module or family of modules) and only intermittent "FULL" builds take such a long time.
Dana Lacoste
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
RE: Multiple CPUs
Posted by "Lacoste, Dana (TSG Software San Diego)" <da...@hp.com>.
> Not sure if i agree here. Why can the building of different artifacts
> not be parallelized if they completely don't depend on each other? If
> my build contains 1 war and 5 completely independent jars that are
> used in the war, why can those 5 jar artifacts not be built in
> parallel ? Sure enough, they can have shared dependencies, but with a
> good artifact download manager it should be trivial to detect
> downloads in progress. Is there another way how completely
> independent artifacts can influence each other, or am i missing
> something ? Why can't constructing an artifact be made an atomic task
> provided it has all required input available ?
It's that last line that I'll leave to the Maven people (I'm a user,
I might even be a power user, but I'm also a perforce administrator,
and the complete lack of atomicity in Maven has always bugged me :)
(and Maven:SCM plugin for perforce makes me shudder :)
Nigel's comment on the repo not being threadsafe is the only other thing
I can think to add at the moment.
> - threading the build output shouldn't be hard, especially if maven
> were to become less verbose. I would be really OK with just reading
> building module ABC.... OK building module DEF.... OK etc etc
I have "-e" turned on in Cruise Control because I find the maven
error messages far too difficult to decipher without more information.
We currently have a workaround preventing the "threaded output" you
describe from our cross platform build: I can't tell (for example) if
that line came from Linux or Solaris (if they're parallel building the
same code at the same time, but on different systems.) So we do the
builds parallel then dump the output sequentially. Ugly, Ugly, Ugly,
but I don't see any way to do threaded output the way Maven works now.
> - I don't see how handling a build failure would cause problems here,
> if an artifact fails to build then the build just stops after the
> completion of those tasks that ran parallel to the failed task. Heck,
> in that way you could even see multiple artifact failures in one
> build, great !
Sometimes you want --fail-at-end, sometimes you don't! :)
With maven parallel, you'd have it always. I can't really express why,
but that bugs me for some reason :)
> - The dependency order should not fundamentally change with parallel
> artifact builds, or did you have a specific case in mind here ?
In our current system (where our parallel builds are done with multiple
OS combinations) we can have this:
Artifact A (Solaris) depends on jar A, B, and C (all -SNAPSHOT, so
they're downloaded and updated with every build)
Artifact A (Linux) has same dependencies.
building in parallel causes both to download at the same time and
install into the same location (NFS mounted home dir) which causes
the issues Nigel mentioned (the repository is NOT threadsafe)
I don't know how to make this work better, but with -SNAPSHOT
in particular it causes huge headaches if you're not paying attention :)
Dana
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Multiple CPUs
Posted by Jorg Heymans <jo...@gmail.com>.
On Thu, Apr 10, 2008 at 7:31 PM, Lacoste, Dana (TSG Software San Diego) <
dana.lacoste@hp.com> wrote:
> The situation below (deploying to multiple J2EE platforms in the build)
> is already easily supported: you're using ant to do it, and ant supports
> a "parallel" section http://ant.apache.org/manual/CoreTasks/parallel.html
>
thanks for the link, i'll have a look at this if it can support my usecase.
>
> Maven was designed with a very heavy emphasis on the "right" way to do
> things, and with most projects (breaking into modules, providing
> single-file
> artifact build results, assumed "src/java" layout, etc.) the only
> "parallel"
> part that could really work would be when compiling the classes from the
> **/*java files. This would theoretically result in "quicker" java
> compiling,
> but I question if the result would be a significant gain, unless you had
> a SERIOUSLY large number of classes in that jar! And with a 6 minute
> build,
> you really don't :)
>
Not sure if i agree here. Why can the building of different artifacts not be
parallelized if they completely don't depend on each other? If my build
contains 1 war and 5 completely independent jars that are used in the war,
why can those 5 jar artifacts not be built in parallel ? Sure enough, they
can have shared dependencies, but with a good artifact download manager it
should be trivial to detect downloads in progress. Is there another way how
completely independent artifacts can influence each other, or am i missing
something ? Why can't constructing an artifact be made an atomic task
provided it has all required input available ?
But that's really just "the way it is" : there's no real way that maven can
> parallelize things without causing a lot of issues. Threading output of
> the build, handling build failures, ensuring dependency order: it would
> add
> a LOT of complexity to the build with not a lot of benefit.
>
- threading the build output shouldn't be hard, especially if maven were to
become less verbose. I would be really OK with just reading
building module ABC.... OK
building module DEF.... OK
etc
etc
- I don't see how handling a build failure would cause problems here, if an
artifact fails to build then the build just stops after the completion of
those tasks that ran parallel to the failed task. Heck, in that way you
could even see multiple artifact failures in one build, great !
- The dependency order should not fundamentally change with parallel
artifact builds, or did you have a specific case in mind here ?
And, just for the record, my maven project has 100 (wow, 100 exactly.
> hadn't
> counted before) modules and a full build of EVERYTHING takes about 330
> minutes
> on my fastest build machine: if there was a way that I could speed this up
That's an interesting build you have there, care to tell us what the longest
running tasks are (hope it's not the compiling :-P)
Cheers,
Jorg
Re: Multiple CPUs
Posted by James William Dumay <ja...@atlassian.com>.
Jorg,
The local repository is not accessed in a non-transactional method.
This means that two processes (or multiple threads for that matter) can
try to write data to the same location on the file system or try to
read/write to a file that is not yet complete.
There is a proposal to fixing this:
http://docs.codehaus.org/display/MAVEN/Local+repository+separation
I don't think we would see any traction on this until Maven 2.1
Thanks
James
On Mon, 2008-04-14 at 14:37 +0200, Jorg Heymans wrote:
> On Mon, Apr 14, 2008 at 12:46 PM, Stephen Connolly <
> stephen.alan.connolly@gmail.com> wrote:
>
> > many problems (as the local repo is not designed to be accessed by
> > multiple
> > threads concurrently) to leave it on by default. The option is still
> > there
> > in the Advanced tab for the m2 project type.
> >
>
> Does anyone actually know more details about the concurrent issues with the
> local repo ? Is it because the repo is accessed in a non-transactional way,
> or is it because the repo access code is not threadsafe ?
>
> Jorg
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Multiple CPUs
Posted by Jorg Heymans <jo...@gmail.com>.
On Mon, Apr 14, 2008 at 12:46 PM, Stephen Connolly <
stephen.alan.connolly@gmail.com> wrote:
> many problems (as the local repo is not designed to be accessed by
> multiple
> threads concurrently) to leave it on by default. The option is still
> there
> in the Advanced tab for the m2 project type.
>
Does anyone actually know more details about the concurrent issues with the
local repo ? Is it because the repo is accessed in a non-transactional way,
or is it because the repo access code is not threadsafe ?
Jorg
Re: Multiple CPUs
Posted by Stephen Connolly <st...@gmail.com>.
Hudson originally had this parallelisation for module builds but hit too
many problems (as the local repo is not designed to be accessed by multiple
threads concurrently) to leave it on by default. The option is still there
in the Advanced tab for the m2 project type.
It can give significantly faster builds... it can break spectacularly if
your project is organised less than optimally!
-Stephen
On Mon, Apr 14, 2008 at 10:45 AM, dimitris kapanidis <d....@gmail.com>
wrote:
>
> hi,
>
> I totally agree about parallizing the build process of unrelated
> artifacts.
>
> my 5cents on this one: I imagine a not so distant future were I can tell
> my
> continuous integration system that I want to build a maven project on a
> cluster. Maven (or the CI) should be able to separate unrelated artifacts
> and build them in parallel, to break down the build time.
>
>
> Simone Gianni-2 wrote:
> >
> > VELO wrote:
> >>> On the build process itself, it has no meaning to parallelize it as
> you
> >>> properly pointed out.
> >>>
> >>>
> >>
> >> May this is true on Java...
> >>
> >> But on flex is not. Flex compilation always use 100% on a single core
> >> computer and never use more them 50% on multi core.
> >>
> >> Well, if we got downloads and tests multithreaded I will be very very
> >> happy
> >> =D
> >>
> > Hi Velo,
> > yes, you are right. I was referring to "build process" as the steps and
> > phases Maven takes while building, one of which is "compile the
> > sources". If the "compile" step is single threaded or multi threaded
> > obviously changes depending on the compiler maven invokes.
> >
> > Simone
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> > For additional commands, e-mail: users-help@maven.apache.org
> >
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Multiple-CPUs-tp16584814s177p16674687.html
> Sent from the Maven - Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
Re: Multiple CPUs
Posted by dimitris kapanidis <d....@gmail.com>.
hi,
I totally agree about parallizing the build process of unrelated artifacts.
my 5cents on this one: I imagine a not so distant future were I can tell my
continuous integration system that I want to build a maven project on a
cluster. Maven (or the CI) should be able to separate unrelated artifacts
and build them in parallel, to break down the build time.
Simone Gianni-2 wrote:
>
> VELO wrote:
>>> On the build process itself, it has no meaning to parallelize it as you
>>> properly pointed out.
>>>
>>>
>>
>> May this is true on Java...
>>
>> But on flex is not. Flex compilation always use 100% on a single core
>> computer and never use more them 50% on multi core.
>>
>> Well, if we got downloads and tests multithreaded I will be very very
>> happy
>> =D
>>
> Hi Velo,
> yes, you are right. I was referring to "build process" as the steps and
> phases Maven takes while building, one of which is "compile the
> sources". If the "compile" step is single threaded or multi threaded
> obviously changes depending on the compiler maven invokes.
>
> Simone
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/Multiple-CPUs-tp16584814s177p16674687.html
Sent from the Maven - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Multiple CPUs
Posted by Simone Gianni <si...@apache.org>.
VELO wrote:
>> On the build process itself, it has no meaning to parallelize it as you
>> properly pointed out.
>>
>>
>
> May this is true on Java...
>
> But on flex is not. Flex compilation always use 100% on a single core
> computer and never use more them 50% on multi core.
>
> Well, if we got downloads and tests multithreaded I will be very very happy
> =D
>
Hi Velo,
yes, you are right. I was referring to "build process" as the steps and
phases Maven takes while building, one of which is "compile the
sources". If the "compile" step is single threaded or multi threaded
obviously changes depending on the compiler maven invokes.
Simone
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Multiple CPUs
Posted by VELO <ve...@gmail.com>.
>
> On the build process itself, it has no meaning to parallelize it as you
> properly pointed out.
>
May this is true on Java...
But on flex is not. Flex compilation always use 100% on a single core
computer and never use more them 50% on multi core.
Well, if we got downloads and tests multithreaded I will be very very happy
=D
VELO
On Fri, Apr 11, 2008 at 2:21 PM, Simone Gianni <si...@apache.org> wrote:
> Hi Dana,
> you are absolutely right, but there are a few things Maven could do to
> improve speed : using multiple threads/cpus :
> - Maven downloads stuff. Now, since the network exists, it exists
> latency, and all net applications work best when multithreaded. Is maven
> currently parallelizing the jar downloads/the connections it makes to
> test if there are new releases of the artifacts? That could speed up a
> lot, expecially on first time users, first time they download stuff,
> first time they see maven as slow :)
> - Unit tests are, by DEFINITION, parallelizable. Maven could create more
> of one instance of the test runner, create a shared list of test
> methods, and have various instances run on different threads and pick a
> test method from the list. Failure of a test does not stop execution of
> other tests in the same package, so it's safe to do so.
>
> On the build process itself, it has no meaning to parallelize it as you
> properly pointed out.
>
> Simone
>
> Lacoste, Dana (TSG Software San Diego) wrote:
> > The situation below (deploying to multiple J2EE platforms in the build)
> > is already easily supported: you're using ant to do it, and ant supports
> > a "parallel" section
> http://ant.apache.org/manual/CoreTasks/parallel.html
> >
> > So, inside a maven-antrun-plugin section you can do a <parallel> with
> > no hesitation.
> >
> > The main issue, though, gets back to one of "The Maven Way"
> >
> > Maven was designed with a very heavy emphasis on the "right" way to do
> > things, and with most projects (breaking into modules, providing
> single-file
> > artifact build results, assumed "src/java" layout, etc.) the only
> "parallel"
> > part that could really work would be when compiling the classes from the
> > **/*java files. This would theoretically result in "quicker" java
> compiling,
> > but I question if the result would be a significant gain, unless you had
> > a SERIOUSLY large number of classes in that jar! And with a 6 minute
> build,
> > you really don't :)
> >
> > For NON-JAVA src files, you can turn on the make flags appropriately
> already
> > (in other words, if maven isn't doing the build, it's just calling ant
> to
> > call make or something, then you can get the make command to
> parallelize)
> > (I remember an old comparison of gcc vs. kylix in this area: gcc
> benefits
> > from the "make -j" to such a large extent because it's really horrible
> at
> > building each file: kylix, with Pascal's simpler compile rules, was so
> much
> > faster they weren't even in the same league!)
> >
> > But that's really just "the way it is" : there's no real way that maven
> can
> > parallelize things without causing a lot of issues. Threading output of
> > the build, handling build failures, ensuring dependency order: it would
> add
> > a LOT of complexity to the build with not a lot of benefit.
> >
> > And, just for the record, my maven project has 100 (wow, 100 exactly.
> hadn't
> > counted before) modules and a full build of EVERYTHING takes about 330
> minutes
> > on my fastest build machine: if there was a way that I could speed this
> up
> > I would GLADLY do so, but there really isn't : any speedup that's caused
> by
> > a change in how maven works would result in serious usability and
> stability
> > issues that would NOT be worth it, IMNSHO.
> >
> > Dana Lacoste
> >
> > -----Original Message (Trimmed for content)-----
> > From: Jorg Heymans [mailto:jorg.heymans@gmail.com]
> > Sent: Thursday, April 10, 2008 8:36 AM
> > To: Maven Users List
> > Subject: Re: Multiple CPUs
> >
> > well imagine that during your integration build you have to deploy 10
> EARs, and each of them to a number of application servers (weblogic,
> websphere etc etc). That adds up pretty quick. I don't know the effort
> involved, but if the ant runner for example could be made to run in a
> different thread for each invocation that could speed things up drastically
> only for this use case.
> >
> > Jorg
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> > For additional commands, e-mail: users-help@maven.apache.org
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
Re: Multiple CPUs
Posted by Jorg Heymans <jo...@gmail.com>.
On Fri, Apr 11, 2008 at 7:21 PM, Simone Gianni <si...@apache.org> wrote:
> Hi Dana,
> you are absolutely right, but there are a few things Maven could do to
> improve speed : using multiple threads/cpus :
> - Maven downloads stuff. Now, since the network exists, it exists
> latency, and all net applications work best when multithreaded. Is maven
> currently parallelizing the jar downloads/the connections it makes to
> test if there are new releases of the artifacts? That could speed up a
> lot, expecially on first time users, first time they download stuff,
> first time they see maven as slow :)
> - Unit tests are, by DEFINITION, parallelizable. Maven could create more
> of one instance of the test runner, create a shared list of test
> methods, and have various instances run on different threads and pick a
> test method from the list. Failure of a test does not stop execution of
> other tests in the same package, so it's safe to do so.
>
+1 on both points.
>
> On the build process itself, it has no meaning to parallelize it as you
> properly pointed out.
>
Sorry to be persistent about this, but the build process is exactly where i
think the most gains are to be made. Parallelizing downloads is for sure a
winner and perhaps potential low-hanging fruit, but after the first clean
install you spend 98% of your time in the build _not_ downloading anything.
Instead, for each artifact you're sequentially going through all of the
lifecycle phases :
validate
generate-sources
process-sources
generate-resources
process-resources
compile
process-classes
generate-test-sources
process-test-sources
generate-test-resources
process-test-resources
test-compile
process-test-classes
test
prepare-package
package
pre-integration-test
integration-test
post-integration-test
verify
install
deploy
So why not run this cycle in parallel for completely unrelated artifacts ?
Yes I know the repo is not threadsafe but for unrelated artifacts that would
not even be an issue as long as you're not invoking 2 separate maven
instances at the same time.
Cheers,
Jorg
Re: Multiple CPUs
Posted by Simone Gianni <si...@apache.org>.
Hi Dana,
you are absolutely right, but there are a few things Maven could do to
improve speed : using multiple threads/cpus :
- Maven downloads stuff. Now, since the network exists, it exists
latency, and all net applications work best when multithreaded. Is maven
currently parallelizing the jar downloads/the connections it makes to
test if there are new releases of the artifacts? That could speed up a
lot, expecially on first time users, first time they download stuff,
first time they see maven as slow :)
- Unit tests are, by DEFINITION, parallelizable. Maven could create more
of one instance of the test runner, create a shared list of test
methods, and have various instances run on different threads and pick a
test method from the list. Failure of a test does not stop execution of
other tests in the same package, so it's safe to do so.
On the build process itself, it has no meaning to parallelize it as you
properly pointed out.
Simone
Lacoste, Dana (TSG Software San Diego) wrote:
> The situation below (deploying to multiple J2EE platforms in the build)
> is already easily supported: you're using ant to do it, and ant supports
> a "parallel" section http://ant.apache.org/manual/CoreTasks/parallel.html
>
> So, inside a maven-antrun-plugin section you can do a <parallel> with
> no hesitation.
>
> The main issue, though, gets back to one of "The Maven Way"
>
> Maven was designed with a very heavy emphasis on the "right" way to do
> things, and with most projects (breaking into modules, providing single-file
> artifact build results, assumed "src/java" layout, etc.) the only "parallel"
> part that could really work would be when compiling the classes from the
> **/*java files. This would theoretically result in "quicker" java compiling,
> but I question if the result would be a significant gain, unless you had
> a SERIOUSLY large number of classes in that jar! And with a 6 minute build,
> you really don't :)
>
> For NON-JAVA src files, you can turn on the make flags appropriately already
> (in other words, if maven isn't doing the build, it's just calling ant to
> call make or something, then you can get the make command to parallelize)
> (I remember an old comparison of gcc vs. kylix in this area: gcc benefits
> from the "make -j" to such a large extent because it's really horrible at
> building each file: kylix, with Pascal's simpler compile rules, was so much
> faster they weren't even in the same league!)
>
> But that's really just "the way it is" : there's no real way that maven can
> parallelize things without causing a lot of issues. Threading output of
> the build, handling build failures, ensuring dependency order: it would add
> a LOT of complexity to the build with not a lot of benefit.
>
> And, just for the record, my maven project has 100 (wow, 100 exactly. hadn't
> counted before) modules and a full build of EVERYTHING takes about 330 minutes
> on my fastest build machine: if there was a way that I could speed this up
> I would GLADLY do so, but there really isn't : any speedup that's caused by
> a change in how maven works would result in serious usability and stability
> issues that would NOT be worth it, IMNSHO.
>
> Dana Lacoste
>
> -----Original Message (Trimmed for content)-----
> From: Jorg Heymans [mailto:jorg.heymans@gmail.com]
> Sent: Thursday, April 10, 2008 8:36 AM
> To: Maven Users List
> Subject: Re: Multiple CPUs
>
> well imagine that during your integration build you have to deploy 10 EARs, and each of them to a number of application servers (weblogic, websphere etc etc). That adds up pretty quick. I don't know the effort involved, but if the ant runner for example could be made to run in a different thread for each invocation that could speed things up drastically only for this use case.
>
> Jorg
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
RE: Multiple CPUs
Posted by "Lacoste, Dana (TSG Software San Diego)" <da...@hp.com>.
The situation below (deploying to multiple J2EE platforms in the build)
is already easily supported: you're using ant to do it, and ant supports
a "parallel" section http://ant.apache.org/manual/CoreTasks/parallel.html
So, inside a maven-antrun-plugin section you can do a <parallel> with
no hesitation.
The main issue, though, gets back to one of "The Maven Way"
Maven was designed with a very heavy emphasis on the "right" way to do
things, and with most projects (breaking into modules, providing single-file
artifact build results, assumed "src/java" layout, etc.) the only "parallel"
part that could really work would be when compiling the classes from the
**/*java files. This would theoretically result in "quicker" java compiling,
but I question if the result would be a significant gain, unless you had
a SERIOUSLY large number of classes in that jar! And with a 6 minute build,
you really don't :)
For NON-JAVA src files, you can turn on the make flags appropriately already
(in other words, if maven isn't doing the build, it's just calling ant to
call make or something, then you can get the make command to parallelize)
(I remember an old comparison of gcc vs. kylix in this area: gcc benefits
from the "make -j" to such a large extent because it's really horrible at
building each file: kylix, with Pascal's simpler compile rules, was so much
faster they weren't even in the same league!)
But that's really just "the way it is" : there's no real way that maven can
parallelize things without causing a lot of issues. Threading output of
the build, handling build failures, ensuring dependency order: it would add
a LOT of complexity to the build with not a lot of benefit.
And, just for the record, my maven project has 100 (wow, 100 exactly. hadn't
counted before) modules and a full build of EVERYTHING takes about 330 minutes
on my fastest build machine: if there was a way that I could speed this up
I would GLADLY do so, but there really isn't : any speedup that's caused by
a change in how maven works would result in serious usability and stability
issues that would NOT be worth it, IMNSHO.
Dana Lacoste
-----Original Message (Trimmed for content)-----
From: Jorg Heymans [mailto:jorg.heymans@gmail.com]
Sent: Thursday, April 10, 2008 8:36 AM
To: Maven Users List
Subject: Re: Multiple CPUs
well imagine that during your integration build you have to deploy 10 EARs, and each of them to a number of application servers (weblogic, websphere etc etc). That adds up pretty quick. I don't know the effort involved, but if the ant runner for example could be made to run in a different thread for each invocation that could speed things up drastically only for this use case.
Jorg
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Multiple CPUs
Posted by Jorg Heymans <jo...@gmail.com>.
On Thu, Apr 10, 2008 at 5:04 PM, Wayne Fay <wa...@gmail.com> wrote:
> I think, if someone was able to hack something together (that worked!)
> and it demonstrated real added value to Maven, then the dev team would
> be more likely to hear this request and consider adding it. But there
> are so many interactions between poms, plugins, etc that I don't
> honestly believe it is worth the effort (which would be enormous).
well imagine that during your integration build you have to deploy 10 EARs,
and each of them to a number of application servers (weblogic, websphere etc
etc). That adds up pretty quick. I don't know the effort involved, but if
the ant runner for example could be made to run in a different thread for
each invocation that could speed things up drastically only for this use
case.
> So until a proof of concept is available, I think this is just a lot
> of chatter on the users list.
See that's the privilege of being a user and not a developer. You can
chatter and dream all you want about seemingly impossible-to-add features
without having to worry about the implementation or getting restricted by
your knowledge of the current codebase. Being able to do this is crucial to
the innovation of any project, so by all means folks please keep the chatter
going :-)
Jorg
Re: Multiple CPUs
Posted by VELO <ve...@gmail.com>.
Very good idea...
The same thing I'm thinking for maven itself, but if we got fastest junit is
good =D
VELO
On Thu, Apr 10, 2008 at 5:22 PM, Jorg Heymans <jo...@gmail.com>
wrote:
> On Thu, Apr 10, 2008 at 6:55 PM, Graham Leggett <mi...@sharp.fm> wrote:
>
> > maven-surefire-plugin could be taught to run tests in parallel: that
> will
> > make a huge difference.
> >
>
> Has anyone looked at using Parallel Junit [1] through surefire ? .
>
> Jorg
>
> [1] https://parallel-junit.dev.java.net/
>
Re: Multiple CPUs
Posted by Jorg Heymans <jo...@gmail.com>.
On Thu, Apr 10, 2008 at 6:55 PM, Graham Leggett <mi...@sharp.fm> wrote:
> maven-surefire-plugin could be taught to run tests in parallel: that will
> make a huge difference.
>
Has anyone looked at using Parallel Junit [1] through surefire ? .
Jorg
[1] https://parallel-junit.dev.java.net/
Re: Multiple CPUs
Posted by Graham Leggett <mi...@sharp.fm>.
VELO wrote:
> Lets just think...
>
> Today someone is reporting maven doesn't your his 2 cores.
>
> Tomorrow 4 cores.
>
> Next year 6 cores. 8, 10, what ever....
>
> Mono threaded maven will not get the max power from a multi-core machine.
>
> So, I still dreaming, may be I can pull a developer to dream together.
GNU make supports parallel building with the -j flag, makes a huge
difference on large projects and isn't really that complicated.
In multi-module builds, maven could optionally build different modules
in parallel, where those modules aren't dependent on each other (like
make -j does).
Or zooming in further, the maven-compile-plugin could compile code by
compiling multiple bits of code in parallel, the same way.
maven-surefire-plugin could be taught to run tests in parallel: that
will make a huge difference.
Regards,
Graham
--
Re: Multiple CPUs
Posted by VELO <ve...@gmail.com>.
Lets just think...
Today someone is reporting maven doesn't your his 2 cores.
Tomorrow 4 cores.
Next year 6 cores. 8, 10, what ever....
Mono threaded maven will not get the max power from a multi-core machine.
So, I still dreaming, may be I can pull a developer to dream together.
VELO
On Thu, Apr 10, 2008 at 12:04 PM, Wayne Fay <wa...@gmail.com> wrote:
> I think, if someone was able to hack something together (that worked!)
> and it demonstrated real added value to Maven, then the dev team would
> be more likely to hear this request and consider adding it. But there
> are so many interactions between poms, plugins, etc that I don't
> honestly believe it is worth the effort (which would be enormous).
>
> So until a proof of concept is available, I think this is just a lot
> of chatter on the users list.
>
> The "answer" to this "problem" for me has always been "break your code
> up into smaller modules, and only re-compile what changed" and poof,
> your compile times will be drastically reduced.
>
> Wayne
>
> On 4/10/08, VELO <ve...@gmail.com> wrote:
> > >
> > > The main problem you would have with a multi-cpu build would be
> > > understanding the console output. Since maven isn't threaded already
> > > the output would be interwoven and impossible to understand.
> > >
> >
> > Agree...
> >
> > Will only be readable on separated text logs.
> >
> > Or multi column output (just a joke =D).
> >
> >
> > VELO
> >
> >
> >
> > On Wed, Apr 9, 2008 at 8:28 PM, Barrie Treloar <ba...@gmail.com>
> wrote:
> >
> > > On Wed, Apr 9, 2008 at 8:51 PM, Benedikt Thelen <th...@gmail.com>
> wrote:
> > > > Hi there, i am sort of a maven newbee,
> > > > At our workplce we have a quite big Coccon project in developenet
> and
> > > we use
> > > > maven to build it. Building takes usually 5-6 minutes which is
> quite a
> > > > while. I noticed using gkrellm and htop that maven only uses one of
> the
> > > two
> > > > processors (Levono Thinkpad with intel core Duo) in my notebook.
> > > > Question: Is there a way to tell maven to use both CPU's while
> > > building? I
> > > > searched gooogle a lot but i didn't find anything.
> > >
> > > If you are building your entire system, including unit tests, in under
> > > 10 minutes that should be good enough.
> > >
> > > 6 minutes is fine. It gives you time to stretch your legs, go to the
> > > loo, grab a drink.
> > >
> > > The alternatives are:
> > >
> > > * manually select which modules to build, (i.e only the ones you
> > > changed) - generally it is faster to run it at the project root than
> > > cd around typing mvn commands
> > >
> > > * setup your IDE to use direct project references instead of
> > > ~/.m2/repository references - then you can develop without running
> > > maven at all !!!! You only run maven just prior to committing the
> > > changes back, which is much less often and you can afford the waste of
> > > 6 minutes.
> > >
> > > * turn off plugins for development and make sure they are on for
> > > continuous development. e.g. you may not need to run checkstyle as
> > > your IDE is already checking this.
> > >
> > >
> > > The main problem you would have with a multi-cpu build would be
> > > understanding the console output. Since maven isn't threaded already
> > > the output would be interwoven and impossible to understand.
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> > > For additional commands, e-mail: users-help@maven.apache.org
> > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
Re: Multiple CPUs
Posted by Wayne Fay <wa...@gmail.com>.
I think, if someone was able to hack something together (that worked!)
and it demonstrated real added value to Maven, then the dev team would
be more likely to hear this request and consider adding it. But there
are so many interactions between poms, plugins, etc that I don't
honestly believe it is worth the effort (which would be enormous).
So until a proof of concept is available, I think this is just a lot
of chatter on the users list.
The "answer" to this "problem" for me has always been "break your code
up into smaller modules, and only re-compile what changed" and poof,
your compile times will be drastically reduced.
Wayne
On 4/10/08, VELO <ve...@gmail.com> wrote:
> >
> > The main problem you would have with a multi-cpu build would be
> > understanding the console output. Since maven isn't threaded already
> > the output would be interwoven and impossible to understand.
> >
>
> Agree...
>
> Will only be readable on separated text logs.
>
> Or multi column output (just a joke =D).
>
>
> VELO
>
>
>
> On Wed, Apr 9, 2008 at 8:28 PM, Barrie Treloar <ba...@gmail.com> wrote:
>
> > On Wed, Apr 9, 2008 at 8:51 PM, Benedikt Thelen <th...@gmail.com> wrote:
> > > Hi there, i am sort of a maven newbee,
> > > At our workplce we have a quite big Coccon project in developenet and
> > we use
> > > maven to build it. Building takes usually 5-6 minutes which is quite a
> > > while. I noticed using gkrellm and htop that maven only uses one of the
> > two
> > > processors (Levono Thinkpad with intel core Duo) in my notebook.
> > > Question: Is there a way to tell maven to use both CPU's while
> > building? I
> > > searched gooogle a lot but i didn't find anything.
> >
> > If you are building your entire system, including unit tests, in under
> > 10 minutes that should be good enough.
> >
> > 6 minutes is fine. It gives you time to stretch your legs, go to the
> > loo, grab a drink.
> >
> > The alternatives are:
> >
> > * manually select which modules to build, (i.e only the ones you
> > changed) - generally it is faster to run it at the project root than
> > cd around typing mvn commands
> >
> > * setup your IDE to use direct project references instead of
> > ~/.m2/repository references - then you can develop without running
> > maven at all !!!! You only run maven just prior to committing the
> > changes back, which is much less often and you can afford the waste of
> > 6 minutes.
> >
> > * turn off plugins for development and make sure they are on for
> > continuous development. e.g. you may not need to run checkstyle as
> > your IDE is already checking this.
> >
> >
> > The main problem you would have with a multi-cpu build would be
> > understanding the console output. Since maven isn't threaded already
> > the output would be interwoven and impossible to understand.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> > For additional commands, e-mail: users-help@maven.apache.org
> >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Multiple CPUs
Posted by VELO <ve...@gmail.com>.
>
> The main problem you would have with a multi-cpu build would be
> understanding the console output. Since maven isn't threaded already
> the output would be interwoven and impossible to understand.
>
Agree...
Will only be readable on separated text logs.
Or multi column output (just a joke =D).
VELO
On Wed, Apr 9, 2008 at 8:28 PM, Barrie Treloar <ba...@gmail.com> wrote:
> On Wed, Apr 9, 2008 at 8:51 PM, Benedikt Thelen <th...@gmail.com> wrote:
> > Hi there, i am sort of a maven newbee,
> > At our workplce we have a quite big Coccon project in developenet and
> we use
> > maven to build it. Building takes usually 5-6 minutes which is quite a
> > while. I noticed using gkrellm and htop that maven only uses one of the
> two
> > processors (Levono Thinkpad with intel core Duo) in my notebook.
> > Question: Is there a way to tell maven to use both CPU's while
> building? I
> > searched gooogle a lot but i didn't find anything.
>
> If you are building your entire system, including unit tests, in under
> 10 minutes that should be good enough.
>
> 6 minutes is fine. It gives you time to stretch your legs, go to the
> loo, grab a drink.
>
> The alternatives are:
>
> * manually select which modules to build, (i.e only the ones you
> changed) - generally it is faster to run it at the project root than
> cd around typing mvn commands
>
> * setup your IDE to use direct project references instead of
> ~/.m2/repository references - then you can develop without running
> maven at all !!!! You only run maven just prior to committing the
> changes back, which is much less often and you can afford the waste of
> 6 minutes.
>
> * turn off plugins for development and make sure they are on for
> continuous development. e.g. you may not need to run checkstyle as
> your IDE is already checking this.
>
>
> The main problem you would have with a multi-cpu build would be
> understanding the console output. Since maven isn't threaded already
> the output would be interwoven and impossible to understand.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
Re: Multiple CPUs
Posted by Barrie Treloar <ba...@gmail.com>.
On Wed, Apr 9, 2008 at 8:51 PM, Benedikt Thelen <th...@gmail.com> wrote:
> Hi there, i am sort of a maven newbee,
> At our workplce we have a quite big Coccon project in developenet and we use
> maven to build it. Building takes usually 5-6 minutes which is quite a
> while. I noticed using gkrellm and htop that maven only uses one of the two
> processors (Levono Thinkpad with intel core Duo) in my notebook.
> Question: Is there a way to tell maven to use both CPU's while building? I
> searched gooogle a lot but i didn't find anything.
If you are building your entire system, including unit tests, in under
10 minutes that should be good enough.
6 minutes is fine. It gives you time to stretch your legs, go to the
loo, grab a drink.
The alternatives are:
* manually select which modules to build, (i.e only the ones you
changed) - generally it is faster to run it at the project root than
cd around typing mvn commands
* setup your IDE to use direct project references instead of
~/.m2/repository references - then you can develop without running
maven at all !!!! You only run maven just prior to committing the
changes back, which is much less often and you can afford the waste of
6 minutes.
* turn off plugins for development and make sure they are on for
continuous development. e.g. you may not need to run checkstyle as
your IDE is already checking this.
The main problem you would have with a multi-cpu build would be
understanding the console output. Since maven isn't threaded already
the output would be interwoven and impossible to understand.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org