You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@maven.apache.org by Ramith Jayasinghe <ra...@yahoo.com> on 2012/04/17 13:34:55 UTC
Selective Parallel Builds with maven 3
Greetings Everyone,
I'm trying to build my project in parallel using Maven 3, so I can
reduce the time taken (Currently it takes around better half of a day to completely build it).
May be this sounds stupid, but the way I want to do is to build
some of the modules in project serially while others in parallel.
So ideally, when I do a 'mvn clean install', I want maven to
parallel build parts of the project that are marked/configured for
that and others sequentially.
Is this possibe with current
capabilities of maven 3?
Thanks in advance,
- Ramith
Jayasinghe
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Selective Parallel Builds with maven 3
Posted by Srinath C <sr...@gmail.com>.
Thanks for the info Ramith. Was wondering why building them in parallel at
the root did not work but if its too much of details to mention about, its
ok.
But I have never built a subset of the modules in parallel. Building the
entire set of modules in parallel is what I have tried successfully.
Regards,
Srinath.
On Wed, Apr 18, 2012 at 11:08 AM, Ramith Jayasinghe
<ra...@opensource.lk>wrote:
> HI John and Srinath,
>
> Thanks for the reply.
>
> We have 2 code bases which I would like to build in parallel. They are [1],
> [2].
>
> [1] https://svn.wso2.org/repos/wso2/carbon/kernel/trunk/
> - Number of Java files : 4700+ (this is with out any generated
> code)
> - Modules: Entire build is structured into multiple modules
> (Hierarchy goes several levels deep)
> [2] https://svn.wso2.org/repos/wso2/carbon/platform/trunk/
> - Number of Java files : around 25000 (without generated
> code)
> - Modules: Same as [1]
>
> To explain the situation further, I tried building the code bases (in
> parallel) at the root but didn't work (due to various reasons). However, I
> figured out parts/modules can be built like this without any modification
> to the setup.
> So my idea is to rather than trying parallelize everything in one attempt (
> which I don't see realistic, considering size of the code base and
> complexity of the setup etc), I like to enable parallel builds to
> parts/modules it will work for certain.
>
> On second thoughts, it takes more than half a day to build it (includes
> tests).
>
> So, has anyone tried something like this before? (or is there a better
> approach)
>
> Thanks,
> - Ramith Jayasinghe
>
> On Tue, Apr 17, 2012 at 8:03 PM, John Patrick <nhoj.patrick@gmail.com
> >wrote:
>
> > Out of interest...
> > How many modules is that?
> > How many java files?
> > Any code generation happening like XMLBeans etc?
> > Also does that half day include or exclude test execution time?
> >
> > I worked somewhere, where the build was 3 hours not including tests,
> > by simply upgrading from maven 2.x to 3 and splitting some of the code
> > into smaller modules I managed to drop the build to 20 minutes.
> > Compiling 10 modules each with 200 java files is much quicker than
> > compiling 2,000 java files at once.
> >
> > On 17 April 2012 15:24, Srinath C <sr...@gmail.com> wrote:
> > > Wow! Half a day sounds like a huge code base!
> > >
> > > Anyways, yes you can do this in Maven 3 but maven takes the call on how
> > the
> > > modules get build. You can enforce a build order among modules using
> > > dependencies (if A depends on B, maven will build B before A) but I'm
> not
> > > aware of any other ways to enforce the order.
> > >
> > > Another notable point is to check if your plugins are compatible with
> > > parallel builds - either check their documentation or simply build with
> > "-T
> > > N" option and watch for warnings (N is the number of threads to build
> in
> > > parallel).
> > >
> > > Regards,
> > > Srinath.
> > >
> > >
> > > On Tue, Apr 17, 2012 at 5:04 PM, Ramith Jayasinghe <
> > ramithrocks@yahoo.com>wrote:
> > >
> > >>
> > >> Greetings Everyone,
> > >> I'm trying to build my project in parallel using Maven 3, so I can
> > >> reduce the time taken (Currently it takes around better half of a day
> to
> > >> completely build it).
> > >> May be this sounds stupid, but the way I want to do is to build
> > >> some of the modules in project serially while others in parallel.
> > >>
> > >> So ideally, when I do a 'mvn clean install', I want maven to
> > >> parallel build parts of the project that are marked/configured for
> > >> that and others sequentially.
> > >>
> > >> Is this possibe with current
> > >> capabilities of maven 3?
> > >>
> > >> Thanks in advance,
> > >> - Ramith
> > >> Jayasinghe
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> > >> For additional commands, e-mail: users-help@maven.apache.org
> > >>
> > >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> > For additional commands, e-mail: users-help@maven.apache.org
> >
> >
>
Re: Selective Parallel Builds with maven 3
Posted by Wayne Fay <wa...@gmail.com>.
> To explain the situation further, I tried building the code bases (in
> parallel) at the root but didn't work (due to various reasons). However, I
> figured out parts/modules can be built like this without any modification
> to the setup.
I've never used parallel builds with any real, substantial projects as yet.
I'd be curious to know more about the "various reasons" you ran into
problems since you haven't told us anything about them. Realistically
I would pick a problem, solve it, then move on to the next, repeat
until all problems are solved. ;-)
If you have a build graph issue, make sure your <dependencies> are
properly declared across the various pom files.
What specific settings were you using for your parallel build? Perhaps
you just need to tweak the parameters a bit.
Wayne
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Selective Parallel Builds with maven 3
Posted by Ramith Jayasinghe <ra...@opensource.lk>.
HI John and Srinath,
Thanks for the reply.
We have 2 code bases which I would like to build in parallel. They are [1],
[2].
[1] https://svn.wso2.org/repos/wso2/carbon/kernel/trunk/
- Number of Java files : 4700+ (this is with out any generated
code)
- Modules: Entire build is structured into multiple modules
(Hierarchy goes several levels deep)
[2] https://svn.wso2.org/repos/wso2/carbon/platform/trunk/
- Number of Java files : around 25000 (without generated
code)
- Modules: Same as [1]
To explain the situation further, I tried building the code bases (in
parallel) at the root but didn't work (due to various reasons). However, I
figured out parts/modules can be built like this without any modification
to the setup.
So my idea is to rather than trying parallelize everything in one attempt (
which I don't see realistic, considering size of the code base and
complexity of the setup etc), I like to enable parallel builds to
parts/modules it will work for certain.
On second thoughts, it takes more than half a day to build it (includes
tests).
So, has anyone tried something like this before? (or is there a better
approach)
Thanks,
- Ramith Jayasinghe
On Tue, Apr 17, 2012 at 8:03 PM, John Patrick <nh...@gmail.com>wrote:
> Out of interest...
> How many modules is that?
> How many java files?
> Any code generation happening like XMLBeans etc?
> Also does that half day include or exclude test execution time?
>
> I worked somewhere, where the build was 3 hours not including tests,
> by simply upgrading from maven 2.x to 3 and splitting some of the code
> into smaller modules I managed to drop the build to 20 minutes.
> Compiling 10 modules each with 200 java files is much quicker than
> compiling 2,000 java files at once.
>
> On 17 April 2012 15:24, Srinath C <sr...@gmail.com> wrote:
> > Wow! Half a day sounds like a huge code base!
> >
> > Anyways, yes you can do this in Maven 3 but maven takes the call on how
> the
> > modules get build. You can enforce a build order among modules using
> > dependencies (if A depends on B, maven will build B before A) but I'm not
> > aware of any other ways to enforce the order.
> >
> > Another notable point is to check if your plugins are compatible with
> > parallel builds - either check their documentation or simply build with
> "-T
> > N" option and watch for warnings (N is the number of threads to build in
> > parallel).
> >
> > Regards,
> > Srinath.
> >
> >
> > On Tue, Apr 17, 2012 at 5:04 PM, Ramith Jayasinghe <
> ramithrocks@yahoo.com>wrote:
> >
> >>
> >> Greetings Everyone,
> >> I'm trying to build my project in parallel using Maven 3, so I can
> >> reduce the time taken (Currently it takes around better half of a day to
> >> completely build it).
> >> May be this sounds stupid, but the way I want to do is to build
> >> some of the modules in project serially while others in parallel.
> >>
> >> So ideally, when I do a 'mvn clean install', I want maven to
> >> parallel build parts of the project that are marked/configured for
> >> that and others sequentially.
> >>
> >> Is this possibe with current
> >> capabilities of maven 3?
> >>
> >> Thanks in advance,
> >> - Ramith
> >> Jayasinghe
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> >> For additional commands, e-mail: users-help@maven.apache.org
> >>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
Re: Selective Parallel Builds with maven 3
Posted by John Patrick <nh...@gmail.com>.
Out of interest...
How many modules is that?
How many java files?
Any code generation happening like XMLBeans etc?
Also does that half day include or exclude test execution time?
I worked somewhere, where the build was 3 hours not including tests,
by simply upgrading from maven 2.x to 3 and splitting some of the code
into smaller modules I managed to drop the build to 20 minutes.
Compiling 10 modules each with 200 java files is much quicker than
compiling 2,000 java files at once.
On 17 April 2012 15:24, Srinath C <sr...@gmail.com> wrote:
> Wow! Half a day sounds like a huge code base!
>
> Anyways, yes you can do this in Maven 3 but maven takes the call on how the
> modules get build. You can enforce a build order among modules using
> dependencies (if A depends on B, maven will build B before A) but I'm not
> aware of any other ways to enforce the order.
>
> Another notable point is to check if your plugins are compatible with
> parallel builds - either check their documentation or simply build with "-T
> N" option and watch for warnings (N is the number of threads to build in
> parallel).
>
> Regards,
> Srinath.
>
>
> On Tue, Apr 17, 2012 at 5:04 PM, Ramith Jayasinghe <ra...@yahoo.com>wrote:
>
>>
>> Greetings Everyone,
>> I'm trying to build my project in parallel using Maven 3, so I can
>> reduce the time taken (Currently it takes around better half of a day to
>> completely build it).
>> May be this sounds stupid, but the way I want to do is to build
>> some of the modules in project serially while others in parallel.
>>
>> So ideally, when I do a 'mvn clean install', I want maven to
>> parallel build parts of the project that are marked/configured for
>> that and others sequentially.
>>
>> Is this possibe with current
>> capabilities of maven 3?
>>
>> Thanks in advance,
>> - Ramith
>> Jayasinghe
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
>> For additional commands, e-mail: users-help@maven.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Selective Parallel Builds with maven 3
Posted by Srinath C <sr...@gmail.com>.
Wow! Half a day sounds like a huge code base!
Anyways, yes you can do this in Maven 3 but maven takes the call on how the
modules get build. You can enforce a build order among modules using
dependencies (if A depends on B, maven will build B before A) but I'm not
aware of any other ways to enforce the order.
Another notable point is to check if your plugins are compatible with
parallel builds - either check their documentation or simply build with "-T
N" option and watch for warnings (N is the number of threads to build in
parallel).
Regards,
Srinath.
On Tue, Apr 17, 2012 at 5:04 PM, Ramith Jayasinghe <ra...@yahoo.com>wrote:
>
> Greetings Everyone,
> I'm trying to build my project in parallel using Maven 3, so I can
> reduce the time taken (Currently it takes around better half of a day to
> completely build it).
> May be this sounds stupid, but the way I want to do is to build
> some of the modules in project serially while others in parallel.
>
> So ideally, when I do a 'mvn clean install', I want maven to
> parallel build parts of the project that are marked/configured for
> that and others sequentially.
>
> Is this possibe with current
> capabilities of maven 3?
>
> Thanks in advance,
> - Ramith
> Jayasinghe
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>