You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by "Wippermann, Peter" <Pe...@talanx.com> on 2015/03/26 10:47:29 UTC
Multi module projects are build "too early" in a multi threaded
build, causing it to fail
Hi everybody,
I have a complex multi module build with 171 modules in total and a nesting depth of up to 4. Too speed up the build, I wanted to use the parallel build feature of Maven 3.0.5 with 4 threads:
https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
The build starts like normal and the reactor build order is correctly identified (and is identical to the single threaded one).
But the build fails shortly after, when the assembly plugin is looking for a submodule's artefact file, which does not (yet!) exist. The final print of the build status reveals, that projects where tried to build from "the middle of the reactor build order". This certainly has to fail, because their dependencies would have been earlier in the build order and haven't been built yet.
The situation is repeatable and always the same: 2 aggregators have been built successfully and 3 aggregator projects have failed.
I debugged Maven 3.0.5 to the point, that I think this is a bug, but wanted to assure with you here first.
I suspect the initialisation of the Directed Acyclic Graph (DAG) in the org.apache.maven.project.ProjectSorter Class to be faulty. I debugged it for one of my multi module projects. There I can see that for all dependencies, build plugins, build extensions and the parent, edges are added to the DAG. But not so for the modules!?
Since the modules are not considered, these aggregator projects are built very early and cause the assembly plugin to fail afterwards.
Any ideas? Is this a bug?
Best regards,
Peter Wippermann
Senior Consultant
IBM Global Business Services
Application Innovation Services - Insurance
--------------------------------------------------------------------------------
Tel: 0511-645 x58384
0151-15 16 1870
E-Mail: peter.wippermann.ext2@talanx.com
IBM Deutschland; Stockholmer Allee 30a; 44269 Dortmund
--------------------------------------------------------------------------------
IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter - Geschäftsführung: Martina Koederitz (Vorsitzende), Reinhard Reschke, Dieter Scholz, Gregor Pillen, Ivo Koerner - Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940
AW: AW: Multi module projects are build "too early" in a multi
threaded build, causing it to fail
Posted by "Wippermann, Peter" <Pe...@talanx.com>.
Hi Karl Heinz,
I used to day to play around with the build and the solution was as easy as you suggested:
In the aggregator's POM I added the modules as dependencies, so the assembly-plugin (which I had commented out before as a workaround) will find them in time.
Thanks for your help again! :-)
Best regards,
Peter Wippermann
Senior Consultant
IBM Global Business Services
Application Innovation Services - Insurance
--------------------------------------------------------------------------------
Tel: 0511-645 x58384
0151-15 16 1870
E-Mail: peter.wippermann.ext2@talanx.com
IBM Deutschland; Stockholmer Allee 30a; 44269 Dortmund
--------------------------------------------------------------------------------
IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter - Geschäftsführung: Martina Koederitz (Vorsitzende), Reinhard Reschke, Dieter Scholz, Gregor Pillen, Ivo Koerner - Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940
-----Ursprüngliche Nachricht-----
Von: Karl Heinz Marbaise [mailto:khmarbaise@gmx.de]
Gesendet: Montag, 30. März 2015 19:45
An: Wippermann, Peter; 'Maven Developers List'; 'info@soebes.de'
Betreff: Re: AW: Multi module projects are build "too early" in a multi threaded build, causing it to fail
Hi Peter,
On 3/30/15 2:54 PM, Wippermann, Peter wrote:
> Hi Karl-Heinz,
>
> thanks for your helpful answer, which brought me further.
> I was able to reproduce the behavior with " mvn clean package" as expected.
>
> Your analysis was correct, the descriptor of my assembly plugin introduced dependencies, which I didn't list in the POM itself.
>
I removed the assembly plugin call and the build is able to run.
What do you mean you have removed the call? Removed it from the pom file ?
But you should add those dependencies correctly to run the assembly correctly...Can you describe what you are doing with Maven-assembly...if you try to build a zip/jar-with-dependencies add them ...and it should work..usually you should define the assembly-plugin in a separate module....
Kind regards
Karl Heinz Marbaise
> (I still run into problems later on in the build, but these seem to
> come from custom plugins. But that's out of scope for now...)
>
> However, this workaround was only valid for testing, I still need the aggregator to build artefacts.
>
I will try to add the modules as dependencies also, to overcome this problem.
> Any other advice?
>
> Best regards,
> Peter Wippermann
>
> Senior Consultant
> IBM Global Business Services
> Application Innovation Services - Insurance
>
> --------------------------------------------------------------------------------
> Tel: 0511-645 x58384
> 0151-15 16 1870
> E-Mail: peter.wippermann.ext2@talanx.com
>
> IBM Deutschland; Stockholmer Allee 30a; 44269 Dortmund
> ----------------------------------------------------------------------
> ---------- IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats:
> Martin Jetter - Geschäftsführung: Martina Koederitz (Vorsitzende),
> Reinhard Reschke, Dieter Scholz, Gregor Pillen, Ivo Koerner - Sitz der
> Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB
> 14562 / WEEE-Reg.-Nr. DE 99369940
>
> -----Ursprüngliche Nachricht-----
> Von: Karl Heinz Marbaise [mailto:khmarbaise@gmx.de]
> Gesendet: Donnerstag, 26. März 2015 12:51
> An: Maven Developers List
> Betreff: Re: Multi module projects are build "too early" in a multi
> threaded build, causing it to fail
>
> Hi,
>
>
> the first thing to check in your build is to make a clean build which
> means empty local repository and make a
>
> mvn clean package
>
>
> to see if all dependencies etc. are correctly done otherwise you will search for phantoms...
>
> Furthermore if it is related to use maven-assembly-plugin which references artifacts in descriptors you must be sure those artifacts must be correctly referenced in the pom file as well...otherwise this can result in such problems...
>
> I assume you will get a failure in the above clean build test...
>
>
>
> Kind regards
> Karl Heinz Marbaise
>
> On 3/26/15 10:47 AM, Wippermann, Peter wrote:
>> Hi everybody,
>>
>> I have a complex multi module build with 171 modules in total and a nesting depth of up to 4. Too speed up the build, I wanted to use the parallel build feature of Maven 3.0.5 with 4 threads:
>> https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+
>> M
>> aven+3
>>
>> The build starts like normal and the reactor build order is correctly identified (and is identical to the single threaded one).
> e
>>
>> The situation is repeatable and always the same: 2 aggregators have been built successfully and 3 aggregator projects have failed.
>>
>> I debugged Maven 3.0.5 to the point, that I think this is a bug, but wanted to assure with you here first.
>> I suspect the initialisation of the Directed Acyclic Graph (DAG) in the org.apache.maven.project.ProjectSorter Class to be faulty. I debugged it for one of my multi module projects. There I can see that for all dependencies, build plugins, build extensions and the parent, edges are added to the DAG. But not so for the modules!?
>>
>> Since the modules are not considered, these aggregator projects are built very early and cause the assembly plugin to fail afterwards.
>> Any ideas? Is this a bug?
>>
>> Best regards,
>> Peter Wippermann
>>
>> Senior Consultant
>> IBM Global Business Services
>> Application Innovation Services - Insurance
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: AW: Multi module projects are build "too early" in a multi threaded
build, causing it to fail
Posted by Karl Heinz Marbaise <kh...@gmx.de>.
Hi Peter,
On 3/30/15 2:54 PM, Wippermann, Peter wrote:
> Hi Karl-Heinz,
>
> thanks for your helpful answer, which brought me further.
> I was able to reproduce the behavior with " mvn clean package" as expected.
>
> Your analysis was correct, the descriptor of my assembly plugin introduced dependencies, which I didn't list in the POM itself.
>
I removed the assembly plugin call and the build is able to run.
What do you mean you have removed the call? Removed it from the pom file ?
But you should add those dependencies correctly to run the assembly
correctly...Can you describe what you are doing with Maven-assembly...if
you try to build a zip/jar-with-dependencies add them ...and it should
work..usually you should define the assembly-plugin in a separate module....
Kind regards
Karl Heinz Marbaise
> (I still run into problems later on in the build, but these seem to come from custom plugins. But that's out of scope for now...)
>
> However, this workaround was only valid for testing, I still need the aggregator to build artefacts.
>
I will try to add the modules as dependencies also, to overcome this
problem.
> Any other advice?
>
> Best regards,
> Peter Wippermann
>
> Senior Consultant
> IBM Global Business Services
> Application Innovation Services - Insurance
>
> --------------------------------------------------------------------------------
> Tel: 0511-645 x58384
> 0151-15 16 1870
> E-Mail: peter.wippermann.ext2@talanx.com
>
> IBM Deutschland; Stockholmer Allee 30a; 44269 Dortmund
> --------------------------------------------------------------------------------
> IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter - Geschäftsführung: Martina Koederitz (Vorsitzende), Reinhard Reschke, Dieter Scholz, Gregor Pillen, Ivo Koerner - Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940
>
> -----Ursprüngliche Nachricht-----
> Von: Karl Heinz Marbaise [mailto:khmarbaise@gmx.de]
> Gesendet: Donnerstag, 26. März 2015 12:51
> An: Maven Developers List
> Betreff: Re: Multi module projects are build "too early" in a multi threaded build, causing it to fail
>
> Hi,
>
>
> the first thing to check in your build is to make a clean build which means empty local repository and make a
>
> mvn clean package
>
>
> to see if all dependencies etc. are correctly done otherwise you will search for phantoms...
>
> Furthermore if it is related to use maven-assembly-plugin which references artifacts in descriptors you must be sure those artifacts must be correctly referenced in the pom file as well...otherwise this can result in such problems...
>
> I assume you will get a failure in the above clean build test...
>
>
>
> Kind regards
> Karl Heinz Marbaise
>
> On 3/26/15 10:47 AM, Wippermann, Peter wrote:
>> Hi everybody,
>>
>> I have a complex multi module build with 171 modules in total and a nesting depth of up to 4. Too speed up the build, I wanted to use the parallel build feature of Maven 3.0.5 with 4 threads:
>> https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+M
>> aven+3
>>
>> The build starts like normal and the reactor build order is correctly identified (and is identical to the single threaded one).
> e
>>
>> The situation is repeatable and always the same: 2 aggregators have been built successfully and 3 aggregator projects have failed.
>>
>> I debugged Maven 3.0.5 to the point, that I think this is a bug, but wanted to assure with you here first.
>> I suspect the initialisation of the Directed Acyclic Graph (DAG) in the org.apache.maven.project.ProjectSorter Class to be faulty. I debugged it for one of my multi module projects. There I can see that for all dependencies, build plugins, build extensions and the parent, edges are added to the DAG. But not so for the modules!?
>>
>> Since the modules are not considered, these aggregator projects are built very early and cause the assembly plugin to fail afterwards.
>> Any ideas? Is this a bug?
>>
>> Best regards,
>> Peter Wippermann
>>
>> Senior Consultant
>> IBM Global Business Services
>> Application Innovation Services - Insurance
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
AW: Multi module projects are build "too early" in a multi threaded
build, causing it to fail
Posted by "Wippermann, Peter" <Pe...@talanx.com>.
Hi Karl-Heinz,
thanks for your helpful answer, which brought me further.
I was able to reproduce the behavior with " mvn clean package" as expected.
Your analysis was correct, the descriptor of my assembly plugin introduced dependencies, which I didn't list in the POM itself. I removed the assembly plugin call and the build is able to run.
(I still run into problems later on in the build, but these seem to come from custom plugins. But that's out of scope for now...)
However, this workaround was only valid for testing, I still need the aggregator to build artefacts. I will try to add the modules as dependencies also, to overcome this problem.
Any other advice?
Best regards,
Peter Wippermann
Senior Consultant
IBM Global Business Services
Application Innovation Services - Insurance
--------------------------------------------------------------------------------
Tel: 0511-645 x58384
0151-15 16 1870
E-Mail: peter.wippermann.ext2@talanx.com
IBM Deutschland; Stockholmer Allee 30a; 44269 Dortmund
--------------------------------------------------------------------------------
IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter - Geschäftsführung: Martina Koederitz (Vorsitzende), Reinhard Reschke, Dieter Scholz, Gregor Pillen, Ivo Koerner - Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940
-----Ursprüngliche Nachricht-----
Von: Karl Heinz Marbaise [mailto:khmarbaise@gmx.de]
Gesendet: Donnerstag, 26. März 2015 12:51
An: Maven Developers List
Betreff: Re: Multi module projects are build "too early" in a multi threaded build, causing it to fail
Hi,
the first thing to check in your build is to make a clean build which means empty local repository and make a
mvn clean package
to see if all dependencies etc. are correctly done otherwise you will search for phantoms...
Furthermore if it is related to use maven-assembly-plugin which references artifacts in descriptors you must be sure those artifacts must be correctly referenced in the pom file as well...otherwise this can result in such problems...
I assume you will get a failure in the above clean build test...
Kind regards
Karl Heinz Marbaise
On 3/26/15 10:47 AM, Wippermann, Peter wrote:
> Hi everybody,
>
> I have a complex multi module build with 171 modules in total and a nesting depth of up to 4. Too speed up the build, I wanted to use the parallel build feature of Maven 3.0.5 with 4 threads:
> https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+M
> aven+3
>
> The build starts like normal and the reactor build order is correctly identified (and is identical to the single threaded one).
e
>
> The situation is repeatable and always the same: 2 aggregators have been built successfully and 3 aggregator projects have failed.
>
> I debugged Maven 3.0.5 to the point, that I think this is a bug, but wanted to assure with you here first.
> I suspect the initialisation of the Directed Acyclic Graph (DAG) in the org.apache.maven.project.ProjectSorter Class to be faulty. I debugged it for one of my multi module projects. There I can see that for all dependencies, build plugins, build extensions and the parent, edges are added to the DAG. But not so for the modules!?
>
> Since the modules are not considered, these aggregator projects are built very early and cause the assembly plugin to fail afterwards.
> Any ideas? Is this a bug?
>
> Best regards,
> Peter Wippermann
>
> Senior Consultant
> IBM Global Business Services
> Application Innovation Services - Insurance
>
> --------------------------------------------------------------------------------
> Tel: 0511-645 x58384
> 0151-15 16 1870
> E-Mail: peter.wippermann.ext2@talanx.com
>
> IBM Deutschland; Stockholmer Allee 30a; 44269 Dortmund
> ----------------------------------------------------------------------
> ---------- IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats:
> Martin Jetter - Geschäftsführung: Martina Koederitz (Vorsitzende),
> Reinhard Reschke, Dieter Scholz, Gregor Pillen, Ivo Koerner - Sitz der
> Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB
> 14562 / WEEE-Reg.-Nr. DE 99369940
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org For additional commands, e-mail: dev-help@maven.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Multi module projects are build "too early" in a multi threaded
build, causing it to fail
Posted by Karl Heinz Marbaise <kh...@gmx.de>.
Hi,
the first thing to check in your build is to make a clean build which
means empty local repository and make a
mvn clean package
to see if all dependencies etc. are correctly done otherwise you will
search for phantoms...
Furthermore if it is related to use maven-assembly-plugin which
references artifacts in descriptors you must be sure those artifacts
must be correctly referenced in the pom file as well...otherwise this
can result in such problems...
I assume you will get a failure in the above clean build test...
Kind regards
Karl Heinz Marbaise
On 3/26/15 10:47 AM, Wippermann, Peter wrote:
> Hi everybody,
>
> I have a complex multi module build with 171 modules in total and a nesting depth of up to 4. Too speed up the build, I wanted to use the parallel build feature of Maven 3.0.5 with 4 threads:
> https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
>
> The build starts like normal and the reactor build order is correctly identified (and is identical to the single threaded one).
e
>
> The situation is repeatable and always the same: 2 aggregators have been built successfully and 3 aggregator projects have failed.
>
> I debugged Maven 3.0.5 to the point, that I think this is a bug, but wanted to assure with you here first.
> I suspect the initialisation of the Directed Acyclic Graph (DAG) in the org.apache.maven.project.ProjectSorter Class to be faulty. I debugged it for one of my multi module projects. There I can see that for all dependencies, build plugins, build extensions and the parent, edges are added to the DAG. But not so for the modules!?
>
> Since the modules are not considered, these aggregator projects are built very early and cause the assembly plugin to fail afterwards.
> Any ideas? Is this a bug?
>
> Best regards,
> Peter Wippermann
>
> Senior Consultant
> IBM Global Business Services
> Application Innovation Services - Insurance
>
> --------------------------------------------------------------------------------
> Tel: 0511-645 x58384
> 0151-15 16 1870
> E-Mail: peter.wippermann.ext2@talanx.com
>
> IBM Deutschland; Stockholmer Allee 30a; 44269 Dortmund
> --------------------------------------------------------------------------------
> IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter - Geschäftsführung: Martina Koederitz (Vorsitzende), Reinhard Reschke, Dieter Scholz, Gregor Pillen, Ivo Koerner - Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Multi module projects are build "too early" in a multi threaded
build, causing it to fail
Posted by ig...@ifedorenko.com.
I still suggest you try the latest version of Maven. At very least this
will tell you if you see new or known-and-fixed problem.
Same for the Smart Builder, you'll know if it actually improves build
times much for your specific project (forgot to mention, Smart Builder
requires java 7 and is silently ignored by Maven if you are on an older
java version).
--
Regards,
Igor
On Thu, Mar 26, 2015, at 07:28 AM, Wippermann, Peter wrote:
> Hi Igor,
>
> thanks for your advice. I also considered upgrading Maven first, but this
> is not possible for us at the moment due to enterprise constraints.
> If the Takari Smart Builder isn't compatible to 3.0.5 I can't try it. :-/
>
> Best regards,
> Peter Wippermann
>
> Senior Consultant
> IBM Global Business Services
> Application Innovation Services - Insurance
>
> --------------------------------------------------------------------------------
> Tel: 0511-645 x58384
> 0151-15 16 1870
> E-Mail: peter.wippermann.ext2@talanx.com
>
> IBM Deutschland; Stockholmer Allee 30a; 44269 Dortmund
> --------------------------------------------------------------------------------
> IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter -
> Geschäftsführung: Martina Koederitz (Vorsitzende), Reinhard Reschke,
> Dieter Scholz, Gregor Pillen, Ivo Koerner - Sitz der Gesellschaft:
> Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 /
> WEEE-Reg.-Nr. DE 99369940
>
>
> -----Ursprüngliche Nachricht-----
> Von: igor@ifedorenko.com [mailto:igor@ifedorenko.com]
> Gesendet: Donnerstag, 26. März 2015 12:24
> An: dev@maven.apache.org
> Betreff: Re: Multi module projects are build "too early" in a multi
> threaded build, causing it to fail
>
> Maven 3.0.5 is kinda old, do you see the same behaviour with the latest
> 3.3.1 release? I believe there were changes in parallel build, but don't
> know if they address your specific problem.
>
> You may also want to try Takari Smart Builder [1]. It schedules build of
> the longest dependency chain first (called "build critical path"), which
> results in significant reduction of overall build time for some projects.
> I can help with setup (which is much easier with Maven 3.3.1,
> btw) and other questions here or on Takari User google group.
>
> [1] https://github.com/takari/takari-smart-builder
>
> --
> Regards,
> Igor
>
> On Thu, Mar 26, 2015, at 05:47 AM, Wippermann, Peter wrote:
> > Hi everybody,
> >
> > I have a complex multi module build with 171 modules in total and a
> > nesting depth of up to 4. Too speed up the build, I wanted to use the
> > parallel build feature of Maven 3.0.5 with 4 threads:
> > https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+M
> > aven+3
> >
> > The build starts like normal and the reactor build order is correctly
> > identified (and is identical to the single threaded one).
> > But the build fails shortly after, when the assembly plugin is looking
> > for a submodule's artefact file, which does not (yet!) exist. The
> > final print of the build status reveals, that projects where tried to
> > build from "the middle of the reactor build order". This certainly has
> > to fail, because their dependencies would have been earlier in the
> > build order and haven't been built yet.
> >
> > The situation is repeatable and always the same: 2 aggregators have
> > been built successfully and 3 aggregator projects have failed.
> >
> > I debugged Maven 3.0.5 to the point, that I think this is a bug, but
> > wanted to assure with you here first.
> > I suspect the initialisation of the Directed Acyclic Graph (DAG) in
> > the org.apache.maven.project.ProjectSorter Class to be faulty. I
> > debugged it for one of my multi module projects. There I can see that
> > for all dependencies, build plugins, build extensions and the parent,
> > edges are added to the DAG. But not so for the modules!?
> >
> > Since the modules are not considered, these aggregator projects are
> > built very early and cause the assembly plugin to fail afterwards.
> > Any ideas? Is this a bug?
> >
> > Best regards,
> > Peter Wippermann
> >
> > Senior Consultant
> > IBM Global Business Services
> > Application Innovation Services - Insurance
> >
> > --------------------------------------------------------------------------------
> > Tel: 0511-645 x58384
> > 0151-15 16 1870
> > E-Mail: peter.wippermann.ext2@talanx.com
> >
> > IBM Deutschland; Stockholmer Allee 30a; 44269 Dortmund
> > ----------------------------------------------------------------------
> > ---------- IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats:
> > Martin Jetter -
> > Geschäftsführung: Martina Koederitz (Vorsitzende), Reinhard Reschke,
> > Dieter Scholz, Gregor Pillen, Ivo Koerner - Sitz der Gesellschaft:
> > Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 /
> > WEEE-Reg.-Nr. DE 99369940
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org For additional
> commands, e-mail: dev-help@maven.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
AW: Multi module projects are build "too early" in a multi threaded
build, causing it to fail
Posted by "Wippermann, Peter" <Pe...@talanx.com>.
Hi Igor,
thanks for your advice. I also considered upgrading Maven first, but this is not possible for us at the moment due to enterprise constraints.
If the Takari Smart Builder isn't compatible to 3.0.5 I can't try it. :-/
Best regards,
Peter Wippermann
Senior Consultant
IBM Global Business Services
Application Innovation Services - Insurance
--------------------------------------------------------------------------------
Tel: 0511-645 x58384
0151-15 16 1870
E-Mail: peter.wippermann.ext2@talanx.com
IBM Deutschland; Stockholmer Allee 30a; 44269 Dortmund
--------------------------------------------------------------------------------
IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter - Geschäftsführung: Martina Koederitz (Vorsitzende), Reinhard Reschke, Dieter Scholz, Gregor Pillen, Ivo Koerner - Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940
-----Ursprüngliche Nachricht-----
Von: igor@ifedorenko.com [mailto:igor@ifedorenko.com]
Gesendet: Donnerstag, 26. März 2015 12:24
An: dev@maven.apache.org
Betreff: Re: Multi module projects are build "too early" in a multi threaded build, causing it to fail
Maven 3.0.5 is kinda old, do you see the same behaviour with the latest
3.3.1 release? I believe there were changes in parallel build, but don't know if they address your specific problem.
You may also want to try Takari Smart Builder [1]. It schedules build of the longest dependency chain first (called "build critical path"), which results in significant reduction of overall build time for some projects. I can help with setup (which is much easier with Maven 3.3.1,
btw) and other questions here or on Takari User google group.
[1] https://github.com/takari/takari-smart-builder
--
Regards,
Igor
On Thu, Mar 26, 2015, at 05:47 AM, Wippermann, Peter wrote:
> Hi everybody,
>
> I have a complex multi module build with 171 modules in total and a
> nesting depth of up to 4. Too speed up the build, I wanted to use the
> parallel build feature of Maven 3.0.5 with 4 threads:
> https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+M
> aven+3
>
> The build starts like normal and the reactor build order is correctly
> identified (and is identical to the single threaded one).
> But the build fails shortly after, when the assembly plugin is looking
> for a submodule's artefact file, which does not (yet!) exist. The
> final print of the build status reveals, that projects where tried to
> build from "the middle of the reactor build order". This certainly has
> to fail, because their dependencies would have been earlier in the
> build order and haven't been built yet.
>
> The situation is repeatable and always the same: 2 aggregators have
> been built successfully and 3 aggregator projects have failed.
>
> I debugged Maven 3.0.5 to the point, that I think this is a bug, but
> wanted to assure with you here first.
> I suspect the initialisation of the Directed Acyclic Graph (DAG) in
> the org.apache.maven.project.ProjectSorter Class to be faulty. I
> debugged it for one of my multi module projects. There I can see that
> for all dependencies, build plugins, build extensions and the parent,
> edges are added to the DAG. But not so for the modules!?
>
> Since the modules are not considered, these aggregator projects are
> built very early and cause the assembly plugin to fail afterwards.
> Any ideas? Is this a bug?
>
> Best regards,
> Peter Wippermann
>
> Senior Consultant
> IBM Global Business Services
> Application Innovation Services - Insurance
>
> --------------------------------------------------------------------------------
> Tel: 0511-645 x58384
> 0151-15 16 1870
> E-Mail: peter.wippermann.ext2@talanx.com
>
> IBM Deutschland; Stockholmer Allee 30a; 44269 Dortmund
> ----------------------------------------------------------------------
> ---------- IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats:
> Martin Jetter -
> Geschäftsführung: Martina Koederitz (Vorsitzende), Reinhard Reschke,
> Dieter Scholz, Gregor Pillen, Ivo Koerner - Sitz der Gesellschaft:
> Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 /
> WEEE-Reg.-Nr. DE 99369940
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org For additional commands, e-mail: dev-help@maven.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Multi module projects are build "too early" in a multi threaded
build, causing it to fail
Posted by ig...@ifedorenko.com.
Maven 3.0.5 is kinda old, do you see the same behaviour with the latest
3.3.1 release? I believe there were changes in parallel build, but don't
know if they address your specific problem.
You may also want to try Takari Smart Builder [1]. It schedules build of
the longest dependency chain first (called "build critical path"), which
results in significant reduction of overall build time for some
projects. I can help with setup (which is much easier with Maven 3.3.1,
btw) and other questions here or on Takari User google group.
[1] https://github.com/takari/takari-smart-builder
--
Regards,
Igor
On Thu, Mar 26, 2015, at 05:47 AM, Wippermann, Peter wrote:
> Hi everybody,
>
> I have a complex multi module build with 171 modules in total and a
> nesting depth of up to 4. Too speed up the build, I wanted to use the
> parallel build feature of Maven 3.0.5 with 4 threads:
> https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
>
> The build starts like normal and the reactor build order is correctly
> identified (and is identical to the single threaded one).
> But the build fails shortly after, when the assembly plugin is looking
> for a submodule's artefact file, which does not (yet!) exist. The final
> print of the build status reveals, that projects where tried to build
> from "the middle of the reactor build order". This certainly has to fail,
> because their dependencies would have been earlier in the build order and
> haven't been built yet.
>
> The situation is repeatable and always the same: 2 aggregators have been
> built successfully and 3 aggregator projects have failed.
>
> I debugged Maven 3.0.5 to the point, that I think this is a bug, but
> wanted to assure with you here first.
> I suspect the initialisation of the Directed Acyclic Graph (DAG) in the
> org.apache.maven.project.ProjectSorter Class to be faulty. I debugged it
> for one of my multi module projects. There I can see that for all
> dependencies, build plugins, build extensions and the parent, edges are
> added to the DAG. But not so for the modules!?
>
> Since the modules are not considered, these aggregator projects are built
> very early and cause the assembly plugin to fail afterwards.
> Any ideas? Is this a bug?
>
> Best regards,
> Peter Wippermann
>
> Senior Consultant
> IBM Global Business Services
> Application Innovation Services - Insurance
>
> --------------------------------------------------------------------------------
> Tel: 0511-645 x58384
> 0151-15 16 1870
> E-Mail: peter.wippermann.ext2@talanx.com
>
> IBM Deutschland; Stockholmer Allee 30a; 44269 Dortmund
> --------------------------------------------------------------------------------
> IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter -
> Geschäftsführung: Martina Koederitz (Vorsitzende), Reinhard Reschke,
> Dieter Scholz, Gregor Pillen, Ivo Koerner - Sitz der Gesellschaft:
> Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 /
> WEEE-Reg.-Nr. DE 99369940
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org