You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "Nigel Magnay (JIRA)" <ji...@codehaus.org> on 2007/05/22 16:47:58 UTC
[jira] Created: (MNG-3004) Allow build lifecycle to execute tasks
in parallel
Allow build lifecycle to execute tasks in parallel
--------------------------------------------------
Key: MNG-3004
URL: http://jira.codehaus.org/browse/MNG-3004
Project: Maven 2
Issue Type: Improvement
Components: Bootstrap & Build, General, Performance
Affects Versions: 2.0.6
Reporter: Nigel Magnay
Attachments: parallel-builds.patch
One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3004) Allow build lifecycle to execute
projects in parallel
Posted by "Arnaud Heritier (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=215204#action_215204 ]
Arnaud Heritier commented on MNG-3004:
--------------------------------------
Nothing new since december ?
> Allow build lifecycle to execute projects in parallel
> -----------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2 & 3
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 3.0-alpha-8
>
> Attachments: MNG-3004-Resurrected-StringSearchModelInterpolatorTest.patch, MNG-3004.increased-testability.patch, MNG3004-SSMI.patch, mng3004.patch, mng3004v2_rev2.patch, parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3004) Allow build lifecycle to execute tasks
in parallel
Posted by "Nigel Magnay (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=159982#action_159982 ]
Nigel Magnay commented on MNG-3004:
-----------------------------------
My original intent was so that all activities could run in parallel, to the granularity of a particular project.
I.E if projects A B and C all depend on project X (and nothing else), then after 'mvn install' finishes on 'X', 'mvn install' can be executed, in parallel, on each of A, B and C.
I stopped trying to get it to work when I realised that the biggest problem is that the local repository isn't safe to be accessed from multiple threads (or even multiple processes for that matter). I think that's a known issue too - I'm guessing that the ability to somehow lock/unlock repository objects (or perhaps use some kind of smarter local repository) is a dependency to getting this to work.
> Allow build lifecycle to execute tasks in parallel
> --------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 2.1.0-M3
>
> Attachments: parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MNG-3004) Allow build lifecycle to execute
projects in parallel
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Rosenvold updated MNG-3004:
------------------------------------
Attachment: mng3004v2_rev2.patch
The enclosed mng39004v2_rev2.patch updates javadoc and removes a little cruft that had managed to creep into the patch.
> Allow build lifecycle to execute projects in parallel
> -----------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2 & 3
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 2.3.x
>
> Attachments: MNG-3004-Resurrected-StringSearchModelInterpolatorTest.patch, MNG-3004.increased-testability.patch, MNG3004-SSMI.patch, mng3004.patch, mng3004v2_rev2.patch, parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3004) Allow build lifecycle to execute
projects in parallel
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=217168#action_217168 ]
Kristian Rosenvold commented on MNG-3004:
-----------------------------------------
r931884:
[MNG-3004] Concurrent building of multimodule projects
This commit consists of a modularization of the DefaultLifecycleExecutor into smaller
components. This is primarily done with the intention of extracting execution strategy
somewhat from the rest of the logic.
It also contains three differenent execution strategies:
A) Linear ("Classic")
B) Parallel (Based on Dan's original implemementation)
C) Weave (Still experimental)
The command line -T option activates parallel running, and the argument
is in the form -T <[nThreads, a float][C][W]|auto>
Notes:
* The output demultiplexer is disabled in this commit, and output will appear intermingled.
* The modello plugin has known problems
* Surefire must be run in forkmode at least once.
> Allow build lifecycle to execute projects in parallel
> -----------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2 & 3
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Assignee: Kristian Rosenvold
> Fix For: 3.0-beta-1
>
> Attachments: MNG-3004-Resurrected-StringSearchModelInterpolatorTest.patch, MNG-3004.increased-testability.patch, MNG3004-SSMI.patch, mng3004.patch, mng3004v2_rev2.patch, parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MNG-3004) Allow build lifecycle to execute tasks
in parallel
Posted by "Brett Porter (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brett Porter updated MNG-3004:
------------------------------
Patch Submitted: [Yes]
> Allow build lifecycle to execute tasks in parallel
> --------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: Reviewed Pending Version Assignment
>
> Attachments: parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (MNG-3004) Allow build lifecycle to
execute tasks in parallel
Posted by "Hans-Peter Störr (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=168455#action_168455 ]
Hans-Peter Störr edited comment on MNG-3004 at 3/11/09 2:17 AM:
----------------------------------------------------------------
Good point. But I am not sure that locking is a good way to do this - locking has its dangers. Perhaps it is possible to make parallel updates harmless. Such as downoading the files under a different name and renaming it to the right name only afterwards.
I think however that parallel builds can be done even now without locking / repository separation by executing a two phase strategy:
1. Update and download all needed dependencies and plugins. (Singlethreaded, although you might do parallel downloads.)
2. Execute a parallel build without any updates / downloads
Perhaps a singlethreaded mvn validate and then a parallel build with maven in offline mode would do this? Maybe you could do a restricted version of parallel builds that only work in offline mode for now, and do the full implementation later when locking or something is done.
I've heard hudson already does parallel builds with maven. Anyone knows how they handle this problem?
was (Author: hstoerr):
Good point. But I am not sure that locking is a good way to do this - locking has its dangers.
Perhaps it is possible to make parallel updates harmless. Such as downoading the files under a different name and renaming it to the right name only afterwards.
For the parallel build however I would suggest a two phase strategy:
1. Update and download all needed dependencies and plugins. (Singlethreaded, perhaps except for the actual downloads.)
2. Execute a parallel build without any updates / downloads
Perhaps a singlethreaded mvn validate and then a parallel build with maven in offline mode would do this?
I've heard hudson already does parallel builds with maven. Anyone knows how they handle this problem?
> Allow build lifecycle to execute tasks in parallel
> --------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 2.2.0-M1
>
> Attachments: parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (MNG-3004) Allow build lifecycle to
execute projects in parallel
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=198467#action_198467 ]
Kristian Rosenvold edited comment on MNG-3004 at 11/17/09 5:08 AM:
-------------------------------------------------------------------
The enclosed patch contains the StringSearchModelInterpolatorTest and its base class I ressurected from the maven 2.2.X build. The test also contains an upgraded concurrency test that fails most of the time unless the patch is applied.
There are a large number of tests in the base class AbstractModelInterpolatorTest that failed. I have commented these tests out, since I have no idea if these should work on the 3.x line.
It would be appreciated if someone took a quick look at the commented-out tests and made a quick assesment if these are still relevant and how they can be made to pass again.
I suspect that some of them fail because I added new File(".") as a don't-know value to the interpolate method
was (Author: krosenvold):
The enclosed patch contains the StringSearchModelInterpolatorTest and its base class I ressurected from the maven 2.2.X build. The test also contains an upgraded concurrency test that fails most of the time unless the patch is applied.
There are a large number of tests in the base class AbstractModelInterpolatorTest that failed. I have commented these tests out, since I have no idea if these should work on the 3.x line.
It would be appreciated if someone took a quick look at the commented-out tests and made a quick assesment if these are still relevant and how they can be made to pass again.
> Allow build lifecycle to execute projects in parallel
> -----------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 2.3.x
>
> Attachments: MNG-3004-Resurrected-StringSearchModelInterpolatorTest.patch, MNG-3004.increased-testability.patch, MNG3004-SSMI.patch, parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3004) Allow build lifecycle to execute tasks
in parallel
Posted by "Hans-Peter Störr (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=170970#action_170970 ]
Hans-Peter Störr commented on MNG-3004:
---------------------------------------
How about implementing a restricted version of parallel builds that only works in offline mode for now? (See my suggestion above.)
I might take a while and more than one release to get something like this right, so it might be better to do at least something now, without waiting for other stuff to happen. And it would be usable right now. If the locking / repository separation / whatever is done later, one can just remove the restriction and - boom - you are there.
> Allow build lifecycle to execute tasks in parallel
> --------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 2.2.0-M1
>
> Attachments: parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MNG-3004) Allow build lifecycle to execute
projects in parallel
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Rosenvold updated MNG-3004:
------------------------------------
Attachment: mng3004.patch
Please accept this patch, that allows "weave" style execution of maven reactor builds.
This patch is the result of a long community discussion and serves as a fully functional patch that is targeted at a specific subset of builds - typically enterprisey reactor builds.
> Allow build lifecycle to execute projects in parallel
> -----------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2 & 3
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 2.3.x
>
> Attachments: MNG-3004-Resurrected-StringSearchModelInterpolatorTest.patch, MNG-3004.increased-testability.patch, MNG3004-SSMI.patch, mng3004.patch, parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MNG-3004) Allow build lifecycle to execute
projects in parallel
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Rosenvold updated MNG-3004:
------------------------------------
Attachment: MNG-3004.increased-testability.patch
After reviewing Dan Fabulich's concurrency update to M3, I extracted some of the functionality into a separate class and wrote a fairly extensive unit test for that class. this patch applies to the MNG-3004 branch.
> Allow build lifecycle to execute projects in parallel
> -----------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 2.3.x
>
> Attachments: MNG-3004-Resurrected-StringSearchModelInterpolatorTest.patch, MNG-3004.increased-testability.patch, MNG3004-SSMI.patch, parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (MNG-3004) Allow build lifecycle to
execute projects in parallel
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=198467#action_198467 ]
Kristian Rosenvold edited comment on MNG-3004 at 11/17/09 5:37 AM:
-------------------------------------------------------------------
The enclosed patch contains the StringSearchModelInterpolatorTest and its base class I ressurected from the maven 2.2.X build. The test also contains an upgraded concurrency test that fails most of the time unless the patch is applied.
There are a large number of tests in the base class AbstractModelInterpolatorTest that failed. I have commented these tests out, since I have no idea if these should work on the 3.x line.
It would be appreciated if someone took a quick look at the commented-out tests and made a quick assesment if these are still relevant and how they can be made to pass again.
I suspect that some of them fail because I added new File(".") as a don't-know value to the interpolate method
If MNG-3004-Resurrected-StringSearchModelInterpolatorTest.patch is applied *before* MNG3004-SSMI.patch, you can have a failing test ;)
was (Author: krosenvold):
The enclosed patch contains the StringSearchModelInterpolatorTest and its base class I ressurected from the maven 2.2.X build. The test also contains an upgraded concurrency test that fails most of the time unless the patch is applied.
There are a large number of tests in the base class AbstractModelInterpolatorTest that failed. I have commented these tests out, since I have no idea if these should work on the 3.x line.
It would be appreciated if someone took a quick look at the commented-out tests and made a quick assesment if these are still relevant and how they can be made to pass again.
I suspect that some of them fail because I added new File(".") as a don't-know value to the interpolate method
> Allow build lifecycle to execute projects in parallel
> -----------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 2.3.x
>
> Attachments: MNG-3004-Resurrected-StringSearchModelInterpolatorTest.patch, MNG-3004.increased-testability.patch, MNG3004-SSMI.patch, parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3004) Allow build lifecycle to execute
projects in parallel
Posted by "Christian Gleissner (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=219340#action_219340 ]
Christian Gleissner commented on MNG-3004:
------------------------------------------
This feature works very well and resulted in a significant build time reduction, depending on the number of cores. Why was the output de-multiplexer commented for Maven 3 Beta 1 and when will this very useful feature be re-enabled?
Also, when performing a parallel build, TeamCity 5's automatic collection of Maven Surefire results no longer finds all results. We are using an external invocation of Maven 3. This may be related to how TeamCity checks for the completion of modules by scanning the log file and could go away once the output de-multiplexing will be re-enabled.
> Allow build lifecycle to execute projects in parallel
> -----------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2 & 3
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Assignee: Kristian Rosenvold
> Fix For: 3.0-beta-1
>
> Attachments: MNG-3004-Resurrected-StringSearchModelInterpolatorTest.patch, MNG-3004.increased-testability.patch, MNG3004-SSMI.patch, mng3004.patch, mng3004v2_rev2.patch, parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MNG-3004) Allow build lifecycle to execute tasks
in parallel
Posted by "Brett Porter (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brett Porter updated MNG-3004:
------------------------------
Fix Version/s: (was: Reviewed Pending Version Assignment)
2.1.0-M3
> Allow build lifecycle to execute tasks in parallel
> --------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 2.1.0-M3
>
> Attachments: parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3004) Allow build lifecycle to execute tasks
in parallel
Posted by "Hans-Peter Störr (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=167398#action_167398 ]
Hans-Peter Störr commented on MNG-3004:
---------------------------------------
Nigel: could you please explain the need for locking if you are doing this? I'd think that each of the builds of A, B and C writes to separate directories - so there is no need for locking. They will want to access directories of their dependencies, but of course those would have been built before. Where are the possible conflicts?
> Allow build lifecycle to execute tasks in parallel
> --------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 2.2.0-M1
>
> Attachments: parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MNG-3004) Allow build lifecycle to execute
projects in parallel
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Rosenvold updated MNG-3004:
------------------------------------
Attachment: MNG-3004-Resurrected-StringSearchModelInterpolatorTest.patch
The enclosed patch contains the StringSearchModelInterpolatorTest and its base class I ressurected from the maven 2.2.X build. The test also contains an upgraded concurrency test that fails most of the time unless the patch is applied.
There are a large number of tests in the base class AbstractModelInterpolatorTest that failed. I have commented these tests out, since I have no idea if these should work on the 3.x line.
It would be appreciated if someone took a quick look at the commented-out tests and made a quick assesment if these are still relevant and how they can be made to pass again.
> Allow build lifecycle to execute projects in parallel
> -----------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 2.3.x
>
> Attachments: MNG-3004-Resurrected-StringSearchModelInterpolatorTest.patch, MNG3004-SSMI.patch, parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3004) Allow build lifecycle to execute tasks
in parallel
Posted by "Nigel Magnay (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=168006#action_168006 ]
Nigel Magnay commented on MNG-3004:
-----------------------------------
If A, B or C all rely on a downloadable artifact X (or, X-SNAPSHOT), then building them in parallel may result in a parallel update of X.
You need to be able to fix it in place for the duration of your build.
> Allow build lifecycle to execute tasks in parallel
> --------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 2.2.0-M1
>
> Attachments: parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3004) Allow build lifecycle to execute tasks
in parallel
Posted by "Nigel Magnay (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=168878#action_168878 ]
Nigel Magnay commented on MNG-3004:
-----------------------------------
> I've heard hudson already does parallel builds with maven. Anyone knows how they handle this problem?
It doesn't.
Hudson initially *only* built in parallel with the native m2 support; compiling serially (like the commandline) was only added later when it was clear that for many projects, parallel compilation breaks, basically, all the time.
What's even worse (from an m2 perspective) is that, because the local repository is not thread- or process- safe, there is a "Use Private Maven Repository" option so that each project has its own, private local repo. Fixes the problem, but this means for us we've something approaching 200Gb of duplicated jar files sitting around in various directories. Yuck.
This is known about - see http://docs.codehaus.org/display/MAVEN/Local+repository+separation .
The simple fact of the matter is that the local repository _ought_ to be threadsafe (either inherently by design, or through some kind of use of locking) but it isn't. Hacks that don't address that fundamental issue aren't really fixing the problem, IMO.
> Allow build lifecycle to execute tasks in parallel
> --------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 2.2.0-M1
>
> Attachments: parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3004) Allow build lifecycle to execute tasks
in parallel
Posted by "Nicolas Frenay (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=195045#action_195045 ]
Nicolas Frenay commented on MNG-3004:
-------------------------------------
I agree with Hans-Peter Störr idea.
If someone is able to attack this problem now, it could be done in a way that it's only available if you're offline. When the thread-safe issue is fixed, this new feature will be ready to go.
It also gives extra-time to test performance with parallelization, as it will probably require some "tweaking". Configurable maximum number of threads comes to my mind.
> Allow build lifecycle to execute tasks in parallel
> --------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 2.3.x
>
> Attachments: parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3004) Allow build lifecycle to execute tasks
in parallel
Posted by "Wendy Smoak (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=168891#action_168891 ]
Wendy Smoak commented on MNG-3004:
----------------------------------
Does the work on parallel artifact resolution in a single build (MNG-3379) help here?
> Allow build lifecycle to execute tasks in parallel
> --------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 2.2.0-M1
>
> Attachments: parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Closed: (MNG-3004) Allow build lifecycle to execute projects
in parallel
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Rosenvold closed MNG-3004.
-----------------------------------
Resolution: Fixed
> Allow build lifecycle to execute projects in parallel
> -----------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2 & 3
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Assignee: Kristian Rosenvold
> Fix For: 3.0-beta-1
>
> Attachments: MNG-3004-Resurrected-StringSearchModelInterpolatorTest.patch, MNG-3004.increased-testability.patch, MNG3004-SSMI.patch, mng3004.patch, mng3004v2_rev2.patch, parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MNG-3004) Allow build lifecycle to execute
projects in parallel
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Rosenvold updated MNG-3004:
------------------------------------
Attachment: MNG3004-SSMI.patch
The attached patch fixes a concurrency issue when running with the maven 3.0 MNG3004 branch. The main issue is that parallel threads will modify the protected/private status of the fields in the class, and concurrent threads will create problems for each other.
If I can find out how to make a failing test I will do this too.
> Allow build lifecycle to execute projects in parallel
> -----------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 2.3.x
>
> Attachments: MNG3004-SSMI.patch, parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MNG-3004) Allow build lifecycle to execute
projects in parallel
Posted by "Brett Porter (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brett Porter updated MNG-3004:
------------------------------
Fix Version/s: (was: 2.3.x)
3.0-alpha-7
this is a patch against trunk.
> Allow build lifecycle to execute projects in parallel
> -----------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2 & 3
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 3.0-alpha-7
>
> Attachments: MNG-3004-Resurrected-StringSearchModelInterpolatorTest.patch, MNG-3004.increased-testability.patch, MNG3004-SSMI.patch, mng3004.patch, mng3004v2_rev2.patch, parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MNG-3004) Allow build lifecycle to execute tasks
in parallel
Posted by "John Casey (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Casey updated MNG-3004:
----------------------------
Fix Version/s: (was: 2.1.0-M3)
2.2.0-M1
We don't have code in place to do file locking in the local repository, nor do we have a design for this...I can't see how we'll get this done ahead of 2.2 at this point.
> Allow build lifecycle to execute tasks in parallel
> --------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 2.2.0-M1
>
> Attachments: parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3004) Allow build lifecycle to execute tasks
in parallel
Posted by "Pi Song (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=165670#action_165670 ]
Pi Song commented on MNG-3004:
------------------------------
Given an assumption that most artifacts aren't changed very often, can't we just say "run in parallel if all the required artifacts are in place?"
Or even can be better if say "run in parallel if two (or more) projects don't share artifacts that have to be downloaded in common"
> Allow build lifecycle to execute tasks in parallel
> --------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 2.2.0-M1
>
> Attachments: parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Work started: (MNG-3004) Allow build lifecycle to execute
projects in parallel
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on MNG-3004 started by Kristian Rosenvold.
> Allow build lifecycle to execute projects in parallel
> -----------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2 & 3
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Assignee: Kristian Rosenvold
> Fix For: 3.0-beta-1
>
> Attachments: MNG-3004-Resurrected-StringSearchModelInterpolatorTest.patch, MNG-3004.increased-testability.patch, MNG3004-SSMI.patch, mng3004.patch, mng3004v2_rev2.patch, parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3004) Allow build lifecycle to execute tasks
in parallel
Posted by "Hans-Peter Störr (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=168455#action_168455 ]
Hans-Peter Störr commented on MNG-3004:
---------------------------------------
Good point. But I am not sure that locking is a good way to do this - locking has its dangers.
Perhaps it is possible to make parallel updates harmless. Such as downoading the files under a different name and renaming it to the right name only afterwards.
For the parallel build however I would suggest a two phase strategy:
1. Update and download all needed dependencies and plugins. (Singlethreaded, perhaps except for the actual downloads.)
2. Execute a parallel build without any updates / downloads
Perhaps a singlethreaded mvn validate and then a parallel build with maven in offline mode would do this?
I've heard hudson already does parallel builds with maven. Anyone knows how they handle this problem?
> Allow build lifecycle to execute tasks in parallel
> --------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 2.2.0-M1
>
> Attachments: parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3004) Allow build lifecycle to execute
projects in parallel
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=219348#action_219348 ]
Kristian Rosenvold commented on MNG-3004:
-----------------------------------------
Thank you, Christian, nice with some feedback.
The output demultiplexer that is present in the code base was only intended as a stop-gap, and is architecturally not the proper solution. MNG-2727 is targeted at beta-2 and we are awaiting its completion. In the event that MNG-2727 does not resolve demultiplexing of the output in parallel build, there will be another issue (with a likely target of beta-2) to fix this.
Regarding your team-city issue I will not be surprised if they're parsing the output somehow, so I can imagine it being slightly confused.
Please also note that there are several other issues being fixed in the maven ecosystem related to stable running of parallel tests, and I'll try to keep this updated in the "relates to section" of this issue. The workaround until updated plugins are released will normally involve adding dependencies to newer library versions to your plugins at the moment.
> Allow build lifecycle to execute projects in parallel
> -----------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2 & 3
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Assignee: Kristian Rosenvold
> Fix For: 3.0-beta-1
>
> Attachments: MNG-3004-Resurrected-StringSearchModelInterpolatorTest.patch, MNG-3004.increased-testability.patch, MNG3004-SSMI.patch, mng3004.patch, mng3004v2_rev2.patch, parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3004) Allow build lifecycle to execute
projects in parallel
Posted by "Paul Benedict (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=217205#action_217205 ]
Paul Benedict commented on MNG-3004:
------------------------------------
With concurrency, is the "downloading dependency" progress still readable? If X downloads are occurring, I hope the output isn't difficult to track.
> Allow build lifecycle to execute projects in parallel
> -----------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2 & 3
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Assignee: Kristian Rosenvold
> Fix For: 3.0-beta-1
>
> Attachments: MNG-3004-Resurrected-StringSearchModelInterpolatorTest.patch, MNG-3004.increased-testability.patch, MNG3004-SSMI.patch, mng3004.patch, mng3004v2_rev2.patch, parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3004) Allow build lifecycle to execute tasks
in parallel
Posted by "THURNER rupert (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=158615#action_158615 ]
THURNER rupert commented on MNG-3004:
-------------------------------------
parallel compiling of files is the main intention of this feature? or does it go further and also would start tests which can be already run?
> Allow build lifecycle to execute tasks in parallel
> --------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 2.1.0-M3
>
> Attachments: parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MNG-3004) Allow build lifecycle to execute
projects in parallel
Posted by "Dan Fabulich (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dan Fabulich updated MNG-3004:
------------------------------
Summary: Allow build lifecycle to execute projects in parallel (was: Allow build lifecycle to execute tasks in parallel)
> Allow build lifecycle to execute projects in parallel
> -----------------------------------------------------
>
> Key: MNG-3004
> URL: http://jira.codehaus.org/browse/MNG-3004
> Project: Maven 2
> Issue Type: Improvement
> Components: Bootstrap & Build, General, Performance
> Affects Versions: 2.0.6
> Reporter: Nigel Magnay
> Fix For: 2.3.x
>
> Attachments: parallel-builds.patch
>
>
> One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.
> Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.
> I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.
> It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira