Re: parallelise not overlapping tasks

[Romain Manni-Bucau <rm...@gmail.com>]

Hi guys,

I did a quick PoC on a small project. It parallelize the mojo per phase
with a config taken from project properties - not sexy but enables to start
"seeing" something.
You can find it at https://github.com/rmannibucau/parallel-mojo-extension

The main issue it enounters is the fact it can't reuse the ExecutorService
(or its CompletionService wrapper) from the MultiThreadedBuilder - since it
is not done in maven core - and the config which is pretty heavy at the
moment.

However the interesting part is that it is doable, the two main challenges
being:

1. ensure the plugins are safe (not sure threadSafe can be reused, I would
assume that yes but some structure can be not thread safe so not fully sure
and it would be ok while mojo are not parallelize in a single module)
2. find a nice configuration - maybe in terms of dependency between mojo?

The 2 is interesting cause it asks the question of phases which would be
less useful in that context.

Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |  Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>


Le sam. 8 déc. 2018 à 13:14, Romain Manni-Bucau <rm...@gmail.com> a
écrit :

> Using the scm is not enough or it is for single module projects.
>
> You have to have a graph of dependencies (inputs/outputs) and save each
> task state in target to have incremental support
>
> But please note incremental build is != parallel build at mojo level.
>
> This last one is easy to do and a quick win IMHO
>
> Le sam. 8 déc. 2018 00:50, Tibor Digana <ti...@apache.org> a écrit :
>
>> In my projects, the most plugins use single execution.
>> External projects also have this kind of principle.
>> Thus we should have a look in those possibilities where the most plugins
>> can gain the performance.
>> Usually the compiler and tests take long.
>> I know that maven-compiler-plugin:3.8.1 will be incremental which is good
>> of course but we should perhaps continue with gaining the build
>> performance.
>> If somebody has an idea on how to develop a compiler which partially
>> compiles a module depending on SCM changes, feel free to bring it to our
>> mailing list. The same with tests where the set of tests is changed
>> depending on SCM changes.
>>
>> Cheers
>> Tibor
>>
>>
>> On Fri, Jan 19, 2018 at 2:21 PM Romain Manni-Bucau <rmannibucau@gmail.com
>> >
>> wrote:
>>
>> > Hi guys,
>> >
>> > there is no way to parallelize not conflicting tasks for a same phase at
>> > the moment right? Any way it gets under the radar?
>> >
>> > A common example is to run all code analyzis concurrently (findbugs,
>> pmd,
>> > checkstyle, ...) at the same time without waiting for one then the other
>> > etc since all can be very long.
>> >
>> > It could be nice an fancy to define part of the reactor
>> parallelisablity in
>> > the pom, like:
>> >
>> > <plan>
>> >     <parallel>
>> >         <phase>process-sources</phase>
>> >         <executions>
>> >             <execution>
>> >               <groupId>...</groupId>
>> >               <artifactId>...</artifactId>
>> >               <version>...</version>
>> >               <id>...</id>
>> >             <execution>
>> >             <execution>
>> >               <groupId>...</groupId>
>> >               <artifactId>...</artifactId>
>> >               <version>...</version>
>> >               <id>...</id>
>> >             <execution>
>> >         </executions>
>> >     </parallel>
>> > </plan>
>> >
>> > anything to enhance it?
>> >
>> > Romain Manni-Bucau
>> > @rmannibucau <https://twitter.com/rmannibucau> |  Blog
>> > <https://rmannibucau.metawerx.net/> | Old Blog
>> > <http://rmannibucau.wordpress.com> | Github <
>> > https://github.com/rmannibucau> |
>> > LinkedIn <https://www.linkedin.com/in/rmannibucau>
>> >
>>
>