You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@maven.apache.org by Francois Marot <fr...@gmail.com> on 2024/11/14 13:32:46 UTC
Multithreaded behavior when building a subset of modules
Hello,
I just faced a problem and would like your input if it is really a problem
or not.
I have a multimodule project where dependencies are like:
A --> B --> C --> D (A depends upon B which depends upon C and so on...)
I would have expected those 2 commands to exhibit the exact same behaviour:
1- mvn package -pl :A,:C -T 1C
2- mvn package -pl :A,:C
but the problem is that when command 1 is run, both projects are built
simultaneously (the logs are interweaved). It should not be the case
because A depends transitively upon C.
When command 2 is run, project C is built before project A which is what I
expect.
I'm using mvn 3.9.9
Best regards
*- - - - -François Marot06 50 91 96 38*
Re: Multithreaded behavior when building a subset of modules
Posted by Francois Marot <fr...@gmail.com>.
THANK YOU Tama, you rock !
I've downloaded latest SNAPSHOT from here (
https://repository.apache.org/content/repositories/snapshots/org/apache/maven/apache-maven/3.9.10-SNAPSHOT/
) and on a very simple example this works as expected.
My test is really simple so I do not pretend to validate the whole thing,
but at least this use-case is now correct.
I think this fix is REALLY important as I expect that some builds may be
corrupted (or the output may not match the code being compiled) in some
case.
Thanks !
Le jeu. 14 nov. 2024 à 15:20, Tamás Cservenák <ta...@cservenak.net> a
écrit :
> Eh, sorry, am a bit overwhelmed....
>
> It seems this is FIXED in Maven-3.9.x branch? :)
>
> https://github.com/apache/maven/commit/45201347c417896b57159221130333f5fa4bbfb6
>
> So once 3.9.10 is out, it should be workable? Can you test the
> maven-3.9.x branch?
>
> Thanks
> T
>
> On Thu, Nov 14, 2024 at 3:14 PM Tamás Cservenák <ta...@cservenak.net>
> wrote:
> >
> > Maven4 fixes this issue, yes, it is "aware" differently than Maven3 is
> > (where excluded projects are fully removed from line of sight).
> >
> > In Maven3 is not doable sadly, at least not with some simple change. I
> > did some experiments about this long time ago, see
> > https://github.com/apache/maven/pull/1406
> >
> > T
> >
> > On Thu, Nov 14, 2024 at 3:12 PM Francois Marot <fr...@gmail.com>
> wrote:
> > >
> > > Thanks for the info Tamás.
> > > Do you know if:
> > > 1- a workaround is planned ?
> > > 2- Maven 4 will solve this problem ?
> > > 3- commands like "-pl :myModule -am " also suffer from this kind of
> problem
> > > ?
> > >
> > > I'm trying to convince everyone at work to move to a large Monorepo
> and my
> > > argument is "learn how to use Maven and your builds will not be
> longer" but
> > > I may have to stop immediately before it is too late !
> > >
> > > Best regards
> > >
> > >
> > > Le jeu. 14 nov. 2024 à 14:56, Tamás Cservenák <ta...@cservenak.net> a
> > > écrit :
> > >
> > > > Howdy,
> > > >
> > > > This is a problem with Maven 3 and how it "reduces" the reactor (when
> > > > using -rf, -pl etc, basically when reactor < full project).
> > > > This was already discussed once on this ML but let me shortly explain
> > > > what happens:
> > > >
> > > > You basically "moved" B out of the reactor, and your "snake" looks
> like
> > > > this:
> > > >
> > > > A (in reactor) -> B (out of reactor, like any external dep) -> C (in
> > > > reactor).
> > > >
> > > > Maven 3 cannot follow these "out then back in" links, and assumes A
> > > > and C are unrelated.
> > > >
> > > > The 1st command simply "loses" the link between A and C, because B
> got
> > > > out of the reactor and built them in parallel.
> > > > The 2nd command works as it is single threaded, but in fact,
> > > > internally, it also does not "see" the relation.
> > > >
> > > >
> > > >
> > > > On Thu, Nov 14, 2024 at 2:33 PM Francois Marot <
> francois.marot@gmail.com>
> > > > wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > I just faced a problem and would like your input if it is really a
> > > > problem
> > > > > or not.
> > > > > I have a multimodule project where dependencies are like:
> > > > >
> > > > > A --> B --> C --> D (A depends upon B which depends upon C and
> so
> > > > on...)
> > > > >
> > > > > I would have expected those 2 commands to exhibit the exact same
> > > > behaviour:
> > > > >
> > > > > 1- mvn package -pl :A,:C -T 1C
> > > > > 2- mvn package -pl :A,:C
> > > > >
> > > > > but the problem is that when command 1 is run, both projects are
> built
> > > > > simultaneously (the logs are interweaved). It should not be the
> case
> > > > > because A depends transitively upon C.
> > > > > When command 2 is run, project C is built before project A which
> is what
> > > > I
> > > > > expect.
> > > > >
> > > > > I'm using mvn 3.9.9
> > > > >
> > > > > Best regards
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *- - - - -François Marot06 50 91 96 38*
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> > > > For additional commands, e-mail: users-help@maven.apache.org
> > > >
> > > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
Re: Multithreaded behavior when building a subset of modules
Posted by Tamás Cservenák <ta...@cservenak.net>.
Eh, sorry, am a bit overwhelmed....
It seems this is FIXED in Maven-3.9.x branch? :)
https://github.com/apache/maven/commit/45201347c417896b57159221130333f5fa4bbfb6
So once 3.9.10 is out, it should be workable? Can you test the
maven-3.9.x branch?
Thanks
T
On Thu, Nov 14, 2024 at 3:14 PM Tamás Cservenák <ta...@cservenak.net> wrote:
>
> Maven4 fixes this issue, yes, it is "aware" differently than Maven3 is
> (where excluded projects are fully removed from line of sight).
>
> In Maven3 is not doable sadly, at least not with some simple change. I
> did some experiments about this long time ago, see
> https://github.com/apache/maven/pull/1406
>
> T
>
> On Thu, Nov 14, 2024 at 3:12 PM Francois Marot <fr...@gmail.com> wrote:
> >
> > Thanks for the info Tamás.
> > Do you know if:
> > 1- a workaround is planned ?
> > 2- Maven 4 will solve this problem ?
> > 3- commands like "-pl :myModule -am " also suffer from this kind of problem
> > ?
> >
> > I'm trying to convince everyone at work to move to a large Monorepo and my
> > argument is "learn how to use Maven and your builds will not be longer" but
> > I may have to stop immediately before it is too late !
> >
> > Best regards
> >
> >
> > Le jeu. 14 nov. 2024 à 14:56, Tamás Cservenák <ta...@cservenak.net> a
> > écrit :
> >
> > > Howdy,
> > >
> > > This is a problem with Maven 3 and how it "reduces" the reactor (when
> > > using -rf, -pl etc, basically when reactor < full project).
> > > This was already discussed once on this ML but let me shortly explain
> > > what happens:
> > >
> > > You basically "moved" B out of the reactor, and your "snake" looks like
> > > this:
> > >
> > > A (in reactor) -> B (out of reactor, like any external dep) -> C (in
> > > reactor).
> > >
> > > Maven 3 cannot follow these "out then back in" links, and assumes A
> > > and C are unrelated.
> > >
> > > The 1st command simply "loses" the link between A and C, because B got
> > > out of the reactor and built them in parallel.
> > > The 2nd command works as it is single threaded, but in fact,
> > > internally, it also does not "see" the relation.
> > >
> > >
> > >
> > > On Thu, Nov 14, 2024 at 2:33 PM Francois Marot <fr...@gmail.com>
> > > wrote:
> > > >
> > > > Hello,
> > > >
> > > > I just faced a problem and would like your input if it is really a
> > > problem
> > > > or not.
> > > > I have a multimodule project where dependencies are like:
> > > >
> > > > A --> B --> C --> D (A depends upon B which depends upon C and so
> > > on...)
> > > >
> > > > I would have expected those 2 commands to exhibit the exact same
> > > behaviour:
> > > >
> > > > 1- mvn package -pl :A,:C -T 1C
> > > > 2- mvn package -pl :A,:C
> > > >
> > > > but the problem is that when command 1 is run, both projects are built
> > > > simultaneously (the logs are interweaved). It should not be the case
> > > > because A depends transitively upon C.
> > > > When command 2 is run, project C is built before project A which is what
> > > I
> > > > expect.
> > > >
> > > > I'm using mvn 3.9.9
> > > >
> > > > Best regards
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *- - - - -François Marot06 50 91 96 38*
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> > > For additional commands, e-mail: users-help@maven.apache.org
> > >
> > >
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Multithreaded behavior when building a subset of modules
Posted by Tamás Cservenák <ta...@cservenak.net>.
Maven4 fixes this issue, yes, it is "aware" differently than Maven3 is
(where excluded projects are fully removed from line of sight).
In Maven3 is not doable sadly, at least not with some simple change. I
did some experiments about this long time ago, see
https://github.com/apache/maven/pull/1406
T
On Thu, Nov 14, 2024 at 3:12 PM Francois Marot <fr...@gmail.com> wrote:
>
> Thanks for the info Tamás.
> Do you know if:
> 1- a workaround is planned ?
> 2- Maven 4 will solve this problem ?
> 3- commands like "-pl :myModule -am " also suffer from this kind of problem
> ?
>
> I'm trying to convince everyone at work to move to a large Monorepo and my
> argument is "learn how to use Maven and your builds will not be longer" but
> I may have to stop immediately before it is too late !
>
> Best regards
>
>
> Le jeu. 14 nov. 2024 à 14:56, Tamás Cservenák <ta...@cservenak.net> a
> écrit :
>
> > Howdy,
> >
> > This is a problem with Maven 3 and how it "reduces" the reactor (when
> > using -rf, -pl etc, basically when reactor < full project).
> > This was already discussed once on this ML but let me shortly explain
> > what happens:
> >
> > You basically "moved" B out of the reactor, and your "snake" looks like
> > this:
> >
> > A (in reactor) -> B (out of reactor, like any external dep) -> C (in
> > reactor).
> >
> > Maven 3 cannot follow these "out then back in" links, and assumes A
> > and C are unrelated.
> >
> > The 1st command simply "loses" the link between A and C, because B got
> > out of the reactor and built them in parallel.
> > The 2nd command works as it is single threaded, but in fact,
> > internally, it also does not "see" the relation.
> >
> >
> >
> > On Thu, Nov 14, 2024 at 2:33 PM Francois Marot <fr...@gmail.com>
> > wrote:
> > >
> > > Hello,
> > >
> > > I just faced a problem and would like your input if it is really a
> > problem
> > > or not.
> > > I have a multimodule project where dependencies are like:
> > >
> > > A --> B --> C --> D (A depends upon B which depends upon C and so
> > on...)
> > >
> > > I would have expected those 2 commands to exhibit the exact same
> > behaviour:
> > >
> > > 1- mvn package -pl :A,:C -T 1C
> > > 2- mvn package -pl :A,:C
> > >
> > > but the problem is that when command 1 is run, both projects are built
> > > simultaneously (the logs are interweaved). It should not be the case
> > > because A depends transitively upon C.
> > > When command 2 is run, project C is built before project A which is what
> > I
> > > expect.
> > >
> > > I'm using mvn 3.9.9
> > >
> > > Best regards
> > >
> > >
> > >
> > >
> > >
> > > *- - - - -François Marot06 50 91 96 38*
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> > For additional commands, e-mail: users-help@maven.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Multithreaded behavior when building a subset of modules
Posted by Francois Marot <fr...@gmail.com>.
Thanks for the info Tamás.
Do you know if:
1- a workaround is planned ?
2- Maven 4 will solve this problem ?
3- commands like "-pl :myModule -am " also suffer from this kind of problem
?
I'm trying to convince everyone at work to move to a large Monorepo and my
argument is "learn how to use Maven and your builds will not be longer" but
I may have to stop immediately before it is too late !
Best regards
Le jeu. 14 nov. 2024 à 14:56, Tamás Cservenák <ta...@cservenak.net> a
écrit :
> Howdy,
>
> This is a problem with Maven 3 and how it "reduces" the reactor (when
> using -rf, -pl etc, basically when reactor < full project).
> This was already discussed once on this ML but let me shortly explain
> what happens:
>
> You basically "moved" B out of the reactor, and your "snake" looks like
> this:
>
> A (in reactor) -> B (out of reactor, like any external dep) -> C (in
> reactor).
>
> Maven 3 cannot follow these "out then back in" links, and assumes A
> and C are unrelated.
>
> The 1st command simply "loses" the link between A and C, because B got
> out of the reactor and built them in parallel.
> The 2nd command works as it is single threaded, but in fact,
> internally, it also does not "see" the relation.
>
>
>
> On Thu, Nov 14, 2024 at 2:33 PM Francois Marot <fr...@gmail.com>
> wrote:
> >
> > Hello,
> >
> > I just faced a problem and would like your input if it is really a
> problem
> > or not.
> > I have a multimodule project where dependencies are like:
> >
> > A --> B --> C --> D (A depends upon B which depends upon C and so
> on...)
> >
> > I would have expected those 2 commands to exhibit the exact same
> behaviour:
> >
> > 1- mvn package -pl :A,:C -T 1C
> > 2- mvn package -pl :A,:C
> >
> > but the problem is that when command 1 is run, both projects are built
> > simultaneously (the logs are interweaved). It should not be the case
> > because A depends transitively upon C.
> > When command 2 is run, project C is built before project A which is what
> I
> > expect.
> >
> > I'm using mvn 3.9.9
> >
> > Best regards
> >
> >
> >
> >
> >
> > *- - - - -François Marot06 50 91 96 38*
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
Re: Multithreaded behavior when building a subset of modules
Posted by Tamás Cservenák <ta...@cservenak.net>.
Howdy,
This is a problem with Maven 3 and how it "reduces" the reactor (when
using -rf, -pl etc, basically when reactor < full project).
This was already discussed once on this ML but let me shortly explain
what happens:
You basically "moved" B out of the reactor, and your "snake" looks like this:
A (in reactor) -> B (out of reactor, like any external dep) -> C (in reactor).
Maven 3 cannot follow these "out then back in" links, and assumes A
and C are unrelated.
The 1st command simply "loses" the link between A and C, because B got
out of the reactor and built them in parallel.
The 2nd command works as it is single threaded, but in fact,
internally, it also does not "see" the relation.
On Thu, Nov 14, 2024 at 2:33 PM Francois Marot <fr...@gmail.com> wrote:
>
> Hello,
>
> I just faced a problem and would like your input if it is really a problem
> or not.
> I have a multimodule project where dependencies are like:
>
> A --> B --> C --> D (A depends upon B which depends upon C and so on...)
>
> I would have expected those 2 commands to exhibit the exact same behaviour:
>
> 1- mvn package -pl :A,:C -T 1C
> 2- mvn package -pl :A,:C
>
> but the problem is that when command 1 is run, both projects are built
> simultaneously (the logs are interweaved). It should not be the case
> because A depends transitively upon C.
> When command 2 is run, project C is built before project A which is what I
> expect.
>
> I'm using mvn 3.9.9
>
> Best regards
>
>
>
>
>
> *- - - - -François Marot06 50 91 96 38*
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org