TomEE maven build analyses: Why only "mvn install" works and what to do about it

[Benjamin Marwell <bm...@apache.org>]

Hello everyone!

Richard asked me a few days ago to take a look at the TomEE maven
build, as it takes up to a few hours to complete. One issue is the
jenkins setup, granted. But I have two findings.

1. shading

Because a few deps, e.g. Apache MQ RA Client, still do not provide
artifacts with the jakarta namespace, they created a module
"deps/activemq-ra-shaded", which does that. A few transformers later
they have a shaded dependency in the Jakarta namespace.

However:
This will mean that modules depending on that will only compile if the
artifact has undergone the packaging phase, as the m-shade-plugin will
operate on packaged artifacts only. Thus, `mvn compile` will fail due
to the required shaded/transformed artifact not existing yet.

Proposed remedy:
Do the required transformations in the compile phase.
Question: Is there already a plugin capable of this?
Possible drawbacks: Re-Packaging gets even more complicated, as the
existing transformers are not available, e.g. for META-INF services
and other resources.
Link: https://github.com/apache/tomee/blob/69fbe3d2e1bc8a268382d2c04ba901919afad3b4/deps/activemq-client-shade/pom.xml#L64-L95
And others.


2. Verification on installed artifact

The TomEE team wants to test the generated .zip archive. For the
tests, it needs to be unpacked multiple times per module. That means,
currently they get the installed artifact from the LOCAL repository
(yes, $HOME/.m2/repository) while in the verify phase of other
modules. This also means that `mvn verify` will not work. It also
means cluttering of the repo, complicated Path finding magic, etc.

Proposed remedy:
Modify / enhance the m-dependency-plugin to make specific artifacts
paths form the reactor available via a system property. Maybe one
chosen property per selected/configured artifact, maybe one global
property, colon-seperated for all of them. Or both. This will make
"mvn verify" work again since no previous copying to the local
repository is needed.

Question: Are there currently alternative strategies to find .zip
files from the reactor while ITs are running via the failsafe plugin?

Links:
a. Path finding:
https://github.com/apache/tomee/blob/69fbe3d2e1bc8a268382d2c04ba901919afad3b4/arquillian/ziplock/src/main/java/org/apache/ziplock/JarLocation.java
b. Custom mvn execution:
https://github.com/apache/tomee/blob/69fbe3d2e1bc8a268382d2c04ba901919afad3b4/arquillian/ziplock/src/main/java/org/apache/ziplock/maven/Mvn.java
c: Maven cache:
https://github.com/apache/tomee/blob/main/arquillian/arquillian-tomee-common/src/main/java/org/apache/openejb/arquillian/common/MavenCache.java
d: Tests: https://github.com/apache/tomee/blob/main/itests/microprofile-jwt-itests/src/test/java/org/apache/tomee/microprofile/jwt/itest/keys/http/HttpKeyInitializationHttp500Test.java
e: Tests 2: https://github.com/apache/tomee/blob/main/itests/failover/src/test/java/org/apache/openejb/itest/failover/RoundRobinConnectionStrategyTest.java


So, what do you think? In my opinion the setup could be "healed" from
not being able to execute compile, test-compile, test, verify by going
the proposed actions. But maybe there are other possibilities you
know. If so, I would be happy to hear them.

If modification of the mentioned plugins is necessary, I would be
willing to code them and cut a 3.x release of those plugins.

Slack channel: #tomee

Have a merry christmas time!
- Ben

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Re: TomEE maven build analyses: Why only "mvn install" works and what to do about it

[Romain Manni-Bucau <rm...@gmail.com>]

Few comments inline

Le ven. 23 déc. 2022 à 13:23, Benjamin Marwell <bm...@apache.org> a
écrit :

> Hello everyone!
>
> Richard asked me a few days ago to take a look at the TomEE maven
> build, as it takes up to a few hours to complete. One issue is the
> jenkins setup, granted. But I have two findings.
>
> 1. shading
>
> Because a few deps, e.g. Apache MQ RA Client, still do not provide
> artifacts with the jakarta namespace, they created a module
> "deps/activemq-ra-shaded", which does that. A few transformers later
> they have a shaded dependency in the Jakarta namespace.
>

Is it an issue?
By the past these deps were done outside tomee project but wefe imported to
release faster so can just be a nasty temporary state but sounds expected
by design if the modules.
Can be done at compile time but will not chnage much things by design again
- we speak of the produced artifact here, we dont want a fork i think.



> However:
> This will mean that modules depending on that will only compile if the
> artifact has undergone the packaging phase, as the m-shade-plugin will
> operate on packaged artifacts only. Thus, `mvn compile` will fail due
> to the required shaded/transformed artifact not existing yet.
>
> Proposed remedy:
> Do the required transformations in the compile phase.
> Question: Is there already a plugin capable of this?
> Possible drawbacks: Re-Packaging gets even more complicated, as the
> existing transformers are not available, e.g. for META-INF services
> and other resources.
> Link:
> https://github.com/apache/tomee/blob/69fbe3d2e1bc8a268382d2c04ba901919afad3b4/deps/activemq-client-shade/pom.xml#L64-L95
> And others.
>
>
> 2. Verification on installed artifact
>
> The TomEE team wants to test the generated .zip archive. For the
> tests, it needs to be unpacked multiple times per module. That means,
> currently they get the installed artifact from the LOCAL repository
> (yes, $HOME/.m2/repository) while in the verify phase of other
> modules. This also means that `mvn verify` will not work. It also
> means cluttering of the repo, complicated Path finding magic, etc.
>

Not really, tomee already handles that is you configure it - it is even
done for some modules but easier to not do so nothing to enhance at maven
level IMO.
Also the magic is required to support other build systems too like gradle
ant and standalone projects.

The other thing is using tomee maven plugin can simplify all assemblies a
lot since it will enable to mix precomoyted distro and oatch them inline
instead of relying on maven assembly which is a bit slower but was never
done by lack of interest afaik.


> Proposed remedy:
> Modify / enhance the m-dependency-plugin to make specific artifacts
> paths form the reactor available via a system property. Maybe one
> chosen property per selected/configured artifact, maybe one global
> property, colon-seperated for all of them. Or both. This will make
> "mvn verify" work again since no previous copying to the local
> repository is needed.
>
> Question: Are there currently alternative strategies to find .zip
> files from the reactor while ITs are running via the failsafe plugin?
>
> Links:
> a. Path finding:
>
> https://github.com/apache/tomee/blob/69fbe3d2e1bc8a268382d2c04ba901919afad3b4/arquillian/ziplock/src/main/java/org/apache/ziplock/JarLocation.java
> b. Custom mvn execution:
>
> https://github.com/apache/tomee/blob/69fbe3d2e1bc8a268382d2c04ba901919afad3b4/arquillian/ziplock/src/main/java/org/apache/ziplock/maven/Mvn.java
> c: Maven cache:
>
> https://github.com/apache/tomee/blob/main/arquillian/arquillian-tomee-common/src/main/java/org/apache/openejb/arquillian/common/MavenCache.java
> d: Tests:
> https://github.com/apache/tomee/blob/main/itests/microprofile-jwt-itests/src/test/java/org/apache/tomee/microprofile/jwt/itest/keys/http/HttpKeyInitializationHttp500Test.java
> e: Tests 2:
> https://github.com/apache/tomee/blob/main/itests/failover/src/test/java/org/apache/openejb/itest/failover/RoundRobinConnectionStrategyTest.java
>
>
> So, what do you think? In my opinion the setup could be "healed" from
> not being able to execute compile, test-compile, test, verify by going
> the proposed actions. But maybe there are other possibilities you
> know. If so, I would be happy to hear them.
>
> If modification of the mentioned plugins is necessary, I would be
> willing to code them and cut a 3.x release of those plugins.
>
> Slack channel: #tomee
>
> Have a merry christmas time!
> - Ben
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>