You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Tamás Cservenák <ta...@cservenak.net> on 2021/02/10 08:50:03 UTC
Plugin dependency resolution
Howdy,
Some time ago I stumbled upon Robert S. remark in "[DISCUSS] MNG-7020
Remove Maven 2 WagonExcluder backward compat code" thread: "Why download,
if they are being removed from the classpath afterwards due to classworld
config". Similarly, there is a thread "maven-site-plugin and
sisu-inject-plexus" discussing that plugin pulls down ancient
plexus-container-default (10+ years ago dropped). And finally, we all know
about "maven downloads the whole internet" maxim :)
Hence, I wanted to check this out. My "experiment" was not about maven
"speed" (whatever you mean by it), but more about it's "bandwidth" usage.
My setup:
- am using (primed) MRM, not interested in bashing Central or measuring my
ISP network speed
- am always nuking local repo (hence, starting state is "get everything
needed for build")
- my test bed project is maven itself
(master, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6)
- build + tests ARE executed, and succeed OK, but I was really interested
in local repository post-build state
- command line am executing from maven checkout root is: rm -r /tmp/repo;
~/bin/maven/apache-maven-XXX/bin/mvn clean install
-Dmaven.repo.local=/tmp/repo
Remarks: times and the whole "experiment' is not scientific, they are just
there for "rough comparison" sake. I did not repeat builds multiple times
(to have some "mean time"), as I was not interested in time, but did repeat
enough times to ensure that local repository state (file count, file sizes)
are consistent.
Results:
maven 3.6.3 (released):
total time 1:45 min (as Maven reports)
total files in local repo: 3743
total bytes in local repo: 162600
count of files having "plexus-container-default" in local repo: 37 (18
POM, 10 JAR)
mvn master (4.0.0-alpha-1, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6):
total time 1:08 min (as Maven reports)
total files in local repo: 3741
total bytes in local repo: 162428
count of files having "plexus-container-default" in local repo: 37 (18
POM, 10 JAR)
so, pretty much the same. Then I went to create a (hacked for now) patch,
as can be seen here:
https://github.com/apache/maven/compare/master...cstamas:plugin-resolution-hack
results with above patched mvn master:
total time 1:04 min (as Maven reports)
total files in local repo: 2992
total bytes in local repo: 149740
count of files having "plexus-container-default" in local repo: 0
Basically patch "cut" almost 1k of unnecessary file downloads, 10+MB byte
downloads in case of Maven being built.
Is there some issue in JIRA covering this behaviour of Maven (I could not
find any)?
Have fun,
Tamas
Re: Plugin dependency resolution
Posted by Tamás Cservenák <ta...@cservenak.net>.
Just for fun, tried it on another, a bit bigger project (137 modules), and
as 3.6.3 and master was "so close", tried master and patch only:
mvn master (4.0.0-alpha-1, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6)
total time 1:49 min
total files in local repo: 7679
total bytes in local repo: 652624
count of files having "plexus-container-default" in local repo: 37
patched mvn master
total time 1:33 min
total files in local repo: 6531
total bytes in local repo: 624568
count of files having "plexus-container-default" in local repo: 0
Some preliminary conclusions: plexus-container-default count is SAME as
with Maven build, so it is most probably some "common" plugin(s) pulling it
in...
Again, around 1.1k unnecessary downloads prevented, which surprises me...
So, to me the "gain" looks constant.... so maybe not really worth the
effort?
T
On Wed, Feb 10, 2021 at 9:50 AM Tamás Cservenák <ta...@cservenak.net> wrote:
> Howdy,
>
> Some time ago I stumbled upon Robert S. remark in "[DISCUSS] MNG-7020
> Remove Maven 2 WagonExcluder backward compat code" thread: "Why download,
> if they are being removed from the classpath afterwards due to classworld
> config". Similarly, there is a thread "maven-site-plugin and
> sisu-inject-plexus" discussing that plugin pulls down ancient
> plexus-container-default (10+ years ago dropped). And finally, we all know
> about "maven downloads the whole internet" maxim :)
>
> Hence, I wanted to check this out. My "experiment" was not about maven
> "speed" (whatever you mean by it), but more about it's "bandwidth" usage.
>
> My setup:
> - am using (primed) MRM, not interested in bashing Central or measuring my
> ISP network speed
> - am always nuking local repo (hence, starting state is "get everything
> needed for build")
> - my test bed project is maven itself
> (master, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6)
> - build + tests ARE executed, and succeed OK, but I was really interested
> in local repository post-build state
> - command line am executing from maven checkout root is: rm -r /tmp/repo;
> ~/bin/maven/apache-maven-XXX/bin/mvn clean install
> -Dmaven.repo.local=/tmp/repo
>
> Remarks: times and the whole "experiment' is not scientific, they are just
> there for "rough comparison" sake. I did not repeat builds multiple times
> (to have some "mean time"), as I was not interested in time, but did repeat
> enough times to ensure that local repository state (file count, file sizes)
> are consistent.
>
> Results:
>
> maven 3.6.3 (released):
> total time 1:45 min (as Maven reports)
> total files in local repo: 3743
> total bytes in local repo: 162600
> count of files having "plexus-container-default" in local repo: 37 (18
> POM, 10 JAR)
>
> mvn master (4.0.0-alpha-1, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6):
> total time 1:08 min (as Maven reports)
> total files in local repo: 3741
> total bytes in local repo: 162428
> count of files having "plexus-container-default" in local repo: 37 (18
> POM, 10 JAR)
>
> so, pretty much the same. Then I went to create a (hacked for now) patch,
> as can be seen here:
>
> https://github.com/apache/maven/compare/master...cstamas:plugin-resolution-hack
>
> results with above patched mvn master:
> total time 1:04 min (as Maven reports)
> total files in local repo: 2992
> total bytes in local repo: 149740
> count of files having "plexus-container-default" in local repo: 0
>
> Basically patch "cut" almost 1k of unnecessary file downloads, 10+MB byte
> downloads in case of Maven being built.
>
> Is there some issue in JIRA covering this behaviour of Maven (I could not
> find any)?
>
> Have fun,
> Tamas
>
Re: Plugin dependency resolution
Posted by Tamás Cservenák <ta...@cservenak.net>.
Created https://issues.apache.org/jira/browse/MNG-7097
On Thu, Feb 11, 2021 at 8:41 AM Tamás Cservenák <ta...@cservenak.net> wrote:
> Hi Robert,
>
> I agree with you: is a really small change.
> Re non-exported packages: will take a look into those a bit more, my POC
> was just at "artifact level"... (but given Maven toss away downloaded
> plugin dependency, is plugin today able to use core artifact's non-exported
> package?)
> Yes, several maven bits, several plexus bits, and so on, it was just
> always annoying me :)
> Yup, will create a JIRA.
>
> Still, I'd really love to have more feedback, have people try the patch on
> more projects than I did (maven itself and one another) ;)
>
> HTH
> Tamas
>
> On Wed, Feb 10, 2021 at 10:33 PM Robert Scholte <rf...@apache.org>
> wrote:
>
>> Hi Tamas,
>>
>> based on the number of lines you wrote versus the amount of downloads
>> that are prevented (and storage) I think it is worth adding.
>> My biggest worry about this solution was about the non exported packages
>> of the exported artifacts. Would such changes have a negative and
>> inconsistent effect?
>> And I think it would make more sense for Maven users
>> If those people take a closer look at the downloaded file, they should
>> now wonder: my is maven-core 2.2.1, 3.0, 3.1.1 etc downloaded, while I'm
>> using Maven 3.6.3?
>>
>> AFAIK there's no ticket for it yet, these were just the ideas I could
>> think of regarding the WagonExcluder issue.
>> Looks like it is time to create it.
>>
>> thanks,
>> Robert
>>
>> On 10-2-2021 09:50:24, Tamás Cservenák <ta...@cservenak.net> wrote:
>> Howdy,
>>
>> Some time ago I stumbled upon Robert S. remark in "[DISCUSS] MNG-7020
>> Remove Maven 2 WagonExcluder backward compat code" thread: "Why download,
>> if they are being removed from the classpath afterwards due to classworld
>> config". Similarly, there is a thread "maven-site-plugin and
>> sisu-inject-plexus" discussing that plugin pulls down ancient
>> plexus-container-default (10+ years ago dropped). And finally, we all know
>> about "maven downloads the whole internet" maxim :)
>>
>> Hence, I wanted to check this out. My "experiment" was not about maven
>> "speed" (whatever you mean by it), but more about it's "bandwidth" usage.
>>
>> My setup:
>> - am using (primed) MRM, not interested in bashing Central or measuring my
>> ISP network speed
>> - am always nuking local repo (hence, starting state is "get everything
>> needed for build")
>> - my test bed project is maven itself
>> (master, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6)
>> - build + tests ARE executed, and succeed OK, but I was really interested
>> in local repository post-build state
>> - command line am executing from maven checkout root is: rm -r /tmp/repo;
>> ~/bin/maven/apache-maven-XXX/bin/mvn clean install
>> -Dmaven.repo.local=/tmp/repo
>>
>> Remarks: times and the whole "experiment' is not scientific, they are just
>> there for "rough comparison" sake. I did not repeat builds multiple times
>> (to have some "mean time"), as I was not interested in time, but did
>> repeat
>> enough times to ensure that local repository state (file count, file
>> sizes)
>> are consistent.
>>
>> Results:
>>
>> maven 3.6.3 (released):
>> total time 1:45 min (as Maven reports)
>> total files in local repo: 3743
>> total bytes in local repo: 162600
>> count of files having "plexus-container-default" in local repo: 37 (18
>> POM, 10 JAR)
>>
>> mvn master (4.0.0-alpha-1, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6):
>> total time 1:08 min (as Maven reports)
>> total files in local repo: 3741
>> total bytes in local repo: 162428
>> count of files having "plexus-container-default" in local repo: 37 (18
>> POM, 10 JAR)
>>
>> so, pretty much the same. Then I went to create a (hacked for now) patch,
>> as can be seen here:
>>
>> https://github.com/apache/maven/compare/master...cstamas:plugin-resolution-hack
>>
>> results with above patched mvn master:
>> total time 1:04 min (as Maven reports)
>> total files in local repo: 2992
>> total bytes in local repo: 149740
>> count of files having "plexus-container-default" in local repo: 0
>>
>> Basically patch "cut" almost 1k of unnecessary file downloads, 10+MB byte
>> downloads in case of Maven being built.
>>
>> Is there some issue in JIRA covering this behaviour of Maven (I could not
>> find any)?
>>
>> Have fun,
>> Tamas
>>
>
Re: Plugin dependency resolution
Posted by Robert Scholte <rf...@apache.org>.
In the past I've tried to set several maven(core) dependencies of a plugin to provided and it broke!
I didn't understand it and didn't had the time to analyze it.
Just a huge warning that you can expect surprises.
I think we can start thinking of "provided" if we have clear APIs and SPIs defined for Maven, now it is too early.
Robert
On 11-2-2021 11:19:03, Romain Manni-Bucau <rm...@gmail.com> wrote:
Hi everyone,
Does it mean we have two action items:
1. make maven plugin plugin warn or fail (with a toggle) if a maven stack
plugin is in scope compile and not provided
2. filter a bit what we "know" (probably using semver in a whitelist of
dependencies)
2 is not perfect from a design point of view but gain looks huge so
probably worth it
Romain Manni-Bucau
@rmannibucau | Blog
| Old Blog
| Github |
LinkedIn | Book
Le jeu. 11 févr. 2021 à 10:20, Tamás Cservenák a
écrit :
> Yup, "provided" scope would make it, but, as you say, it would require all
> (ours and non-ours) to be "fixed".
>
> Just for example, in tooling we did for Nexus 2, the NX plugin packaging
> (equivalent of maven plugin packaging)
> was ENFORCING that any NX artifact used as NX plugin dependency be declared
> as "provided", otherwise
> it was failing the build.
>
> HTH
> T
>
> On Thu, Feb 11, 2021 at 9:36 AM Hervé BOUTEMY
> wrote:
>
> > thinking about plexus-utils case: only XPP3 class is shared by Maven core
> > Then I imagine that it that dependency is filtered at runtime, many calls
> > from
> > plugins to other features of that lib will fail...
> > If you don't beat at me, I'll try to test tonight
> >
> > Another idea: would marking libraries as "provided" in plugins just do
> the
> > job? Of course, each plugin would require to apply this best practice,
> > which
> > will take longer time...
> > need to test also...
> >
> > Regards,
> >
> > Hervé
> >
> > Le jeudi 11 février 2021, 08:41:21 CET Tamás Cservenák a écrit :
> > > Hi Robert,
> > >
> > > I agree with you: is a really small change.
> > > Re non-exported packages: will take a look into those a bit more, my
> POC
> > > was just at "artifact level"... (but given Maven toss away downloaded
> > > plugin dependency, is plugin today able to use core artifact's
> > non-exported
> > > package?)
> > > Yes, several maven bits, several plexus bits, and so on, it was just
> > always
> > > annoying me :)
> > > Yup, will create a JIRA.
> > >
> > > Still, I'd really love to have more feedback, have people try the patch
> > on
> > > more projects than I did (maven itself and one another) ;)
> > >
> > > HTH
> > > Tamas
> > >
> > > On Wed, Feb 10, 2021 at 10:33 PM Robert Scholte
> > >
> > > wrote:
> > > > Hi Tamas,
> > > >
> > > > based on the number of lines you wrote versus the amount of downloads
> > that
> > > > are prevented (and storage) I think it is worth adding.
> > > > My biggest worry about this solution was about the non exported
> > packages
> > > > of the exported artifacts. Would such changes have a negative and
> > > > inconsistent effect?
> > > > And I think it would make more sense for Maven users
> > > > If those people take a closer look at the downloaded file, they
> should
> > now
> > > > wonder: my is maven-core 2.2.1, 3.0, 3.1.1 etc downloaded, while I'm
> > using
> > > > Maven 3.6.3?
> > > >
> > > > AFAIK there's no ticket for it yet, these were just the ideas I could
> > > > think of regarding the WagonExcluder issue.
> > > > Looks like it is time to create it.
> > > >
> > > > thanks,
> > > > Robert
> > > >
> > > > On 10-2-2021 09:50:24, Tamás Cservenák wrote:
> > > > Howdy,
> > > >
> > > > Some time ago I stumbled upon Robert S. remark in "[DISCUSS] MNG-7020
> > > > Remove Maven 2 WagonExcluder backward compat code" thread: "Why
> > download,
> > > > if they are being removed from the classpath afterwards due to
> > classworld
> > > > config". Similarly, there is a thread "maven-site-plugin and
> > > > sisu-inject-plexus" discussing that plugin pulls down ancient
> > > > plexus-container-default (10+ years ago dropped). And finally, we all
> > know
> > > > about "maven downloads the whole internet" maxim :)
> > > >
> > > > Hence, I wanted to check this out. My "experiment" was not about
> maven
> > > > "speed" (whatever you mean by it), but more about it's "bandwidth"
> > usage.
> > > >
> > > > My setup:
> > > > - am using (primed) MRM, not interested in bashing Central or
> > measuring my
> > > > ISP network speed
> > > > - am always nuking local repo (hence, starting state is "get
> everything
> > > > needed for build")
> > > > - my test bed project is maven itself
> > > > (master, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6)
> > > > - build + tests ARE executed, and succeed OK, but I was really
> > interested
> > > > in local repository post-build state
> > > > - command line am executing from maven checkout root is: rm -r
> > /tmp/repo;
> > > > ~/bin/maven/apache-maven-XXX/bin/mvn clean install
> > > > -Dmaven.repo.local=/tmp/repo
> > > >
> > > > Remarks: times and the whole "experiment' is not scientific, they are
> > just
> > > > there for "rough comparison" sake. I did not repeat builds multiple
> > times
> > > > (to have some "mean time"), as I was not interested in time, but did
> > > > repeat
> > > > enough times to ensure that local repository state (file count, file
> > > > sizes)
> > > > are consistent.
> > > >
> > > > Results:
> > > >
> > > > maven 3.6.3 (released):
> > > > total time 1:45 min (as Maven reports)
> > > > total files in local repo: 3743
> > > > total bytes in local repo: 162600
> > > > count of files having "plexus-container-default" in local repo: 37
> (18
> > > > POM, 10 JAR)
> > > >
> > > > mvn master (4.0.0-alpha-1, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6):
> > > > total time 1:08 min (as Maven reports)
> > > > total files in local repo: 3741
> > > > total bytes in local repo: 162428
> > > > count of files having "plexus-container-default" in local repo: 37
> (18
> > > > POM, 10 JAR)
> > > >
> > > > so, pretty much the same. Then I went to create a (hacked for now)
> > patch,
> > > > as can be seen here:
> > > >
> > > >
> >
> https://github.com/apache/maven/compare/master...cstamas:plugin-resolution
> > > > -hack
> > > >
> > > > results with above patched mvn master:
> > > > total time 1:04 min (as Maven reports)
> > > > total files in local repo: 2992
> > > > total bytes in local repo: 149740
> > > > count of files having "plexus-container-default" in local repo: 0
> > > >
> > > > Basically patch "cut" almost 1k of unnecessary file downloads, 10+MB
> > byte
> > > > downloads in case of Maven being built.
> > > >
> > > > Is there some issue in JIRA covering this behaviour of Maven (I could
> > not
> > > > find any)?
> > > >
> > > > Have fun,
> > > > Tamas
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > For additional commands, e-mail: dev-help@maven.apache.org
> >
> >
>
Re: Plugin dependency resolution
Posted by Tamás Cservenák <ta...@cservenak.net>.
Yup,
this can be approached on "two ends": from Maven and from Plugins....
Note: let's name artifacts having GA org.apache.maven:maven-* as "maven
runtime", artifacts being "the maven", in short: MR.
From plugin end, I never understood why MR dependencies are NOT declared as
"provided".
maven-plugin packaged artifacts CANNOT "live" outside of a MR. Is really
the same as "servlet-api" for web apps.
Moreover (am now just opening up a random Apache maven plugin, am not
picking at any of those, just giving an example), look at this:
https://github.com/apache/maven-deploy-plugin/blob/master/pom.xml#L72-L91
The dependencies in this POM are declared using "best practices" (defines
mavenVersion property), BUT let's assume someone does not do it like that.
Hence, it becomes possible that a plugin depends on maven-plugin-api 3.0,
maven-core:3.1 and maven-model:3.2.
Moreover, let's assume that the plugin builds PASSES and is deployed
despite this -- possibly human oversight -- mistake.
My point is: at maven plugin build time
- if MR artifact is NOT provided fail/warn (toggle)
- make sure all MR artifacts are SAME version, otherwise fail/warm (toggle)
And yes, the "filter a bit we know" part is still hazy (at plugin build
time), especially as developers may develop it's maven plugin for Maven
3.0, 3.1, 3.2.... and those all have slightly different MR...
But again, we could draw the line, and make the latest maven-plugin-plugin
raise the bar of "lowest maven version" it can build a plugin for.....
(and for those publish some "BOM" or "inventory", analogy of signature for
animal sniffer?)
And this could be maven-plugin-plugin or maybe even some enforcer rule? (if
there is some like this, am unaware of it, sorry)
HTH
T
On Thu, Feb 11, 2021 at 11:19 AM Romain Manni-Bucau <rm...@gmail.com>
wrote:
> Hi everyone,
>
> Does it mean we have two action items:
>
> 1. make maven plugin plugin warn or fail (with a toggle) if a maven stack
> plugin is in scope compile and not provided
> 2. filter a bit what we "know" (probably using semver in a whitelist of
> dependencies)
>
> 2 is not perfect from a design point of view but gain looks huge so
> probably worth it
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github <
> https://github.com/rmannibucau> |
> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
>
>
> Le jeu. 11 févr. 2021 à 10:20, Tamás Cservenák <ta...@cservenak.net> a
> écrit :
>
> > Yup, "provided" scope would make it, but, as you say, it would require
> all
> > (ours and non-ours) to be "fixed".
> >
> > Just for example, in tooling we did for Nexus 2, the NX plugin packaging
> > (equivalent of maven plugin packaging)
> > was ENFORCING that any NX artifact used as NX plugin dependency be
> declared
> > as "provided", otherwise
> > it was failing the build.
> >
> > HTH
> > T
> >
> > On Thu, Feb 11, 2021 at 9:36 AM Hervé BOUTEMY <he...@free.fr>
> > wrote:
> >
> > > thinking about plexus-utils case: only XPP3 class is shared by Maven
> core
> > > Then I imagine that it that dependency is filtered at runtime, many
> calls
> > > from
> > > plugins to other features of that lib will fail...
> > > If you don't beat at me, I'll try to test tonight
> > >
> > > Another idea: would marking libraries as "provided" in plugins just do
> > the
> > > job? Of course, each plugin would require to apply this best practice,
> > > which
> > > will take longer time...
> > > need to test also...
> > >
> > > Regards,
> > >
> > > Hervé
> > >
> > > Le jeudi 11 février 2021, 08:41:21 CET Tamás Cservenák a écrit :
> > > > Hi Robert,
> > > >
> > > > I agree with you: is a really small change.
> > > > Re non-exported packages: will take a look into those a bit more, my
> > POC
> > > > was just at "artifact level"... (but given Maven toss away downloaded
> > > > plugin dependency, is plugin today able to use core artifact's
> > > non-exported
> > > > package?)
> > > > Yes, several maven bits, several plexus bits, and so on, it was just
> > > always
> > > > annoying me :)
> > > > Yup, will create a JIRA.
> > > >
> > > > Still, I'd really love to have more feedback, have people try the
> patch
> > > on
> > > > more projects than I did (maven itself and one another) ;)
> > > >
> > > > HTH
> > > > Tamas
> > > >
> > > > On Wed, Feb 10, 2021 at 10:33 PM Robert Scholte <
> rfscholte@apache.org>
> > > >
> > > > wrote:
> > > > > Hi Tamas,
> > > > >
> > > > > based on the number of lines you wrote versus the amount of
> downloads
> > > that
> > > > > are prevented (and storage) I think it is worth adding.
> > > > > My biggest worry about this solution was about the non exported
> > > packages
> > > > > of the exported artifacts. Would such changes have a negative and
> > > > > inconsistent effect?
> > > > > And I think it would make more sense for Maven users
> > > > > If those people take a closer look at the downloaded file, they
> > should
> > > now
> > > > > wonder: my is maven-core 2.2.1, 3.0, 3.1.1 etc downloaded, while
> I'm
> > > using
> > > > > Maven 3.6.3?
> > > > >
> > > > > AFAIK there's no ticket for it yet, these were just the ideas I
> could
> > > > > think of regarding the WagonExcluder issue.
> > > > > Looks like it is time to create it.
> > > > >
> > > > > thanks,
> > > > > Robert
> > > > >
> > > > > On 10-2-2021 09:50:24, Tamás Cservenák <ta...@cservenak.net>
> wrote:
> > > > > Howdy,
> > > > >
> > > > > Some time ago I stumbled upon Robert S. remark in "[DISCUSS]
> MNG-7020
> > > > > Remove Maven 2 WagonExcluder backward compat code" thread: "Why
> > > download,
> > > > > if they are being removed from the classpath afterwards due to
> > > classworld
> > > > > config". Similarly, there is a thread "maven-site-plugin and
> > > > > sisu-inject-plexus" discussing that plugin pulls down ancient
> > > > > plexus-container-default (10+ years ago dropped). And finally, we
> all
> > > know
> > > > > about "maven downloads the whole internet" maxim :)
> > > > >
> > > > > Hence, I wanted to check this out. My "experiment" was not about
> > maven
> > > > > "speed" (whatever you mean by it), but more about it's "bandwidth"
> > > usage.
> > > > >
> > > > > My setup:
> > > > > - am using (primed) MRM, not interested in bashing Central or
> > > measuring my
> > > > > ISP network speed
> > > > > - am always nuking local repo (hence, starting state is "get
> > everything
> > > > > needed for build")
> > > > > - my test bed project is maven itself
> > > > > (master, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6)
> > > > > - build + tests ARE executed, and succeed OK, but I was really
> > > interested
> > > > > in local repository post-build state
> > > > > - command line am executing from maven checkout root is: rm -r
> > > /tmp/repo;
> > > > > ~/bin/maven/apache-maven-XXX/bin/mvn clean install
> > > > > -Dmaven.repo.local=/tmp/repo
> > > > >
> > > > > Remarks: times and the whole "experiment' is not scientific, they
> are
> > > just
> > > > > there for "rough comparison" sake. I did not repeat builds multiple
> > > times
> > > > > (to have some "mean time"), as I was not interested in time, but
> did
> > > > > repeat
> > > > > enough times to ensure that local repository state (file count,
> file
> > > > > sizes)
> > > > > are consistent.
> > > > >
> > > > > Results:
> > > > >
> > > > > maven 3.6.3 (released):
> > > > > total time 1:45 min (as Maven reports)
> > > > > total files in local repo: 3743
> > > > > total bytes in local repo: 162600
> > > > > count of files having "plexus-container-default" in local repo: 37
> > (18
> > > > > POM, 10 JAR)
> > > > >
> > > > > mvn master (4.0.0-alpha-1,
> ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6):
> > > > > total time 1:08 min (as Maven reports)
> > > > > total files in local repo: 3741
> > > > > total bytes in local repo: 162428
> > > > > count of files having "plexus-container-default" in local repo: 37
> > (18
> > > > > POM, 10 JAR)
> > > > >
> > > > > so, pretty much the same. Then I went to create a (hacked for now)
> > > patch,
> > > > > as can be seen here:
> > > > >
> > > > >
> > >
> >
> https://github.com/apache/maven/compare/master...cstamas:plugin-resolution
> > > > > -hack
> > > > >
> > > > > results with above patched mvn master:
> > > > > total time 1:04 min (as Maven reports)
> > > > > total files in local repo: 2992
> > > > > total bytes in local repo: 149740
> > > > > count of files having "plexus-container-default" in local repo: 0
> > > > >
> > > > > Basically patch "cut" almost 1k of unnecessary file downloads,
> 10+MB
> > > byte
> > > > > downloads in case of Maven being built.
> > > > >
> > > > > Is there some issue in JIRA covering this behaviour of Maven (I
> could
> > > not
> > > > > find any)?
> > > > >
> > > > > Have fun,
> > > > > Tamas
> > >
> > >
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > For additional commands, e-mail: dev-help@maven.apache.org
> > >
> > >
> >
>
Re: Plugin dependency resolution
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Hi everyone,
Does it mean we have two action items:
1. make maven plugin plugin warn or fail (with a toggle) if a maven stack
plugin is in scope compile and not provided
2. filter a bit what we "know" (probably using semver in a whitelist of
dependencies)
2 is not perfect from a design point of view but gain looks huge so
probably worth it
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
Le jeu. 11 févr. 2021 à 10:20, Tamás Cservenák <ta...@cservenak.net> a
écrit :
> Yup, "provided" scope would make it, but, as you say, it would require all
> (ours and non-ours) to be "fixed".
>
> Just for example, in tooling we did for Nexus 2, the NX plugin packaging
> (equivalent of maven plugin packaging)
> was ENFORCING that any NX artifact used as NX plugin dependency be declared
> as "provided", otherwise
> it was failing the build.
>
> HTH
> T
>
> On Thu, Feb 11, 2021 at 9:36 AM Hervé BOUTEMY <he...@free.fr>
> wrote:
>
> > thinking about plexus-utils case: only XPP3 class is shared by Maven core
> > Then I imagine that it that dependency is filtered at runtime, many calls
> > from
> > plugins to other features of that lib will fail...
> > If you don't beat at me, I'll try to test tonight
> >
> > Another idea: would marking libraries as "provided" in plugins just do
> the
> > job? Of course, each plugin would require to apply this best practice,
> > which
> > will take longer time...
> > need to test also...
> >
> > Regards,
> >
> > Hervé
> >
> > Le jeudi 11 février 2021, 08:41:21 CET Tamás Cservenák a écrit :
> > > Hi Robert,
> > >
> > > I agree with you: is a really small change.
> > > Re non-exported packages: will take a look into those a bit more, my
> POC
> > > was just at "artifact level"... (but given Maven toss away downloaded
> > > plugin dependency, is plugin today able to use core artifact's
> > non-exported
> > > package?)
> > > Yes, several maven bits, several plexus bits, and so on, it was just
> > always
> > > annoying me :)
> > > Yup, will create a JIRA.
> > >
> > > Still, I'd really love to have more feedback, have people try the patch
> > on
> > > more projects than I did (maven itself and one another) ;)
> > >
> > > HTH
> > > Tamas
> > >
> > > On Wed, Feb 10, 2021 at 10:33 PM Robert Scholte <rf...@apache.org>
> > >
> > > wrote:
> > > > Hi Tamas,
> > > >
> > > > based on the number of lines you wrote versus the amount of downloads
> > that
> > > > are prevented (and storage) I think it is worth adding.
> > > > My biggest worry about this solution was about the non exported
> > packages
> > > > of the exported artifacts. Would such changes have a negative and
> > > > inconsistent effect?
> > > > And I think it would make more sense for Maven users
> > > > If those people take a closer look at the downloaded file, they
> should
> > now
> > > > wonder: my is maven-core 2.2.1, 3.0, 3.1.1 etc downloaded, while I'm
> > using
> > > > Maven 3.6.3?
> > > >
> > > > AFAIK there's no ticket for it yet, these were just the ideas I could
> > > > think of regarding the WagonExcluder issue.
> > > > Looks like it is time to create it.
> > > >
> > > > thanks,
> > > > Robert
> > > >
> > > > On 10-2-2021 09:50:24, Tamás Cservenák <ta...@cservenak.net> wrote:
> > > > Howdy,
> > > >
> > > > Some time ago I stumbled upon Robert S. remark in "[DISCUSS] MNG-7020
> > > > Remove Maven 2 WagonExcluder backward compat code" thread: "Why
> > download,
> > > > if they are being removed from the classpath afterwards due to
> > classworld
> > > > config". Similarly, there is a thread "maven-site-plugin and
> > > > sisu-inject-plexus" discussing that plugin pulls down ancient
> > > > plexus-container-default (10+ years ago dropped). And finally, we all
> > know
> > > > about "maven downloads the whole internet" maxim :)
> > > >
> > > > Hence, I wanted to check this out. My "experiment" was not about
> maven
> > > > "speed" (whatever you mean by it), but more about it's "bandwidth"
> > usage.
> > > >
> > > > My setup:
> > > > - am using (primed) MRM, not interested in bashing Central or
> > measuring my
> > > > ISP network speed
> > > > - am always nuking local repo (hence, starting state is "get
> everything
> > > > needed for build")
> > > > - my test bed project is maven itself
> > > > (master, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6)
> > > > - build + tests ARE executed, and succeed OK, but I was really
> > interested
> > > > in local repository post-build state
> > > > - command line am executing from maven checkout root is: rm -r
> > /tmp/repo;
> > > > ~/bin/maven/apache-maven-XXX/bin/mvn clean install
> > > > -Dmaven.repo.local=/tmp/repo
> > > >
> > > > Remarks: times and the whole "experiment' is not scientific, they are
> > just
> > > > there for "rough comparison" sake. I did not repeat builds multiple
> > times
> > > > (to have some "mean time"), as I was not interested in time, but did
> > > > repeat
> > > > enough times to ensure that local repository state (file count, file
> > > > sizes)
> > > > are consistent.
> > > >
> > > > Results:
> > > >
> > > > maven 3.6.3 (released):
> > > > total time 1:45 min (as Maven reports)
> > > > total files in local repo: 3743
> > > > total bytes in local repo: 162600
> > > > count of files having "plexus-container-default" in local repo: 37
> (18
> > > > POM, 10 JAR)
> > > >
> > > > mvn master (4.0.0-alpha-1, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6):
> > > > total time 1:08 min (as Maven reports)
> > > > total files in local repo: 3741
> > > > total bytes in local repo: 162428
> > > > count of files having "plexus-container-default" in local repo: 37
> (18
> > > > POM, 10 JAR)
> > > >
> > > > so, pretty much the same. Then I went to create a (hacked for now)
> > patch,
> > > > as can be seen here:
> > > >
> > > >
> >
> https://github.com/apache/maven/compare/master...cstamas:plugin-resolution
> > > > -hack
> > > >
> > > > results with above patched mvn master:
> > > > total time 1:04 min (as Maven reports)
> > > > total files in local repo: 2992
> > > > total bytes in local repo: 149740
> > > > count of files having "plexus-container-default" in local repo: 0
> > > >
> > > > Basically patch "cut" almost 1k of unnecessary file downloads, 10+MB
> > byte
> > > > downloads in case of Maven being built.
> > > >
> > > > Is there some issue in JIRA covering this behaviour of Maven (I could
> > not
> > > > find any)?
> > > >
> > > > Have fun,
> > > > Tamas
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > For additional commands, e-mail: dev-help@maven.apache.org
> >
> >
>
Re: Plugin dependency resolution
Posted by Tamás Cservenák <ta...@cservenak.net>.
Yup, "provided" scope would make it, but, as you say, it would require all
(ours and non-ours) to be "fixed".
Just for example, in tooling we did for Nexus 2, the NX plugin packaging
(equivalent of maven plugin packaging)
was ENFORCING that any NX artifact used as NX plugin dependency be declared
as "provided", otherwise
it was failing the build.
HTH
T
On Thu, Feb 11, 2021 at 9:36 AM Hervé BOUTEMY <he...@free.fr> wrote:
> thinking about plexus-utils case: only XPP3 class is shared by Maven core
> Then I imagine that it that dependency is filtered at runtime, many calls
> from
> plugins to other features of that lib will fail...
> If you don't beat at me, I'll try to test tonight
>
> Another idea: would marking libraries as "provided" in plugins just do the
> job? Of course, each plugin would require to apply this best practice,
> which
> will take longer time...
> need to test also...
>
> Regards,
>
> Hervé
>
> Le jeudi 11 février 2021, 08:41:21 CET Tamás Cservenák a écrit :
> > Hi Robert,
> >
> > I agree with you: is a really small change.
> > Re non-exported packages: will take a look into those a bit more, my POC
> > was just at "artifact level"... (but given Maven toss away downloaded
> > plugin dependency, is plugin today able to use core artifact's
> non-exported
> > package?)
> > Yes, several maven bits, several plexus bits, and so on, it was just
> always
> > annoying me :)
> > Yup, will create a JIRA.
> >
> > Still, I'd really love to have more feedback, have people try the patch
> on
> > more projects than I did (maven itself and one another) ;)
> >
> > HTH
> > Tamas
> >
> > On Wed, Feb 10, 2021 at 10:33 PM Robert Scholte <rf...@apache.org>
> >
> > wrote:
> > > Hi Tamas,
> > >
> > > based on the number of lines you wrote versus the amount of downloads
> that
> > > are prevented (and storage) I think it is worth adding.
> > > My biggest worry about this solution was about the non exported
> packages
> > > of the exported artifacts. Would such changes have a negative and
> > > inconsistent effect?
> > > And I think it would make more sense for Maven users
> > > If those people take a closer look at the downloaded file, they should
> now
> > > wonder: my is maven-core 2.2.1, 3.0, 3.1.1 etc downloaded, while I'm
> using
> > > Maven 3.6.3?
> > >
> > > AFAIK there's no ticket for it yet, these were just the ideas I could
> > > think of regarding the WagonExcluder issue.
> > > Looks like it is time to create it.
> > >
> > > thanks,
> > > Robert
> > >
> > > On 10-2-2021 09:50:24, Tamás Cservenák <ta...@cservenak.net> wrote:
> > > Howdy,
> > >
> > > Some time ago I stumbled upon Robert S. remark in "[DISCUSS] MNG-7020
> > > Remove Maven 2 WagonExcluder backward compat code" thread: "Why
> download,
> > > if they are being removed from the classpath afterwards due to
> classworld
> > > config". Similarly, there is a thread "maven-site-plugin and
> > > sisu-inject-plexus" discussing that plugin pulls down ancient
> > > plexus-container-default (10+ years ago dropped). And finally, we all
> know
> > > about "maven downloads the whole internet" maxim :)
> > >
> > > Hence, I wanted to check this out. My "experiment" was not about maven
> > > "speed" (whatever you mean by it), but more about it's "bandwidth"
> usage.
> > >
> > > My setup:
> > > - am using (primed) MRM, not interested in bashing Central or
> measuring my
> > > ISP network speed
> > > - am always nuking local repo (hence, starting state is "get everything
> > > needed for build")
> > > - my test bed project is maven itself
> > > (master, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6)
> > > - build + tests ARE executed, and succeed OK, but I was really
> interested
> > > in local repository post-build state
> > > - command line am executing from maven checkout root is: rm -r
> /tmp/repo;
> > > ~/bin/maven/apache-maven-XXX/bin/mvn clean install
> > > -Dmaven.repo.local=/tmp/repo
> > >
> > > Remarks: times and the whole "experiment' is not scientific, they are
> just
> > > there for "rough comparison" sake. I did not repeat builds multiple
> times
> > > (to have some "mean time"), as I was not interested in time, but did
> > > repeat
> > > enough times to ensure that local repository state (file count, file
> > > sizes)
> > > are consistent.
> > >
> > > Results:
> > >
> > > maven 3.6.3 (released):
> > > total time 1:45 min (as Maven reports)
> > > total files in local repo: 3743
> > > total bytes in local repo: 162600
> > > count of files having "plexus-container-default" in local repo: 37 (18
> > > POM, 10 JAR)
> > >
> > > mvn master (4.0.0-alpha-1, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6):
> > > total time 1:08 min (as Maven reports)
> > > total files in local repo: 3741
> > > total bytes in local repo: 162428
> > > count of files having "plexus-container-default" in local repo: 37 (18
> > > POM, 10 JAR)
> > >
> > > so, pretty much the same. Then I went to create a (hacked for now)
> patch,
> > > as can be seen here:
> > >
> > >
> https://github.com/apache/maven/compare/master...cstamas:plugin-resolution
> > > -hack
> > >
> > > results with above patched mvn master:
> > > total time 1:04 min (as Maven reports)
> > > total files in local repo: 2992
> > > total bytes in local repo: 149740
> > > count of files having "plexus-container-default" in local repo: 0
> > >
> > > Basically patch "cut" almost 1k of unnecessary file downloads, 10+MB
> byte
> > > downloads in case of Maven being built.
> > >
> > > Is there some issue in JIRA covering this behaviour of Maven (I could
> not
> > > find any)?
> > >
> > > Have fun,
> > > Tamas
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: Plugin dependency resolution
Posted by Hervé BOUTEMY <he...@free.fr>.
thinking about plexus-utils case: only XPP3 class is shared by Maven core
Then I imagine that it that dependency is filtered at runtime, many calls from
plugins to other features of that lib will fail...
If you don't beat at me, I'll try to test tonight
Another idea: would marking libraries as "provided" in plugins just do the
job? Of course, each plugin would require to apply this best practice, which
will take longer time...
need to test also...
Regards,
Hervé
Le jeudi 11 février 2021, 08:41:21 CET Tamás Cservenák a écrit :
> Hi Robert,
>
> I agree with you: is a really small change.
> Re non-exported packages: will take a look into those a bit more, my POC
> was just at "artifact level"... (but given Maven toss away downloaded
> plugin dependency, is plugin today able to use core artifact's non-exported
> package?)
> Yes, several maven bits, several plexus bits, and so on, it was just always
> annoying me :)
> Yup, will create a JIRA.
>
> Still, I'd really love to have more feedback, have people try the patch on
> more projects than I did (maven itself and one another) ;)
>
> HTH
> Tamas
>
> On Wed, Feb 10, 2021 at 10:33 PM Robert Scholte <rf...@apache.org>
>
> wrote:
> > Hi Tamas,
> >
> > based on the number of lines you wrote versus the amount of downloads that
> > are prevented (and storage) I think it is worth adding.
> > My biggest worry about this solution was about the non exported packages
> > of the exported artifacts. Would such changes have a negative and
> > inconsistent effect?
> > And I think it would make more sense for Maven users
> > If those people take a closer look at the downloaded file, they should now
> > wonder: my is maven-core 2.2.1, 3.0, 3.1.1 etc downloaded, while I'm using
> > Maven 3.6.3?
> >
> > AFAIK there's no ticket for it yet, these were just the ideas I could
> > think of regarding the WagonExcluder issue.
> > Looks like it is time to create it.
> >
> > thanks,
> > Robert
> >
> > On 10-2-2021 09:50:24, Tamás Cservenák <ta...@cservenak.net> wrote:
> > Howdy,
> >
> > Some time ago I stumbled upon Robert S. remark in "[DISCUSS] MNG-7020
> > Remove Maven 2 WagonExcluder backward compat code" thread: "Why download,
> > if they are being removed from the classpath afterwards due to classworld
> > config". Similarly, there is a thread "maven-site-plugin and
> > sisu-inject-plexus" discussing that plugin pulls down ancient
> > plexus-container-default (10+ years ago dropped). And finally, we all know
> > about "maven downloads the whole internet" maxim :)
> >
> > Hence, I wanted to check this out. My "experiment" was not about maven
> > "speed" (whatever you mean by it), but more about it's "bandwidth" usage.
> >
> > My setup:
> > - am using (primed) MRM, not interested in bashing Central or measuring my
> > ISP network speed
> > - am always nuking local repo (hence, starting state is "get everything
> > needed for build")
> > - my test bed project is maven itself
> > (master, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6)
> > - build + tests ARE executed, and succeed OK, but I was really interested
> > in local repository post-build state
> > - command line am executing from maven checkout root is: rm -r /tmp/repo;
> > ~/bin/maven/apache-maven-XXX/bin/mvn clean install
> > -Dmaven.repo.local=/tmp/repo
> >
> > Remarks: times and the whole "experiment' is not scientific, they are just
> > there for "rough comparison" sake. I did not repeat builds multiple times
> > (to have some "mean time"), as I was not interested in time, but did
> > repeat
> > enough times to ensure that local repository state (file count, file
> > sizes)
> > are consistent.
> >
> > Results:
> >
> > maven 3.6.3 (released):
> > total time 1:45 min (as Maven reports)
> > total files in local repo: 3743
> > total bytes in local repo: 162600
> > count of files having "plexus-container-default" in local repo: 37 (18
> > POM, 10 JAR)
> >
> > mvn master (4.0.0-alpha-1, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6):
> > total time 1:08 min (as Maven reports)
> > total files in local repo: 3741
> > total bytes in local repo: 162428
> > count of files having "plexus-container-default" in local repo: 37 (18
> > POM, 10 JAR)
> >
> > so, pretty much the same. Then I went to create a (hacked for now) patch,
> > as can be seen here:
> >
> > https://github.com/apache/maven/compare/master...cstamas:plugin-resolution
> > -hack
> >
> > results with above patched mvn master:
> > total time 1:04 min (as Maven reports)
> > total files in local repo: 2992
> > total bytes in local repo: 149740
> > count of files having "plexus-container-default" in local repo: 0
> >
> > Basically patch "cut" almost 1k of unnecessary file downloads, 10+MB byte
> > downloads in case of Maven being built.
> >
> > Is there some issue in JIRA covering this behaviour of Maven (I could not
> > find any)?
> >
> > Have fun,
> > Tamas
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Plugin dependency resolution
Posted by Tamás Cservenák <ta...@cservenak.net>.
Hi Robert,
I agree with you: is a really small change.
Re non-exported packages: will take a look into those a bit more, my POC
was just at "artifact level"... (but given Maven toss away downloaded
plugin dependency, is plugin today able to use core artifact's non-exported
package?)
Yes, several maven bits, several plexus bits, and so on, it was just always
annoying me :)
Yup, will create a JIRA.
Still, I'd really love to have more feedback, have people try the patch on
more projects than I did (maven itself and one another) ;)
HTH
Tamas
On Wed, Feb 10, 2021 at 10:33 PM Robert Scholte <rf...@apache.org>
wrote:
> Hi Tamas,
>
> based on the number of lines you wrote versus the amount of downloads that
> are prevented (and storage) I think it is worth adding.
> My biggest worry about this solution was about the non exported packages
> of the exported artifacts. Would such changes have a negative and
> inconsistent effect?
> And I think it would make more sense for Maven users
> If those people take a closer look at the downloaded file, they should now
> wonder: my is maven-core 2.2.1, 3.0, 3.1.1 etc downloaded, while I'm using
> Maven 3.6.3?
>
> AFAIK there's no ticket for it yet, these were just the ideas I could
> think of regarding the WagonExcluder issue.
> Looks like it is time to create it.
>
> thanks,
> Robert
>
> On 10-2-2021 09:50:24, Tamás Cservenák <ta...@cservenak.net> wrote:
> Howdy,
>
> Some time ago I stumbled upon Robert S. remark in "[DISCUSS] MNG-7020
> Remove Maven 2 WagonExcluder backward compat code" thread: "Why download,
> if they are being removed from the classpath afterwards due to classworld
> config". Similarly, there is a thread "maven-site-plugin and
> sisu-inject-plexus" discussing that plugin pulls down ancient
> plexus-container-default (10+ years ago dropped). And finally, we all know
> about "maven downloads the whole internet" maxim :)
>
> Hence, I wanted to check this out. My "experiment" was not about maven
> "speed" (whatever you mean by it), but more about it's "bandwidth" usage.
>
> My setup:
> - am using (primed) MRM, not interested in bashing Central or measuring my
> ISP network speed
> - am always nuking local repo (hence, starting state is "get everything
> needed for build")
> - my test bed project is maven itself
> (master, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6)
> - build + tests ARE executed, and succeed OK, but I was really interested
> in local repository post-build state
> - command line am executing from maven checkout root is: rm -r /tmp/repo;
> ~/bin/maven/apache-maven-XXX/bin/mvn clean install
> -Dmaven.repo.local=/tmp/repo
>
> Remarks: times and the whole "experiment' is not scientific, they are just
> there for "rough comparison" sake. I did not repeat builds multiple times
> (to have some "mean time"), as I was not interested in time, but did repeat
> enough times to ensure that local repository state (file count, file sizes)
> are consistent.
>
> Results:
>
> maven 3.6.3 (released):
> total time 1:45 min (as Maven reports)
> total files in local repo: 3743
> total bytes in local repo: 162600
> count of files having "plexus-container-default" in local repo: 37 (18
> POM, 10 JAR)
>
> mvn master (4.0.0-alpha-1, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6):
> total time 1:08 min (as Maven reports)
> total files in local repo: 3741
> total bytes in local repo: 162428
> count of files having "plexus-container-default" in local repo: 37 (18
> POM, 10 JAR)
>
> so, pretty much the same. Then I went to create a (hacked for now) patch,
> as can be seen here:
>
> https://github.com/apache/maven/compare/master...cstamas:plugin-resolution-hack
>
> results with above patched mvn master:
> total time 1:04 min (as Maven reports)
> total files in local repo: 2992
> total bytes in local repo: 149740
> count of files having "plexus-container-default" in local repo: 0
>
> Basically patch "cut" almost 1k of unnecessary file downloads, 10+MB byte
> downloads in case of Maven being built.
>
> Is there some issue in JIRA covering this behaviour of Maven (I could not
> find any)?
>
> Have fun,
> Tamas
>
Re: Plugin dependency resolution
Posted by Robert Scholte <rf...@apache.org>.
Hi Tamas,
based on the number of lines you wrote versus the amount of downloads that are prevented (and storage) I think it is worth adding.
My biggest worry about this solution was about the non exported packages of the exported artifacts. Would such changes have a negative and inconsistent effect?
And I think it would make more sense for Maven users
If those people take a closer look at the downloaded file, they should now wonder: my is maven-core 2.2.1, 3.0, 3.1.1 etc downloaded, while I'm using Maven 3.6.3?
AFAIK there's no ticket for it yet, these were just the ideas I could think of regarding the WagonExcluder issue.
Looks like it is time to create it.
thanks,
Robert
On 10-2-2021 09:50:24, Tamás Cservenák <ta...@cservenak.net> wrote:
Howdy,
Some time ago I stumbled upon Robert S. remark in "[DISCUSS] MNG-7020
Remove Maven 2 WagonExcluder backward compat code" thread: "Why download,
if they are being removed from the classpath afterwards due to classworld
config". Similarly, there is a thread "maven-site-plugin and
sisu-inject-plexus" discussing that plugin pulls down ancient
plexus-container-default (10+ years ago dropped). And finally, we all know
about "maven downloads the whole internet" maxim :)
Hence, I wanted to check this out. My "experiment" was not about maven
"speed" (whatever you mean by it), but more about it's "bandwidth" usage.
My setup:
- am using (primed) MRM, not interested in bashing Central or measuring my
ISP network speed
- am always nuking local repo (hence, starting state is "get everything
needed for build")
- my test bed project is maven itself
(master, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6)
- build + tests ARE executed, and succeed OK, but I was really interested
in local repository post-build state
- command line am executing from maven checkout root is: rm -r /tmp/repo;
~/bin/maven/apache-maven-XXX/bin/mvn clean install
-Dmaven.repo.local=/tmp/repo
Remarks: times and the whole "experiment' is not scientific, they are just
there for "rough comparison" sake. I did not repeat builds multiple times
(to have some "mean time"), as I was not interested in time, but did repeat
enough times to ensure that local repository state (file count, file sizes)
are consistent.
Results:
maven 3.6.3 (released):
total time 1:45 min (as Maven reports)
total files in local repo: 3743
total bytes in local repo: 162600
count of files having "plexus-container-default" in local repo: 37 (18
POM, 10 JAR)
mvn master (4.0.0-alpha-1, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6):
total time 1:08 min (as Maven reports)
total files in local repo: 3741
total bytes in local repo: 162428
count of files having "plexus-container-default" in local repo: 37 (18
POM, 10 JAR)
so, pretty much the same. Then I went to create a (hacked for now) patch,
as can be seen here:
https://github.com/apache/maven/compare/master...cstamas:plugin-resolution-hack
results with above patched mvn master:
total time 1:04 min (as Maven reports)
total files in local repo: 2992
total bytes in local repo: 149740
count of files having "plexus-container-default" in local repo: 0
Basically patch "cut" almost 1k of unnecessary file downloads, 10+MB byte
downloads in case of Maven being built.
Is there some issue in JIRA covering this behaviour of Maven (I could not
find any)?
Have fun,
Tamas