You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by GitBox <gi...@apache.org> on 2022/01/07 09:47:45 UTC
[GitHub] [maven-mvnd] KatyConnor opened a new issue #570: not fast
KatyConnor opened a new issue #570:
URL: https://github.com/apache/maven-mvnd/issues/570
The project build speed is not fast, the same as mvn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-mvnd] gnodet commented on issue #570: not fast
Posted by GitBox <gi...@apache.org>.
gnodet commented on issue #570:
URL: https://github.com/apache/maven-mvnd/issues/570#issuecomment-1008743063
> I also did a benchmark with two scala projects, which have about 30 subprojects:
>
> * flow-all: without maven-shade-plugin,
> * offline-all: with maven-shade-plugin to build **fat-jar** which required for spark task.
>
> using commands for both mvn/mvnd: `mvn -pl $PROJECT -am spotless:apply clean --no-snapshot-updates $PHASE`, while extra opts for:
>
> * `mvn`: `-T 1C`
> * `mvnd`: `-Dquickly -Dmvnd.maxHeapSize=8G`
>
> ## reuslt
> PROJECT PHASE mvn(s) mvnd(s)
> flow-all install 215/253/238/219/208/205(avg=223) 254/259/245/205/230/209(avg=233)
> offline-all package 416/433/431/443(avg=430) 412/431/418/398/411/409(avg=413)
> the performance of compilation and build is quite close to each other.
>
> by the way, it is time-consuming to build fat-jar with maven-shade-plugin, is there better way to create fat-jar for spark jobs?
How come the duration for `mvn` is decreasing run after run ? Is that a public project ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-mvnd] gnodet commented on issue #570: not fast
Posted by GitBox <gi...@apache.org>.
gnodet commented on issue #570:
URL: https://github.com/apache/maven-mvnd/issues/570#issuecomment-1009657052
> > So mvnd speeds up by
> > ```
> > * enabling concurrent build by default
> >
> > * cacheing a few things and help speeding up the time spent by maven _itself_ when loading the poms, the plugins
> >
> > * keeping the JVM warm
> > If if the plugins themselves use 95% of the build time and you're comparing to a multithreaded build, `mvnd` help much...
> > ```
>
> Thanks for your time.
>
> * Considering "enabling concurrent build by default", it seems `mvn` with `-T 1C` opts can also achieve multi-threaded build?
Yes, though the output is imho much more usable in mvnd.
> * only for `offline-all` project which requires `maven-shade-plugin` to build fat-jar, the `shade-plugin` uses most of the build time. for `flow-all`, there are no time-consuming plugins observed.
Sure, but again, `mvnd` does not speed up plugin execution (or only to some degree because the JVM can be kept warm and JIT can kick in). You can also investigate your build adding `-Dmvnd.buildTime=true` on the command line, the report at the end should tell you where time was spent in plugins.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-mvnd] gnodet commented on issue #570: not fast
Posted by GitBox <gi...@apache.org>.
gnodet commented on issue #570:
URL: https://github.com/apache/maven-mvnd/issues/570#issuecomment-1008750142
> How come the duration for `mvn` is decreasing run after run ?
Well, it doesn't really. At least it's not really meaningful.
Do you have lots of test execution ? `mvnd` is not magic so if most of your time is already spent compiling and running tests and if you compare with a concurrent maven build, the only remaining benefit is to warm-up the JVM. The benefit becomes nearly insignificant if you spend 5 minutes running tests...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-mvnd] gnodet edited a comment on issue #570: not fast
Posted by GitBox <gi...@apache.org>.
gnodet edited a comment on issue #570:
URL: https://github.com/apache/maven-mvnd/issues/570#issuecomment-1008870741
So mvnd speeds up by
* enabling concurrent build by default
* cacheing a few things and help speeding up the time spent by maven *itself* when loading the poms, the plugins
* keeping the JVM warm
If if the plugins themselves use 95% of the build time and you're comparing to a multithreaded build, `mvnd` won't help much...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-mvnd] conderls edited a comment on issue #570: not fast
Posted by GitBox <gi...@apache.org>.
conderls edited a comment on issue #570:
URL: https://github.com/apache/maven-mvnd/issues/570#issuecomment-1009520138
> So mvnd speeds up by
>
> * enabling concurrent build by default
>
> * cacheing a few things and help speeding up the time spent by maven _itself_ when loading the poms, the plugins
>
> * keeping the JVM warm
> If if the plugins themselves use 95% of the build time and you're comparing to a multithreaded build, `mvnd` help much...
Thanks for your time.
- Considering "enabling concurrent build by default", it seems `mvn` with `-T 1C` opts can also achieve multi-threaded build?
- only for `offline-all` project which requires `maven-shade-plugin` to build fat-jar, the `shade-plugin` uses most of the build time. for `flow-all`, there are no time-consuming plugins observed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-mvnd] gnodet commented on issue #570: not fast
Posted by GitBox <gi...@apache.org>.
gnodet commented on issue #570:
URL: https://github.com/apache/maven-mvnd/issues/570#issuecomment-1009759496
Then there may be a problem. Can you provide a project for investigation ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-mvnd] conderls commented on issue #570: not fast
Posted by GitBox <gi...@apache.org>.
conderls commented on issue #570:
URL: https://github.com/apache/maven-mvnd/issues/570#issuecomment-1009520138
> So mvnd speeds up by
>
> * enabling concurrent build by default
>
> * cacheing a few things and help speeding up the time spent by maven _itself_ when loading the poms, the plugins
>
> * keeping the JVM warm
> If if the plugins themselves use 95% of the build time and you're comparing to a multithreaded build, `mvnd` help much...
Thanks for your time.
Considering "enabling concurrent build by default", it seems `mvn` with `-T 1C` opts can also achieve multi-threaded build?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-mvnd] gnodet closed issue #570: not fast
Posted by GitBox <gi...@apache.org>.
gnodet closed issue #570:
URL: https://github.com/apache/maven-mvnd/issues/570
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-mvnd] gnodet edited a comment on issue #570: not fast
Posted by GitBox <gi...@apache.org>.
gnodet edited a comment on issue #570:
URL: https://github.com/apache/maven-mvnd/issues/570#issuecomment-1007289044
Could you measure the time of the first, second and third build with `mvn` and `mvnd` and report ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-mvnd] gnodet closed issue #570: not fast
Posted by GitBox <gi...@apache.org>.
gnodet closed issue #570:
URL: https://github.com/apache/maven-mvnd/issues/570
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-mvnd] conderls commented on issue #570: not fast
Posted by GitBox <gi...@apache.org>.
conderls commented on issue #570:
URL: https://github.com/apache/maven-mvnd/issues/570#issuecomment-1009779410
> Then there may be a problem. Can you provide a project for investigation ?
sorry for that, the projects are accessible in intranet only now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-mvnd] gnodet commented on issue #570: not fast
Posted by GitBox <gi...@apache.org>.
gnodet commented on issue #570:
URL: https://github.com/apache/maven-mvnd/issues/570#issuecomment-1007289044
Could you measure the time of the first, second and third build and report ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-mvnd] gnodet commented on issue #570: not fast
Posted by GitBox <gi...@apache.org>.
gnodet commented on issue #570:
URL: https://github.com/apache/maven-mvnd/issues/570#issuecomment-1007289044
Could you measure the time of the first, second and third build and report ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-mvnd] conderls edited a comment on issue #570: not fast
Posted by GitBox <gi...@apache.org>.
conderls edited a comment on issue #570:
URL: https://github.com/apache/maven-mvnd/issues/570#issuecomment-1008721913
I also did a benchmark with two scala projects, which have about 30 subprojects:
- flow-all: without maven-shade-plugin,
- offline-all: with maven-shade-plugin to build **fat-jar** which required for spark task.
using commands for both mvn/mvnd: `mvn -pl $PROJECT -am spotless:apply clean --no-snapshot-updates $PHASE`, while extra opts for:
- `mvn`: `-T 1C`
- `mvnd`: `-Dquickly -Dmvnd.maxHeapSize=8G`
## reuslt
| PROJECT | PHASE | mvn(s) | mvnd(s) |
|:-----------:|:--------:|:-----------:|:------------|
| flow-all | install | 215/253/238/219/208/205(avg=223) | 254/259/245/205/230/209(avg=233) |
| offline-all | package | 416/433/431/443(avg=430) | 412/431/418/398/411/409(avg=413) |
the performance of compilation and build is quite close to each other.
by the way, it is time-consuming to build fat-jar with maven-shade-plugin, is there better way to create fat-jar for spark jobs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-mvnd] gnodet edited a comment on issue #570: not fast
Posted by GitBox <gi...@apache.org>.
gnodet edited a comment on issue #570:
URL: https://github.com/apache/maven-mvnd/issues/570#issuecomment-1007289044
Could you measure the time of the first, second and third build with `mvn` and `mvnd` and report ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-mvnd] conderls commented on issue #570: not fast
Posted by GitBox <gi...@apache.org>.
conderls commented on issue #570:
URL: https://github.com/apache/maven-mvnd/issues/570#issuecomment-1008721913
I also did a benchmark with two scala projects, which have about 30 subprojects:
- flow-all: without maven-shade-plugin,
- offline-all: with maven-shade-plugin to build **fat-jar** which required for spark task.
using commands for both mvn/mvnd: `mvn -pl $PROJECT -am spotless:apply clean --no-snapshot-updates $PHASE`, while extra opts for:
- `mvn`: `-T 1C`
- `mvnd`: `-Dquickly -Dmvnd.maxHeapSize=8G`
## reuslt
| PROJECT | PHASE | mvn(s) | mvnd(s) |
|:-----------:|:--------:|:-----------:|:------------|
| flow-all | install | 215/253/238/219/208/205(avg=223) | 254/259/245/205/230/209(avg=233) |
| offline-all | package | 416/433/431/443(avg=430) | 412/431/418/398/411/409(avg=413) |
by the way, it is time-consuming to build fat-jar with maven-shade-plugin, is there better way to create fat-jar for spark jobs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-mvnd] gnodet commented on issue #570: not fast
Posted by GitBox <gi...@apache.org>.
gnodet commented on issue #570:
URL: https://github.com/apache/maven-mvnd/issues/570#issuecomment-1009790291
Well, I can't do much without any data. Please open a new issue when/if you have something that can be reproduced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-mvnd] conderls commented on issue #570: not fast
Posted by GitBox <gi...@apache.org>.
conderls commented on issue #570:
URL: https://github.com/apache/maven-mvnd/issues/570#issuecomment-1008796157
>
by default, I turned off test during compiling and packaging by
```xml
<scoverage.skip>true</scoverage.skip>
<scalatest.skip>true</scalatest.skip>
<surefire.skip>true</surefire.skip>
```
for offline-all, it tooks about 50% time spent to build fat-jar for maven-shade-plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-mvnd] gnodet commented on issue #570: not fast
Posted by GitBox <gi...@apache.org>.
gnodet commented on issue #570:
URL: https://github.com/apache/maven-mvnd/issues/570#issuecomment-1008870741
So mvnd speeds up by
* enabling concurrent build by default
* cacheing a few things and help speeding up the time spent by maven *itself* when loading the poms, the plugins
* keeping the JVM warm
If if the plugins themselves use 95% of the build time and you're comparing to a multithreaded build, `mvnd` help much...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-mvnd] conderls commented on issue #570: not fast
Posted by GitBox <gi...@apache.org>.
conderls commented on issue #570:
URL: https://github.com/apache/maven-mvnd/issues/570#issuecomment-1009753047
with `-Dmvnd.buildTime=true` options, the report shows that, the time spent in plugins is negligible, e.g.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org