You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@netbeans.apache.org by Jaroslav Tulach <ja...@gmail.com> on 2021/11/03 03:20:24 UTC
Speeding up CI by running unit tests in a single JVM
Hi.
One of the reasons why our CI takes so long is the old habit to execute
each of our tests in a separate JVM. I am aware (now after five years
working on GraalVM) that JVM doesn't do anything useful for the first two
seconds. It starts in interpreter mode, collects profiles, invokes C1
compiler and (in the worst) case also C2 compiler. Only then the JVM
reaches the point of "peak performance" when it is actually useful. However
before reaching the point many of our tests are over. As a result running:
$ ant -f java/java.source.base/ test
just took 9.5 minutes on my computer. The cure is simple, if I change the
forkMode:
diff --git a/nbbuild/templates/common.xml b/nbbuild/templates/common.xml
index 377b6db0d1a4..ec063c3ba7b4 100644
--- a/nbbuild/templates/common.xml
+++ b/nbbuild/templates/common.xml
@@ -740,7 +740,7 @@
<attribute name="test.type"/>
<attribute name="disable.apple.ui" default="false"/>
<sequential>
- <junit showoutput="true" fork="true"
failureproperty="tests.failed" errorproperty="tests.failed"
filtertrace="${test.filter.trace}"
tempdir="${build.test.@{test.type}.results.dir}"
timeout=
"${test.timeout}" jvm="${test.nbjdk.java}">
+ <junit showoutput="true" fork="true" forkmode="perBatch"
failureproperty="tests.failed" errorproperty="tests.failed"
filtertrace="${test.filter.trace}" tempdir="${build.test.@{test.type}.re
sults.dir}" timeout="${test.timeout}" jvm="${test.nbjdk.java}">
<batchtest todir="${build.test.@{test.type}.results.dir}">
<fileset dir="${build.test.@{test.type}.classes.dir}"
includes="${test.includes}" excludes="${test.excludes}"/>
</batchtest>
then the test suite runs in 4.5 minutes. I believe similar speed up could
be achieved in other test suites as well.
Of course, there is a drawback. By running the tests in a single JVM they
may start to influence each other. Sure, unit tests shouldn't do that, but
that is easier said than done, especially with [Injectable Singletons](
http://wiki.apidesign.org/wiki/Injectable_Singleton) - used everywhere in
NetBeans - and their limited co-existence capabilities. As such we probably
need an opt-in for modules/tests to turn the single JVM mode on.
Does making CI faster worth it? Anyone willing to give it a try?
-jt
PS: Please avoid [Pentium-like jokes](
https://twitter.com/chainq/status/948725922642976768) that it is "faster
but incorrect". With opt-in, it can be faster and correct.
Re: Speeding up CI by running unit tests in a single JVM
Posted by Eric Bresie <eb...@gmail.com>.
Silly question, when indicating CI, does this mean Travis CI, the Github
Actions/builds, or Apache CI (Jenkins) pipeline? Assume in any case the
results of change in the specific project tests might be similar but may
require different implementations of CI side configuration if needed.
The single thread seems appealing to speed things up, but assume cases like
testing different jdk versions would still fall under the scope of needing
a new JVM.
In past troubleshooting on one CI issue (believe it was one of the Gradle
Project tests during one of the Travis CI tests), it seemed like a possible
concurrency issue at the time, so assume in that case having single thread
might reduce the potential multithreads issues while different tests
working with same lock/shared files as the same time.
Eric Bresie
ebresie@gmail.com
On Wed, Nov 3, 2021 at 12:48 AM Michael Bien <mb...@gmail.com> wrote:
> if test suites can be identified which only run true functional unit
> tests without booting up the whole module system, sure why not. But lets
> try to be careful there and not add more failure modes :)
>
> CI needs more reliability at the moment, not less. Devs second guessing
> the test results is not a good situation to be in - i don't even have a
> restart button, i usually just ignore a failure and hope someone
> restarts it.
>
>
> As a sidenote: build itself is a big slice of the total time.
>
> If you take a look at:
> https://github.com/mbien/netbeans/actions/runs/1388038850
>
> You see build + commit validation takes together about 17 mins (rest is
> setup, tar.gz and upload).
>
> The secondary jobs which only do extract + test (using the same tar from
> the primary job) are fairly fast and can run in parallel (they don't do
> full testing to not waste apache resources on pipeline experiments). JDK
> 8 is also not the fastest starting JVM around the block - secondary test
> jobs should be faster by default.
>
> (JDK 16+17 tests are the fastest since they all fail early ;))
>
> regards,
> michael
>
>
> On 03.11.21 04:20, Jaroslav Tulach wrote:
> > Hi.
> > One of the reasons why our CI takes so long is the old habit to execute
> > each of our tests in a separate JVM. I am aware (now after five years
> > working on GraalVM) that JVM doesn't do anything useful for the first two
> > seconds. It starts in interpreter mode, collects profiles, invokes C1
> > compiler and (in the worst) case also C2 compiler. Only then the JVM
> > reaches the point of "peak performance" when it is actually useful.
> However
> > before reaching the point many of our tests are over. As a result
> running:
> >
> > $ ant -f java/java.source.base/ test
> >
> > just took 9.5 minutes on my computer. The cure is simple, if I change the
> > forkMode:
> >
> > diff --git a/nbbuild/templates/common.xml b/nbbuild/templates/common.xml
> > index 377b6db0d1a4..ec063c3ba7b4 100644
> > --- a/nbbuild/templates/common.xml
> > +++ b/nbbuild/templates/common.xml
> > @@ -740,7 +740,7 @@
> > <attribute name="test.type"/>
> > <attribute name="disable.apple.ui" default="false"/>
> > <sequential>
> > - <junit showoutput="true" fork="true"
> > failureproperty="tests.failed" errorproperty="tests.failed"
> > filtertrace="${test.filter.trace}"
> > tempdir="${build.test.@{test.type}.results.dir}"
> > timeout=
> > "${test.timeout}" jvm="${test.nbjdk.java}">
> > + <junit showoutput="true" fork="true" forkmode="perBatch"
> > failureproperty="tests.failed" errorproperty="tests.failed"
> > filtertrace="${test.filter.trace}" tempdir="${build.test.@{test.type}.re
> > sults.dir}" timeout="${test.timeout}" jvm="${test.nbjdk.java}">
> > <batchtest todir="${build.test.@
> {test.type}.results.dir}">
> >
> > <fileset dir="${build.test.@
> {test.type}.classes.dir}"
> > includes="${test.includes}" excludes="${test.excludes}"/>
> > </batchtest>
> >
> > then the test suite runs in 4.5 minutes. I believe similar speed up could
> > be achieved in other test suites as well.
> >
> > Of course, there is a drawback. By running the tests in a single JVM they
> > may start to influence each other. Sure, unit tests shouldn't do that,
> but
> > that is easier said than done, especially with [Injectable Singletons](
> > http://wiki.apidesign.org/wiki/Injectable_Singleton) - used everywhere
> in
> > NetBeans - and their limited co-existence capabilities. As such we
> probably
> > need an opt-in for modules/tests to turn the single JVM mode on.
> >
> > Does making CI faster worth it? Anyone willing to give it a try?
> > -jt
> >
> > PS: Please avoid [Pentium-like jokes](
> > https://twitter.com/chainq/status/948725922642976768) that it is "faster
> > but incorrect". With opt-in, it can be faster and correct.
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> For additional commands, e-mail: dev-help@netbeans.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
>
Re: Speeding up CI by running unit tests in a single JVM
Posted by Michael Bien <mb...@gmail.com>.
if test suites can be identified which only run true functional unit
tests without booting up the whole module system, sure why not. But lets
try to be careful there and not add more failure modes :)
CI needs more reliability at the moment, not less. Devs second guessing
the test results is not a good situation to be in - i don't even have a
restart button, i usually just ignore a failure and hope someone
restarts it.
As a sidenote: build itself is a big slice of the total time.
If you take a look at:
https://github.com/mbien/netbeans/actions/runs/1388038850
You see build + commit validation takes together about 17 mins (rest is
setup, tar.gz and upload).
The secondary jobs which only do extract + test (using the same tar from
the primary job) are fairly fast and can run in parallel (they don't do
full testing to not waste apache resources on pipeline experiments). JDK
8 is also not the fastest starting JVM around the block - secondary test
jobs should be faster by default.
(JDK 16+17 tests are the fastest since they all fail early ;))
regards,
michael
On 03.11.21 04:20, Jaroslav Tulach wrote:
> Hi.
> One of the reasons why our CI takes so long is the old habit to execute
> each of our tests in a separate JVM. I am aware (now after five years
> working on GraalVM) that JVM doesn't do anything useful for the first two
> seconds. It starts in interpreter mode, collects profiles, invokes C1
> compiler and (in the worst) case also C2 compiler. Only then the JVM
> reaches the point of "peak performance" when it is actually useful. However
> before reaching the point many of our tests are over. As a result running:
>
> $ ant -f java/java.source.base/ test
>
> just took 9.5 minutes on my computer. The cure is simple, if I change the
> forkMode:
>
> diff --git a/nbbuild/templates/common.xml b/nbbuild/templates/common.xml
> index 377b6db0d1a4..ec063c3ba7b4 100644
> --- a/nbbuild/templates/common.xml
> +++ b/nbbuild/templates/common.xml
> @@ -740,7 +740,7 @@
> <attribute name="test.type"/>
> <attribute name="disable.apple.ui" default="false"/>
> <sequential>
> - <junit showoutput="true" fork="true"
> failureproperty="tests.failed" errorproperty="tests.failed"
> filtertrace="${test.filter.trace}"
> tempdir="${build.test.@{test.type}.results.dir}"
> timeout=
> "${test.timeout}" jvm="${test.nbjdk.java}">
> + <junit showoutput="true" fork="true" forkmode="perBatch"
> failureproperty="tests.failed" errorproperty="tests.failed"
> filtertrace="${test.filter.trace}" tempdir="${build.test.@{test.type}.re
> sults.dir}" timeout="${test.timeout}" jvm="${test.nbjdk.java}">
> <batchtest todir="${build.test.@{test.type}.results.dir}">
>
> <fileset dir="${build.test.@{test.type}.classes.dir}"
> includes="${test.includes}" excludes="${test.excludes}"/>
> </batchtest>
>
> then the test suite runs in 4.5 minutes. I believe similar speed up could
> be achieved in other test suites as well.
>
> Of course, there is a drawback. By running the tests in a single JVM they
> may start to influence each other. Sure, unit tests shouldn't do that, but
> that is easier said than done, especially with [Injectable Singletons](
> http://wiki.apidesign.org/wiki/Injectable_Singleton) - used everywhere in
> NetBeans - and their limited co-existence capabilities. As such we probably
> need an opt-in for modules/tests to turn the single JVM mode on.
>
> Does making CI faster worth it? Anyone willing to give it a try?
> -jt
>
> PS: Please avoid [Pentium-like jokes](
> https://twitter.com/chainq/status/948725922642976768) that it is "faster
> but incorrect". With opt-in, it can be faster and correct.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists