You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by Chris Hostetter <ho...@fucit.org> on 2022/06/08 21:35:47 UTC
Obscenely long gradle compile times? (related to error-prone plugin?)
It's probably been at least a month (maybe more?) since I've done much
solr compilation, but today when I tried to test out a trivial local patch
(for SOLR-16241) I'm finding that compiling with gradle is obscenely
slow...
After explicitly removing my ~/.gradle/gradle.properties (in case it
wasn't playing nice w/some recent changes) i tried the following against
main/HEAD (using OpenJDK 11.0.4) ...
$ git clean -fxd && ./gradlew tasks && ./gradlew compileTestJava
The first gradlew command (to force it to download a fresh
gradle-wrapper.jar and run ':localSettings' only took 21s -- but the
second invocation (to pick up the local settings in a new daemon) took
"19m 31s" ! (note that's just compilation - not running any tests)
My CPU never got above 15%, and looking at occasional jstack dumps seemed
to show it always deep in errorprone classes. (Which i know isn't a new
thing in our build, and hasn't been upgraded since the last time i'm 100%
confident I ran the build to commit a bug fix)
When I hacked ./gradle/validation/error-prone.gradle to hardcode 'def
skipErrorProne = true' the entire "git clean -fxd ... && ./gradlew
compileTestJava" command started taking ~1 minute.
Jenkins builds don't seem to be this slow; which at first I assumed was
because error-prone.gradle ties skipErrorProne to rootProject.usesAltJvm
... but I don't see the WARN message from alternative-jdk-support.gradle
in any of the jenkins logs to suggest that they are being run in a way to
leverage the usesAltJvm support.
Has anyone else seen ridiculously long compilation times like this?
-Hoss
http://www.lucidworks.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Obscenely long gradle compile times? (related to error-prone plugin?)
Posted by Chris Hostetter <ho...@fucit.org>.
Thanks Uwe, jira filed...
https://issues.apache.org/jira/browse/SOLR-16258
: Date: Thu, 9 Jun 2022 10:52:03 +0200
: From: Uwe Schindler <uw...@thetaphi.de>
: Reply-To: dev@solr.apache.org
: To: dev@solr.apache.org
: Subject: Re: Obscenely long gradle compile times? (related to error-prone
: plugin?)
:
: Hi Chris,
:
: in later PRs done by Dawid and me we improved the whole thing even more.
: Instead of linking this to property tests.nightly (which would run it too
: seldom), we added another project property. In addition the Lucene build now
: detects if it is ran in a CI environment (it looks for env vars) and then
: enables it. This makes sure the error-prone is running always on Github pull
: requests or on Jenkins, but normal developers can enable it explicitly and its
: not ran by default:
:
: https://github.com/apache/lucene/blob/1dceff12c8bb764255927be55e46f4d435f47aed/gradle/globals.gradle
:
: https://github.com/apache/lucene/blob/1dceff12c8bb764255927be55e46f4d435f47aed/gradle/validation/error-prone.gradle
:
: Uwe
:
: Am 09.06.2022 um 01:31 schrieb Chris Hostetter:
: > FYI: https://issues.apache.org/jira/browse/LUCENE-9879
: >
: > This doesn't explin why I don't remember compilation being dog ass slow
: > before (even though checking out the SHAs of commits i've done a few
: > months ago shows comilation is dog ass slow against those points in time)
: > ...
: >
: > ... but it does explain why it's dog ass slow.
: >
: > Seems like Solr should probably follow Lucene's lead here ?
: >
: >
: >
: > : Date: Wed, 8 Jun 2022 14:35:47 -0700 (MST)
: > : From: Chris Hostetter <ho...@fucit.org>
: > : To: Solr Dev <de...@solr.apache.org>
: > : Subject: Obscenely long gradle compile times? (related to error-prone
: > plugin?)
: > :
: > :
: > : It's probably been at least a month (maybe more?) since I've done much
: > solr
: > : compilation, but today when I tried to test out a trivial local patch (for
: > : SOLR-16241) I'm finding that compiling with gradle is obscenely slow...
: > :
: > : After explicitly removing my ~/.gradle/gradle.properties (in case it
: > wasn't
: > : playing nice w/some recent changes) i tried the following against
: > main/HEAD
: > : (using OpenJDK 11.0.4) ...
: > :
: > : $ git clean -fxd && ./gradlew tasks && ./gradlew compileTestJava
: > :
: > : The first gradlew command (to force it to download a fresh
: > gradle-wrapper.jar
: > : and run ':localSettings' only took 21s -- but the second invocation (to
: > pick
: > : up the local settings in a new daemon) took "19m 31s" ! (note that's just
: > : compilation - not running any tests)
: > :
: > : My CPU never got above 15%, and looking at occasional jstack dumps seemed
: > to
: > : show it always deep in errorprone classes. (Which i know isn't a new
: > thing in
: > : our build, and hasn't been upgraded since the last time i'm 100% confident
: > I
: > : ran the build to commit a bug fix)
: > :
: > : When I hacked ./gradle/validation/error-prone.gradle to hardcode 'def
: > : skipErrorProne = true' the entire "git clean -fxd ... && ./gradlew
: > : compileTestJava" command started taking ~1 minute.
: > :
: > : Jenkins builds don't seem to be this slow; which at first I assumed was
: > : because error-prone.gradle ties skipErrorProne to rootProject.usesAltJvm
: > ...
: > : but I don't see the WARN message from alternative-jdk-support.gradle in
: > any of
: > : the jenkins logs to suggest that they are being run in a way to leverage
: > the
: > : usesAltJvm support.
: > :
: > :
: > : Has anyone else seen ridiculously long compilation times like this?
: > :
: > :
: > :
: > : -Hoss
: > : http://www.lucidworks.com/
: > :
: >
: > -Hoss
: > http://www.lucidworks.com/
: >
: > ---------------------------------------------------------------------
: > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
: > For additional commands, e-mail: dev-help@solr.apache.org
: >
: --
: Uwe Schindler
: Achterdiek 19, D-28357 Bremen
: https://www.thetaphi.de
: eMail: uwe@thetaphi.de
:
:
: ---------------------------------------------------------------------
: To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
: For additional commands, e-mail: dev-help@solr.apache.org
:
:
-Hoss
http://www.lucidworks.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Obscenely long gradle compile times? (related to error-prone plugin?)
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi Chris,
in later PRs done by Dawid and me we improved the whole thing even more.
Instead of linking this to property tests.nightly (which would run it
too seldom), we added another project property. In addition the Lucene
build now detects if it is ran in a CI environment (it looks for env
vars) and then enables it. This makes sure the error-prone is running
always on Github pull requests or on Jenkins, but normal developers can
enable it explicitly and its not ran by default:
https://github.com/apache/lucene/blob/1dceff12c8bb764255927be55e46f4d435f47aed/gradle/globals.gradle
https://github.com/apache/lucene/blob/1dceff12c8bb764255927be55e46f4d435f47aed/gradle/validation/error-prone.gradle
Uwe
Am 09.06.2022 um 01:31 schrieb Chris Hostetter:
> FYI: https://issues.apache.org/jira/browse/LUCENE-9879
>
> This doesn't explin why I don't remember compilation being dog ass slow
> before (even though checking out the SHAs of commits i've done a few
> months ago shows comilation is dog ass slow against those points in time)
> ...
>
> ... but it does explain why it's dog ass slow.
>
> Seems like Solr should probably follow Lucene's lead here ?
>
>
>
> : Date: Wed, 8 Jun 2022 14:35:47 -0700 (MST)
> : From: Chris Hostetter <ho...@fucit.org>
> : To: Solr Dev <de...@solr.apache.org>
> : Subject: Obscenely long gradle compile times? (related to error-prone plugin?)
> :
> :
> : It's probably been at least a month (maybe more?) since I've done much solr
> : compilation, but today when I tried to test out a trivial local patch (for
> : SOLR-16241) I'm finding that compiling with gradle is obscenely slow...
> :
> : After explicitly removing my ~/.gradle/gradle.properties (in case it wasn't
> : playing nice w/some recent changes) i tried the following against main/HEAD
> : (using OpenJDK 11.0.4) ...
> :
> : $ git clean -fxd && ./gradlew tasks && ./gradlew compileTestJava
> :
> : The first gradlew command (to force it to download a fresh gradle-wrapper.jar
> : and run ':localSettings' only took 21s -- but the second invocation (to pick
> : up the local settings in a new daemon) took "19m 31s" ! (note that's just
> : compilation - not running any tests)
> :
> : My CPU never got above 15%, and looking at occasional jstack dumps seemed to
> : show it always deep in errorprone classes. (Which i know isn't a new thing in
> : our build, and hasn't been upgraded since the last time i'm 100% confident I
> : ran the build to commit a bug fix)
> :
> : When I hacked ./gradle/validation/error-prone.gradle to hardcode 'def
> : skipErrorProne = true' the entire "git clean -fxd ... && ./gradlew
> : compileTestJava" command started taking ~1 minute.
> :
> : Jenkins builds don't seem to be this slow; which at first I assumed was
> : because error-prone.gradle ties skipErrorProne to rootProject.usesAltJvm ...
> : but I don't see the WARN message from alternative-jdk-support.gradle in any of
> : the jenkins logs to suggest that they are being run in a way to leverage the
> : usesAltJvm support.
> :
> :
> : Has anyone else seen ridiculously long compilation times like this?
> :
> :
> :
> : -Hoss
> : http://www.lucidworks.com/
> :
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: uwe@thetaphi.de
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Obscenely long gradle compile times? (related to error-prone plugin?)
Posted by Chris Hostetter <ho...@fucit.org>.
FYI: https://issues.apache.org/jira/browse/LUCENE-9879
This doesn't explin why I don't remember compilation being dog ass slow
before (even though checking out the SHAs of commits i've done a few
months ago shows comilation is dog ass slow against those points in time)
...
... but it does explain why it's dog ass slow.
Seems like Solr should probably follow Lucene's lead here ?
: Date: Wed, 8 Jun 2022 14:35:47 -0700 (MST)
: From: Chris Hostetter <ho...@fucit.org>
: To: Solr Dev <de...@solr.apache.org>
: Subject: Obscenely long gradle compile times? (related to error-prone plugin?)
:
:
: It's probably been at least a month (maybe more?) since I've done much solr
: compilation, but today when I tried to test out a trivial local patch (for
: SOLR-16241) I'm finding that compiling with gradle is obscenely slow...
:
: After explicitly removing my ~/.gradle/gradle.properties (in case it wasn't
: playing nice w/some recent changes) i tried the following against main/HEAD
: (using OpenJDK 11.0.4) ...
:
: $ git clean -fxd && ./gradlew tasks && ./gradlew compileTestJava
:
: The first gradlew command (to force it to download a fresh gradle-wrapper.jar
: and run ':localSettings' only took 21s -- but the second invocation (to pick
: up the local settings in a new daemon) took "19m 31s" ! (note that's just
: compilation - not running any tests)
:
: My CPU never got above 15%, and looking at occasional jstack dumps seemed to
: show it always deep in errorprone classes. (Which i know isn't a new thing in
: our build, and hasn't been upgraded since the last time i'm 100% confident I
: ran the build to commit a bug fix)
:
: When I hacked ./gradle/validation/error-prone.gradle to hardcode 'def
: skipErrorProne = true' the entire "git clean -fxd ... && ./gradlew
: compileTestJava" command started taking ~1 minute.
:
: Jenkins builds don't seem to be this slow; which at first I assumed was
: because error-prone.gradle ties skipErrorProne to rootProject.usesAltJvm ...
: but I don't see the WARN message from alternative-jdk-support.gradle in any of
: the jenkins logs to suggest that they are being run in a way to leverage the
: usesAltJvm support.
:
:
: Has anyone else seen ridiculously long compilation times like this?
:
:
:
: -Hoss
: http://www.lucidworks.com/
:
-Hoss
http://www.lucidworks.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org