You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Dawid Weiss <da...@gmail.com> on 2019/12/02 15:07:26 UTC
Gradle build effort (respin, please read)
Hi folks.
For those who prefer the short version:
git clone https://gitbox.apache.org/repos/asf/lucene-solr.git -b gradle-master
gradlew :help
gradlew -p lucene test
gradlew -p solr/packaging assemble
A longer explanation follows.
I realize many of you probably got tired of the discussions and issues
concerning the gradle build. I don't know why this turned so awkward
instead of exciting (which it is) but I wanted to try to reflect on
the lessons learned and retry the effort, perhaps with a better
outcome this time.
Erick and I spent some time trying to revive what Mark did but we
failed -- there were just too many mixed concerns and work-in-progress
changes to reconcile. I wanted something that would be:
- usable from the first minute,
- simple (no custom gradle plugins, etc.),
- divisible into smaller logical chunks so that contributions could be
smaller and incremental; without the need to work on the whole thing
for months,
- backward compatible with ant (because we are forced to use it until
everything is ported).
I took some of the things from Cao and Mark but I essentially rewrote
the build file and split it into logical fragments that configure
different build aspects:
https://github.com/apache/lucene-solr/blob/gradle-master/build.gradle#L18-L48
A good example of the "smaller fragment" approach is a developer aid
to display the slowest tests at the end of the run -- it's
self-contained and gracefully separated from anything else in the
build:
https://github.com/apache/lucene-solr/blob/gradle-master/gradle/testing/slowest-tests-at-end.gradle
It was, in fact, possible to isolate all the sore points and problems
present with the current ant-based build into separate files that
provide "workarounds" or "hacks" so that gradle works on the same file
structure as ant. All of these files indicate a potential problem with
the build itself (dependencies on test classes, non-conventional
folder naming, etc.) but until we get rid of ant we can't easily fix
these:
https://github.com/apache/lucene-solr/tree/gradle-master/gradle/ant-compat
The gradle-master branch is essentially a fork from the master branch
with gradle build files layered on top. My (most controversial)
decision was to embrace transitive dependencies -- I agree with Mark
that they just can't be managed manually anymore, especially for Solr.
Gradle has vast possibilities of excluding and configuring
dependencies in any way we like without the need to enumerate each and
every package manually. When something is added for the first time or
upgraded we will have a safety trigger of the lock file and jar
checksums (I still need to add this particular section though).
The build works for me just fine: tests run, assembly works, solr
packaging works. IntelliJ imports the project as it is, without any
need for custom tuning.
The question is how do we proceed from here. I can merge master fairly
often but I think it would make most sense if we folded this in to
master _as soon as possible_ so that people start doing actual
development using gradle (and provide real life feedback). Please
shout out if you have something against it.
In the meantime, if you're familiar with gradle and would like to help
then here is a list of things that are missing from the build:
https://issues.apache.org/jira/browse/LUCENE-9077
Some of them are simple, some are more difficult. The list is very
likely not close to complete. All of these items require some
understanding of how gradle works. Gradle's documentation is very good
but my experience tells me gradle can be confusing as hell at the
beginning nonetheless - if you don't understand something in the
current build let me know, I'll try to help.
Silent consensus is great but we'd really love to hear from you. Try
it and let us know what you think.
(Erick &) Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Gradle build effort (respin, please read)
Posted by Dawid Weiss <da...@gmail.com>.
This may help you understand the hierarchy of how properties are
shadowing each other.
https://docs.gradle.org/current/userguide/build_environment.html
The discussion concerning project properties that people could
"override" locally concerned the settings
set in the global gradle.properties file -- this was something I
wasn't too happy with if they clashed with other projects.
For now the gradle build doesn't have any means to populate certain
properties with custom values (other than command-line). If there is a
need for it we can add it.
Dawid
On Tue, Dec 3, 2019 at 11:48 PM Erick Erickson <er...@gmail.com> wrote:
>
> Thanks, I’m still untangling the various bits Mark put in and
> relying on people who actually, you know, understand
> things ;)
>
> > On Dec 3, 2019, at 5:11 PM, Dawid Weiss <da...@gmail.com> wrote:
> >
> >> // the project version
> >> version=9.0.0
> >
> > You shouldn't have it in your global configuration. In fact, only
> > these make sense in there:
> >
> > org.gradle.daemon=true
> > org.gradle.workers.max = 6
> >
> > The cache and daemon settings I'd leave to the project's config.
> >
> > Dawid
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: dev-help@lucene.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Gradle build effort (respin, please read)
Posted by Erick Erickson <er...@gmail.com>.
Thanks, I’m still untangling the various bits Mark put in and
relying on people who actually, you know, understand
things ;)
> On Dec 3, 2019, at 5:11 PM, Dawid Weiss <da...@gmail.com> wrote:
>
>> // the project version
>> version=9.0.0
>
> You shouldn't have it in your global configuration. In fact, only
> these make sense in there:
>
> org.gradle.daemon=true
> org.gradle.workers.max = 6
>
> The cache and daemon settings I'd leave to the project's config.
>
> Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Gradle build effort (respin, please read)
Posted by Dawid Weiss <da...@gmail.com>.
> // the project version
> version=9.0.0
You shouldn't have it in your global configuration. In fact, only
these make sense in there:
org.gradle.daemon=true
org.gradle.workers.max = 6
The cache and daemon settings I'd leave to the project's config.
Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Gradle build effort (respin, please read)
Posted by Erick Erickson <er...@gmail.com>.
Here’s what I have in ~/.gradle/gradle.properties that seems to at least be in the right ballpark. Based on Mark’s “intro to the gradle build” page:
// the project version
version=9.0.0
// whether or not to use the Gradle daemon - if true, keeps the build process around for reuse for up to 3 hours so that startup times are removed and hotspot has a chance to rock - you should generally set this to true in your ~/.gradle/gradle.properties
// we set to false so that a CI system with no setting will not use a daemon by default
org.gradle.daemon=true
// allows tasks to be executed in parallel, across modules
org.gradle.parallel=true
// max parallel jobs to run at once, including both tasks and tests.
// default is number of CPU cores which is often too high - you should start by setting it to half the number of cores, especially if you have hyperthreading
org.gradle.workers.max = 6
// number of jvms to distribute tests across in parallel
// defaults to number of CPU cores / 2 - you should just set this the same as org.gradle.workers.max
// NOTE: gradle does not try to balance tests across jvms yet: https://github.com/gradle/gradle/issues/2669
tests_jvms = 6
// enables gradles build cache, which will reuse cached build outputs when it can
org.gradle.caching=true
// experimental gradle feature - does not currently work with our version constraints plugin: https://github.com/palantir/gradle-consistent-versions/pull/145
// also known to have other issues and not known to really speed anything up anyhow
org.gradle.configureondemand=false
// how much ram the gradle daemon or process can use
org.gradle.jvmargs=-Xmx1g
> On Dec 3, 2019, at 3:57 PM, Dawid Weiss <da...@gmail.com> wrote:
>
> That's aligned with my intuition -- logical cores / 2. Sadly I don't
> see how we may compute the number of workers dynamically (you can do
> this with test forks per project but not with the overall number of
> workers). I asked about it [1]. This is perhaps one of the few
> settings you could override globally via ~/.gradle/gradle.properties
> because it really applies to your hardware (corrects the overestimated
> default by Gradle).
>
> org.gradle.workers.max=[cpu cures]/2
>
> D.
>
> [1] https://discuss.gradle.org/t/set-max-workers-dynamically/34087
>
> On Tue, Dec 3, 2019 at 9:19 PM David Smiley <da...@gmail.com> wrote:
>>
>> FWIW on a 6 year old MacBook Pro with a Quad-core i7, it seems max-workers of 2 is about right, clocking in at 21:32. 3 took 20:17; not much better.
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>>
>>
>> On Tue, Dec 3, 2019 at 9:13 AM Erick Erickson <er...@gmail.com> wrote:
>>>
>>> David Smiley:
>>>
>>> gradlew -p solr/packaging assemble
>>>
>>> couple of things:
>>> 1> The place you run bin/solr from is different
>>> 2> I didn’t need to specify the -p parameter and it defaulted to ‘...solr/packaging/build/solr-9.0.0-SNAPSHOT', FWIW.
>>>
>>> Once I got over having to switch to a different dir than I was accustomed to, I realized that by not mixing the build output with source, things are _much_ cleaner. After a “gradlew clean”, the packaging directory only contains a build.gradle file.
>>>
>>>> On Dec 3, 2019, at 4:48 AM, Jan Høydahl <ja...@cominvent.com> wrote:
>>>>
>>>> gradlew -p solr/packaging assemble
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Gradle build effort (respin, please read)
Posted by Dawid Weiss <da...@gmail.com>.
That's aligned with my intuition -- logical cores / 2. Sadly I don't
see how we may compute the number of workers dynamically (you can do
this with test forks per project but not with the overall number of
workers). I asked about it [1]. This is perhaps one of the few
settings you could override globally via ~/.gradle/gradle.properties
because it really applies to your hardware (corrects the overestimated
default by Gradle).
org.gradle.workers.max=[cpu cures]/2
D.
[1] https://discuss.gradle.org/t/set-max-workers-dynamically/34087
On Tue, Dec 3, 2019 at 9:19 PM David Smiley <da...@gmail.com> wrote:
>
> FWIW on a 6 year old MacBook Pro with a Quad-core i7, it seems max-workers of 2 is about right, clocking in at 21:32. 3 took 20:17; not much better.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Tue, Dec 3, 2019 at 9:13 AM Erick Erickson <er...@gmail.com> wrote:
>>
>> David Smiley:
>>
>> gradlew -p solr/packaging assemble
>>
>> couple of things:
>> 1> The place you run bin/solr from is different
>> 2> I didn’t need to specify the -p parameter and it defaulted to ‘...solr/packaging/build/solr-9.0.0-SNAPSHOT', FWIW.
>>
>> Once I got over having to switch to a different dir than I was accustomed to, I realized that by not mixing the build output with source, things are _much_ cleaner. After a “gradlew clean”, the packaging directory only contains a build.gradle file.
>>
>> > On Dec 3, 2019, at 4:48 AM, Jan Høydahl <ja...@cominvent.com> wrote:
>> >
>> > gradlew -p solr/packaging assemble
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Gradle build effort (respin, please read)
Posted by David Smiley <da...@gmail.com>.
FWIW on a 6 year old MacBook Pro with a Quad-core i7, it seems max-workers
of 2 is about right, clocking in at 21:32. 3 took 20:17; not much better.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Tue, Dec 3, 2019 at 9:13 AM Erick Erickson <er...@gmail.com>
wrote:
> David Smiley:
>
> gradlew -p solr/packaging assemble
>
> couple of things:
> 1> The place you run bin/solr from is different
> 2> I didn’t need to specify the -p parameter and it defaulted to
> ‘...solr/packaging/build/solr-9.0.0-SNAPSHOT', FWIW.
>
> Once I got over having to switch to a different dir than I was accustomed
> to, I realized that by not mixing the build output with source, things are
> _much_ cleaner. After a “gradlew clean”, the packaging directory only
> contains a build.gradle file.
>
> > On Dec 3, 2019, at 4:48 AM, Jan Høydahl <ja...@cominvent.com> wrote:
> >
> > gradlew -p solr/packaging assemble
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: Gradle build effort (respin, please read)
Posted by Erick Erickson <er...@gmail.com>.
David Smiley:
gradlew -p solr/packaging assemble
couple of things:
1> The place you run bin/solr from is different
2> I didn’t need to specify the -p parameter and it defaulted to ‘...solr/packaging/build/solr-9.0.0-SNAPSHOT', FWIW.
Once I got over having to switch to a different dir than I was accustomed to, I realized that by not mixing the build output with source, things are _much_ cleaner. After a “gradlew clean”, the packaging directory only contains a build.gradle file.
> On Dec 3, 2019, at 4:48 AM, Jan Høydahl <ja...@cominvent.com> wrote:
>
> gradlew -p solr/packaging assemble
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Gradle build effort (respin, please read)
Posted by Dawid Weiss <da...@gmail.com>.
Here's an example why I think toying with this gradle build is fun.
This adds a build fragment that cleans up the application of forbidden API
rules (they're applied automatically when a module consumes a certain
dependency).
Note how this is isolated from everything else, yet cross-cuts across
all modules to apply
it consistently and without too much boilerplate.
https://github.com/apache/lucene-solr/commit/6461909129c806a8c71d5e4d053061640fce4554
Dawid
D.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Gradle build effort (respin, please read)
Posted by Dawid Weiss <da...@gmail.com>.
> Great work Dawid. Just trying out solr tests now.
A full suite of solr and lucene takes ~50 minutes... And I think it's
pretty much the same with ant now (?).
> I see that it is not very smart in parallel scheduling. But that is probably something we can improve over time?
Maybe. I don't think it's going to improve things significantly
though. It'd take some deeper restructuring of the codebase to get
them to run faster.
> Is there some env.var or global setting to force e.g. -Ptests.seed=deadbeef for faster runs?
Fixing the seed should be only done if you want to run two consecutive
times with exactly the same combination of components. Don't fix it
locally forever, please. Those randomizations are really effective to
discover odd stuff (even at the JVM bugs level...).
> We probably don’t need to randomize everything everywhere every time?
Perhaps some of the stuff we do statically at test suite level
(LuceneTestCase etc.) is too eager... I didn't look into this. But
overall I disagree with Mark here -- I still have faith in that
randomization of components has value to it.
> Also, I like your moving of build files from the source tree. Will soon get used to gradlew -p solr/packaging assemble.
> Wonder how this will work with Admin UI changes. Hopefully you can just run a new assemble and reload browser on each change.
If you wish to rerun assembly with a server running in the background
it may be tricky because some files may be locked by live processes
(applies to Windows in particular) and some files generated by the
server (logs, etc.) may be removed by the sync (since it's a sync, not
a copy). Working on "live" files served via http is indeed going to be
tricky. You could always work on those files inside the assembled
distribution and copy them over to the sources once done... not a
solution but a workaround.
D.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Gradle build effort (respin, please read)
Posted by Jan Høydahl <ja...@cominvent.com>.
Great work Dawid. Just trying out solr tests now.
I see that it is not very smart in parallel scheduling. But that is probably something we can improve over time?
Is there some env.var or global setting to force e.g. -Ptests.seed=deadbeef for faster runs?
We probably don’t need to randomize everything everywhere every time?
Also, I like your moving of build files from the source tree. Will soon get used to gradlew -p solr/packaging assemble.
Wonder how this will work with Admin UI changes. Hopefully you can just run a new assemble and reload browser on each change.
Jan
> 3. des. 2019 kl. 09:35 skrev Dawid Weiss <da...@gmail.com>:
>
>> gradlew -p lucene test
>> That took my machine 27 minutes! I can see it used like 7 threads when I'd rather it use 3-4. That's probably why.
>
> I capped the execution of parallel tests to this
> (gradle/testing/defaults-tests.gradle):
>
> // Set up default parallel execution limits.
> maxParallelForks = (int) Math.max(1,
> Math.min(Runtime.runtime.availableProcessors() / 2.0, 3.0))
>
> but this limit applies per-project and gradle runs with N workers in
> parallel mode (where N equals the number of cores). For example, on my
> 8-core Linux machine (SSD drive) the test time for
>
> ./gradlew -p lucene cleanTest test -Ptests.seed=28BC12BA7C2CD894
>
> is 11m 32s. If I limit max workers to 4:
>
> ./gradlew -p lucene cleanTest test -Ptests.seed=28BC12BA7C2CD894 --max-workers=4
>
> the build time is XXX.
>
> It is obvious we need to fine-tune this but it's not obvious what the
> defaults should be because there are many dimensions to performance:
> I/O congestion, CPU cores, overall memory bandwidth and operating
> system (underlying filesystem implementation mostly, I believe). Also,
> as Mark pointed out before, the test runner is slightly different in
> Gradle and doesn't load-balance tests too efficiently (no work
> stealing).
>
> Finally, all the above also doesn't mean we can only think of
> improving gradle performance and not tests themselves... Some of them
> are just (very) slow. :)
>
> If you're willing to experiment then try to run with a varying number
> of workers (and identical seed to keep the tests the same). Then see
> which one turns out to be the best for you (and report back). I think
> it'll be half the number of cores (effectively physical cores) unless
> you have a very beefy machine when memory bandwidth and I/O comes into
> play.
>
> gradlew --max-workers N -Ptests.seed=deadbeef
>
>> gradlew :helpAnt
>> This failed: -- java.io.FileNotFoundException
>
> Corrected, apologies.
>
> Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Gradle build effort (respin, please read)
Posted by Dawid Weiss <da...@gmail.com>.
> ./gradlew -p lucene cleanTest test -Ptests.seed=28BC12BA7C2CD894 --max-workers=4
>
> the build time is XXX.
Yeah... you can tell I'm a non-linear e-mail writer, huh? s/XXX/11m 43s/.
For what it's worth, the "ant test" on Lucene takes an even longer
time on that machine:
BUILD SUCCESSFUL
Total time: 14 minutes 1 second
And I think it's a similar situation with Solr tests (I checked a
while ago, didn't try now).
Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Gradle build effort (respin, please read)
Posted by Dawid Weiss <da...@gmail.com>.
> gradlew -p lucene test
> That took my machine 27 minutes! I can see it used like 7 threads when I'd rather it use 3-4. That's probably why.
I capped the execution of parallel tests to this
(gradle/testing/defaults-tests.gradle):
// Set up default parallel execution limits.
maxParallelForks = (int) Math.max(1,
Math.min(Runtime.runtime.availableProcessors() / 2.0, 3.0))
but this limit applies per-project and gradle runs with N workers in
parallel mode (where N equals the number of cores). For example, on my
8-core Linux machine (SSD drive) the test time for
./gradlew -p lucene cleanTest test -Ptests.seed=28BC12BA7C2CD894
is 11m 32s. If I limit max workers to 4:
./gradlew -p lucene cleanTest test -Ptests.seed=28BC12BA7C2CD894 --max-workers=4
the build time is XXX.
It is obvious we need to fine-tune this but it's not obvious what the
defaults should be because there are many dimensions to performance:
I/O congestion, CPU cores, overall memory bandwidth and operating
system (underlying filesystem implementation mostly, I believe). Also,
as Mark pointed out before, the test runner is slightly different in
Gradle and doesn't load-balance tests too efficiently (no work
stealing).
Finally, all the above also doesn't mean we can only think of
improving gradle performance and not tests themselves... Some of them
are just (very) slow. :)
If you're willing to experiment then try to run with a varying number
of workers (and identical seed to keep the tests the same). Then see
which one turns out to be the best for you (and report back). I think
it'll be half the number of cores (effectively physical cores) unless
you have a very beefy machine when memory bandwidth and I/O comes into
play.
gradlew --max-workers N -Ptests.seed=deadbeef
> gradlew :helpAnt
> This failed: -- java.io.FileNotFoundException
Corrected, apologies.
Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Gradle build effort (respin, please read)
Posted by David Smiley <da...@gmail.com>.
Really impressive Dawid! I like how you were able to organize the build
files, especially the hacks and keeping the build.gradle files relatively
straightforward.
The dependency approach is totally good with me. It addresses the
essential problem with pure Maven which is unknowingly sucking in
transitive dependencies. Code reviews will continue to make new/updated
dependencies visible as they are today by forcing the commit to include
checksums 'n such.
gradlew -p lucene test
That took my machine 27 minutes! I can see it used like 7 threads when I'd
rather it use 3-4. That's probably why.
gradlew :helpAnt
This failed: -- java.io.FileNotFoundException:
/Users/dsmiley/DevSearch/lucene-solr/help/ant-gradle.txt (No such file or
directory) seems like a simple error related to an incomplete rename or
other refactor
What gradlew command is the equivalent of "ant server" for Solr? I wish to
use "bin/solr".
I'm +1 with you committing to master as soon as you are comfortable.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Mon, Dec 2, 2019 at 11:36 AM Dawid Weiss <da...@gmail.com> wrote:
> > - run a single unit test (NOTE: one of the TODO’s is running the test
> multiple times. Gradle defaults to marking a test as “good” once it passes
> and doesn’t re-run it until there are code changes. This can be fixed)
>
> See gradlew :helpTests, Erick -- I corrected some of it today. In short:
>
> You can already run a single test multiple times with tests.iters
> (passed via -P or -D). This works from IDEs as well (because it's the
> runner's code, not gradle). Full 'beasting' mode is more difficult
> because current working directories will overlap. I have some ideas
> how this can be solved.
>
> I don't think we should force tests to re-run if nothing has changed.
> If everything is up-to-date and no properties have changed, the tests
> should skip. It really helps with incremental runs. Please note that
> "nothing has changed" is tricky with gradle because it takes all the
> properties, arguments, etc. passed to the test runner as input -- we
> pick a random test seed on each run so tests should re-run
> automatically because the seed will be different. However, if you run
> the same seed multiple times over (gradlew test -Ptests.seed=deadbeef)
> then it will indeed skip them. Finally, if you wish to force a re-run
> of any task gradle gives you many options to do so (the simplest is to
> add a forced "cleanXXX" of a task named XXX - try it: gradlew
> cleanTest test -Ptests.seed=deadbeef).
>
> I will add this explanation to the tests help file.
>
> I've warned gradle can be frustratingly complex! (but once you learn
> it it's fun ;).
>
> D.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: Gradle build effort (respin, please read)
Posted by Dawid Weiss <da...@gmail.com>.
> - run a single unit test (NOTE: one of the TODO’s is running the test multiple times. Gradle defaults to marking a test as “good” once it passes and doesn’t re-run it until there are code changes. This can be fixed)
See gradlew :helpTests, Erick -- I corrected some of it today. In short:
You can already run a single test multiple times with tests.iters
(passed via -P or -D). This works from IDEs as well (because it's the
runner's code, not gradle). Full 'beasting' mode is more difficult
because current working directories will overlap. I have some ideas
how this can be solved.
I don't think we should force tests to re-run if nothing has changed.
If everything is up-to-date and no properties have changed, the tests
should skip. It really helps with incremental runs. Please note that
"nothing has changed" is tricky with gradle because it takes all the
properties, arguments, etc. passed to the test runner as input -- we
pick a random test seed on each run so tests should re-run
automatically because the seed will be different. However, if you run
the same seed multiple times over (gradlew test -Ptests.seed=deadbeef)
then it will indeed skip them. Finally, if you wish to force a re-run
of any task gradle gives you many options to do so (the simplest is to
add a forced "cleanXXX" of a task named XXX - try it: gradlew
cleanTest test -Ptests.seed=deadbeef).
I will add this explanation to the tests help file.
I've warned gradle can be frustratingly complex! (but once you learn
it it's fun ;).
D.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Gradle build effort (respin, please read)
Posted by Erick Erickson <er...@gmail.com>.
First, a slight correction. I deserve very little credit here,
Dawid did the heavy lifting, a huge shout-out to him! Cheering
from the sidelines is easy ;)
I played around a little with this last night, and AFAIC let’s just
fold it into master. From my perspective, it’s important that we
be able to try using this in baby steps, falling back to ant
whenever there’s something we can’t do in Gradle (yet).
Having more people contribute now that it’s functional will speed
this process.
Last night I was able to:
- build Solr
- run a single unit test (NOTE: one of the TODO’s is running the test multiple times. Gradle defaults to marking a test as “good” once it passes and doesn’t re-run it until there are code changes. This can be fixed)
- run the full test suite
- start solr with bin/solr start… (NOTE: this is in a slightly different place now)
- start Solr and attach a debugger and hit breakpoints.
IOW, I can do my regular work just using Gradle.
+1 to merging with master.
About merging into 8x. Straw-man proposal: we don’t. There are going to be a zillion commits on this before we stop using Ant, and merging into 8x after they’re all done will be “interesting”. So I propose we target 9.0 for using Gradle rather than Ant and do _NOT_ merge any of this into 8x. If the Gradle build isn’t ready by the time 9.0 comes out, we can still use Ant and switch over later.
> On Dec 2, 2019, at 10:07 AM, Dawid Weiss <da...@gmail.com> wrote:
>
> Hi folks.
>
> For those who prefer the short version:
> git clone https://gitbox.apache.org/repos/asf/lucene-solr.git -b gradle-master
> gradlew :help
> gradlew -p lucene test
> gradlew -p solr/packaging assemble
>
> A longer explanation follows.
>
> I realize many of you probably got tired of the discussions and issues
> concerning the gradle build. I don't know why this turned so awkward
> instead of exciting (which it is) but I wanted to try to reflect on
> the lessons learned and retry the effort, perhaps with a better
> outcome this time.
>
> Erick and I spent some time trying to revive what Mark did but we
> failed -- there were just too many mixed concerns and work-in-progress
> changes to reconcile. I wanted something that would be:
>
> - usable from the first minute,
> - simple (no custom gradle plugins, etc.),
> - divisible into smaller logical chunks so that contributions could be
> smaller and incremental; without the need to work on the whole thing
> for months,
> - backward compatible with ant (because we are forced to use it until
> everything is ported).
>
> I took some of the things from Cao and Mark but I essentially rewrote
> the build file and split it into logical fragments that configure
> different build aspects:
>
> https://github.com/apache/lucene-solr/blob/gradle-master/build.gradle#L18-L48
>
> A good example of the "smaller fragment" approach is a developer aid
> to display the slowest tests at the end of the run -- it's
> self-contained and gracefully separated from anything else in the
> build:
>
> https://github.com/apache/lucene-solr/blob/gradle-master/gradle/testing/slowest-tests-at-end.gradle
>
> It was, in fact, possible to isolate all the sore points and problems
> present with the current ant-based build into separate files that
> provide "workarounds" or "hacks" so that gradle works on the same file
> structure as ant. All of these files indicate a potential problem with
> the build itself (dependencies on test classes, non-conventional
> folder naming, etc.) but until we get rid of ant we can't easily fix
> these:
>
> https://github.com/apache/lucene-solr/tree/gradle-master/gradle/ant-compat
>
> The gradle-master branch is essentially a fork from the master branch
> with gradle build files layered on top. My (most controversial)
> decision was to embrace transitive dependencies -- I agree with Mark
> that they just can't be managed manually anymore, especially for Solr.
> Gradle has vast possibilities of excluding and configuring
> dependencies in any way we like without the need to enumerate each and
> every package manually. When something is added for the first time or
> upgraded we will have a safety trigger of the lock file and jar
> checksums (I still need to add this particular section though).
>
> The build works for me just fine: tests run, assembly works, solr
> packaging works. IntelliJ imports the project as it is, without any
> need for custom tuning.
>
> The question is how do we proceed from here. I can merge master fairly
> often but I think it would make most sense if we folded this in to
> master _as soon as possible_ so that people start doing actual
> development using gradle (and provide real life feedback). Please
> shout out if you have something against it.
>
> In the meantime, if you're familiar with gradle and would like to help
> then here is a list of things that are missing from the build:
> https://issues.apache.org/jira/browse/LUCENE-9077
>
> Some of them are simple, some are more difficult. The list is very
> likely not close to complete. All of these items require some
> understanding of how gradle works. Gradle's documentation is very good
> but my experience tells me gradle can be confusing as hell at the
> beginning nonetheless - if you don't understand something in the
> current build let me know, I'll try to help.
>
> Silent consensus is great but we'd really love to hear from you. Try
> it and let us know what you think.
>
> (Erick &) Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org