You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Mark Miller <ma...@gmail.com> on 2019/09/25 08:52:32 UTC
Solr Test Improvement PR From the Gradle Branch.
So my Gradle landing might be a little delayed, I've burned myself out a
little trying to get Solr tests to perform at top speed due to Gradle's
"dumb" parallel test distribution.
I extended the Gradle test task and made my own testFast target that is
"smart" and has been fun to play with, but that only drew me back into the
test quagmire I've been in and out of before. As soon as I saw tests start
to move faster, I was hooked again. The worst kind of addiction, every
additional diminishing savings costing more blood.
In the end I fixed a bunch more bs, pulled in a bunch of improvements and
fixes around tests from the Starburst branch that had not come over yet,
and have generally been wrapping my magic lasso around this test suite. A
bunch of that includes stuff in the code that was causing test problems or
slowness. I've put a lot of emphasis on moving actual slow stuff
to @Nightly runs and fixing not actual slow stuff.
On my 18 core dev machine, I've gotten down to about 7 minutes for all the
Solr modules currently in the Gradle build (there might be a new one not
there).
There is a bunch of clean up I have to do, and review of some things, and
again it's a rather large ''patch".
I'd like to get it out of my gradle branch and into master before moving in
Gradle, so I plan to put up a PR when I can in the near future and am
hoping for some help with review.
Thanks,
--
- Mark
http://about.me/markrmiller
Re: Solr Test Improvement PR From the Gradle Branch.
Posted by Mark Miller <ma...@gmail.com>.
But hey, we will start using the configset specified in the test on master
again!
On Thu, Sep 26, 2019 at 3:36 PM Mark Miller <ma...@gmail.com> wrote:
> Yes, it's all in the gradle branch, locally. I would like to take it out
> and put in master before moving forward with the gradle branch.
>
> It's not a tiny amount to digest unfortunately, but the results are pretty
> compelling.
>
> I've opened an issue here: SOLR-13796: Fix Solr Test Performance
>
> https://issues.apache.org/jira/browse/SOLR-13796#
>
> I have a bunch to finish up and cleanup and I've taken the approach of
> paring back hard so I can add back slowly and see what needs to stay out of
> fast test runs.
>
> I'd like to not have this stay outstanding for long though - which puts me
> in a tough spot, but what can you do. I plan to push a new branch next to
> the Gradle branch even sooner so maybe I can start getting feedback earlier.
>
> I'll be honest, I might have cut a sacred cow or two, you'll probably
> freak out about a couple things. We can address those, I would have done
> more but I'm more limited by the speed of input and number of classes than
> will to do.
>
> On Thu, Sep 26, 2019 at 12:22 PM Chris Hostetter <ho...@fucit.org>
> wrote:
>
>>
>> : In the end I fixed a bunch more bs, pulled in a bunch of improvements
>> and
>> : fixes around tests from the Starburst branch that had not come over yet,
>> : and have generally been wrapping my magic lasso around this test suite.
>> A
>> : bunch of that includes stuff in the code that was causing test problems
>> or
>> : slowness. I've put a lot of emphasis on moving actual slow stuff
>> : to @Nightly runs and fixing not actual slow stuff.
>>
>> ...
>>
>> : I'd like to get it out of my gradle branch and into master before
>> moving in
>> : Gradle, so I plan to put up a PR when I can in the near future and am
>> : hoping for some help with review.
>>
>> So if i understand correctly: You've got a bunch of stuff on your gradle
>> branch that is general test improvements/refactoring not dependent on or
>> directly related to the ant->gradle migration at all; and as such you
>> (wisely) want to review & commit it independently of the gradle migration.
>>
>> correct?
>>
>> I'm 100% on board helping to review & test those changes if you wnat to
>> tease it out into it's own branch / PR ... just ping me once you have a
>> jira so I don't miss it.
>>
>>
>> -Hoss
>> http://www.lucidworks.com/
>>
>
>
> --
> - Mark
>
> http://about.me/markrmiller
>
--
- Mark
http://about.me/markrmiller
Re: Solr Test Improvement PR From the Gradle Branch.
Posted by Mark Miller <ma...@gmail.com>.
Yes, it's all in the gradle branch, locally. I would like to take it out
and put in master before moving forward with the gradle branch.
It's not a tiny amount to digest unfortunately, but the results are pretty
compelling.
I've opened an issue here: SOLR-13796: Fix Solr Test Performance
https://issues.apache.org/jira/browse/SOLR-13796#
I have a bunch to finish up and cleanup and I've taken the approach of
paring back hard so I can add back slowly and see what needs to stay out of
fast test runs.
I'd like to not have this stay outstanding for long though - which puts me
in a tough spot, but what can you do. I plan to push a new branch next to
the Gradle branch even sooner so maybe I can start getting feedback earlier.
I'll be honest, I might have cut a sacred cow or two, you'll probably freak
out about a couple things. We can address those, I would have done more but
I'm more limited by the speed of input and number of classes than will to
do.
On Thu, Sep 26, 2019 at 12:22 PM Chris Hostetter <ho...@fucit.org>
wrote:
>
> : In the end I fixed a bunch more bs, pulled in a bunch of improvements and
> : fixes around tests from the Starburst branch that had not come over yet,
> : and have generally been wrapping my magic lasso around this test suite. A
> : bunch of that includes stuff in the code that was causing test problems
> or
> : slowness. I've put a lot of emphasis on moving actual slow stuff
> : to @Nightly runs and fixing not actual slow stuff.
>
> ...
>
> : I'd like to get it out of my gradle branch and into master before moving
> in
> : Gradle, so I plan to put up a PR when I can in the near future and am
> : hoping for some help with review.
>
> So if i understand correctly: You've got a bunch of stuff on your gradle
> branch that is general test improvements/refactoring not dependent on or
> directly related to the ant->gradle migration at all; and as such you
> (wisely) want to review & commit it independently of the gradle migration.
>
> correct?
>
> I'm 100% on board helping to review & test those changes if you wnat to
> tease it out into it's own branch / PR ... just ping me once you have a
> jira so I don't miss it.
>
>
> -Hoss
> http://www.lucidworks.com/
>
--
- Mark
http://about.me/markrmiller
Re: Solr Test Improvement PR From the Gradle Branch.
Posted by Chris Hostetter <ho...@fucit.org>.
: In the end I fixed a bunch more bs, pulled in a bunch of improvements and
: fixes around tests from the Starburst branch that had not come over yet,
: and have generally been wrapping my magic lasso around this test suite. A
: bunch of that includes stuff in the code that was causing test problems or
: slowness. I've put a lot of emphasis on moving actual slow stuff
: to @Nightly runs and fixing not actual slow stuff.
...
: I'd like to get it out of my gradle branch and into master before moving in
: Gradle, so I plan to put up a PR when I can in the near future and am
: hoping for some help with review.
So if i understand correctly: You've got a bunch of stuff on your gradle
branch that is general test improvements/refactoring not dependent on or
directly related to the ant->gradle migration at all; and as such you
(wisely) want to review & commit it independently of the gradle migration.
correct?
I'm 100% on board helping to review & test those changes if you wnat to
tease it out into it's own branch / PR ... just ping me once you have a
jira so I don't miss it.
-Hoss
http://www.lucidworks.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org