Non nightly test times

[Mark Miller <ma...@gmail.com>]

I can’t handle some of these test outliers anymore - the ones that run for
1-3 minutes with no added value and are easy to address. So I’m going to
address the ones that are annoying me most.

Please, after this, when changing or adding tests, if 95-99% of the non
nightly run comes in under a minute, and your test comes in at 2 minutes,
figure out how it can not be that test. I don’t want to spend all day
waiting for tests or cleaning them up.

MRM
-- 
- Mark

http://about.me/markrmiller

Re: Non nightly test times

[Mark Miller <ma...@gmail.com>]

I’ve got a number of jira issues, I’ll gather a list at some point.l this
week.

Without yet looking at a couple very long tests, it took a day or two to
bring the runs from my branch from the 12-13 min range to the 8-9 range.

As long as I’m feeling forward momentum, once I get those issues isolated
and committed, I’ll keep trimming the long tail. I didn’t get to a few of
the longest tests.

The reason it would be frustrating for tests to be added to the long tail
and that it’s very impactful to hit those tests is that the Gradle test
runner does not have the same capabilities as the ant runner. The ant test
framework will track test lengths with the ant build and launch longer
tests first. This helps it so that when using multiple test jvms, you don’t
end up at the end running a bunch of long tests on fewer jvms than you
specified. Gradle devs have discussed this kind of feature, it doesn’t seem
it’s coming any time soon. So long tail test lengths have a much greater
impact on test runs on the Gradle build than they do on the ant build.

The other thing I will do is remove the 10 second per test thread leak
linger and address thread leaks that we can’t control individually.

The test framework does a great job letting you know when you introduce
undesirable threads leaks. However, if you keep a universal linger as we
have or you let the framework try and compensate for leaky tests by
interrupting threads, developers can add poor production and test behavior
and not be alerted by the framework to address things.

I’ve done this on the branch with these test fixes as well, but I won’t
pull that over until all the individual test speeds up I currently have are
in.

Mark

On Sun, Sep 26, 2021 at 7:47 AM David Smiley <ds...@apache.org> wrote:

> Sounds good Mark.  Can you share a sample of test names on your mind?
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Tue, Sep 21, 2021 at 6:59 PM Mark Miller <ma...@gmail.com> wrote:
>
>> I can’t handle some of these test outliers anymore - the ones that run
>> for 1-3 minutes with no added value and are easy to address. So I’m going
>> to address the ones that are annoying me most.
>>
>> Please, after this, when changing or adding tests, if 95-99% of the non
>> nightly run comes in under a minute, and your test comes in at 2 minutes,
>> figure out how it can not be that test. I don’t want to spend all day
>> waiting for tests or cleaning them up.
>>
>> MRM
>> --
>> - Mark
>>
>> http://about.me/markrmiller
>>
> --
- Mark

http://about.me/markrmiller

Re: Non nightly test times

[David Smiley <ds...@apache.org>]

Sounds good Mark.  Can you share a sample of test names on your mind?

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Tue, Sep 21, 2021 at 6:59 PM Mark Miller <ma...@gmail.com> wrote:

> I can’t handle some of these test outliers anymore - the ones that run for
> 1-3 minutes with no added value and are easy to address. So I’m going to
> address the ones that are annoying me most.
>
> Please, after this, when changing or adding tests, if 95-99% of the non
> nightly run comes in under a minute, and your test comes in at 2 minutes,
> figure out how it can not be that test. I don’t want to spend all day
> waiting for tests or cleaning them up.
>
> MRM
> --
> - Mark
>
> http://about.me/markrmiller
>