The Wild World of Solr Tests

[Mark Miller <ma...@gmail.com>]

Just an FYI on a plan I have to try and improve Solr’s test suite:

Many years back, a few changes happened that made getting a handle on
Solr’s tests quite difficult.

We started running tests in parallel.

We added a random testing framework and mentality.

We started running a few Jenkins all the time rather than one or few runs
per day.

We added a lot of distributed code and tests with timeouts and many
complicated interactions.

The results have been great on many fronts, but the number of fails
produced and the current Jenkins reporting has made the Solr test suite
quite hard to get a handle on.

It is much too hard to tell even some basic things: what tests are flakey
right now? which test fails actually affect devs most (fails that happen
even in a clean, well resourced environment)? did I break it? was that test
already flakey? is that test still flakey? what are our worst tests right
now? is that test getting better or worse? Is it a bad test or just a bad
test under low resources? How many tests are flakey?

The stream of email fails is easy to ignore and hard to follow. Even if you
do follow, the information is an always changing stream that is difficult
to summarize. (though we can do things here too, a while back I sent a
couple test reports by doing simple regexes on emails and trying to count
up test fails).


Our nightly tests also get little to no visibility due to all the fails.
Sometimes some of those tests are simply broken because changes were made
that didn't account for them. We need a way to highlight @Nightly only test
fails.

I, like many others, have spent a lot of time trying to improve our test
stability over the years. But it’s whack a mole and often feels like it’s
hard to make real progress on hardening the whole suite.

Part of the issue is that Solr has almost 900 tests. If even just 10-20 of
them fail 1 out 30 or 1 out of 60 runs, that is going to produce plenty of
‘ant test’ fails when many Jenkins machines are blasting all day.


For individual tests, one way we have found that works well for hardening
is to ‘beast’ the test. Or run it many times, ideally in parallel. It’s
been in my head a long time, but I have finally built upon that strategy
and extended it out to a full ‘test run’ of beasting.

By beasting every test, I am able to generate a test report based on a
single commit point that objectively scores each test. Currently I am doing
30 test runs, 10 at a time. Eventually I’d like to up that to at least 50,
but of course I can also just go higher for the now growing list of known
flakey tests. We also will have some easy to reference history though.
Tests will carry a checkable reputation.

Also, unlike many Jenkins failures, reproducing is usually quite simple.
Just beast it again. Sometimes you might have to do it more than once or
for more runs, but generally it's easy to pop up the fail again or test a
fix. So far over the years, I have never hit a fail while beasting that I
couldn't hit again, even very rare chaos monkey fails that would take
100-300 runs to hit.

Anyway, I’m working on this strategy here: SOLR-10032: Create report to
assess Solr test quality at a commit point.

I am about to put up my 3rd report, and I’m going to soon start summarizing
these reports and pinging the dev list with the results. We will see where
it goes, but I think we have few enough troublesome tests that we can make
a very significant improvement over the next couple months.


It nicely highlights the @Nightly only test results, which tests are
Ignored, BadAppled and AwaitsFixed, and new tests that enter the report as
failing will also be highlighted and can be pushed back against. The
cleaner we get things, the more strict we can try and be.

Right now I’m mainly working on the ugliest tests. There are only a
handful. Once all of the tests are just flakey (fail < 10% out of the 30
runs), I’m going to start pushing on those individual tests harder and will
try and encourage authors of those tests to help.


- Mark
-- 
- Mark
about.me/markrmiller

Re: The Wild World of Solr Tests

[David Smiley <da...@gmail.com>]

Thanks so much Mark for doing this -- it is so important!  The current
state is a sad state.

Another thing related to Solr tests I've been thinking about is trying to
make tests more self contained instead of needing custom schema or configs,
or running the risk of adding yet more info to a shared config.  In many
(but not all) cases, I believe it's possible to write tests using
ManagedSchema (and some techniques I'll share later), to avoid it.  Once I
finish a feature I'm working on that uses this technique I'll share it more
broadly to get public comment.

~ David

On Wed, Feb 8, 2017 at 7:02 PM Mark Miller <ma...@gmail.com> wrote:

> Just an FYI on a plan I have to try and improve Solr’s test suite:
>
> Many years back, a few changes happened that made getting a handle on
> Solr’s tests quite difficult.
>
> We started running tests in parallel.
>
> We added a random testing framework and mentality.
>
> We started running a few Jenkins all the time rather than one or few runs
> per day.
>
> We added a lot of distributed code and tests with timeouts and many
> complicated interactions.
>
> The results have been great on many fronts, but the number of fails
> produced and the current Jenkins reporting has made the Solr test suite
> quite hard to get a handle on.
>
> It is much too hard to tell even some basic things: what tests are flakey
> right now? which test fails actually affect devs most (fails that happen
> even in a clean, well resourced environment)? did I break it? was that test
> already flakey? is that test still flakey? what are our worst tests right
> now? is that test getting better or worse? Is it a bad test or just a bad
> test under low resources? How many tests are flakey?
>
> The stream of email fails is easy to ignore and hard to follow. Even if
> you do follow, the information is an always changing stream that is
> difficult to summarize. (though we can do things here too, a while back I
> sent a couple test reports by doing simple regexes on emails and trying to
> count up test fails).
>
>
> Our nightly tests also get little to no visibility due to all the fails.
> Sometimes some of those tests are simply broken because changes were made
> that didn't account for them. We need a way to highlight @Nightly only test
> fails.
>
> I, like many others, have spent a lot of time trying to improve our test
> stability over the years. But it’s whack a mole and often feels like it’s
> hard to make real progress on hardening the whole suite.
>
> Part of the issue is that Solr has almost 900 tests. If even just 10-20 of
> them fail 1 out 30 or 1 out of 60 runs, that is going to produce plenty of
> ‘ant test’ fails when many Jenkins machines are blasting all day.
>
>
> For individual tests, one way we have found that works well for hardening
> is to ‘beast’ the test. Or run it many times, ideally in parallel. It’s
> been in my head a long time, but I have finally built upon that strategy
> and extended it out to a full ‘test run’ of beasting.
>
> By beasting every test, I am able to generate a test report based on a
> single commit point that objectively scores each test. Currently I am doing
> 30 test runs, 10 at a time. Eventually I’d like to up that to at least 50,
> but of course I can also just go higher for the now growing list of known
> flakey tests. We also will have some easy to reference history though.
> Tests will carry a checkable reputation.
>
> Also, unlike many Jenkins failures, reproducing is usually quite simple.
> Just beast it again. Sometimes you might have to do it more than once or
> for more runs, but generally it's easy to pop up the fail again or test a
> fix. So far over the years, I have never hit a fail while beasting that I
> couldn't hit again, even very rare chaos monkey fails that would take
> 100-300 runs to hit.
>
> Anyway, I’m working on this strategy here: SOLR-10032: Create report to
> assess Solr test quality at a commit point.
>
> I am about to put up my 3rd report, and I’m going to soon start
> summarizing these reports and pinging the dev list with the results. We
> will see where it goes, but I think we have few enough troublesome tests
> that we can make a very significant improvement over the next couple months.
>
>
> It nicely highlights the @Nightly only test results, which tests are
> Ignored, BadAppled and AwaitsFixed, and new tests that enter the report
> as failing will also be highlighted and can be pushed back against. The
> cleaner we get things, the more strict we can try and be.
>
> Right now I’m mainly working on the ugliest tests. There are only a
> handful. Once all of the tests are just flakey (fail < 10% out of the 30
> runs), I’m going to start pushing on those individual tests harder and will
> try and encourage authors of those tests to help.
>
>
> - Mark
> --
> - Mark
> about.me/markrmiller
>
-- 
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com

RE: The Wild World of Solr Tests

[Uwe Schindler <uw...@thetaphi.de>]

Thanks Mark!

 

We can make nightly tests more visible by changing their Subject in the e-mail and maybe add some other priority flags. I did this for the Java 9 EA builds to mark them with “JENKINS-EA” and also set their priority to “low”.

 

Uwe

 

-----

Uwe Schindler

Achterdiek 19, D-28357 Bremen

http://www.thetaphi.de <http://www.thetaphi.de/> 

eMail: uwe@thetaphi.de

 

From: Mark Miller [mailto:markrmiller@gmail.com] 
Sent: Thursday, February 9, 2017 1:02 AM
To: dev@lucene.apache.org
Subject: The Wild World of Solr Tests

 

Just an FYI on a plan I have to try and improve Solr’s test suite:

 

Many years back, a few changes happened that made getting a handle on Solr’s tests quite difficult.

 

We started running tests in parallel.

We added a random testing framework and mentality.

We started running a few Jenkins all the time rather than one or few runs per day.

We added a lot of distributed code and tests with timeouts and many complicated interactions.

 

The results have been great on many fronts, but the number of fails produced and the current Jenkins reporting has made the Solr test suite quite hard to get a handle on.

 

It is much too hard to tell even some basic things: what tests are flakey right now? which test fails actually affect devs most (fails that happen even in a clean, well resourced environment)? did I break it? was that test already flakey? is that test still flakey? what are our worst tests right now? is that test getting better or worse? Is it a bad test or just a bad test under low resources? How many tests are flakey?

 

The stream of email fails is easy to ignore and hard to follow. Even if you do follow, the information is an always changing stream that is difficult to summarize. (though we can do things here too, a while back I sent a couple test reports by doing simple regexes on emails and trying to count up test fails).

 

Our nightly tests also get little to no visibility due to all the fails. Sometimes some of those tests are simply broken because changes were made that didn't account for them. We need a way to highlight @Nightly only test fails.

 

I, like many others, have spent a lot of time trying to improve our test stability over the years. But it’s whack a mole and often feels like it’s hard to make real progress on hardening the whole suite.

 

Part of the issue is that Solr has almost 900 tests. If even just 10-20 of them fail 1 out 30 or 1 out of 60 runs, that is going to produce plenty of ‘ant test’ fails when many Jenkins machines are blasting all day. 

 

For individual tests, one way we have found that works well for hardening is to ‘beast’ the test. Or run it many times, ideally in parallel. It’s been in my head a long time, but I have finally built upon that strategy and extended it out to a full ‘test run’ of beasting.

 

By beasting every test, I am able to generate a test report based on a single commit point that objectively scores each test. Currently I am doing 30 test runs, 10 at a time. Eventually I’d like to up that to at least 50, but of course I can also just go higher for the now growing list of known flakey tests. We also will have some easy to reference history though. Tests will carry a checkable reputation.

 

Also, unlike many Jenkins failures, reproducing is usually quite simple. Just beast it again. Sometimes you might have to do it more than once or for more runs, but generally it's easy to pop up the fail again or test a fix. So far over the years, I have never hit a fail while beasting that I couldn't hit again, even very rare chaos monkey fails that would take 100-300 runs to hit.

 

Anyway, I’m working on this strategy here: SOLR-10032: Create report to assess Solr test quality at a commit point.

 

I am about to put up my 3rd report, and I’m going to soon start summarizing these reports and pinging the dev list with the results. We will see where it goes, but I think we have few enough troublesome tests that we can make a very significant improvement over the next couple months.

 

It nicely highlights the @Nightly only test results, which tests are Ignored, BadAppled and AwaitsFixed, and new tests that enter the report as failing will also be highlighted and can be pushed back against. The cleaner we get things, the more strict we can try and be.

 

Right now I’m mainly working on the ugliest tests. There are only a handful. Once all of the tests are just flakey (fail < 10% out of the 30 runs), I’m going to start pushing on those individual tests harder and will try and encourage authors of those tests to help.

 

- Mark

-- 

- Mark 

about.me/markrmiller <http://about.me/markrmiller> 

Re: The Wild World of Solr Tests

[Alan Woodward <al...@flax.co.uk>]

+1, this is great work, thanks Mark.

> On 9 Feb 2017, at 20:39, Dawid Weiss <da...@gmail.com> wrote:
> 
> This is a very important, hard and ungrateful task. Thanks for doing
> this Mark. As you know I tried (with your help!) to clean some of this
> mess a long time ago and sadly failed. It'd be great to speed those
> tests up and make them more robust.
> 
> Dawid
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: The Wild World of Solr Tests

[Mark Miller <ma...@gmail.com>]

Once thing that any contributor can help with is beasting their own tests
before committing as well. I didn't mention that some amount of this can
also be done as part of the dev process ;) There is the built in ant beast
target that is super simple to use. I like to use my beasting script
because of how it handles logs and results and customizations, and
independent test runs, but for your average dev, a little ant beast usage
could go a long way with minimal effort. Automation is always best, but
some good test culture ain't a bad idea to push either.

- Mark

On Thu, Feb 9, 2017 at 6:53 PM Mark Miller <ma...@gmail.com> wrote:

> bq. I feel it would certainly help contributors like me to improve the
> quality of patch before it is committed to trunk. There are many apache
> projects which provide this type of infrastructure support e.g. when we
> submit a patch in hadoop, an automated jenkins bot provides feedback about
> various aspects of the patch e.g. check-style errors, unit tests failures,
> javadocs etc.
>
> Well, at some point, if this is going to work or survive over the longer
> term, it will have to evolve and automate and improve like that to some
> degree.
>
> We will see though. One of the biggest problems is the hardware. We have
> access to some Apache Jenkins machines that are already quite busy and
> probably not the greatest candidates for that job (if it was even possible
> to steal one from the jenkins cluster). Meanwhile, last I knew Apache does
> not allow companies to donate hardware to Apache for a specific purpose or
> project. So hardware for full automation is one issue. Beasting a test well
> can be done in 5-30 in most cases on decent hardware with enough RAM (tests
> potentially need 512MB for heap alone each, so a decent amount of RAM or
> fast swap is required for lots to go in parallel. More in parallel tends to
> produce fails faster (though too many will obviously overload the hardware
> and not be very useful either).
>
> Back when I was talking about a full beasting test run to Greg Chanan a
> couple years back, my main thought was we don't have the hardware for it.
> Some company could do it, but how do you actually integrate into
> our automated process that way over the long term? Companies come and go,
> interest and money comes and goes, etc. So what is the sustainable plan?
> I'm not sure, but I think driving in the right direction might present some
> kind of new possibilities.
>
> When I brought up the unlikeliness of the build system or reporting or
> hardware showing up in a way that works for the project anytime soon, Greg
> brought up the same type of idea for a post commit job type thing that
> could aluto track and beast new tests (and perhaps altered tests)
> automatically and then post the results to JIRA. I still think that is a
> great idea if we could figure out a hardware side.
>
> For a while I'm just going to produce reports in the cloud on GCE** and
> provide the initial hand holding and support this needs to try and get off
> the ground and reliable. If it get's any steam or helps produce any
> results, perhaps others will have further ideas on how to automate, find
> available hardware/resource and improve the strategy.
>
> Once something is fully automated, beasting all the tests *could*
> theoretically  become a much less common exercise. You could have pre or
> post commit hooks to look at tests in patches or recent commits, you could
> have a job that randomly selects tests to beast every day or other day,
> some of those beastings could be for a lot of runs, and of course you could
> provide extra coverage to tests that have a poor history.
>
> Anyhow, I'll fake a bit of that for a while once I get a little more
> ramped up. But yeah, eventually more automation and strategy will be
> important.
>
> - Mark
>
>
> ** Thank you Cloudera! That last report was probably $24 excluding my
> labor with 10 machine - the first report on a single machine cost much more.
>
>
>
> On Thu, Feb 9, 2017 at 6:02 PM Hrishikesh Gadre <ga...@gmail.com>
> wrote:
>
> Hi Mark,
>
> Thanks for taking care of this.
>
> >>preventing new tests that are not solid.
>
> What are your thoughts on this? While keeping track of recently introduced
> and flaky tests is one way to go forward, would it make sense to have some
> sort of automated test run *before* committing the changes ? I feel it
> would certainly help contributors like me to improve the quality of patch
> before it is committed to trunk. There are many apache projects which
> provide this type of infrastructure support e.g. when we submit a patch in
> hadoop, an automated jenkins bot provides feedback about various aspects of
> the patch e.g. check-style errors, unit tests failures, javadocs etc.
>
> Thoughts?
>
> Thanks
> -Hrishikesh
>
>
>
>
> On Thu, Feb 9, 2017 at 2:45 PM, Mark Miller <ma...@gmail.com> wrote:
>
> bq. a long time ago and sadly failed.
>
> It's really a quite difficult problem. Other than cutting out tons of test
> coverage, there has been no easy way to get to the top of the hill and stay
> there.
>
> I've gone after this with mostly sweat and time in the past. I still
> remember one xmas day 4 or 5 years ago when I had all the apache and
> policeman and my jenkins green for the main branches I tracked (like 10-14
> jobs?) for the first (and only time I ended up seeing) time at once. I've
> also set up my own jenkins machine and jobs at least 3 or 4 times for
> months on end, with simple job percentage passing plugins and other things
> that many others have done. It's all such a narrow tiny view into the real
> data though. You can just try and tread water or go out and try and
> physically burn down tests.
>
> It's really not so simple as just donating time. Not unless everyone did a
> lot more of it. First off, you can spend a ton of time taking many, many
> tests go from failing 5 in 25 times to something like 1 in 60 or 1 in 100,
> but dozens of tests failing even that often being run all the time is still
> a huge problem that will grow and festure in our current situation. So how
> do you even see where you are or ensure any progress made is kept? Random
> changes from many other areas bring once solid tests into worse shape,
> other tests accumulate new problems because no one is looking into tests
> that commonly fail, etc, etc. Many of theses tests provide critical tests
> coverage.
>
> You really have to know where to focus the time. Sometimes hardening a
> test is pretty quick, but often it's hours or even days of time
> fighting mischievous little issues.
>
> The only way I can see we can get anywhere that we can hold is by
> generating the proper data to tell us what the situation is and to make it
> very simple to track that situation over time.
>
> It's also something a bit more authoritative than one commiters
> opinion when it comes to pushing on authors to harden tests. Some tests
> mainly fail on jenkins, some tests mainly fail for this guy or on that
> platform, "how can you bug me about my test, I see your test fail", etc.
> It's because there are just so many ways for tests to have a problem and so
> many ways to run the unit tests (I run ant tests with 10 jvms and 6 cores,
> others 2 and 2 and or even more). But in a fair and reasonable environment,
> running a test 100 times, 10 times in parallel, is a fantastic neutral and
> probably useful data point. If a test fails something 40% of the large
> majority of other tests can survive then "I can't see it in my env" loses
> most of it's weight. Improve or @BadApple.
>
> I've banged my head against this wall more than once, and maybe this
> amounts to about as much, but I've been thinking of this 'beasting test run
> report' for many years and I really think current and reliable data is a
> required step to get this right.
>
> Beyond that, there are not too many people comfortable just hopping around
> all areas of the code and hardening a broad range of tests, but I'll put in
> my fair share again and see if something sparks.
>
> If we can get to a better point, I'll be happy to help police
> the appearance of new flakey tests.
>
> Even now I will place extra focus on preventing new tests that are not
> solid. The report helps me out with monitoring that by trying to use git to
> get a create date for each test to display and listing the last 3 failure
> percentage results (newer tests will have 0 or 1 or 2 entries).
>
> I've spent a lot of time just getting it all automated, building my
> confidence in its results, and driving down the time it takes to generate a
> report so that I can do it more frequently and/or for much longer
> iterations.
>
> Over time there is a lot of little ways to improve efficiency on that
> front though. For example, new tests can be run many more iterations, tests
> that have failed before can be run higher iterations, etc. Linking the
> reports together gives us some ammo for choosing how much time and beasting
> we should spend on a test, and if a test is improving, getting worse, etc.
> Data we can act on with confidence. I also have all these reports as tsv
> files so they can be easily imported into just about any system for longer
> or more intense cross report tracking or something.
>
> - Mark
>
>
> On Thu, Feb 9, 2017 at 3:39 PM Dawid Weiss <da...@gmail.com> wrote:
>
> This is a very important, hard and ungrateful task. Thanks for doing
> this Mark. As you know I tried (with your help!) to clean some of this
> mess a long time ago and sadly failed. It'd be great to speed those
> tests up and make them more robust.
>
> Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
> --
> - Mark
> about.me/markrmiller
>
>
> --
> - Mark
> about.me/markrmiller
>
-- 
- Mark
about.me/markrmiller

Re: The Wild World of Solr Tests

[Mark Miller <ma...@gmail.com>]

bq. I feel it would certainly help contributors like me to improve the
quality of patch before it is committed to trunk. There are many apache
projects which provide this type of infrastructure support e.g. when we
submit a patch in hadoop, an automated jenkins bot provides feedback about
various aspects of the patch e.g. check-style errors, unit tests failures,
javadocs etc.

Well, at some point, if this is going to work or survive over the longer
term, it will have to evolve and automate and improve like that to some
degree.

We will see though. One of the biggest problems is the hardware. We have
access to some Apache Jenkins machines that are already quite busy and
probably not the greatest candidates for that job (if it was even possible
to steal one from the jenkins cluster). Meanwhile, last I knew Apache does
not allow companies to donate hardware to Apache for a specific purpose or
project. So hardware for full automation is one issue. Beasting a test well
can be done in 5-30 in most cases on decent hardware with enough RAM (tests
potentially need 512MB for heap alone each, so a decent amount of RAM or
fast swap is required for lots to go in parallel. More in parallel tends to
produce fails faster (though too many will obviously overload the hardware
and not be very useful either).

Back when I was talking about a full beasting test run to Greg Chanan a
couple years back, my main thought was we don't have the hardware for it.
Some company could do it, but how do you actually integrate into
our automated process that way over the long term? Companies come and go,
interest and money comes and goes, etc. So what is the sustainable plan?
I'm not sure, but I think driving in the right direction might present some
kind of new possibilities.

When I brought up the unlikeliness of the build system or reporting or
hardware showing up in a way that works for the project anytime soon, Greg
brought up the same type of idea for a post commit job type thing that
could aluto track and beast new tests (and perhaps altered tests)
automatically and then post the results to JIRA. I still think that is a
great idea if we could figure out a hardware side.

For a while I'm just going to produce reports in the cloud on GCE** and
provide the initial hand holding and support this needs to try and get off
the ground and reliable. If it get's any steam or helps produce any
results, perhaps others will have further ideas on how to automate, find
available hardware/resource and improve the strategy.

Once something is fully automated, beasting all the tests *could*
theoretically  become a much less common exercise. You could have pre or
post commit hooks to look at tests in patches or recent commits, you could
have a job that randomly selects tests to beast every day or other day,
some of those beastings could be for a lot of runs, and of course you could
provide extra coverage to tests that have a poor history.

Anyhow, I'll fake a bit of that for a while once I get a little more ramped
up. But yeah, eventually more automation and strategy will be important.

- Mark


** Thank you Cloudera! That last report was probably $24 excluding my labor
with 10 machine - the first report on a single machine cost much more.



On Thu, Feb 9, 2017 at 6:02 PM Hrishikesh Gadre <ga...@gmail.com>
wrote:

> Hi Mark,
>
> Thanks for taking care of this.
>
> >>preventing new tests that are not solid.
>
> What are your thoughts on this? While keeping track of recently introduced
> and flaky tests is one way to go forward, would it make sense to have some
> sort of automated test run *before* committing the changes ? I feel it
> would certainly help contributors like me to improve the quality of patch
> before it is committed to trunk. There are many apache projects which
> provide this type of infrastructure support e.g. when we submit a patch in
> hadoop, an automated jenkins bot provides feedback about various aspects of
> the patch e.g. check-style errors, unit tests failures, javadocs etc.
>
> Thoughts?
>
> Thanks
> -Hrishikesh
>
>
>
>
> On Thu, Feb 9, 2017 at 2:45 PM, Mark Miller <ma...@gmail.com> wrote:
>
> bq. a long time ago and sadly failed.
>
> It's really a quite difficult problem. Other than cutting out tons of test
> coverage, there has been no easy way to get to the top of the hill and stay
> there.
>
> I've gone after this with mostly sweat and time in the past. I still
> remember one xmas day 4 or 5 years ago when I had all the apache and
> policeman and my jenkins green for the main branches I tracked (like 10-14
> jobs?) for the first (and only time I ended up seeing) time at once. I've
> also set up my own jenkins machine and jobs at least 3 or 4 times for
> months on end, with simple job percentage passing plugins and other things
> that many others have done. It's all such a narrow tiny view into the real
> data though. You can just try and tread water or go out and try and
> physically burn down tests.
>
> It's really not so simple as just donating time. Not unless everyone did a
> lot more of it. First off, you can spend a ton of time taking many, many
> tests go from failing 5 in 25 times to something like 1 in 60 or 1 in 100,
> but dozens of tests failing even that often being run all the time is still
> a huge problem that will grow and festure in our current situation. So how
> do you even see where you are or ensure any progress made is kept? Random
> changes from many other areas bring once solid tests into worse shape,
> other tests accumulate new problems because no one is looking into tests
> that commonly fail, etc, etc. Many of theses tests provide critical tests
> coverage.
>
> You really have to know where to focus the time. Sometimes hardening a
> test is pretty quick, but often it's hours or even days of time
> fighting mischievous little issues.
>
> The only way I can see we can get anywhere that we can hold is by
> generating the proper data to tell us what the situation is and to make it
> very simple to track that situation over time.
>
> It's also something a bit more authoritative than one commiters
> opinion when it comes to pushing on authors to harden tests. Some tests
> mainly fail on jenkins, some tests mainly fail for this guy or on that
> platform, "how can you bug me about my test, I see your test fail", etc.
> It's because there are just so many ways for tests to have a problem and so
> many ways to run the unit tests (I run ant tests with 10 jvms and 6 cores,
> others 2 and 2 and or even more). But in a fair and reasonable environment,
> running a test 100 times, 10 times in parallel, is a fantastic neutral and
> probably useful data point. If a test fails something 40% of the large
> majority of other tests can survive then "I can't see it in my env" loses
> most of it's weight. Improve or @BadApple.
>
> I've banged my head against this wall more than once, and maybe this
> amounts to about as much, but I've been thinking of this 'beasting test run
> report' for many years and I really think current and reliable data is a
> required step to get this right.
>
> Beyond that, there are not too many people comfortable just hopping around
> all areas of the code and hardening a broad range of tests, but I'll put in
> my fair share again and see if something sparks.
>
> If we can get to a better point, I'll be happy to help police
> the appearance of new flakey tests.
>
> Even now I will place extra focus on preventing new tests that are not
> solid. The report helps me out with monitoring that by trying to use git to
> get a create date for each test to display and listing the last 3 failure
> percentage results (newer tests will have 0 or 1 or 2 entries).
>
> I've spent a lot of time just getting it all automated, building my
> confidence in its results, and driving down the time it takes to generate a
> report so that I can do it more frequently and/or for much longer
> iterations.
>
> Over time there is a lot of little ways to improve efficiency on that
> front though. For example, new tests can be run many more iterations, tests
> that have failed before can be run higher iterations, etc. Linking the
> reports together gives us some ammo for choosing how much time and beasting
> we should spend on a test, and if a test is improving, getting worse, etc.
> Data we can act on with confidence. I also have all these reports as tsv
> files so they can be easily imported into just about any system for longer
> or more intense cross report tracking or something.
>
> - Mark
>
>
> On Thu, Feb 9, 2017 at 3:39 PM Dawid Weiss <da...@gmail.com> wrote:
>
> This is a very important, hard and ungrateful task. Thanks for doing
> this Mark. As you know I tried (with your help!) to clean some of this
> mess a long time ago and sadly failed. It'd be great to speed those
> tests up and make them more robust.
>
> Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
> --
> - Mark
> about.me/markrmiller
>
>
> --
- Mark
about.me/markrmiller

Re: The Wild World of Solr Tests

[Hrishikesh Gadre <ga...@gmail.com>]

Hi Mark,

Thanks for taking care of this.

>>preventing new tests that are not solid.

What are your thoughts on this? While keeping track of recently introduced
and flaky tests is one way to go forward, would it make sense to have some
sort of automated test run *before* committing the changes ? I feel it
would certainly help contributors like me to improve the quality of patch
before it is committed to trunk. There are many apache projects which
provide this type of infrastructure support e.g. when we submit a patch in
hadoop, an automated jenkins bot provides feedback about various aspects of
the patch e.g. check-style errors, unit tests failures, javadocs etc.

Thoughts?

Thanks
-Hrishikesh




On Thu, Feb 9, 2017 at 2:45 PM, Mark Miller <ma...@gmail.com> wrote:

> bq. a long time ago and sadly failed.
>
> It's really a quite difficult problem. Other than cutting out tons of test
> coverage, there has been no easy way to get to the top of the hill and stay
> there.
>
> I've gone after this with mostly sweat and time in the past. I still
> remember one xmas day 4 or 5 years ago when I had all the apache and
> policeman and my jenkins green for the main branches I tracked (like 10-14
> jobs?) for the first (and only time I ended up seeing) time at once. I've
> also set up my own jenkins machine and jobs at least 3 or 4 times for
> months on end, with simple job percentage passing plugins and other things
> that many others have done. It's all such a narrow tiny view into the real
> data though. You can just try and tread water or go out and try and
> physically burn down tests.
>
> It's really not so simple as just donating time. Not unless everyone did a
> lot more of it. First off, you can spend a ton of time taking many, many
> tests go from failing 5 in 25 times to something like 1 in 60 or 1 in 100,
> but dozens of tests failing even that often being run all the time is still
> a huge problem that will grow and festure in our current situation. So how
> do you even see where you are or ensure any progress made is kept? Random
> changes from many other areas bring once solid tests into worse shape,
> other tests accumulate new problems because no one is looking into tests
> that commonly fail, etc, etc. Many of theses tests provide critical tests
> coverage.
>
> You really have to know where to focus the time. Sometimes hardening a
> test is pretty quick, but often it's hours or even days of time
> fighting mischievous little issues.
>
> The only way I can see we can get anywhere that we can hold is by
> generating the proper data to tell us what the situation is and to make it
> very simple to track that situation over time.
>
> It's also something a bit more authoritative than one commiters
> opinion when it comes to pushing on authors to harden tests. Some tests
> mainly fail on jenkins, some tests mainly fail for this guy or on that
> platform, "how can you bug me about my test, I see your test fail", etc.
> It's because there are just so many ways for tests to have a problem and so
> many ways to run the unit tests (I run ant tests with 10 jvms and 6 cores,
> others 2 and 2 and or even more). But in a fair and reasonable environment,
> running a test 100 times, 10 times in parallel, is a fantastic neutral and
> probably useful data point. If a test fails something 40% of the large
> majority of other tests can survive then "I can't see it in my env" loses
> most of it's weight. Improve or @BadApple.
>
> I've banged my head against this wall more than once, and maybe this
> amounts to about as much, but I've been thinking of this 'beasting test run
> report' for many years and I really think current and reliable data is a
> required step to get this right.
>
> Beyond that, there are not too many people comfortable just hopping around
> all areas of the code and hardening a broad range of tests, but I'll put in
> my fair share again and see if something sparks.
>
> If we can get to a better point, I'll be happy to help police
> the appearance of new flakey tests.
>
> Even now I will place extra focus on preventing new tests that are not
> solid. The report helps me out with monitoring that by trying to use git to
> get a create date for each test to display and listing the last 3 failure
> percentage results (newer tests will have 0 or 1 or 2 entries).
>
> I've spent a lot of time just getting it all automated, building my
> confidence in its results, and driving down the time it takes to generate a
> report so that I can do it more frequently and/or for much longer
> iterations.
>
> Over time there is a lot of little ways to improve efficiency on that
> front though. For example, new tests can be run many more iterations, tests
> that have failed before can be run higher iterations, etc. Linking the
> reports together gives us some ammo for choosing how much time and beasting
> we should spend on a test, and if a test is improving, getting worse, etc.
> Data we can act on with confidence. I also have all these reports as tsv
> files so they can be easily imported into just about any system for longer
> or more intense cross report tracking or something.
>
> - Mark
>
>
> On Thu, Feb 9, 2017 at 3:39 PM Dawid Weiss <da...@gmail.com> wrote:
>
>> This is a very important, hard and ungrateful task. Thanks for doing
>> this Mark. As you know I tried (with your help!) to clean some of this
>> mess a long time ago and sadly failed. It'd be great to speed those
>> tests up and make them more robust.
>>
>> Dawid
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>> --
> - Mark
> about.me/markrmiller
>

Re: The Wild World of Solr Tests

[Mark Miller <ma...@gmail.com>]

bq. a long time ago and sadly failed.

It's really a quite difficult problem. Other than cutting out tons of test
coverage, there has been no easy way to get to the top of the hill and stay
there.

I've gone after this with mostly sweat and time in the past. I still
remember one xmas day 4 or 5 years ago when I had all the apache and
policeman and my jenkins green for the main branches I tracked (like 10-14
jobs?) for the first (and only time I ended up seeing) time at once. I've
also set up my own jenkins machine and jobs at least 3 or 4 times for
months on end, with simple job percentage passing plugins and other things
that many others have done. It's all such a narrow tiny view into the real
data though. You can just try and tread water or go out and try and
physically burn down tests.

It's really not so simple as just donating time. Not unless everyone did a
lot more of it. First off, you can spend a ton of time taking many, many
tests go from failing 5 in 25 times to something like 1 in 60 or 1 in 100,
but dozens of tests failing even that often being run all the time is still
a huge problem that will grow and festure in our current situation. So how
do you even see where you are or ensure any progress made is kept? Random
changes from many other areas bring once solid tests into worse shape,
other tests accumulate new problems because no one is looking into tests
that commonly fail, etc, etc. Many of theses tests provide critical tests
coverage.

You really have to know where to focus the time. Sometimes hardening a test
is pretty quick, but often it's hours or even days of time
fighting mischievous little issues.

The only way I can see we can get anywhere that we can hold is by
generating the proper data to tell us what the situation is and to make it
very simple to track that situation over time.

It's also something a bit more authoritative than one commiters
opinion when it comes to pushing on authors to harden tests. Some tests
mainly fail on jenkins, some tests mainly fail for this guy or on that
platform, "how can you bug me about my test, I see your test fail", etc.
It's because there are just so many ways for tests to have a problem and so
many ways to run the unit tests (I run ant tests with 10 jvms and 6 cores,
others 2 and 2 and or even more). But in a fair and reasonable environment,
running a test 100 times, 10 times in parallel, is a fantastic neutral and
probably useful data point. If a test fails something 40% of the large
majority of other tests can survive then "I can't see it in my env" loses
most of it's weight. Improve or @BadApple.

I've banged my head against this wall more than once, and maybe this
amounts to about as much, but I've been thinking of this 'beasting test run
report' for many years and I really think current and reliable data is a
required step to get this right.

Beyond that, there are not too many people comfortable just hopping around
all areas of the code and hardening a broad range of tests, but I'll put in
my fair share again and see if something sparks.

If we can get to a better point, I'll be happy to help police
the appearance of new flakey tests.

Even now I will place extra focus on preventing new tests that are not
solid. The report helps me out with monitoring that by trying to use git to
get a create date for each test to display and listing the last 3 failure
percentage results (newer tests will have 0 or 1 or 2 entries).

I've spent a lot of time just getting it all automated, building my
confidence in its results, and driving down the time it takes to generate a
report so that I can do it more frequently and/or for much longer
iterations.

Over time there is a lot of little ways to improve efficiency on that front
though. For example, new tests can be run many more iterations, tests that
have failed before can be run higher iterations, etc. Linking the reports
together gives us some ammo for choosing how much time and beasting we
should spend on a test, and if a test is improving, getting worse, etc.
Data we can act on with confidence. I also have all these reports as tsv
files so they can be easily imported into just about any system for longer
or more intense cross report tracking or something.

- Mark


On Thu, Feb 9, 2017 at 3:39 PM Dawid Weiss <da...@gmail.com> wrote:

> This is a very important, hard and ungrateful task. Thanks for doing
> this Mark. As you know I tried (with your help!) to clean some of this
> mess a long time ago and sadly failed. It'd be great to speed those
> tests up and make them more robust.
>
> Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
> --
- Mark
about.me/markrmiller

Re: The Wild World of Solr Tests

[Dawid Weiss <da...@gmail.com>]

This is a very important, hard and ungrateful task. Thanks for doing
this Mark. As you know I tried (with your help!) to clean some of this
mess a long time ago and sadly failed. It'd be great to speed those
tests up and make them more robust.

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org