You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by Jan Høydahl <ja...@cominvent.com> on 2022/01/14 12:44:42 UTC
Re: Unit tests
Reviving this thread from last November.
I see that e.g. TestCollectionAPI.test has failed on Solr-check-main lately. Failure rate 6%. http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.cloud.api.collections.TestCollectionAPI.test
Think it's time to do a @BadApple and test-fixing raid to get rid of all the noise.
Jan
> 21. nov. 2021 kl. 05:40 skrev Gus Heck <gu...@gmail.com>:
>
> So in trying to assess whether my changes for https://issues.apache.org/jira/browse/SOLR-15590 <https://issues.apache.org/jira/browse/SOLR-15590> had any effect on tests I ran the full clean check 10 times on main and branch (from the date of this email). The results were not suggestive of a difference but they were depressing:
>
> TEST FAILS:
>
> Main:
>
> Failed on run 4,5,7,9,10, hung on run 2 (4 runs succeeded, 7 total failures)
> Ran for over 1.5h (killed manually):
> 1 @ :solr:core:test > Executing test org.apache.solr.cloud.CollectionPropsTest (run 2) (message on screen when killed)
> Thread leaks:
> 1 @ org.apache.solr.cloud.TestLeaderElectionZkExpiry.classMethod (:solr:core) (run 10)
> 2 @ org.apache.solr.schema.TestBulkSchemaConcurrent.classMethod (:solr:core) (run 4,7)
> Others:
> 2 @ org.apache.solr.cloud.api.collections.TestCollectionAPI.test (:solr:core) (run 7,9) (not repro)
> 1 @ org.apache.solr.common.util.TestJsonRecordReader.testArrayOfRootObjects (:solr:solrj) (run 5) (not repro)
> 1 @ org.apache.solr.cloud.TestPullReplica.testRemoveAllWriterReplicas (:solr:core) (run4) (not repro)
>
> Branch:
>
> Failed on run 1,2,6,8,9,10 (4 runs succeeded, 6 total failures)
> Thread Leaks
> 1 @ org.apache.solr.schema.TestBulkSchemaConcurrent.classMethod (:solr:core) (run 1)
> 2 @ org.apache.solr.handler.designer.TestSchemaDesignerSettingsDAO.classMethod (:solr:core) (run 9,10)
> 1 @ org.apache.solr.client.solrj.impl.CloudHttp2SolrClientBuilderTest.classMethod (:solr:solrj) (run 2)
> 1 @ org.apache.solr.cloud.TestLeaderElectionZkExpiry.classMethod (:solr:core) (run 8)
> Other:
> 1 @ org.apache.solr.cloud.OverseerTest.testShardLeaderChange (:solr:core) (run 6) (not repro)
>
> There is nothing suggestive of a difference here, and both branch and main are in terrible shape, each passing less than 50%, though I got curious about how much it does tell me and the sample is so small that this really only gives us good confidence that the fail rate is over 30% on each branch, Fiddling with R Studio a bit gives me depressingly large numbers of runs to reliably detect anything like a 20% change in build failure rate. For example (assuming I didn't make an error interpreting this, stats class was several decades ago) if we take the 60% just observed as current, and introduce a bad change that causes 20% increase in overall build flakiness... to 80% then if we allow ourselves 10% chance of type 1 error and a 10% chance of type 2 error we need to run the build 135 times... (output from R version 3.6.3, chi-square)
>
> power.prop.test(n=NULL, p1=0.6, p2=0.8, power=0.9, sig.level=0.1, alternative="one.sided")
>
> Two-sample comparison of proportions power calculation
>
> n = 67.32733
> p1 = 0.6
> p2 = 0.8
> sig.level = 0.1
> power = 0.9
> alternative = one.sided
>
> NOTE: n is number in *each* group
>
> Right now there is basically no practical way to be confident if you have not introduced or caused a change in the flakiness of the build. This also more or less proves what I had suspected... The only way to control flakey tests is to completely kill them since any sane number of builds has a good likelihood of misleading you unless you do something that leads to a virtual 100% fail rate.
>
> On Mon, Nov 15, 2021 at 7:45 PM Gus Heck <gus.heck@gmail.com <ma...@gmail.com>> wrote:
> Things seem to be not great.
>
> I was testing my branch and saw one class with a non-reproducing failure twice in 9 runs (7 successes)
>
> So I checked out main to see what the failure rates looked like there:
>
> I've now run main it 5 times and had 6 test failures. All of these are different tests and all failing tests pass repeatedly if run alone. Things seem to have gone downhill at the end of april:
>
> http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html <http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html>
>
> As I write check and nightly are failing on jenkins (and amusingly badapples is passing)
>
> https://ci-builds.apache.org/job/Solr/ <https://ci-builds.apache.org/job/Solr/>
>
> As it stands there's even a chance that some of what I've done in reworking how CoreContainer starts up, may have improved the test situation, for many tests (but the one that's failing is doing so with a thread leak detected from httpclient)
>
> I'm really a bit alarmed at the number of failures I'm seeing on main.
>
> Let me know if you have been building main with no issues recently. (then I need to figure out what's wrong with my setup, or why it's more sensitive) Or, let me know if you have an idea why main fails so much.
>
> -Gus
>
> --
> http://www.needhamsoftware.com <http://www.needhamsoftware.com/> (work)
> http://www.the111shift.com <http://www.the111shift.com/> (play)
>
>
> --
> http://www.needhamsoftware.com <http://www.needhamsoftware.com/> (work)
> http://www.the111shift.com <http://www.the111shift.com/> (play)
Re: Unit tests
Posted by Gus Heck <gu...@gmail.com>.
On Fri, Jan 14, 2022 at 10:36 PM Mark Miller <ma...@gmail.com> wrote:
> The true nature and state of those tests lie far deeper than pretty much
> anyone occasionally scratches with their trowel. To really take a peak, you
> have to do at minimum, something like setup a Jenkins farm with half a
> dozen, a dozen machines with varying low to high need specs, randomize
> parallel overlap and test order and actually shake the Jenga tower to see
> what falls out.
>
Yeah, Random variation is a big enemy here. I once took the time to figure
out what number of tests is required to generate enough pass fail data
points to detect (with statistics) if the pass fail rate has changed by
something like 25%... winds up being over 100 test runs before *and* after
the change. The only thing detectable (with p=0.95 significance) in 10 runs
is like going from 90% fail to 0% fail. Anything less and you don't have
the statistical power necessary. That's why 1/10 fails had me
slightly hopeful.... IIRC what I found something like that might be
significant at p=0.6 or so... i.e. > 50% chance that something changed.
(didn't actually calculate that specific case) of not being a random event.
.... But since my last mail I've run curent main 5 times and got 4 fails so
random is looking like the answer.
>
> That will expose a real view rather then a narrow slit into a shifting,
> opaque, but “relatively balancing from a view point”perspective at least
> view.
>
> Just from a practical squeeze, many projects just push on narrowing that
> slit view and leaning into more efforts on keeping the structure balanced
> in that far. Perhaps going as far as, run in a known Docker environment,
> minimize disturbances and test recording with light parallel at most and
> even, just a master Jenkins run is the real deal, developers, your luck
> will vary, outside of adventuring, you’ll have an easier time letting the
> test source of truth Jenkins instance dictate your hat fails or not.
>
My thinking is that laptop sized runs are what we care about, and failure
with excess resources is less likely. So Running on smaller machines with
high parallelism to simulate a developer trying to get through the build as
fast as their laptop will go would be my focus.
--
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)
Re: Unit tests
Posted by Jan Høydahl <ja...@cominvent.com>.
15. jan. 2022 kl. 18:28 skrev Gus Heck <gu...@gmail.com>:
>
> Also the efforts to reduce which tests run by default on dev laptops worries me, it amounts to head in the sand.
Are you referring to factoring out integration tests from unit tests? Or the default of running @Slow, @Nightly, @Wekly, @BadApples (don't remember what is on and off by default)?
I was not suggesting we stop running integration tests - far from it. But the way it is now, I fear that many are tempted to merge to main with just precommit and perhaps randomly running a few obviously relevant tests, since gettinga SUCCESS on the full suite both takes forever and needs to repeat 5 times to get there. So we could actually increase daily testing by adding this filter.
Jan
Re: Unit tests
Posted by Gus Heck <gu...@gmail.com>.
On Fri, Jan 14, 2022 at 10:45 PM Mark Miller <ma...@gmail.com> wrote:
> “Your test fails or not” *
>
> I did see some time back, that thread leaks did need some attention. I
> don’t know if it’s gotten any worse, but I did have some offenders
> addressed.
>
The case I was looking at was zkevent threads firing after shutdown begins.
There is some attempt to let those threads drain which I guess is a thought
towards's graceful shutdown, but I'm increasingly feeling that such is a
hopeless idea and we should just design everything to assume that the power
cord will get unplugged. The change that seemed to be doing some good was
synchronizing the shutdown() method in corecontainer with ZkController,
which lead to some deadlocks, but I seemed to have got that worked out (at
least well enough to complete 10 times in a row)
I'll fiddle a little more and post it ...
>
> Really though, the whole idea of removing the reliance on the test
> framework to sweep leaks and poor test behavior under the table is
> predicated on the idea that those issues would instead become viable and
> rather than accumulate and exist forever, they would be fixed and good
> tests and often production behavior would be enforced like it was
> originally intended to be.
>
> I think that’s an unlikely outcome, which is probably putting it
> generously looking historically. And so the more beneficial move may just
> to be to revert back to the test framework burying that stuff to the degree
> that it did.
>
The primary thing that bugs me there is that we run in partial, mocked up
jetty instead of the real thing so we may be observing things that don't
relate to the real running solr, but that's a whole other thing wherein we
should either kick Jetty out entirely, turn our test harness into the live
server or better optimize standard jetty startup so that It doesn't kill
test times as much (and yes I recall your prior notes on the difficulty of
that!). I favor the later course by default but haven't looked deeply.
Also the efforts to reduce which tests run by default on dev laptops
worries me, it amounts to head in the sand. This sort of thing is
reasonable in a non-volunteer environment where the metrics coming out of
the build server are obsessed over by managers, and they assign someone to
go fix things up every now and then. I'm not sure failures would get enough
attention in a volunteer oriented space if it doesn't irritate us in our
daily work.
-Gus
--
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)
Re: Unit tests
Posted by Mark Miller <ma...@gmail.com>.
“Your test fails or not” *
I did see some time back, that thread leaks did need some attention. I
don’t know if it’s gotten any worse, but I did have some offenders
addressed.
Really though, the whole idea of removing the reliance on the test
framework to sweep leaks and poor test behavior under the table is
predicated on the idea that those issues would instead become viable and
rather than accumulate and exist forever, they would be fixed and good
tests and often production behavior would be enforced like it was
originally intended to be.
I think that’s an unlikely outcome, which is probably putting it generously
looking historically. And so the more beneficial move may just to be to
revert back to the test framework burying that stuff to the degree that it
did.
Re: Unit tests
Posted by Mark Miller <ma...@gmail.com>.
The true nature and state of those tests lie far deeper than pretty much
anyone occasionally scratches with their trowel. To really take a peak, you
have to do at minimum, something like setup a Jenkins farm with half a
dozen, a dozen machines with varying low to high need specs, randomize
parallel overlap and test order and actually shake the Jenga tower to see
what falls out.
That will expose a real view rather then a narrow slit into a shifting,
opaque, but “relatively balancing from a view point”perspective at least
view.
Just from a practical squeeze, many projects just push on narrowing that
slit view and leaning into more efforts on keeping the structure balanced
in that far. Perhaps going as far as, run in a known Docker environment,
minimize disturbances and test recording with light parallel at most and
even, just a master Jenkins run is the real deal, developers, your luck
will vary, outside of adventuring, you’ll have an easier time letting the
test source of truth Jenkins instance dictate your hat fails or not.
Re: Unit tests
Posted by Gus Heck <gu...@gmail.com>.
Actually, I spent a bunch of time fiddling with tests last weekend which is
why I didn't get to the things I, on a checkout that wasn't too terribly
old (maybe 2 weeks at that time? I hadn't seen anything that looked like it
should fix the tests and Fucit still looked on fire so I just continued
with what I had) on main, I did some fixes mostly centering on shutdown
race conditions. I got to a point with 9/10 runs passing.
However, then I stashed and updated as a check and ran the tests 10 times
and got 1 failure, so I'm now pondering if my changes were worth it. I
haven't had time to check if the fact I pulled in one fix I saw go by in
email can account for these improvements... (
https://github.com/apache/solr/pull/505/commits/dfb887201e65f4e2a0b4aecadaba0f46d5e527df)
The original thing that set me off was the fact I was seeing lots of thread
leak suite failures.
It's also entirely possible that my 10 runs with one fail after the update
were just luck... previously 4-6 fails in 10 runs was common.
On Fri, Jan 14, 2022 at 7:44 AM Jan Høydahl <ja...@cominvent.com> wrote:
> Reviving this thread from last November.
>
> I see that e.g. TestCollectionAPI.test has failed on Solr-check-main
> lately. Failure rate 6%.
> http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.cloud.api.collections.TestCollectionAPI.test
> Think it's time to do a @BadApple and test-fixing raid to get rid of all
> the noise.
>
> Jan
>
> 21. nov. 2021 kl. 05:40 skrev Gus Heck <gu...@gmail.com>:
>
> So in trying to assess whether my changes for
> https://issues.apache.org/jira/browse/SOLR-15590 had any effect on tests
> I ran the full clean check 10 times on main and branch (from the date of
> this email). The results were not suggestive of a difference but they were
> depressing:
>
> TEST FAILS:
>
> Main:
>
> Failed on run 4,5,7,9,10, hung on run 2 (4 runs succeeded, 7 total
> failures)
> Ran for over 1.5h (killed manually):
> 1 @ :solr:core:test > Executing test
> org.apache.solr.cloud.CollectionPropsTest (run 2) (message on screen when
> killed)
> Thread leaks:
> 1 @ org.apache.solr.cloud.TestLeaderElectionZkExpiry.classMethod
> (:solr:core) (run 10)
> 2 @ org.apache.solr.schema.TestBulkSchemaConcurrent.classMethod
> (:solr:core) (run 4,7)
> Others:
> 2 @ org.apache.solr.cloud.api.collections.TestCollectionAPI.test
> (:solr:core) (run 7,9) (not repro)
> 1 @
> org.apache.solr.common.util.TestJsonRecordReader.testArrayOfRootObjects
> (:solr:solrj) (run 5) (not repro)
> 1 @ org.apache.solr.cloud.TestPullReplica.testRemoveAllWriterReplicas
> (:solr:core) (run4) (not repro)
>
> Branch:
>
> Failed on run 1,2,6,8,9,10 (4 runs succeeded, 6 total failures)
> Thread Leaks
> 1 @ org.apache.solr.schema.TestBulkSchemaConcurrent.classMethod
> (:solr:core) (run 1)
> 2 @
> org.apache.solr.handler.designer.TestSchemaDesignerSettingsDAO.classMethod
> (:solr:core) (run 9,10)
> 1 @
> org.apache.solr.client.solrj.impl.CloudHttp2SolrClientBuilderTest.classMethod
> (:solr:solrj) (run 2)
> 1 @ org.apache.solr.cloud.TestLeaderElectionZkExpiry.classMethod
> (:solr:core) (run 8)
> Other:
> 1 @ org.apache.solr.cloud.OverseerTest.testShardLeaderChange
> (:solr:core) (run 6) (not repro)
>
> There is nothing suggestive of a difference here, and both branch and main
> are in terrible shape, each passing less than 50%, though I got curious
> about how much it does tell me and the sample is so small that this really
> only gives us good confidence that the fail rate is over 30% on each
> branch, Fiddling with R Studio a bit gives me depressingly large numbers of
> runs to reliably detect anything like a 20% change in build failure rate.
> For example (assuming I didn't make an error interpreting this, stats class
> was several decades ago) if we take the 60% just observed as current, and
> introduce a bad change that causes 20% increase in overall build
> flakiness... to 80% then if we allow ourselves 10% chance of type 1 error
> and a 10% chance of type 2 error we need to run the build 135 times...
> (output from R version 3.6.3, chi-square)
>
> power.prop.test(n=NULL, p1=0.6, p2=0.8, power=0.9, sig.level=0.1,
>> alternative="one.sided")
>>
>> Two-sample comparison of proportions power calculation
>>
>> n = 67.32733
>> p1 = 0.6
>> p2 = 0.8
>> sig.level = 0.1
>> power = 0.9
>> alternative = one.sided
>>
>> NOTE: n is number in *each* group
>>
>
> Right now there is basically no practical way to be confident if you have
> not introduced or caused a change in the flakiness of the build. This also
> more or less proves what I had suspected... The only way to control flakey
> tests is to completely kill them since any sane number of builds has a good
> likelihood of misleading you unless you do something that leads to a
> virtual 100% fail rate.
>
> On Mon, Nov 15, 2021 at 7:45 PM Gus Heck <gu...@gmail.com> wrote:
>
>> Things seem to be not great.
>>
>> I was testing my branch and saw one class with a non-reproducing failure
>> twice in 9 runs (7 successes)
>>
>> So I checked out main to see what the failure rates looked like there:
>>
>> I've now run main it 5 times and had 6 test failures. All of these are
>> different tests and all failing tests pass repeatedly if run alone. Things
>> seem to have gone downhill at the end of april:
>>
>>
>> http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html
>>
>> As I write check and nightly are failing on jenkins (and amusingly
>> badapples is passing)
>>
>> https://ci-builds.apache.org/job/Solr/
>>
>> As it stands there's even a chance that some of what I've done in
>> reworking how CoreContainer starts up, may have improved the test
>> situation, for many tests (but the one that's failing is doing so with a
>> thread leak detected from httpclient)
>>
>> I'm really a bit alarmed at the number of failures I'm seeing on main.
>>
>> Let me know if you have been building main with no issues recently. (then
>> I need to figure out what's wrong with my setup, or why it's more
>> sensitive) Or, let me know if you have an idea why main fails so much.
>>
>> -Gus
>>
>> --
>> http://www.needhamsoftware.com (work)
>> http://www.the111shift.com (play)
>>
>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)
>
>
>
--
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)