You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by Uwe Schindler <uw...@thetaphi.de> on 2025/11/06 14:23:18 UTC
Please help: Checksum and memory problems with builds on Policeman Jenkins => builds disabled
Hi,
There seems to be something completely wrong with Solr builds on main
branch on Policeman Jenkins. Can anybody explain why we see this error
all the time (since weeks), but on ASF Jenkins this resolved recently
(around November 1st -- suddenly without any related change in git!!!):
Caused by: org.gradle.api.GradleException: Dependency checksum validation failed:
- Dependency checksum mismatch ('com.nvidia.cuvs.lucene:cuvs-lucene:25.10.0'), expected it to be: 735cac75bcf3b55763941cfd1e473dfcf47fbf97, but was: b5a458458255bfdfa6688a571b5bee032428f2d4
To me it looks like the artifacts changed while keeping the same version
number of cuvs. As gradle caches downloaded artifacts forever (because
maven artifacts are not intended to be changed) it will of course keep
the old artifacts with no-longer the new/expected checksum of same
artifact versions.
Can anybody explain what is happening here? Is there a way to fix the
build? Cleaning the workspace does not help so I disabled the builds for
now (we no longer test Solr on Linux, Windows, macOs). *In addition the
reason why I killed the builds is some issue with the execution at all.
It happens many times that "dead" test runner JVMs keep alive and are
not killed at end of build. After some time the machine ran out of
memory and OOM killer started to kill virtual machines (windows, macos).*
So next to the checksum issue, we should figure out why the build's test
seems to have stuck JVMs dangling as dead processes after the build has
finished.
Uwe
Am 06.11.2025 um 02:48 schrieb Policeman Jenkins Server:
> Build:https://jenkins.thetaphi.de/job/Solr-main-Linux/29472/
> Java: 64bit/hotspot/jdk-23.0.2 -XX:-UseCompressedOops -XX:+UseParallelGC
>
> 4 tests failed.
> FAILED: org.apache.solr.cloud.TestRandomFlRTGCloud.testRandomizedUpdatesAndRTGs
>
> Error Message:
> java.lang.AssertionError: qt=/get&wt=xml&subq.q={!field+f%3Dnext_2_ids_i+v%3D$row.id}&fl=id,_root_:id,a*,my_escaped_json_field_alias:eee_s,my_geo_alias:[geo+f%3Dgeo_2_srpt],my_xml_field_alias:[xml+f%3Dggg_s],my_core_alias:[core],[shard],eee_s,my_int_field_alias2:bbb_i,[json+f%3Dfff_s],[core],my_geo_alias2:geo_2_srpt,[explain],bbb_i,score_alias:score,func_aaa_alias:add(1.3,sub(aaa_i,aaa_i)),my_docid_alias:[docid],ccc_s,id,subq:[subquery],*_s,my_json_field_alias:[json+f%3Deee_s],bogus_alias:other_bogus_field_i,[value+v%3D42+t%3Dint],my_shard_alias:[shard],aaa_i,score,ddd_s,*_i,geo_2_srpt&fl=[xml+f%3Dhhh_s],my_str_field_alias:ddd_s,explain_alias:[explain],my_int_field_alias:bbb_i&ids=312,312,265&ids=&ids=322 => {stream=org.apache.http.conn.EofSensorInputStream@67e71ce5, responseStatus=500, closeableResponse=HttpResponseProxy{HTTP/1.1 500 Server Error [Server: Jetty(12.0.27), Date: Thu, 06 Nov 2025 01:44:35 GMT, Server: Jetty(12.0.27), Date: Thu, 06 Nov 2025 01:44:35 GMT, Vary: Accept-Encoding, Last-Modified: Thu, 06 Nov 2025 01:44:37 GMT, ETag: "19a56d650b8", Cache-Control: no-cache, no-store, Pragma: no-cache, Expires: Sat, 01 Jan 2000 01:00:00 GMT, Content-Type: application/xml; charset=UTF-8, Content-Length: 3375] ResponseEntityProxy{[Content-Type: application/xml; charset=UTF-8,Content-Length: 3375,Chunked: false]}}}
>
> Stack Trace:
> java.lang.AssertionError: qt=/get&wt=xml&subq.q={!field+f%3Dnext_2_ids_i+v%3D$row.id}&fl=id,_root_:id,a*,my_escaped_json_field_alias:eee_s,my_geo_alias:[geo+f%3Dgeo_2_srpt],my_xml_field_alias:[xml+f%3Dggg_s],my_core_alias:[core],[shard],eee_s,my_int_field_alias2:bbb_i,[json+f%3Dfff_s],[core],my_geo_alias2:geo_2_srpt,[explain],bbb_i,score_alias:score,func_aaa_alias:add(1.3,sub(aaa_i,aaa_i)),my_docid_alias:[docid],ccc_s,id,subq:[subquery],*_s,my_json_field_alias:[json+f%3Deee_s],bogus_alias:other_bogus_field_i,[value+v%3D42+t%3Dint],my_shard_alias:[shard],aaa_i,score,ddd_s,*_i,geo_2_srpt&fl=[xml+f%3Dhhh_s],my_str_field_alias:ddd_s,explain_alias:[explain],my_int_field_alias:bbb_i&ids=312,312,265&ids=&ids=322 => {stream=org.apache.http.conn.EofSensorInputStream@67e71ce5, responseStatus=500, closeableResponse=HttpResponseProxy{HTTP/1.1 500 Server Error [Server: Jetty(12.0.27), Date: Thu, 06 Nov 2025 01:44:35 GMT, Server: Jetty(12.0.27), Date: Thu, 06 Nov 2025 01:44:35 GMT, Vary: Accept-Encoding, Last-Modified: Thu, 06 Nov 2025 01:44:37 GMT, ETag: "19a56d650b8", Cache-Control: no-cache, no-store, Pragma: no-cache, Expires: Sat, 01 Jan 2000 01:00:00 GMT, Content-Type: application/xml; charset=UTF-8, Content-Length: 3375] ResponseEntityProxy{[Content-Type: application/xml; charset=UTF-8,Content-Length: 3375,Chunked: false]}}}
> at __randomizedtesting.SeedInfo.seed([C602858C07426F25:78DE4D70C0E5E709]:0)
> at org.junit.Assert.fail(Assert.java:89)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertNotNull(Assert.java:713)
> at org.apache.solr.cloud.TestRandomFlRTGCloud.assertRTG(TestRandomFlRTGCloud.java:544)
> at org.apache.solr.cloud.TestRandomFlRTGCloud.assertOneIter(TestRandomFlRTGCloud.java:311)
> at org.apache.solr.cloud.TestRandomFlRTGCloud.testRandomizedUpdatesAndRTGs(TestRandomFlRTGCloud.java:262)
> at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> at java.base/java.lang.reflect.Method.invoke(Method.java:580)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1763)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
> at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
> at java.base/java.lang.Thread.run(Thread.java:1575)
>
>
> FAILED: org.apache.solr.cloud.TestRandomFlRTGCloud.testRandomizedUpdatesAndRTGs
>
> Error Message:
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> Stack Trace:
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at __randomizedtesting.SeedInfo.seed([C602858C07426F25:78DE4D70C0E5E709]:0)
> at java.base/java.math.BigInteger.shiftRightImpl(BigInteger.java:3601)
> at java.base/java.math.BigInteger.shiftRight(BigInteger.java:3566)
> at java.base/sun.security.util.math.intpoly.IntegerPolynomial.setLimbsValuePositive(IntegerPolynomial.java:340)
> at java.base/sun.security.util.math.intpoly.MontgomeryIntegerPolynomialP256.getElement(MontgomeryIntegerPolynomialP256.java:104)
> at java.base/sun.security.util.math.intpoly.MontgomeryIntegerPolynomialP256.getElement(MontgomeryIntegerPolynomialP256.java:40)
> at java.base/sun.security.ec.ECOperations.forParameters(ECOperations.java:94)
> at java.base/sun.security.ec.ECKeyPairGenerator.generateKeyPairImpl(ECKeyPairGenerator.java:182)
> at java.base/sun.security.ec.ECKeyPairGenerator.generateKeyPair(ECKeyPairGenerator.java:146)
> at java.base/java.security.KeyPairGenerator$Delegate.generateKeyPair(KeyPairGenerator.java:726)
> at java.base/sun.security.ssl.ECDHKeyExchange$ECDHEPossession.<init>(ECDHKeyExchange.java:118)
> at java.base/sun.security.ssl.NamedGroup$ECDHEScheme.createPossession(NamedGroup.java:706)
> at java.base/sun.security.ssl.NamedGroup$NamedGroupSpec.createPossession(NamedGroup.java:642)
> at java.base/sun.security.ssl.NamedGroup.createPossession(NamedGroup.java:550)
> at java.base/sun.security.ssl.SSLKeyExchange$T13KeyAgreement.createPossession(SSLKeyExchange.java:573)
> at java.base/sun.security.ssl.SSLKeyExchange.createPossessions(SSLKeyExchange.java:92)
> at java.base/sun.security.ssl.KeyShareExtension$CHKeyShareProducer.getShare(KeyShareExtension.java:295)
> at java.base/sun.security.ssl.KeyShareExtension$CHKeyShareProducer.produce(KeyShareExtension.java:257)
> at java.base/sun.security.ssl.SSLExtension.produce(SSLExtension.java:599)
> at java.base/sun.security.ssl.SSLExtensions.produce(SSLExtensions.java:265)
> at java.base/sun.security.ssl.ClientHello$ClientHelloKickstartProducer.produce(ClientHello.java:635)
> at java.base/sun.security.ssl.SSLHandshake.kickstart(SSLHandshake.java:526)
> at java.base/sun.security.ssl.ClientHandshakeContext.kickstart(ClientHandshakeContext.java:112)
> at java.base/sun.security.ssl.TransportContext.kickstart(TransportContext.java:263)
> at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:448)
> at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
> at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
> at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
> at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
> at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
> at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
> at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
> at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
>
>
> FAILED: org.apache.solr.cloud.TestRandomFlRTGCloud.testRandomizedUpdatesAndRTGs
>
> Error Message:
> java.lang.AssertionError: num docs mismatch: qt=/get&wt=xml&subq.q={!field+f%3Dnext_2_ids_i+v%3D$row.id}&fl=id,_root_:id,score,ggg_s,my_geo_alias:[geo+f%3Dgeo_2_srpt],my_shard_alias:[shard],bogus_raw_alias:[xml+f%3Dbogus_raw_field_ss],[xml+f%3Dhhh_s],explain_alias:[explain],[shard],eee_s,my_str_field_alias:ddd_s,my_str_field_alias2:ddd_s,[explain],[value+v%3D42+t%3Dint]&fl=bogus_unused_field_ss,geo_2_srpt,ccc_s,bbb_i,bogus_alias:other_bogus_field_i,[docid],my_core_alias:[core],*_i,*,id,[json+f%3Dfff_s],my_json_field_alias:[json+f%3Deee_s],my_escaped_json_field_alias:eee_s,val_alias:[value+v%3D1976+t%3Dint],my_xml_field_alias:[xml+f%3Dggg_s],subq:[subquery],my_geo_alias2:geo_2_srpt&fl=score_alias:score,my_int_field_alias2:bbb_i,[core],[geo+f%3Dgeo_1_srpt],*_s&fl=my_escaped_xml_field_alias:ggg_s,aaa_i,a*&fl=&fl=add(1.3,sub(aaa_i,aaa_i))&fl=my_docid_alias:[docid]&id=181 => [SolrInputDocument(fields: [id=181, aaa_i=-1976406138, bbb_i=2099255939, ccc_s=qzltjban, ddd_s=tivallpxa, eee_s={"njfsy":"njfsy"}, fff_s="", ggg_s=<root>gzrgylphkh</root>, hhh_s=tabfpe, geo_1_srpt=POINT( 42 23 ), geo_2_srpt=POINT( 42 -16 ), next_2_ids_ss=[182, 183], axx_i=1617014719, ayy_i=1062547001, azz_s=ixxbsnxsdj])] vs {numFound=0,numFoundExact=true,start=0,docs=[]} expected:<1> but was:<0>
>
> Stack Trace:
> java.lang.AssertionError: num docs mismatch: qt=/get&wt=xml&subq.q={!field+f%3Dnext_2_ids_i+v%3D$row.id}&fl=id,_root_:id,score,ggg_s,my_geo_alias:[geo+f%3Dgeo_2_srpt],my_shard_alias:[shard],bogus_raw_alias:[xml+f%3Dbogus_raw_field_ss],[xml+f%3Dhhh_s],explain_alias:[explain],[shard],eee_s,my_str_field_alias:ddd_s,my_str_field_alias2:ddd_s,[explain],[value+v%3D42+t%3Dint]&fl=bogus_unused_field_ss,geo_2_srpt,ccc_s,bbb_i,bogus_alias:other_bogus_field_i,[docid],my_core_alias:[core],*_i,*,id,[json+f%3Dfff_s],my_json_field_alias:[json+f%3Deee_s],my_escaped_json_field_alias:eee_s,val_alias:[value+v%3D1976+t%3Dint],my_xml_field_alias:[xml+f%3Dggg_s],subq:[subquery],my_geo_alias2:geo_2_srpt&fl=score_alias:score,my_int_field_alias2:bbb_i,[core],[geo+f%3Dgeo_1_srpt],*_s&fl=my_escaped_xml_field_alias:ggg_s,aaa_i,a*&fl=&fl=add(1.3,sub(aaa_i,aaa_i))&fl=my_docid_alias:[docid]&id=181 => [SolrInputDocument(fields: [id=181, aaa_i=-1976406138, bbb_i=2099255939, ccc_s=qzltjban, ddd_s=tivallpxa, eee_s={"njfsy":"njfsy"}, fff_s="", ggg_s=<root>gzrgylphkh</root>, hhh_s=tabfpe, geo_1_srpt=POINT( 42 23 ), geo_2_srpt=POINT( 42 -16 ), next_2_ids_ss=[182, 183], axx_i=1617014719, ayy_i=1062547001, azz_s=ixxbsnxsdj])] vs {numFound=0,numFoundExact=true,start=0,docs=[]} expected:<1> but was:<0>
> at __randomizedtesting.SeedInfo.seed([C602858C07426F25:78DE4D70C0E5E709]:0)
> at org.junit.Assert.fail(Assert.java:89)
> at org.junit.Assert.failNotEquals(Assert.java:835)
> at org.junit.Assert.assertEquals(Assert.java:647)
> at org.apache.solr.cloud.TestRandomFlRTGCloud.assertRTG(TestRandomFlRTGCloud.java:546)
> at org.apache.solr.cloud.TestRandomFlRTGCloud.assertOneIter(TestRandomFlRTGCloud.java:311)
> at org.apache.solr.cloud.TestRandomFlRTGCloud.testRandomizedUpdatesAndRTGs(TestRandomFlRTGCloud.java:262)
> at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> at java.base/java.lang.reflect.Method.invoke(Method.java:580)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1763)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
> at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
> at java.base/java.lang.Thread.run(Thread.java:1575)
>
>
> FAILED: org.apache.solr.cloud.TestRandomFlRTGCloud.testRandomizedUpdatesAndRTGs
>
> Error Message:
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> Stack Trace:
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at __randomizedtesting.SeedInfo.seed([C602858C07426F25:78DE4D70C0E5E709]:0)
> at java.base/sun.security.ssl.NewSessionTicket$T13NewSessionTicketMessage.<init>(NewSessionTicket.java:221)
> at java.base/sun.security.ssl.NewSessionTicket$T13NewSessionTicketConsumer.consume(NewSessionTicket.java:563)
> at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393)
> at java.base/sun.security.ssl.PostHandshakeContext.dispatch(PostHandshakeContext.java:82)
> at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:447)
> at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:206)
> at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
> at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
> at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1481)
> at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1068)
> at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
> at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
> at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
> at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
> at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
> at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
> at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
> at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
> at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
> at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
> at org.apache.solr.client.solrj.apache.HttpSolrClient.executeMethod(HttpSolrClient.java:571)
> at org.apache.solr.client.solrj.apache.HttpSolrClient.request(HttpSolrClient.java:257)
> at org.apache.solr.client.solrj.apache.HttpSolrClient.request(HttpSolrClient.java:236)
> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:283)
> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:166)
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:builds-unsubscribe@solr.apache.org
> For additional commands, e-mail:builds-help@solr.apache.org
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail:uwe@thetaphi.de
Re: Please help: Checksum and memory problems with builds on Policeman Jenkins => builds disabled
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
are there any updates regarding this? I'd like to test Solr with Java 26
EA (see mail on Lucene mailing list), but the checkum problem prevents
running Solr tests. Is there any way to work around this?
Uwe
Am 06.11.2025 um 17:45 schrieb Ishan Chattopadhyaya:
> I have an idea, I'll respond in detail shortly.
>
> On Thu, 6 Nov, 2025, 7:54 pm Uwe Schindler, <uw...@thetaphi.de> wrote:
>
>> Hi,
>>
>> There seems to be something completely wrong with Solr builds on main
>> branch on Policeman Jenkins. Can anybody explain why we see this error
>> all the time (since weeks), but on ASF Jenkins this resolved recently
>> (around November 1st -- suddenly without any related change in git!!!):
>>
>> Caused by: org.gradle.api.GradleException: Dependency checksum validation
>> failed:
>> - Dependency checksum mismatch
>> ('com.nvidia.cuvs.lucene:cuvs-lucene:25.10.0'), expected it to be:
>> 735cac75bcf3b55763941cfd1e473dfcf47fbf97, but was:
>> b5a458458255bfdfa6688a571b5bee032428f2d4
>>
>> To me it looks like the artifacts changed while keeping the same version
>> number of cuvs. As gradle caches downloaded artifacts forever (because
>> maven artifacts are not intended to be changed) it will of course keep
>> the old artifacts with no-longer the new/expected checksum of same
>> artifact versions.
>>
>> Can anybody explain what is happening here? Is there a way to fix the
>> build? Cleaning the workspace does not help so I disabled the builds for
>> now (we no longer test Solr on Linux, Windows, macOs). *In addition the
>> reason why I killed the builds is some issue with the execution at all.
>> It happens many times that "dead" test runner JVMs keep alive and are
>> not killed at end of build. After some time the machine ran out of
>> memory and OOM killer started to kill virtual machines (windows, macos).*
>>
>> So next to the checksum issue, we should figure out why the build's test
>> seems to have stuck JVMs dangling as dead processes after the build has
>> finished.
>>
>> Uwe
>>
>> Am 06.11.2025 um 02:48 schrieb Policeman Jenkins Server:
>>> Build:https://jenkins.thetaphi.de/job/Solr-main-Linux/29472/
>>> Java: 64bit/hotspot/jdk-23.0.2 -XX:-UseCompressedOops -XX:+UseParallelGC
>>>
>>> 4 tests failed.
>>> FAILED:
>> org.apache.solr.cloud.TestRandomFlRTGCloud.testRandomizedUpdatesAndRTGs
>>> Error Message:
>>> java.lang.AssertionError:
>> qt=/get&wt=xml&subq.q={!field+f%3Dnext_2_ids_i+v%3D$row.id}&fl=id,_root_:id,a*,my_escaped_json_field_alias:eee_s,my_geo_alias:[geo+f%3Dgeo_2_srpt],my_xml_field_alias:[xml+f%3Dggg_s],my_core_alias:[core],[shard],eee_s,my_int_field_alias2:bbb_i,[json+f%3Dfff_s],[core],my_geo_alias2:geo_2_srpt,[explain],bbb_i,score_alias:score,func_aaa_alias:add(1.3,sub(aaa_i,aaa_i)),my_docid_alias:[docid],ccc_s,id,subq:[subquery],*_s,my_json_field_alias:[json+f%3Deee_s],bogus_alias:other_bogus_field_i,[value+v%3D42+t%3Dint],my_shard_alias:[shard],aaa_i,score,ddd_s,*_i,geo_2_srpt&fl=[xml+f%3Dhhh_s],my_str_field_alias:ddd_s,explain_alias:[explain],my_int_field_alias:bbb_i&ids=312,312,265&ids=&ids=322
>> => {stream=org.apache.http.conn.EofSensorInputStream@67e71ce5,
>> responseStatus=500, closeableResponse=HttpResponseProxy{HTTP/1.1 500 Server
>> Error [Server: Jetty(12.0.27), Date: Thu, 06 Nov 2025 01:44:35 GMT, Server:
>> Jetty(12.0.27), Date: Thu, 06 Nov 2025 01:44:35 GMT, Vary: Accept-Encoding,
>> Last-Modified: Thu, 06 Nov 2025 01:44:37 GMT, ETag: "19a56d650b8",
>> Cache-Control: no-cache, no-store, Pragma: no-cache, Expires: Sat, 01 Jan
>> 2000 01:00:00 GMT, Content-Type: application/xml; charset=UTF-8,
>> Content-Length: 3375] ResponseEntityProxy{[Content-Type: application/xml;
>> charset=UTF-8,Content-Length: 3375,Chunked: false]}}}
>>> Stack Trace:
>>> java.lang.AssertionError:
>> qt=/get&wt=xml&subq.q={!field+f%3Dnext_2_ids_i+v%3D$row.id}&fl=id,_root_:id,a*,my_escaped_json_field_alias:eee_s,my_geo_alias:[geo+f%3Dgeo_2_srpt],my_xml_field_alias:[xml+f%3Dggg_s],my_core_alias:[core],[shard],eee_s,my_int_field_alias2:bbb_i,[json+f%3Dfff_s],[core],my_geo_alias2:geo_2_srpt,[explain],bbb_i,score_alias:score,func_aaa_alias:add(1.3,sub(aaa_i,aaa_i)),my_docid_alias:[docid],ccc_s,id,subq:[subquery],*_s,my_json_field_alias:[json+f%3Deee_s],bogus_alias:other_bogus_field_i,[value+v%3D42+t%3Dint],my_shard_alias:[shard],aaa_i,score,ddd_s,*_i,geo_2_srpt&fl=[xml+f%3Dhhh_s],my_str_field_alias:ddd_s,explain_alias:[explain],my_int_field_alias:bbb_i&ids=312,312,265&ids=&ids=322
>> => {stream=org.apache.http.conn.EofSensorInputStream@67e71ce5,
>> responseStatus=500, closeableResponse=HttpResponseProxy{HTTP/1.1 500 Server
>> Error [Server: Jetty(12.0.27), Date: Thu, 06 Nov 2025 01:44:35 GMT, Server:
>> Jetty(12.0.27), Date: Thu, 06 Nov 2025 01:44:35 GMT, Vary: Accept-Encoding,
>> Last-Modified: Thu, 06 Nov 2025 01:44:37 GMT, ETag: "19a56d650b8",
>> Cache-Control: no-cache, no-store, Pragma: no-cache, Expires: Sat, 01 Jan
>> 2000 01:00:00 GMT, Content-Type: application/xml; charset=UTF-8,
>> Content-Length: 3375] ResponseEntityProxy{[Content-Type: application/xml;
>> charset=UTF-8,Content-Length: 3375,Chunked: false]}}}
>>> at
>> __randomizedtesting.SeedInfo.seed([C602858C07426F25:78DE4D70C0E5E709]:0)
>>> at org.junit.Assert.fail(Assert.java:89)
>>> at org.junit.Assert.assertTrue(Assert.java:42)
>>> at org.junit.Assert.assertNotNull(Assert.java:713)
>>> at
>> org.apache.solr.cloud.TestRandomFlRTGCloud.assertRTG(TestRandomFlRTGCloud.java:544)
>>> at
>> org.apache.solr.cloud.TestRandomFlRTGCloud.assertOneIter(TestRandomFlRTGCloud.java:311)
>>> at
>> org.apache.solr.cloud.TestRandomFlRTGCloud.testRandomizedUpdatesAndRTGs(TestRandomFlRTGCloud.java:262)
>>> at
>> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
>>> at java.base/java.lang.reflect.Method.invoke(Method.java:580)
>>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1763)
>>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
>>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
>>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
>>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
>>> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>>> at
>> org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
>>> at
>> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>>> at
>> org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
>>> at
>> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
>>> at
>> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
>>> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
>>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
>>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
>>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
>>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
>>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
>>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
>>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
>>> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>>> at
>> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>> org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
>>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>> org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>>> at
>> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>>> at
>> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
>>> at
>> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
>>> at
>> org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
>>> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
>>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
>>> at java.base/java.lang.Thread.run(Thread.java:1575)
>>>
>>>
>>> FAILED:
>> org.apache.solr.cloud.TestRandomFlRTGCloud.testRandomizedUpdatesAndRTGs
>>> Error Message:
>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>
>>> Stack Trace:
>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>> at
>> __randomizedtesting.SeedInfo.seed([C602858C07426F25:78DE4D70C0E5E709]:0)
>>> at
>> java.base/java.math.BigInteger.shiftRightImpl(BigInteger.java:3601)
>>> at java.base/java.math.BigInteger.shiftRight(BigInteger.java:3566)
>>> at
>> java.base/sun.security.util.math.intpoly.IntegerPolynomial.setLimbsValuePositive(IntegerPolynomial.java:340)
>>> at
>> java.base/sun.security.util.math.intpoly.MontgomeryIntegerPolynomialP256.getElement(MontgomeryIntegerPolynomialP256.java:104)
>>> at
>> java.base/sun.security.util.math.intpoly.MontgomeryIntegerPolynomialP256.getElement(MontgomeryIntegerPolynomialP256.java:40)
>>> at java.base/sun.security.ec
>> .ECOperations.forParameters(ECOperations.java:94)
>>> at java.base/sun.security.ec
>> .ECKeyPairGenerator.generateKeyPairImpl(ECKeyPairGenerator.java:182)
>>> at java.base/sun.security.ec
>> .ECKeyPairGenerator.generateKeyPair(ECKeyPairGenerator.java:146)
>>> at
>> java.base/java.security.KeyPairGenerator$Delegate.generateKeyPair(KeyPairGenerator.java:726)
>>> at
>> java.base/sun.security.ssl.ECDHKeyExchange$ECDHEPossession.<init>(ECDHKeyExchange.java:118)
>>> at
>> java.base/sun.security.ssl.NamedGroup$ECDHEScheme.createPossession(NamedGroup.java:706)
>>> at
>> java.base/sun.security.ssl.NamedGroup$NamedGroupSpec.createPossession(NamedGroup.java:642)
>>> at
>> java.base/sun.security.ssl.NamedGroup.createPossession(NamedGroup.java:550)
>>> at
>> java.base/sun.security.ssl.SSLKeyExchange$T13KeyAgreement.createPossession(SSLKeyExchange.java:573)
>>> at
>> java.base/sun.security.ssl.SSLKeyExchange.createPossessions(SSLKeyExchange.java:92)
>>> at
>> java.base/sun.security.ssl.KeyShareExtension$CHKeyShareProducer.getShare(KeyShareExtension.java:295)
>>> at
>> java.base/sun.security.ssl.KeyShareExtension$CHKeyShareProducer.produce(KeyShareExtension.java:257)
>>> at
>> java.base/sun.security.ssl.SSLExtension.produce(SSLExtension.java:599)
>>> at
>> java.base/sun.security.ssl.SSLExtensions.produce(SSLExtensions.java:265)
>>> at
>> java.base/sun.security.ssl.ClientHello$ClientHelloKickstartProducer.produce(ClientHello.java:635)
>>> at
>> java.base/sun.security.ssl.SSLHandshake.kickstart(SSLHandshake.java:526)
>>> at
>> java.base/sun.security.ssl.ClientHandshakeContext.kickstart(ClientHandshakeContext.java:112)
>>> at
>> java.base/sun.security.ssl.TransportContext.kickstart(TransportContext.java:263)
>>> at
>> java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:448)
>>> at
>> java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
>>> at
>> org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
>>> at
>> org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
>>> at
>> org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
>>> at
>> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
>>> at
>> org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
>>> at
>> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
>>> at
>> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
>>>
>>> FAILED:
>> org.apache.solr.cloud.TestRandomFlRTGCloud.testRandomizedUpdatesAndRTGs
>>> Error Message:
>>> java.lang.AssertionError: num docs mismatch:
>> qt=/get&wt=xml&subq.q={!field+f%3Dnext_2_ids_i+v%3D$row.id}&fl=id,_root_:id,score,ggg_s,my_geo_alias:[geo+f%3Dgeo_2_srpt],my_shard_alias:[shard],bogus_raw_alias:[xml+f%3Dbogus_raw_field_ss],[xml+f%3Dhhh_s],explain_alias:[explain],[shard],eee_s,my_str_field_alias:ddd_s,my_str_field_alias2:ddd_s,[explain],[value+v%3D42+t%3Dint]&fl=bogus_unused_field_ss,geo_2_srpt,ccc_s,bbb_i,bogus_alias:other_bogus_field_i,[docid],my_core_alias:[core],*_i,*,id,[json+f%3Dfff_s],my_json_field_alias:[json+f%3Deee_s],my_escaped_json_field_alias:eee_s,val_alias:[value+v%3D1976+t%3Dint],my_xml_field_alias:[xml+f%3Dggg_s],subq:[subquery],my_geo_alias2:geo_2_srpt&fl=score_alias:score,my_int_field_alias2:bbb_i,[core],[geo+f%3Dgeo_1_srpt],*_s&fl=my_escaped_xml_field_alias:ggg_s,aaa_i,a*&fl=&fl=add(1.3,sub(aaa_i,aaa_i))&fl=my_docid_alias:[docid]&id=181
>> => [SolrInputDocument(fields: [id=181, aaa_i=-1976406138, bbb_i=2099255939,
>> ccc_s=qzltjban, ddd_s=tivallpxa, eee_s={"njfsy":"njfsy"}, fff_s="",
>> ggg_s=<root>gzrgylphkh</root>, hhh_s=tabfpe, geo_1_srpt=POINT( 42 23 ),
>> geo_2_srpt=POINT( 42 -16 ), next_2_ids_ss=[182, 183], axx_i=1617014719,
>> ayy_i=1062547001, azz_s=ixxbsnxsdj])] vs
>> {numFound=0,numFoundExact=true,start=0,docs=[]} expected:<1> but was:<0>
>>> Stack Trace:
>>> java.lang.AssertionError: num docs mismatch:
>> qt=/get&wt=xml&subq.q={!field+f%3Dnext_2_ids_i+v%3D$row.id}&fl=id,_root_:id,score,ggg_s,my_geo_alias:[geo+f%3Dgeo_2_srpt],my_shard_alias:[shard],bogus_raw_alias:[xml+f%3Dbogus_raw_field_ss],[xml+f%3Dhhh_s],explain_alias:[explain],[shard],eee_s,my_str_field_alias:ddd_s,my_str_field_alias2:ddd_s,[explain],[value+v%3D42+t%3Dint]&fl=bogus_unused_field_ss,geo_2_srpt,ccc_s,bbb_i,bogus_alias:other_bogus_field_i,[docid],my_core_alias:[core],*_i,*,id,[json+f%3Dfff_s],my_json_field_alias:[json+f%3Deee_s],my_escaped_json_field_alias:eee_s,val_alias:[value+v%3D1976+t%3Dint],my_xml_field_alias:[xml+f%3Dggg_s],subq:[subquery],my_geo_alias2:geo_2_srpt&fl=score_alias:score,my_int_field_alias2:bbb_i,[core],[geo+f%3Dgeo_1_srpt],*_s&fl=my_escaped_xml_field_alias:ggg_s,aaa_i,a*&fl=&fl=add(1.3,sub(aaa_i,aaa_i))&fl=my_docid_alias:[docid]&id=181
>> => [SolrInputDocument(fields: [id=181, aaa_i=-1976406138, bbb_i=2099255939,
>> ccc_s=qzltjban, ddd_s=tivallpxa, eee_s={"njfsy":"njfsy"}, fff_s="",
>> ggg_s=<root>gzrgylphkh</root>, hhh_s=tabfpe, geo_1_srpt=POINT( 42 23 ),
>> geo_2_srpt=POINT( 42 -16 ), next_2_ids_ss=[182, 183], axx_i=1617014719,
>> ayy_i=1062547001, azz_s=ixxbsnxsdj])] vs
>> {numFound=0,numFoundExact=true,start=0,docs=[]} expected:<1> but was:<0>
>>> at
>> __randomizedtesting.SeedInfo.seed([C602858C07426F25:78DE4D70C0E5E709]:0)
>>> at org.junit.Assert.fail(Assert.java:89)
>>> at org.junit.Assert.failNotEquals(Assert.java:835)
>>> at org.junit.Assert.assertEquals(Assert.java:647)
>>> at
>> org.apache.solr.cloud.TestRandomFlRTGCloud.assertRTG(TestRandomFlRTGCloud.java:546)
>>> at
>> org.apache.solr.cloud.TestRandomFlRTGCloud.assertOneIter(TestRandomFlRTGCloud.java:311)
>>> at
>> org.apache.solr.cloud.TestRandomFlRTGCloud.testRandomizedUpdatesAndRTGs(TestRandomFlRTGCloud.java:262)
>>> at
>> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
>>> at java.base/java.lang.reflect.Method.invoke(Method.java:580)
>>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1763)
>>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
>>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
>>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
>>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
>>> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>>> at
>> org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
>>> at
>> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>>> at
>> org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
>>> at
>> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
>>> at
>> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
>>> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
>>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
>>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
>>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
>>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
>>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
>>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
>>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
>>> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>>> at
>> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>> org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
>>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>> org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>>> at
>> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>>> at
>> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
>>> at
>> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
>>> at
>> org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
>>> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
>>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
>>> at java.base/java.lang.Thread.run(Thread.java:1575)
>>>
>>>
>>> FAILED:
>> org.apache.solr.cloud.TestRandomFlRTGCloud.testRandomizedUpdatesAndRTGs
>>> Error Message:
>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>
>>> Stack Trace:
>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>> at
>> __randomizedtesting.SeedInfo.seed([C602858C07426F25:78DE4D70C0E5E709]:0)
>>> at
>> java.base/sun.security.ssl.NewSessionTicket$T13NewSessionTicketMessage.<init>(NewSessionTicket.java:221)
>>> at
>> java.base/sun.security.ssl.NewSessionTicket$T13NewSessionTicketConsumer.consume(NewSessionTicket.java:563)
>>> at
>> java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393)
>>> at
>> java.base/sun.security.ssl.PostHandshakeContext.dispatch(PostHandshakeContext.java:82)
>>> at
>> java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:447)
>>> at
>> java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:206)
>>> at
>> java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
>>> at
>> java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
>>> at
>> java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1481)
>>> at
>> java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1068)
>>> at org.apache.http.impl.io
>> .SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
>>> at org.apache.http.impl.io
>> .SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
>>> at org.apache.http.impl.io
>> .SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
>>> at
>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
>>> at
>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
>>> at org.apache.http.impl.io
>> .AbstractMessageParser.parse(AbstractMessageParser.java:259)
>>> at
>> org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
>>> at
>> org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
>>> at
>> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
>>> at
>> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
>>> at
>> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
>>> at
>> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
>>> at
>> org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
>>> at
>> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
>>> at
>> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
>>> at
>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
>>> at
>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
>>> at
>> org.apache.solr.client.solrj.apache.HttpSolrClient.executeMethod(HttpSolrClient.java:571)
>>> at
>> org.apache.solr.client.solrj.apache.HttpSolrClient.request(HttpSolrClient.java:257)
>>> at
>> org.apache.solr.client.solrj.apache.HttpSolrClient.request(HttpSolrClient.java:236)
>>> at
>> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:283)
>>> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:166)
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail:builds-unsubscribe@solr.apache.org
>>> For additional commands, e-mail:builds-help@solr.apache.org
>> --
>> Uwe Schindler
>> Achterdiek 19, D-28357 Bremen
>> https://www.thetaphi.de
>> eMail:uwe@thetaphi.de
>>
>>
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: uwe@thetaphi.de
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Please help: Checksum and memory problems with builds on Policeman Jenkins => builds disabled
Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
I have an idea, I'll respond in detail shortly.
On Thu, 6 Nov, 2025, 7:54 pm Uwe Schindler, <uw...@thetaphi.de> wrote:
> Hi,
>
> There seems to be something completely wrong with Solr builds on main
> branch on Policeman Jenkins. Can anybody explain why we see this error
> all the time (since weeks), but on ASF Jenkins this resolved recently
> (around November 1st -- suddenly without any related change in git!!!):
>
> Caused by: org.gradle.api.GradleException: Dependency checksum validation
> failed:
> - Dependency checksum mismatch
> ('com.nvidia.cuvs.lucene:cuvs-lucene:25.10.0'), expected it to be:
> 735cac75bcf3b55763941cfd1e473dfcf47fbf97, but was:
> b5a458458255bfdfa6688a571b5bee032428f2d4
>
> To me it looks like the artifacts changed while keeping the same version
> number of cuvs. As gradle caches downloaded artifacts forever (because
> maven artifacts are not intended to be changed) it will of course keep
> the old artifacts with no-longer the new/expected checksum of same
> artifact versions.
>
> Can anybody explain what is happening here? Is there a way to fix the
> build? Cleaning the workspace does not help so I disabled the builds for
> now (we no longer test Solr on Linux, Windows, macOs). *In addition the
> reason why I killed the builds is some issue with the execution at all.
> It happens many times that "dead" test runner JVMs keep alive and are
> not killed at end of build. After some time the machine ran out of
> memory and OOM killer started to kill virtual machines (windows, macos).*
>
> So next to the checksum issue, we should figure out why the build's test
> seems to have stuck JVMs dangling as dead processes after the build has
> finished.
>
> Uwe
>
> Am 06.11.2025 um 02:48 schrieb Policeman Jenkins Server:
> > Build:https://jenkins.thetaphi.de/job/Solr-main-Linux/29472/
> > Java: 64bit/hotspot/jdk-23.0.2 -XX:-UseCompressedOops -XX:+UseParallelGC
> >
> > 4 tests failed.
> > FAILED:
> org.apache.solr.cloud.TestRandomFlRTGCloud.testRandomizedUpdatesAndRTGs
> >
> > Error Message:
> > java.lang.AssertionError:
> qt=/get&wt=xml&subq.q={!field+f%3Dnext_2_ids_i+v%3D$row.id}&fl=id,_root_:id,a*,my_escaped_json_field_alias:eee_s,my_geo_alias:[geo+f%3Dgeo_2_srpt],my_xml_field_alias:[xml+f%3Dggg_s],my_core_alias:[core],[shard],eee_s,my_int_field_alias2:bbb_i,[json+f%3Dfff_s],[core],my_geo_alias2:geo_2_srpt,[explain],bbb_i,score_alias:score,func_aaa_alias:add(1.3,sub(aaa_i,aaa_i)),my_docid_alias:[docid],ccc_s,id,subq:[subquery],*_s,my_json_field_alias:[json+f%3Deee_s],bogus_alias:other_bogus_field_i,[value+v%3D42+t%3Dint],my_shard_alias:[shard],aaa_i,score,ddd_s,*_i,geo_2_srpt&fl=[xml+f%3Dhhh_s],my_str_field_alias:ddd_s,explain_alias:[explain],my_int_field_alias:bbb_i&ids=312,312,265&ids=&ids=322
> => {stream=org.apache.http.conn.EofSensorInputStream@67e71ce5,
> responseStatus=500, closeableResponse=HttpResponseProxy{HTTP/1.1 500 Server
> Error [Server: Jetty(12.0.27), Date: Thu, 06 Nov 2025 01:44:35 GMT, Server:
> Jetty(12.0.27), Date: Thu, 06 Nov 2025 01:44:35 GMT, Vary: Accept-Encoding,
> Last-Modified: Thu, 06 Nov 2025 01:44:37 GMT, ETag: "19a56d650b8",
> Cache-Control: no-cache, no-store, Pragma: no-cache, Expires: Sat, 01 Jan
> 2000 01:00:00 GMT, Content-Type: application/xml; charset=UTF-8,
> Content-Length: 3375] ResponseEntityProxy{[Content-Type: application/xml;
> charset=UTF-8,Content-Length: 3375,Chunked: false]}}}
> >
> > Stack Trace:
> > java.lang.AssertionError:
> qt=/get&wt=xml&subq.q={!field+f%3Dnext_2_ids_i+v%3D$row.id}&fl=id,_root_:id,a*,my_escaped_json_field_alias:eee_s,my_geo_alias:[geo+f%3Dgeo_2_srpt],my_xml_field_alias:[xml+f%3Dggg_s],my_core_alias:[core],[shard],eee_s,my_int_field_alias2:bbb_i,[json+f%3Dfff_s],[core],my_geo_alias2:geo_2_srpt,[explain],bbb_i,score_alias:score,func_aaa_alias:add(1.3,sub(aaa_i,aaa_i)),my_docid_alias:[docid],ccc_s,id,subq:[subquery],*_s,my_json_field_alias:[json+f%3Deee_s],bogus_alias:other_bogus_field_i,[value+v%3D42+t%3Dint],my_shard_alias:[shard],aaa_i,score,ddd_s,*_i,geo_2_srpt&fl=[xml+f%3Dhhh_s],my_str_field_alias:ddd_s,explain_alias:[explain],my_int_field_alias:bbb_i&ids=312,312,265&ids=&ids=322
> => {stream=org.apache.http.conn.EofSensorInputStream@67e71ce5,
> responseStatus=500, closeableResponse=HttpResponseProxy{HTTP/1.1 500 Server
> Error [Server: Jetty(12.0.27), Date: Thu, 06 Nov 2025 01:44:35 GMT, Server:
> Jetty(12.0.27), Date: Thu, 06 Nov 2025 01:44:35 GMT, Vary: Accept-Encoding,
> Last-Modified: Thu, 06 Nov 2025 01:44:37 GMT, ETag: "19a56d650b8",
> Cache-Control: no-cache, no-store, Pragma: no-cache, Expires: Sat, 01 Jan
> 2000 01:00:00 GMT, Content-Type: application/xml; charset=UTF-8,
> Content-Length: 3375] ResponseEntityProxy{[Content-Type: application/xml;
> charset=UTF-8,Content-Length: 3375,Chunked: false]}}}
> > at
> __randomizedtesting.SeedInfo.seed([C602858C07426F25:78DE4D70C0E5E709]:0)
> > at org.junit.Assert.fail(Assert.java:89)
> > at org.junit.Assert.assertTrue(Assert.java:42)
> > at org.junit.Assert.assertNotNull(Assert.java:713)
> > at
> org.apache.solr.cloud.TestRandomFlRTGCloud.assertRTG(TestRandomFlRTGCloud.java:544)
> > at
> org.apache.solr.cloud.TestRandomFlRTGCloud.assertOneIter(TestRandomFlRTGCloud.java:311)
> > at
> org.apache.solr.cloud.TestRandomFlRTGCloud.testRandomizedUpdatesAndRTGs(TestRandomFlRTGCloud.java:262)
> > at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:580)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1763)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
> > at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
> > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> > at
> org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
> > at
> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> > at
> org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
> > at
> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> > at
> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> > at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
> > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> > at
> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> > at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> > at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> > at
> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> > at
> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> > at
> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> > at
> org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
> > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
> > at java.base/java.lang.Thread.run(Thread.java:1575)
> >
> >
> > FAILED:
> org.apache.solr.cloud.TestRandomFlRTGCloud.testRandomizedUpdatesAndRTGs
> >
> > Error Message:
> > java.lang.OutOfMemoryError: GC overhead limit exceeded
> >
> > Stack Trace:
> > java.lang.OutOfMemoryError: GC overhead limit exceeded
> > at
> __randomizedtesting.SeedInfo.seed([C602858C07426F25:78DE4D70C0E5E709]:0)
> > at
> java.base/java.math.BigInteger.shiftRightImpl(BigInteger.java:3601)
> > at java.base/java.math.BigInteger.shiftRight(BigInteger.java:3566)
> > at
> java.base/sun.security.util.math.intpoly.IntegerPolynomial.setLimbsValuePositive(IntegerPolynomial.java:340)
> > at
> java.base/sun.security.util.math.intpoly.MontgomeryIntegerPolynomialP256.getElement(MontgomeryIntegerPolynomialP256.java:104)
> > at
> java.base/sun.security.util.math.intpoly.MontgomeryIntegerPolynomialP256.getElement(MontgomeryIntegerPolynomialP256.java:40)
> > at java.base/sun.security.ec
> .ECOperations.forParameters(ECOperations.java:94)
> > at java.base/sun.security.ec
> .ECKeyPairGenerator.generateKeyPairImpl(ECKeyPairGenerator.java:182)
> > at java.base/sun.security.ec
> .ECKeyPairGenerator.generateKeyPair(ECKeyPairGenerator.java:146)
> > at
> java.base/java.security.KeyPairGenerator$Delegate.generateKeyPair(KeyPairGenerator.java:726)
> > at
> java.base/sun.security.ssl.ECDHKeyExchange$ECDHEPossession.<init>(ECDHKeyExchange.java:118)
> > at
> java.base/sun.security.ssl.NamedGroup$ECDHEScheme.createPossession(NamedGroup.java:706)
> > at
> java.base/sun.security.ssl.NamedGroup$NamedGroupSpec.createPossession(NamedGroup.java:642)
> > at
> java.base/sun.security.ssl.NamedGroup.createPossession(NamedGroup.java:550)
> > at
> java.base/sun.security.ssl.SSLKeyExchange$T13KeyAgreement.createPossession(SSLKeyExchange.java:573)
> > at
> java.base/sun.security.ssl.SSLKeyExchange.createPossessions(SSLKeyExchange.java:92)
> > at
> java.base/sun.security.ssl.KeyShareExtension$CHKeyShareProducer.getShare(KeyShareExtension.java:295)
> > at
> java.base/sun.security.ssl.KeyShareExtension$CHKeyShareProducer.produce(KeyShareExtension.java:257)
> > at
> java.base/sun.security.ssl.SSLExtension.produce(SSLExtension.java:599)
> > at
> java.base/sun.security.ssl.SSLExtensions.produce(SSLExtensions.java:265)
> > at
> java.base/sun.security.ssl.ClientHello$ClientHelloKickstartProducer.produce(ClientHello.java:635)
> > at
> java.base/sun.security.ssl.SSLHandshake.kickstart(SSLHandshake.java:526)
> > at
> java.base/sun.security.ssl.ClientHandshakeContext.kickstart(ClientHandshakeContext.java:112)
> > at
> java.base/sun.security.ssl.TransportContext.kickstart(TransportContext.java:263)
> > at
> java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:448)
> > at
> java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
> > at
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
> > at
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
> > at
> org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
> > at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
> > at
> org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
> > at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
> > at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
> >
> >
> > FAILED:
> org.apache.solr.cloud.TestRandomFlRTGCloud.testRandomizedUpdatesAndRTGs
> >
> > Error Message:
> > java.lang.AssertionError: num docs mismatch:
> qt=/get&wt=xml&subq.q={!field+f%3Dnext_2_ids_i+v%3D$row.id}&fl=id,_root_:id,score,ggg_s,my_geo_alias:[geo+f%3Dgeo_2_srpt],my_shard_alias:[shard],bogus_raw_alias:[xml+f%3Dbogus_raw_field_ss],[xml+f%3Dhhh_s],explain_alias:[explain],[shard],eee_s,my_str_field_alias:ddd_s,my_str_field_alias2:ddd_s,[explain],[value+v%3D42+t%3Dint]&fl=bogus_unused_field_ss,geo_2_srpt,ccc_s,bbb_i,bogus_alias:other_bogus_field_i,[docid],my_core_alias:[core],*_i,*,id,[json+f%3Dfff_s],my_json_field_alias:[json+f%3Deee_s],my_escaped_json_field_alias:eee_s,val_alias:[value+v%3D1976+t%3Dint],my_xml_field_alias:[xml+f%3Dggg_s],subq:[subquery],my_geo_alias2:geo_2_srpt&fl=score_alias:score,my_int_field_alias2:bbb_i,[core],[geo+f%3Dgeo_1_srpt],*_s&fl=my_escaped_xml_field_alias:ggg_s,aaa_i,a*&fl=&fl=add(1.3,sub(aaa_i,aaa_i))&fl=my_docid_alias:[docid]&id=181
> => [SolrInputDocument(fields: [id=181, aaa_i=-1976406138, bbb_i=2099255939,
> ccc_s=qzltjban, ddd_s=tivallpxa, eee_s={"njfsy":"njfsy"}, fff_s="",
> ggg_s=<root>gzrgylphkh</root>, hhh_s=tabfpe, geo_1_srpt=POINT( 42 23 ),
> geo_2_srpt=POINT( 42 -16 ), next_2_ids_ss=[182, 183], axx_i=1617014719,
> ayy_i=1062547001, azz_s=ixxbsnxsdj])] vs
> {numFound=0,numFoundExact=true,start=0,docs=[]} expected:<1> but was:<0>
> >
> > Stack Trace:
> > java.lang.AssertionError: num docs mismatch:
> qt=/get&wt=xml&subq.q={!field+f%3Dnext_2_ids_i+v%3D$row.id}&fl=id,_root_:id,score,ggg_s,my_geo_alias:[geo+f%3Dgeo_2_srpt],my_shard_alias:[shard],bogus_raw_alias:[xml+f%3Dbogus_raw_field_ss],[xml+f%3Dhhh_s],explain_alias:[explain],[shard],eee_s,my_str_field_alias:ddd_s,my_str_field_alias2:ddd_s,[explain],[value+v%3D42+t%3Dint]&fl=bogus_unused_field_ss,geo_2_srpt,ccc_s,bbb_i,bogus_alias:other_bogus_field_i,[docid],my_core_alias:[core],*_i,*,id,[json+f%3Dfff_s],my_json_field_alias:[json+f%3Deee_s],my_escaped_json_field_alias:eee_s,val_alias:[value+v%3D1976+t%3Dint],my_xml_field_alias:[xml+f%3Dggg_s],subq:[subquery],my_geo_alias2:geo_2_srpt&fl=score_alias:score,my_int_field_alias2:bbb_i,[core],[geo+f%3Dgeo_1_srpt],*_s&fl=my_escaped_xml_field_alias:ggg_s,aaa_i,a*&fl=&fl=add(1.3,sub(aaa_i,aaa_i))&fl=my_docid_alias:[docid]&id=181
> => [SolrInputDocument(fields: [id=181, aaa_i=-1976406138, bbb_i=2099255939,
> ccc_s=qzltjban, ddd_s=tivallpxa, eee_s={"njfsy":"njfsy"}, fff_s="",
> ggg_s=<root>gzrgylphkh</root>, hhh_s=tabfpe, geo_1_srpt=POINT( 42 23 ),
> geo_2_srpt=POINT( 42 -16 ), next_2_ids_ss=[182, 183], axx_i=1617014719,
> ayy_i=1062547001, azz_s=ixxbsnxsdj])] vs
> {numFound=0,numFoundExact=true,start=0,docs=[]} expected:<1> but was:<0>
> > at
> __randomizedtesting.SeedInfo.seed([C602858C07426F25:78DE4D70C0E5E709]:0)
> > at org.junit.Assert.fail(Assert.java:89)
> > at org.junit.Assert.failNotEquals(Assert.java:835)
> > at org.junit.Assert.assertEquals(Assert.java:647)
> > at
> org.apache.solr.cloud.TestRandomFlRTGCloud.assertRTG(TestRandomFlRTGCloud.java:546)
> > at
> org.apache.solr.cloud.TestRandomFlRTGCloud.assertOneIter(TestRandomFlRTGCloud.java:311)
> > at
> org.apache.solr.cloud.TestRandomFlRTGCloud.testRandomizedUpdatesAndRTGs(TestRandomFlRTGCloud.java:262)
> > at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:580)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1763)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
> > at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
> > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> > at
> org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
> > at
> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> > at
> org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
> > at
> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> > at
> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> > at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
> > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> > at
> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> > at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> > at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> > at
> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> > at
> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> > at
> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> > at
> org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
> > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
> > at java.base/java.lang.Thread.run(Thread.java:1575)
> >
> >
> > FAILED:
> org.apache.solr.cloud.TestRandomFlRTGCloud.testRandomizedUpdatesAndRTGs
> >
> > Error Message:
> > java.lang.OutOfMemoryError: GC overhead limit exceeded
> >
> > Stack Trace:
> > java.lang.OutOfMemoryError: GC overhead limit exceeded
> > at
> __randomizedtesting.SeedInfo.seed([C602858C07426F25:78DE4D70C0E5E709]:0)
> > at
> java.base/sun.security.ssl.NewSessionTicket$T13NewSessionTicketMessage.<init>(NewSessionTicket.java:221)
> > at
> java.base/sun.security.ssl.NewSessionTicket$T13NewSessionTicketConsumer.consume(NewSessionTicket.java:563)
> > at
> java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393)
> > at
> java.base/sun.security.ssl.PostHandshakeContext.dispatch(PostHandshakeContext.java:82)
> > at
> java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:447)
> > at
> java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:206)
> > at
> java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
> > at
> java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
> > at
> java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1481)
> > at
> java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1068)
> > at org.apache.http.impl.io
> .SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
> > at org.apache.http.impl.io
> .SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
> > at org.apache.http.impl.io
> .SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
> > at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
> > at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
> > at org.apache.http.impl.io
> .AbstractMessageParser.parse(AbstractMessageParser.java:259)
> > at
> org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
> > at
> org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
> > at
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
> > at
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
> > at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
> > at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
> > at
> org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
> > at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
> > at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> > at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> > at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
> > at
> org.apache.solr.client.solrj.apache.HttpSolrClient.executeMethod(HttpSolrClient.java:571)
> > at
> org.apache.solr.client.solrj.apache.HttpSolrClient.request(HttpSolrClient.java:257)
> > at
> org.apache.solr.client.solrj.apache.HttpSolrClient.request(HttpSolrClient.java:236)
> > at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:283)
> > at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:166)
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail:builds-unsubscribe@solr.apache.org
> > For additional commands, e-mail:builds-help@solr.apache.org
>
> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail:uwe@thetaphi.de
>
>