You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2018/03/02 05:39:04 UTC
[JENKINS] Lucene-Solr-NightlyTests-6.6 - Build # 32 - Still
Unstable
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.6/32/
3 tests failed.
FAILED: org.apache.solr.cloud.ShardSplitTest.testSplitWithChaosMonkey
Error Message:
There are still nodes recoverying - waited for 330 seconds
Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
at __randomizedtesting.SeedInfo.seed([34BB004ACFA3C0F3:BF9CD39B8EA56B77]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:187)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:144)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:139)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:856)
at org.apache.solr.cloud.ShardSplitTest.testSplitWithChaosMonkey(ShardSplitTest.java:437)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv
Error Message:
java.lang.RuntimeException: Error from server at http://127.0.0.1:33720/solr/test_col: Async exception during distributed update: Error from server at http://127.0.0.1:43289/solr/test_col_shard1_replica2: Server Error request: http://127.0.0.1:43289/solr/test_col_shard1_replica2/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A33720%2Fsolr%2Ftest_col_shard3_replica2%2F&wt=javabin&version=2 Remote error message: Failed synchronous update on shard StdNode: http://127.0.0.1:45221/solr/test_col_shard1_replica1/ update: org.apache.solr.client.solrj.request.UpdateRequest@772b5bf6
Stack Trace:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error from server at http://127.0.0.1:33720/solr/test_col: Async exception during distributed update: Error from server at http://127.0.0.1:43289/solr/test_col_shard1_replica2: Server Error
request: http://127.0.0.1:43289/solr/test_col_shard1_replica2/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A33720%2Fsolr%2Ftest_col_shard3_replica2%2F&wt=javabin&version=2
Remote error message: Failed synchronous update on shard StdNode: http://127.0.0.1:45221/solr/test_col_shard1_replica1/ update: org.apache.solr.client.solrj.request.UpdateRequest@772b5bf6
at __randomizedtesting.SeedInfo.seed([34BB004ACFA3C0F3:2AF620C45FEFAE2]:0)
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:281)
at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv(TestStressCloudBlindAtomicUpdates.java:193)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Error from server at http://127.0.0.1:33720/solr/test_col: Async exception during distributed update: Error from server at http://127.0.0.1:43289/solr/test_col_shard1_replica2: Server Error
request: http://127.0.0.1:43289/solr/test_col_shard1_replica2/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A33720%2Fsolr%2Ftest_col_shard3_replica2%2F&wt=javabin&version=2
Remote error message: Failed synchronous update on shard StdNode: http://127.0.0.1:45221/solr/test_col_shard1_replica1/ update: org.apache.solr.client.solrj.request.UpdateRequest@772b5bf6
at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.run(TestStressCloudBlindAtomicUpdates.java:409)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:33720/solr/test_col: Async exception during distributed update: Error from server at http://127.0.0.1:43289/solr/test_col_shard1_replica2: Server Error
request: http://127.0.0.1:43289/solr/test_col_shard1_replica2/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A33720%2Fsolr%2Ftest_col_shard3_replica2%2F&wt=javabin&version=2
Remote error message: Failed synchronous update on shard StdNode: http://127.0.0.1:45221/solr/test_col_shard1_replica1/ update: org.apache.solr.client.solrj.request.UpdateRequest@772b5bf6
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:612)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:177)
at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.doRandomAtomicUpdate(TestStressCloudBlindAtomicUpdates.java:368)
at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.run(TestStressCloudBlindAtomicUpdates.java:397)
... 6 more
FAILED: org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test
Error Message:
expected:<2> but was:<3>
Stack Trace:
java.lang.AssertionError: expected:<2> but was:<3>
at __randomizedtesting.SeedInfo.seed([34BB004ACFA3C0F3:BCEF3F90615FAD0B]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.assertSliceAndReplicaCount(SharedFSAutoReplicaFailoverTest.java:419)
at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:264)
at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 12973 lines...]
[junit4] Suite: org.apache.solr.cloud.ShardSplitTest
[junit4] 2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/init-core-data-001
[junit4] 2> 2751978 WARN (SUITE-ShardSplitTest-seed#[34BB004ACFA3C0F3]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
[junit4] 2> 2751979 INFO (SUITE-ShardSplitTest-seed#[34BB004ACFA3C0F3]-worker) [ ] o.a.s.SolrTestCaseJ4 Using TrieFields
[junit4] 2> 2751980 INFO (SUITE-ShardSplitTest-seed#[34BB004ACFA3C0F3]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
[junit4] 2> 2751980 INFO (SUITE-ShardSplitTest-seed#[34BB004ACFA3C0F3]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
[junit4] 2> 2751995 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 2751995 INFO (Thread-3803) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 2751995 INFO (Thread-3803) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 2752009 ERROR (Thread-3803) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 2752095 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.ZkTestServer start zk server on port:33069
[junit4] 2> 2752161 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 2752181 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 2752182 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 2752183 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 2752193 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 2752194 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 2752195 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 2752217 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 2752219 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 2752237 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 2752238 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 2752684 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/control-001/cores/collection1
[junit4] 2> 2752685 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 2752686 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@188c2521{/,null,AVAILABLE}
[junit4] 2> 2752726 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@5cf6f4e6{HTTP/1.1,[http/1.1]}{127.0.0.1:36075}
[junit4] 2> 2752727 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.e.j.s.Server Started @2764404ms
[junit4] 2> 2752727 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/tempDir-001/control/data, hostContext=/, hostPort=36075, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/control-001/cores}
[junit4] 2> 2752727 ERROR (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2752727 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.6.3
[junit4] 2> 2752727 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2752727 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2752727 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-02T03:22:19.408Z
[junit4] 2> 2752751 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 2752751 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/control-001/solr.xml
[junit4] 2> 2752802 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 2752825 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33069/solr
[junit4] 2> 2753037 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [n:127.0.0.1:36075_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 2753038 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [n:127.0.0.1:36075_ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:36075_
[junit4] 2> 2753053 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [n:127.0.0.1:36075_ ] o.a.s.c.Overseer Overseer (id=99612160077135876-127.0.0.1:36075_-n_0000000000) starting
[junit4] 2> 2753110 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [n:127.0.0.1:36075_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36075_
[junit4] 2> 2753125 INFO (zkCallback-2553-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2753387 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [n:127.0.0.1:36075_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/control-001/cores
[junit4] 2> 2753387 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [n:127.0.0.1:36075_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 2753388 INFO (coreLoadExecutor-5297-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 2753446 INFO (OverseerStateUpdate-99612160077135876-127.0.0.1:36075_-n_0000000000) [n:127.0.0.1:36075_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 2754570 WARN (coreLoadExecutor-5297-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ c:control_collection x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 2754571 INFO (coreLoadExecutor-5297-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ c:control_collection x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.3
[junit4] 2> 2754644 INFO (coreLoadExecutor-5297-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 2755099 INFO (coreLoadExecutor-5297-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ c:control_collection x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 2755159 INFO (coreLoadExecutor-5297-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ c:control_collection x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection, trusted=true
[junit4] 2> 2755159 INFO (coreLoadExecutor-5297-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ c:control_collection x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2755159 INFO (coreLoadExecutor-5297-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ c:control_collection x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/control-001/cores/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/control-001/cores/collection1/data/]
[junit4] 2> 2755160 INFO (coreLoadExecutor-5297-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ c:control_collection x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@608cc2ff
[junit4] 2> 2755174 INFO (coreLoadExecutor-5297-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ c:control_collection x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@7b154237
[junit4] 2> 2755282 WARN (coreLoadExecutor-5297-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ c:control_collection x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 2755497 INFO (coreLoadExecutor-5297-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ c:control_collection x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 2755497 INFO (coreLoadExecutor-5297-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ c:control_collection x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 2755511 INFO (coreLoadExecutor-5297-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ c:control_collection x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2755511 INFO (coreLoadExecutor-5297-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ c:control_collection x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2755511 INFO (coreLoadExecutor-5297-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ c:control_collection x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=1.0341796875, floorSegmentMB=0.845703125, forceMergeDeletesPctAllowed=14.694259749291467, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6735098025561761
[junit4] 2> 2755531 INFO (coreLoadExecutor-5297-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ c:control_collection x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1bc9afc8[collection1] main]
[junit4] 2> 2755532 INFO (coreLoadExecutor-5297-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 2755533 INFO (coreLoadExecutor-5297-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2755533 INFO (coreLoadExecutor-5297-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ c:control_collection x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 2755534 INFO (searcherExecutor-5298-thread-1-processing-n:127.0.0.1:36075_ x:collection1 c:control_collection) [n:127.0.0.1:36075_ c:control_collection x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1bc9afc8[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2755547 INFO (coreLoadExecutor-5297-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ c:control_collection x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1593794564957667328
[junit4] 2> 2755585 INFO (coreZkRegister-5290-thread-1-processing-n:127.0.0.1:36075_ x:collection1 c:control_collection) [n:127.0.0.1:36075_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 2755586 INFO (coreZkRegister-5290-thread-1-processing-n:127.0.0.1:36075_ x:collection1 c:control_collection) [n:127.0.0.1:36075_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 2755586 INFO (coreZkRegister-5290-thread-1-processing-n:127.0.0.1:36075_ x:collection1 c:control_collection) [n:127.0.0.1:36075_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:36075/collection1/
[junit4] 2> 2755586 INFO (coreZkRegister-5290-thread-1-processing-n:127.0.0.1:36075_ x:collection1 c:control_collection) [n:127.0.0.1:36075_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 2755586 INFO (coreZkRegister-5290-thread-1-processing-n:127.0.0.1:36075_ x:collection1 c:control_collection) [n:127.0.0.1:36075_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:36075/collection1/ has no replicas
[junit4] 2> 2755586 INFO (coreZkRegister-5290-thread-1-processing-n:127.0.0.1:36075_ x:collection1 c:control_collection) [n:127.0.0.1:36075_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 2755604 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2755606 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33069/solr ready
[junit4] 2> 2755606 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 2755606 INFO (coreZkRegister-5290-thread-1-processing-n:127.0.0.1:36075_ x:collection1 c:control_collection) [n:127.0.0.1:36075_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:36075/collection1/ shard1
[junit4] 2> 2755606 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
[junit4] 2> 2755768 INFO (coreZkRegister-5290-thread-1-processing-n:127.0.0.1:36075_ x:collection1 c:control_collection) [n:127.0.0.1:36075_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 2756138 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/shard-1-001/cores/collection1
[junit4] 2> 2756139 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/shard-1-001
[junit4] 2> 2756139 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 2756154 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@25789070{/,null,AVAILABLE}
[junit4] 2> 2756155 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@784272ce{HTTP/1.1,[http/1.1]}{127.0.0.1:34046}
[junit4] 2> 2756155 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.e.j.s.Server Started @2767833ms
[junit4] 2> 2756155 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=34046, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/shard-1-001/cores}
[junit4] 2> 2756156 ERROR (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2756156 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.6.3
[junit4] 2> 2756174 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2756174 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2756174 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-02T03:22:22.855Z
[junit4] 2> 2756199 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 2756199 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/shard-1-001/solr.xml
[junit4] 2> 2756246 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 2756247 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33069/solr
[junit4] 2> 2756275 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [n:127.0.0.1:34046_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2756278 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [n:127.0.0.1:34046_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 2756293 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [n:127.0.0.1:34046_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34046_
[junit4] 2> 2756307 INFO (zkCallback-2557-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 2756307 INFO (zkCallback-2563-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 2756308 INFO (zkCallback-2553-thread-2-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 2756503 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [n:127.0.0.1:34046_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/shard-1-001/cores
[junit4] 2> 2756503 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [n:127.0.0.1:34046_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 2756515 INFO (coreLoadExecutor-5308-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 2756517 INFO (OverseerStateUpdate-99612160077135876-127.0.0.1:36075_-n_0000000000) [n:127.0.0.1:36075_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
[junit4] 2> 2756652 INFO (zkCallback-2563-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2757569 WARN (coreLoadExecutor-5308-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ c:collection1 x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 2757569 INFO (coreLoadExecutor-5308-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.3
[junit4] 2> 2757614 INFO (coreLoadExecutor-5308-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 2757955 INFO (coreLoadExecutor-5308-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 2758038 INFO (coreLoadExecutor-5308-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
[junit4] 2> 2758039 INFO (coreLoadExecutor-5308-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ c:collection1 x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2758039 INFO (coreLoadExecutor-5308-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/shard-1-001/cores/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/shard-1-001/cores/collection1/data/]
[junit4] 2> 2758039 INFO (coreLoadExecutor-5308-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@608cc2ff
[junit4] 2> 2758041 INFO (coreLoadExecutor-5308-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@3bb647df
[junit4] 2> 2758083 WARN (coreLoadExecutor-5308-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 2758262 INFO (coreLoadExecutor-5308-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 2758262 INFO (coreLoadExecutor-5308-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 2758262 INFO (coreLoadExecutor-5308-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2758263 INFO (coreLoadExecutor-5308-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2758264 INFO (coreLoadExecutor-5308-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=1.0341796875, floorSegmentMB=0.845703125, forceMergeDeletesPctAllowed=14.694259749291467, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6735098025561761
[junit4] 2> 2758264 INFO (coreLoadExecutor-5308-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@56379b9f[collection1] main]
[junit4] 2> 2758278 INFO (coreLoadExecutor-5308-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 2758278 INFO (coreLoadExecutor-5308-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2758278 INFO (coreLoadExecutor-5308-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 2758280 INFO (searcherExecutor-5309-thread-1-processing-n:127.0.0.1:34046_ x:collection1 c:collection1) [n:127.0.0.1:34046_ c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@56379b9f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2758280 INFO (coreLoadExecutor-5308-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1593794567823425536
[junit4] 2> 2758300 INFO (coreZkRegister-5303-thread-1-processing-n:127.0.0.1:34046_ x:collection1 c:collection1) [n:127.0.0.1:34046_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 2758300 INFO (coreZkRegister-5303-thread-1-processing-n:127.0.0.1:34046_ x:collection1 c:collection1) [n:127.0.0.1:34046_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 2758300 INFO (coreZkRegister-5303-thread-1-processing-n:127.0.0.1:34046_ x:collection1 c:collection1) [n:127.0.0.1:34046_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:34046/collection1/
[junit4] 2> 2758300 INFO (coreZkRegister-5303-thread-1-processing-n:127.0.0.1:34046_ x:collection1 c:collection1) [n:127.0.0.1:34046_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 2758300 INFO (coreZkRegister-5303-thread-1-processing-n:127.0.0.1:34046_ x:collection1 c:collection1) [n:127.0.0.1:34046_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:34046/collection1/ has no replicas
[junit4] 2> 2758300 INFO (coreZkRegister-5303-thread-1-processing-n:127.0.0.1:34046_ x:collection1 c:collection1) [n:127.0.0.1:34046_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 2758311 INFO (coreZkRegister-5303-thread-1-processing-n:127.0.0.1:34046_ x:collection1 c:collection1) [n:127.0.0.1:34046_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:34046/collection1/ shard2
[junit4] 2> 2758413 INFO (zkCallback-2563-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2758463 INFO (coreZkRegister-5303-thread-1-processing-n:127.0.0.1:34046_ x:collection1 c:collection1) [n:127.0.0.1:34046_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 2758579 INFO (zkCallback-2563-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2758910 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/shard-2-001/cores/collection1
[junit4] 2> 2758910 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/shard-2-001
[junit4] 2> 2758927 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 2758928 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5439a048{/,null,AVAILABLE}
[junit4] 2> 2758929 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@35d6ab91{HTTP/1.1,[http/1.1]}{127.0.0.1:37622}
[junit4] 2> 2758929 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.e.j.s.Server Started @2770607ms
[junit4] 2> 2758929 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=37622, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/shard-2-001/cores}
[junit4] 2> 2758930 ERROR (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2758930 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.6.3
[junit4] 2> 2758930 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2758930 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2758930 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-02T03:22:25.611Z
[junit4] 2> 2758949 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 2758949 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/shard-2-001/solr.xml
[junit4] 2> 2758955 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 2758955 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33069/solr
[junit4] 2> 2759004 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [n:127.0.0.1:37622_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 2759006 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [n:127.0.0.1:37622_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 2759008 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [n:127.0.0.1:37622_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37622_
[junit4] 2> 2759053 INFO (zkCallback-2563-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 2759094 INFO (zkCallback-2569-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 2759098 INFO (zkCallback-2553-thread-2-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 2759106 INFO (zkCallback-2557-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 2759263 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [n:127.0.0.1:37622_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/shard-2-001/cores
[junit4] 2> 2759263 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [n:127.0.0.1:37622_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 2759305 INFO (coreLoadExecutor-5319-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 2759330 INFO (OverseerStateUpdate-99612160077135876-127.0.0.1:36075_-n_0000000000) [n:127.0.0.1:36075_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 2759434 INFO (zkCallback-2569-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 2759435 INFO (zkCallback-2563-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 2760412 WARN (coreLoadExecutor-5319-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ c:collection1 x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 2760413 INFO (coreLoadExecutor-5319-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.3
[junit4] 2> 2760480 INFO (coreLoadExecutor-5319-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 2760807 INFO (coreLoadExecutor-5319-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 2760918 INFO (coreLoadExecutor-5319-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
[junit4] 2> 2760919 INFO (coreLoadExecutor-5319-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ c:collection1 x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2760919 INFO (coreLoadExecutor-5319-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/shard-2-001/cores/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/shard-2-001/cores/collection1/data/]
[junit4] 2> 2760919 INFO (coreLoadExecutor-5319-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@608cc2ff
[junit4] 2> 2760921 INFO (coreLoadExecutor-5319-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@527c86fb
[junit4] 2> 2761009 WARN (coreLoadExecutor-5319-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 2761185 INFO (coreLoadExecutor-5319-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 2761185 INFO (coreLoadExecutor-5319-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 2761186 INFO (coreLoadExecutor-5319-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2761186 INFO (coreLoadExecutor-5319-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2761199 INFO (coreLoadExecutor-5319-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=1.0341796875, floorSegmentMB=0.845703125, forceMergeDeletesPctAllowed=14.694259749291467, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6735098025561761
[junit4] 2> 2761200 INFO (coreLoadExecutor-5319-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@40f3827a[collection1] main]
[junit4] 2> 2761201 INFO (coreLoadExecutor-5319-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 2761227 INFO (coreLoadExecutor-5319-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2761228 INFO (coreLoadExecutor-5319-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 2761244 INFO (searcherExecutor-5320-thread-1-processing-n:127.0.0.1:37622_ x:collection1 c:collection1) [n:127.0.0.1:37622_ c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@40f3827a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2761244 INFO (coreLoadExecutor-5319-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1593794570931404800
[junit4] 2> 2761276 INFO (coreZkRegister-5314-thread-1-processing-n:127.0.0.1:37622_ x:collection1 c:collection1) [n:127.0.0.1:37622_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 2761276 INFO (coreZkRegister-5314-thread-1-processing-n:127.0.0.1:37622_ x:collection1 c:collection1) [n:127.0.0.1:37622_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 2761276 INFO (coreZkRegister-5314-thread-1-processing-n:127.0.0.1:37622_ x:collection1 c:collection1) [n:127.0.0.1:37622_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:37622/collection1/
[junit4] 2> 2761277 INFO (coreZkRegister-5314-thread-1-processing-n:127.0.0.1:37622_ x:collection1 c:collection1) [n:127.0.0.1:37622_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 2761277 INFO (coreZkRegister-5314-thread-1-processing-n:127.0.0.1:37622_ x:collection1 c:collection1) [n:127.0.0.1:37622_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:37622/collection1/ has no replicas
[junit4] 2> 2761277 INFO (coreZkRegister-5314-thread-1-processing-n:127.0.0.1:37622_ x:collection1 c:collection1) [n:127.0.0.1:37622_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 2761290 INFO (coreZkRegister-5314-thread-1-processing-n:127.0.0.1:37622_ x:collection1 c:collection1) [n:127.0.0.1:37622_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:37622/collection1/ shard1
[junit4] 2> 2761398 INFO (zkCallback-2569-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 2761398 INFO (zkCallback-2563-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 2761409 INFO (coreZkRegister-5314-thread-1-processing-n:127.0.0.1:37622_ x:collection1 c:collection1) [n:127.0.0.1:37622_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 2761520 INFO (zkCallback-2569-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 2761520 INFO (zkCallback-2563-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 2761784 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/shard-3-001/cores/collection1
[junit4] 2> 2761785 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/shard-3-001
[junit4] 2> 2761785 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 2761787 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@32dd3317{/,null,AVAILABLE}
[junit4] 2> 2761799 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@12513c83{HTTP/1.1,[http/1.1]}{127.0.0.1:36591}
[junit4] 2> 2761799 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.e.j.s.Server Started @2773477ms
[junit4] 2> 2761799 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/, hostPort=36591, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/shard-3-001/cores}
[junit4] 2> 2761800 ERROR (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2761800 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.6.3
[junit4] 2> 2761801 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2761801 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2761801 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-02T03:22:28.482Z
[junit4] 2> 2761805 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 2761805 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/shard-3-001/solr.xml
[junit4] 2> 2761823 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 2761824 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33069/solr
[junit4] 2> 2761852 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [n:127.0.0.1:36591_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 2761855 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [n:127.0.0.1:36591_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 2761869 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [n:127.0.0.1:36591_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36591_
[junit4] 2> 2761870 INFO (zkCallback-2557-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 2761870 INFO (zkCallback-2569-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 2761871 INFO (zkCallback-2553-thread-1-processing-n:127.0.0.1:36075_) [n:127.0.0.1:36075_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 2761874 INFO (zkCallback-2575-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 2761881 INFO (zkCallback-2563-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 2762115 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [n:127.0.0.1:36591_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/shard-3-001/cores
[junit4] 2> 2762115 INFO (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[34BB004ACFA3C0F3]) [n:127.0.0.1:36591_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 2762151 INFO (coreLoadExecutor-5330-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 2762165 INFO (OverseerStateUpdate-99612160077135876-127.0.0.1:36075_-n_0000000000) [n:127.0.0.1:36075_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
[junit4] 2> 2762276 INFO (zkCallback-2575-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 2762276 INFO (zkCallback-2563-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 2762276 INFO (zkCallback-2569-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 2763197 WARN (coreLoadExecutor-5330-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ c:collection1 x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 2763197 INFO (coreLoadExecutor-5330-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.3
[junit4] 2> 2763249 INFO (coreLoadExecutor-5330-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 2763506 INFO (coreLoadExecutor-5330-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 2763604 INFO (coreLoadExecutor-5330-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
[junit4] 2> 2763605 INFO (coreLoadExecutor-5330-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ c:collection1 x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2763605 INFO (coreLoadExecutor-5330-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/shard-3-001/cores/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ShardSplitTest_34BB004ACFA3C0F3-001/shard-3-001/cores/collection1/data/]
[junit4] 2> 2763605 INFO (coreLoadExecutor-5330-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@608cc2ff
[junit4] 2> 2763680 INFO (coreLoadExecutor-5330-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@2b8ef00c
[junit4] 2> 2763778 WARN (coreLoadExecutor-5330-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 2763966 INFO (coreLoadExecutor-5330-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 2763966 INFO (coreLoadExecutor-5330-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 2763967 INFO (coreLoadExecutor-5330-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2763967 INFO (coreLoadExecutor-5330-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2763990 INFO (coreLoadExecutor-5330-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=1.0341796875, floorSegmentMB=0.845703125, forceMergeDeletesPctAllowed=14.694259749291467, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6735098025561761
[junit4] 2> 2763991 INFO (coreLoadExecutor-5330-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@ca68a4e[collection1] main]
[junit4] 2> 2763993 INFO (coreLoadExecutor-5330-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 2763994 INFO (coreLoadExecutor-5330-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2763994 INFO (coreLoadExecutor-5330-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 2763995 INFO (searcherExecutor-5331-thread-1-processing-n:127.0.0.1:36591_ x:collection1 c:collection1) [n:127.0.0.1:36591_ c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@ca68a4e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2763996 INFO (coreLoadExecutor-5330-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1593794573817085952
[junit4] 2> 2764012 INFO (coreZkRegister-5325-thread-1-processing-n:127.0.0.1:36591_ x:collection1 c:collection1) [n:127.0.0.1:36591_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
[junit4] 2> 2764012 INFO (updateExecutor-2572-thread-1-processing-n:127.0.0.1:36591_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:36591_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
[junit4] 2> 2764012 INFO (recoveryExecutor-2573-thread-1-processing-n:127.0.0.1:36591_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:36591_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
[junit4] 2> 2764041 INFO (recoveryExecutor-2573-thread-1-processing-n:127.0.0.1:36591_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:36591_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
[junit4] 2> 2764041 INFO (recoveryExecutor-2573-thread-1-processing-n:127.0.0.1:36591_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:36591_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
[junit4] 2> 2764041 INFO (recoveryExecutor-2573-thread-1-processing-n:127.0.0.1:36591_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:36591_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 2764041 INFO (recoveryExecutor-2573-thread-1-processing-n:127.0.0.1:36591_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:36591_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:34046/collection1/] and I am [http://127.0.0.1:36591/collection1/]
[junit4] 2> 2764043 INFO (recoveryExecutor-2573-thread-1-processing-n:127.0.0.1:36591_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:36591_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:34046]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:36591_&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
[junit4] 2> 2764079 INFO (qtp1820166628-14961) [n:127.0.0.1:34046_ ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
[junit4] 2> 2764080 INFO (qtp1820166628-14961) [n:127.0.0.1:34046_ ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard2 of collection1) have state: recovering
[junit4] 2> 2764080 INFO (qtp1820166628-14961) [n:127.0.0.1:34046_ ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard2, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:36591_, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"http://127.0.0.1:36591","node_name":"127.0.0.1:36591_","state":"down"}
[junit4] 2> 2764180 INFO (zkCallback-2575-thread-1-processing-n:127.0.0.1:36591_) [n:127.0.0.1:36591_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 2764180 INFO (zkCallback-2569-thread-1-processing-n:127.0.0.1:37622_) [n:127.0.0.1:37622_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 2764180 INFO (zkCallback-2563-thread-1-processing-n:127.0.0.1:34046_) [n:127.0.0.1:34046_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 2765085 INFO (qtp1820166628-14961) [n:127.0.0.1:34046_ ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard2, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:36591_, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"http://127.0.0.1:36591","node_name":"127.0.0.1:36591_","state":"recovering"}
[junit4] 2> 2765085 INFO (qtp1820166628-14961) [n:127.0.0.1:34046_ ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4] 2> 2765085 INFO (qtp1820166628-14961) [n:127.0.0.1:34046_ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:36591_&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node3&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1006
[junit4] 2> 2765597 INFO (recoveryExecutor-2573-thread-1-processing-n:127.0.0.1:36591_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:36591_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://127.0.0.1:34046/collection1/] - recoveringAfterStartup=[true]
[junit4] 2> 2765597 INFO (recoveryExecutor-2573-thread-1-processing-n:127.0.0.1:36591_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:36591_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=http://127.0.0.1:36591 START replicas=[http://127.0.0.1:34046/collection1/] nUpdates=100
[junit4] 2> 2765645 INFO (qtp1820166628-14959) [n:127.0.0.1:34046_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:1.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 2765645 INFO (qtp1820166628-14959) [n:127.0.0.1:34046_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 2765775 INFO (recoveryExecutor-2573-thread-1-processing-n:127.0.0.1:36591_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:36591_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:36.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 2765775 INFO (recoveryExecutor-2573-thread-1-processing-n:127.0.0.1:36591_ x:collection1 s:shard2 c:collection1 r:core_node3) [n
[...truncated too long message...]
ng-n:127.0.0.1:33720_solr) [n:127.0.0.1:33720_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 6881554 INFO (jetty-closer-15029-thread-3) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4eeab602{/solr,null,UNAVAILABLE}
[junit4] 2> 6881704 WARN (zkCallback-15076-thread-3-processing-n:127.0.0.1:36461_solr) [n:127.0.0.1:36461_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 6881704 INFO (jetty-closer-15029-thread-6) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@78c1e86b{/solr,null,UNAVAILABLE}
[junit4] 2> 6882028 WARN (zkCallback-15059-thread-2-processing-n:127.0.0.1:43845_solr) [n:127.0.0.1:43845_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 6882029 INFO (jetty-closer-15029-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@3e018eaa{/solr,null,UNAVAILABLE}
[junit4] 2> 6882255 WARN (zkCallback-15083-thread-2-processing-n:127.0.0.1:41903_solr) [n:127.0.0.1:41903_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 6882255 INFO (jetty-closer-15029-thread-5) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5bf1a9ad{/solr,null,UNAVAILABLE}
[junit4] 2> 6882276 ERROR (SUITE-TestStressCloudBlindAtomicUpdates-seed#[34BB004ACFA3C0F3]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 6882277 INFO (SUITE-TestStressCloudBlindAtomicUpdates-seed#[34BB004ACFA3C0F3]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:45514 45514
[junit4] 2> 6882393 INFO (Thread-66670) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:45514 45514
[junit4] 2> 6882394 WARN (Thread-66670) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 11 /solr/aliases.json
[junit4] 2> 11 /solr/clusterprops.json
[junit4] 2> 10 /solr/configs/org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set
[junit4] 2> 10 /solr/security.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 11 /solr/clusterstate.json
[junit4] 2> 10 /solr/collections/test_col/state.json
[junit4] 2> 2 /solr/overseer_elect/election/99612404653031433-127.0.0.1:41509_solr-n_0000000000
[junit4] 2> 2 /solr/overseer_elect/election/99612404653031439-127.0.0.1:43289_solr-n_0000000003
[junit4] 2> 2 /solr/overseer_elect/election/99612404653031452-127.0.0.1:41903_solr-n_0000000006
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 11 /solr/live_nodes
[junit4] 2> 11 /solr/collections
[junit4] 2>
[junit4] 2> 6882394 INFO (SUITE-TestStressCloudBlindAtomicUpdates-seed#[34BB004ACFA3C0F3]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_34BB004ACFA3C0F3-001
[junit4] 2> Mar 02, 2018 4:31:08 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene62): {long_stored_idx=PostingsFormat(name=LuceneVarGapDocFreqInterval), long_dv_idx=Lucene50(blocksize=128), id=FST50, long_dv_stored_idx=Lucene50(blocksize=128)}, docValues:{long_stored_idx=DocValuesFormat(name=Lucene54), _version_=DocValuesFormat(name=Lucene54), long_dv_idx=DocValuesFormat(name=Asserting), long_dv_stored=DocValuesFormat(name=Direct), id=DocValuesFormat(name=Direct), long_dv=DocValuesFormat(name=Asserting), long_dv_stored_idx=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=1923, maxMBSortInHeap=5.189996557949406, sim=RandomSimilarity(queryNorm=false,coord=no): {}, locale=bg-BG, timezone=CTT
[junit4] 2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=130046464,total=531628032
[junit4] 2> NOTE: All tests run in this JVM: [TestGeoJSONResponseWriter, TestClusterStateMutator, ScriptEngineTest, ManagedSchemaRoundRobinCloudTest, TestBinaryResponseWriter, TestRandomCollapseQParserPlugin, TestFieldCacheSanityChecker, BufferStoreTest, TestSolrConfigHandler, SolrMetricReporterTest, BadCopyFieldTest, BJQParserTest, TestApiFramework, TestManagedResourceStorage, TestFieldCacheVsDocValues, ExitableDirectoryReaderTest, TestCloudSchemaless, TestReplicationHandlerBackup, TestExactStatsCache, LeaderElectionContextKeyTest, TestLeaderElectionWithEmptyReplica, TestZkAclsWithHadoopAuth, TestValueSourceCache, TestNumericTerms32, LeaderElectionIntegrationTest, TestStressUserVersions, TestBlendedInfixSuggestions, TestPushWriter, TestCloudJSONFacetJoinDomain, TestOmitPositions, TestPseudoReturnFields, TestFilteredDocIdSet, TestMultiWordSynonyms, WrapperMergePolicyFactoryTest, TestFunctionQuery, TestStressRecovery, TestLocalFSCloudBackupRestore, TestDelegationWithHadoopAuth, DeleteStatusTest, AliasIntegrationTest, CurrencyFieldOpenExchangeTest, TestCodecSupport, RecoveryAfterSoftCommitTest, DistributedQueryElevationComponentTest, SolrSlf4jReporterTest, TestLazyCores, DocValuesNotIndexedTest, TestComplexPhraseQParserPlugin, TestCustomStream, DistributedFacetPivotSmallAdvancedTest, TestPathTrie, PolyFieldTest, ClassificationUpdateProcessorTest, OverseerCollectionConfigSetProcessorTest, TestCloudRecovery, InfoHandlerTest, StandardRequestHandlerTest, ReturnFieldsTest, SolrCoreTest, MissingSegmentRecoveryTest, TestDynamicFieldResource, TestHighFrequencyDictionaryFactory, SecurityConfHandlerTest, TestDistributedSearch, UniqFieldsUpdateProcessorFactoryTest, TestJsonRequest, LeaderInitiatedRecoveryOnCommitTest, TestConfigSetsAPI, BasicDistributedZkTest, HdfsSyncSliceTest, AddBlockUpdateTest, ConfigSetsAPITest, OutputWriterTest, ZkNodePropsTest, BaseCdcrDistributedZkTest, TestFoldingMultitermQuery, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestJettySolrRunner, ChaosMonkeyNothingIsSafeTest, TestDocBasedVersionConstraints, SyncSliceTest, ExternalFileFieldSortTest, FileBasedSpellCheckerTest, MBeansHandlerTest, ShufflingReplicaListTransformerTest, TestReplicaProperties, HighlighterMaxOffsetTest, TestClassicSimilarityFactory, TestInitQParser, TestDocSet, TestPerFieldSimilarity, TestSchemaResource, TestInPlaceUpdatesDistrib, SignatureUpdateProcessorFactoryTest, TestPayloadCheckQParserPlugin, DeleteNodeTest, TestSolrCoreParser, TestSSLRandomization, TestBulkSchemaConcurrent, ExplicitHLLTest, TestJmxIntegration, PeerSyncTest, DistribJoinFromCollectionTest, DataDrivenBlockJoinTest, DistributedQueryComponentCustomSortTest, DisMaxRequestHandlerTest, TestMinMaxOnMultiValuedField, VersionInfoTest, TestPivotHelperCode, TestSolrCLIRunExample, TestClassNameShortening, TestReloadAndDeleteDocs, QueryEqualityTest, HdfsRecoverLeaseTest, TestPHPSerializedResponseWriter, BasicDistributedZk2Test, UnloadDistributedZkTest, OpenCloseCoreStressTest, BasicZkTest, TestReplicationHandler, ZkCLITest, TestDistributedGrouping, DistributedSpellCheckComponentTest, ZkControllerTest, TestRealTimeGet, TestJoin, TestStressVersions, DistributedTermsComponentTest, TestSolr4Spatial, QueryElevationComponentTest, TestFiltering, CurrencyFieldXmlFileTest, AnalysisAfterCoreReloadTest, TestExtendedDismaxParser, SolrRequestParserTest, SuggesterTest, TestStressLucene, TestCSVLoader, SolrCoreCheckLockOnStartupTest, TestUpdate, TestAtomicUpdateErrorCases, DirectUpdateHandlerOptimizeTest, StatelessScriptUpdateProcessorFactoryTest, TestSolrDeletionPolicy1, XsltUpdateRequestHandlerTest, CacheHeaderTest, TestQueryUtils, DirectSolrSpellCheckerTest, DocumentAnalysisRequestHandlerTest, PrimitiveFieldTypeTest, PathHierarchyTokenizerFactoryTest, FastVectorHighlighterTest, LoggingHandlerTest, SolrPluginUtilsTest, UpdateRequestProcessorFactoryTest, TestCSVResponseWriter, TestAnalyzedSuggestions, TestPartialUpdateDeduplication, PingRequestHandlerTest, BadComponentTest, TestSolrDeletionPolicy2, SampleTest, TestSearchPerf, MinimalSchemaTest, TestConfig, TestLuceneMatchVersion, TestCharFilters, TestIBSimilarityFactory, TimeZoneUtilsTest, URLClassifyProcessorTest, DateMathParserTest, TestSystemIdResolver, DateFieldTest, SpellingQueryConverterTest, RAMDirectoryFactoryTest, ClusterStateTest, TestLRUCache, TestUtils, SliceStateTest, SystemInfoHandlerTest, FileUtilsTest, DistributedMLTComponentTest, DistributedIntervalFacetingTest, CdcrRequestHandlerTest, CollectionsAPISolrJTest, DeleteInactiveReplicaTest, DeleteReplicaTest, DistribDocExpirationUpdateProcessorTest, DistributedQueueTest, DistributedVersionInfoTest, LeaderFailureAfterFreshStartTest, MigrateRouteKeyTest, RemoteQueryErrorTest, ReplicationFactorTest, RestartWhileUpdatingTest, SolrCLIZkUtilsTest, SolrCloudExampleTest, SolrXmlInZkTest, TestAuthenticationFramework, TestCloudDeleteByQuery, TestCloudPseudoReturnFields, TestClusterProperties, TestConfigSetsAPIExclusivity, TestConfigSetsAPIZkFailure, TestDistribDocBasedVersion, TestDownShardTolerantSearch, TestHdfsCloudBackupRestore, TestMiniSolrCloudClusterSSL, TestSizeLimitedDistributedMap, TestSolrCloudWithKerberosAlt, TestSolrCloudWithSecureImpersonation, TestStressCloudBlindAtomicUpdates]
[junit4] Completed [538/713 (3!)] on J0 in 398.66s, 5 tests, 1 error <<< FAILURES!
[...truncated 46891 lines...]
[JENKINS] Lucene-Solr-NightlyTests-6.6 - Build # 38 - Still
Unstable
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.6/38/
3 tests failed.
FAILED: org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.testCollectionsAPI
Error Message:
Something is broken in the assert for no shards using the same indexDir - probably something was changed in the attributes published in the MBean of SolrCore : {}
Stack Trace:
java.lang.AssertionError: Something is broken in the assert for no shards using the same indexDir - probably something was changed in the attributes published in the MBean of SolrCore : {}
at __randomizedtesting.SeedInfo.seed([5FBDCDA8573442E0:17C8B91C51076D75]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.checkNoTwoShardsUseTheSameIndexDir(CollectionsAPIDistributedZkTest.java:646)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:524)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.hdfs.StressHdfsTest.test
Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:47796/_pto
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:47796/_pto
at __randomizedtesting.SeedInfo.seed([5FBDCDA8573442E0:D7E9F272F9C82F18]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:623)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:447)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:388)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1383)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1134)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at org.apache.solr.cloud.hdfs.StressHdfsTest.createAndDeleteCollection(StressHdfsTest.java:220)
at org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:103)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515)
... 51 more
FAILED: org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.test
Error Message:
Captured an uncaught exception in thread: Thread[id=15220, name=testExecutor-2507-thread-10, state=RUNNABLE, group=TGRP-HdfsUnloadDistributedZkTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=15220, name=testExecutor-2507-thread-10, state=RUNNABLE, group=TGRP-HdfsUnloadDistributedZkTest]
at __randomizedtesting.SeedInfo.seed([5FBDCDA8573442E0:D7E9F272F9C82F18]:0)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:51812: Error CREATEing SolrCore 'multiunload9': Unable to create core [multiunload9] Caused by: Direct buffer memory
at __randomizedtesting.SeedInfo.seed([5FBDCDA8573442E0]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:612)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at org.apache.solr.cloud.BasicDistributedZkTest.lambda$createCollectionInOneInstance$0(BasicDistributedZkTest.java:589)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 12464 lines...]
[junit4] Suite: org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/init-core-data-001
[junit4] 2> 840095 WARN (SUITE-HdfsUnloadDistributedZkTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=4 numCloses=4
[junit4] 2> 840096 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.a.s.SolrTestCaseJ4 Using TrieFields
[junit4] 2> 840097 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
[junit4] 2> 840097 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
[junit4] 1> Formatting using clusterid: testClusterID
[junit4] 2> 840148 WARN (SUITE-HdfsUnloadDistributedZkTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
[junit4] 2> 840174 WARN (SUITE-HdfsUnloadDistributedZkTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 840181 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.m.log jetty-6.1.26
[junit4] 2> 840206 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_60296_hdfs____.k3gbk1/webapp
[junit4] 2> 840665 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:60296
[junit4] 2> 840775 WARN (SUITE-HdfsUnloadDistributedZkTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 840776 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.m.log jetty-6.1.26
[junit4] 2> 840795 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_59666_datanode____tudoop/webapp
[junit4] 2> 841272 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:59666
[junit4] 2> 841349 WARN (SUITE-HdfsUnloadDistributedZkTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 841351 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.m.log jetty-6.1.26
[junit4] 2> 841365 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_37376_datanode____.a5pjmz/webapp
[junit4] 2> 841834 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37376
[junit4] 2> 841979 ERROR (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/tempDir-001/hdfsBaseDir/data/data2/]] heartbeating to localhost/127.0.0.1:50475) [ ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
[junit4] 2> 841993 ERROR (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/tempDir-001/hdfsBaseDir/data/data4/]] heartbeating to localhost/127.0.0.1:50475) [ ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
[junit4] 2> 842007 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x49c75e8d85a701: from storage DS-1c9d5cb1-5254-4efe-bd60-28d2595886bb node DatanodeRegistration(127.0.0.1:43167, datanodeUuid=81bcc8e7-3d01-448a-9d83-8aa88358d350, infoPort=43749, infoSecurePort=0, ipcPort=47142, storageInfo=lv=-56;cid=testClusterID;nsid=746833506;c=0), blocks: 0, hasStaleStorage: true, processing time: 1 msecs
[junit4] 2> 842007 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x49c75e8d85f96d: from storage DS-b59e5c3f-1dff-4764-9bf2-322b33d05f46 node DatanodeRegistration(127.0.0.1:36367, datanodeUuid=380b4549-d68f-4b69-85ec-418c677229eb, infoPort=38437, infoSecurePort=0, ipcPort=55971, storageInfo=lv=-56;cid=testClusterID;nsid=746833506;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
[junit4] 2> 842007 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x49c75e8d85a701: from storage DS-f41f3bb6-b5cd-4078-8203-8b333fac5fc3 node DatanodeRegistration(127.0.0.1:43167, datanodeUuid=81bcc8e7-3d01-448a-9d83-8aa88358d350, infoPort=43749, infoSecurePort=0, ipcPort=47142, storageInfo=lv=-56;cid=testClusterID;nsid=746833506;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
[junit4] 2> 842007 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x49c75e8d85f96d: from storage DS-41fb63d0-80d6-461e-8463-0a2d5b3f9fb0 node DatanodeRegistration(127.0.0.1:36367, datanodeUuid=380b4549-d68f-4b69-85ec-418c677229eb, infoPort=38437, infoSecurePort=0, ipcPort=55971, storageInfo=lv=-56;cid=testClusterID;nsid=746833506;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
[junit4] 2> 842480 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 842493 INFO (Thread-9244) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 842493 INFO (Thread-9244) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 842503 ERROR (Thread-9244) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 842593 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.ZkTestServer start zk server on port:35511
[junit4] 2> 842606 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 842608 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 842610 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 842611 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 842613 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 842614 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 842616 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 842619 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 842620 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 842621 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 842623 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 842739 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/control-001/cores/collection1
[junit4] 2> 842741 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 842742 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4e2593b0{/,null,AVAILABLE}
[junit4] 2> 842743 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@51f17164{HTTP/1.1,[http/1.1]}{127.0.0.1:46394}
[junit4] 2> 842747 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.e.j.s.Server Started @845876ms
[junit4] 2> 842747 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:50475/hdfs__localhost_50475__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.6_checkout_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001_tempDir-002_control_data, hostContext=/, hostPort=46394, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/control-001/cores}
[junit4] 2> 842748 ERROR (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 842748 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.6.4
[junit4] 2> 842748 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 842748 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 842748 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-08T13:16:20.360Z
[junit4] 2> 842752 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 842752 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/control-001/solr.xml
[junit4] 2> 842763 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 842765 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35511/solr
[junit4] 2> 842794 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [n:127.0.0.1:46394_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 842794 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [n:127.0.0.1:46394_ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:46394_
[junit4] 2> 842795 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [n:127.0.0.1:46394_ ] o.a.s.c.Overseer Overseer (id=99648469736620036-127.0.0.1:46394_-n_0000000000) starting
[junit4] 2> 842806 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [n:127.0.0.1:46394_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46394_
[junit4] 2> 842813 INFO (zkCallback-1285-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 842893 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [n:127.0.0.1:46394_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/control-001/cores
[junit4] 2> 842893 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [n:127.0.0.1:46394_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 842895 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 842905 INFO (OverseerStateUpdate-99648469736620036-127.0.0.1:46394_-n_0000000000) [n:127.0.0.1:46394_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 843921 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.4
[junit4] 2> 843955 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 844106 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 844123 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection, trusted=true
[junit4] 2> 844123 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:50475/solr_hdfs_home
[junit4] 2> 844123 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
[junit4] 2> 844123 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 844123 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/control-001/cores/collection1], dataDir=[hdfs://localhost:50475/solr_hdfs_home/control_collection/core_node1/data/]
[junit4] 2> 844124 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@33d2f39d
[junit4] 2> 844124 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:50475/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
[junit4] 2> 844135 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 844135 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 844135 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 844144 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 844145 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:50475/solr_hdfs_home/control_collection/core_node1/data
[junit4] 2> 844165 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:50475/solr_hdfs_home/control_collection/core_node1/data/index
[junit4] 2> 844172 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 844172 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 844172 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 844178 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 844178 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=29.3125, floorSegmentMB=0.25390625, forceMergeDeletesPctAllowed=0.9764929896714247, segmentsPerTier=50.0, maxCFSSegmentSizeMB=1.5185546875, noCFSRatio=0.30202736206383585
[junit4] 2> 844194 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:36367 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f41f3bb6-b5cd-4078-8203-8b333fac5fc3:NORMAL:127.0.0.1:43167|RBW], ReplicaUC[[DISK]DS-41fb63d0-80d6-461e-8463-0a2d5b3f9fb0:NORMAL:127.0.0.1:36367|FINALIZED]]} size 0
[junit4] 2> 844196 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43167 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f41f3bb6-b5cd-4078-8203-8b333fac5fc3:NORMAL:127.0.0.1:43167|RBW], ReplicaUC[[DISK]DS-41fb63d0-80d6-461e-8463-0a2d5b3f9fb0:NORMAL:127.0.0.1:36367|FINALIZED]]} size 0
[junit4] 2> 844210 WARN (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 844254 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
[junit4] 2> 844254 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 844254 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
[junit4] 2> 844274 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 844274 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 844280 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=7, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=32.333984375, floorSegmentMB=1.826171875, forceMergeDeletesPctAllowed=21.043811099767726, segmentsPerTier=14.0, maxCFSSegmentSizeMB=0.912109375, noCFSRatio=1.0
[junit4] 2> 844290 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@584668d9[collection1] main]
[junit4] 2> 844292 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 844292 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 844292 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 844295 INFO (searcherExecutor-2430-thread-1-processing-n:127.0.0.1:46394_ x:collection1 c:control_collection) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@584668d9[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 844300 INFO (coreLoadExecutor-2429-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ c:control_collection x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1594375517673357312
[junit4] 2> 844307 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 844308 INFO (coreZkRegister-2422-thread-1-processing-n:127.0.0.1:46394_ x:collection1 c:control_collection) [n:127.0.0.1:46394_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 844308 INFO (coreZkRegister-2422-thread-1-processing-n:127.0.0.1:46394_ x:collection1 c:control_collection) [n:127.0.0.1:46394_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 844308 INFO (coreZkRegister-2422-thread-1-processing-n:127.0.0.1:46394_ x:collection1 c:control_collection) [n:127.0.0.1:46394_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:46394/collection1/
[junit4] 2> 844308 INFO (coreZkRegister-2422-thread-1-processing-n:127.0.0.1:46394_ x:collection1 c:control_collection) [n:127.0.0.1:46394_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 844308 INFO (coreZkRegister-2422-thread-1-processing-n:127.0.0.1:46394_ x:collection1 c:control_collection) [n:127.0.0.1:46394_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:46394/collection1/ has no replicas
[junit4] 2> 844308 INFO (coreZkRegister-2422-thread-1-processing-n:127.0.0.1:46394_ x:collection1 c:control_collection) [n:127.0.0.1:46394_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 844309 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35511/solr ready
[junit4] 2> 844309 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 844309 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
[junit4] 2> 844314 INFO (coreZkRegister-2422-thread-1-processing-n:127.0.0.1:46394_ x:collection1 c:control_collection) [n:127.0.0.1:46394_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:46394/collection1/ shard1
[junit4] 2> 844428 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/shard-1-001/cores/collection1
[junit4] 2> 844429 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/shard-1-001
[junit4] 2> 844429 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 844431 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7500575f{/,null,AVAILABLE}
[junit4] 2> 844431 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@2bc1ce0e{HTTP/1.1,[http/1.1]}{127.0.0.1:51812}
[junit4] 2> 844432 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.e.j.s.Server Started @847560ms
[junit4] 2> 844432 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:50475/hdfs__localhost_50475__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.6_checkout_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=51812, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/shard-1-001/cores}
[junit4] 2> 844432 ERROR (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 844433 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.6.4
[junit4] 2> 844433 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 844433 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 844433 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-08T13:16:22.045Z
[junit4] 2> 844436 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 844436 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/shard-1-001/solr.xml
[junit4] 2> 844443 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 844443 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35511/solr
[junit4] 2> 844455 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [n:127.0.0.1:51812_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 844457 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [n:127.0.0.1:51812_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 844460 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [n:127.0.0.1:51812_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51812_
[junit4] 2> 844461 INFO (zkCallback-1289-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 844463 INFO (zkCallback-1285-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 844463 INFO (zkCallback-1295-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 844465 INFO (coreZkRegister-2422-thread-1-processing-n:127.0.0.1:46394_ x:collection1 c:control_collection) [n:127.0.0.1:46394_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 844548 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [n:127.0.0.1:51812_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/shard-1-001/cores
[junit4] 2> 844548 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [n:127.0.0.1:51812_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 844549 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 844551 INFO (OverseerStateUpdate-99648469736620036-127.0.0.1:46394_-n_0000000000) [n:127.0.0.1:46394_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
[junit4] 2> 844655 INFO (zkCallback-1295-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 845567 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.4
[junit4] 2> 845582 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 845733 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 845746 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
[junit4] 2> 845747 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:50475/solr_hdfs_home
[junit4] 2> 845747 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
[junit4] 2> 845747 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 845747 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/shard-1-001/cores/collection1], dataDir=[hdfs://localhost:50475/solr_hdfs_home/collection1/core_node1/data/]
[junit4] 2> 845747 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@33d2f39d
[junit4] 2> 845747 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:50475/solr_hdfs_home/collection1/core_node1/data/snapshot_metadata
[junit4] 2> 845756 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 845756 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 845756 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 845763 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 845764 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:50475/solr_hdfs_home/collection1/core_node1/data
[junit4] 2> 845782 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:50475/solr_hdfs_home/collection1/core_node1/data/index
[junit4] 2> 845789 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 845789 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 845789 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 845796 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 845797 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=29.3125, floorSegmentMB=0.25390625, forceMergeDeletesPctAllowed=0.9764929896714247, segmentsPerTier=50.0, maxCFSSegmentSizeMB=1.5185546875, noCFSRatio=0.30202736206383585
[junit4] 2> 845816 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:36367 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1c9d5cb1-5254-4efe-bd60-28d2595886bb:NORMAL:127.0.0.1:43167|RBW], ReplicaUC[[DISK]DS-b59e5c3f-1dff-4764-9bf2-322b33d05f46:NORMAL:127.0.0.1:36367|RBW]]} size 0
[junit4] 2> 845819 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43167 is added to blk_1073741826_1002 size 71
[junit4] 2> 845833 WARN (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 845887 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
[junit4] 2> 845887 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 845887 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
[junit4] 2> 845901 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 845901 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 845903 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=7, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=32.333984375, floorSegmentMB=1.826171875, forceMergeDeletesPctAllowed=21.043811099767726, segmentsPerTier=14.0, maxCFSSegmentSizeMB=0.912109375, noCFSRatio=1.0
[junit4] 2> 845908 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7f150de[collection1] main]
[junit4] 2> 845909 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 845910 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 845910 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 845923 INFO (searcherExecutor-2441-thread-1-processing-n:127.0.0.1:51812_ x:collection1 c:collection1) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7f150de[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 845926 INFO (coreLoadExecutor-2440-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1594375519378341888
[junit4] 2> 845942 INFO (coreZkRegister-2435-thread-1-processing-n:127.0.0.1:51812_ x:collection1 c:collection1) [n:127.0.0.1:51812_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 845942 INFO (coreZkRegister-2435-thread-1-processing-n:127.0.0.1:51812_ x:collection1 c:collection1) [n:127.0.0.1:51812_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 845942 INFO (coreZkRegister-2435-thread-1-processing-n:127.0.0.1:51812_ x:collection1 c:collection1) [n:127.0.0.1:51812_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:51812/collection1/
[junit4] 2> 845942 INFO (coreZkRegister-2435-thread-1-processing-n:127.0.0.1:51812_ x:collection1 c:collection1) [n:127.0.0.1:51812_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 845942 INFO (coreZkRegister-2435-thread-1-processing-n:127.0.0.1:51812_ x:collection1 c:collection1) [n:127.0.0.1:51812_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:51812/collection1/ has no replicas
[junit4] 2> 845943 INFO (coreZkRegister-2435-thread-1-processing-n:127.0.0.1:51812_ x:collection1 c:collection1) [n:127.0.0.1:51812_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 845957 INFO (coreZkRegister-2435-thread-1-processing-n:127.0.0.1:51812_ x:collection1 c:collection1) [n:127.0.0.1:51812_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:51812/collection1/ shard2
[junit4] 2> 846054 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/shard-2-001/cores/collection1
[junit4] 2> 846054 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/shard-2-001
[junit4] 2> 846055 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 846057 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@59302569{/,null,AVAILABLE}
[junit4] 2> 846057 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@34090090{HTTP/1.1,[http/1.1]}{127.0.0.1:55160}
[junit4] 2> 846058 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.e.j.s.Server Started @849187ms
[junit4] 2> 846058 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:50475/hdfs__localhost_50475__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.6_checkout_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001_tempDir-002_jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=55160, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/shard-2-001/cores}
[junit4] 2> 846058 ERROR (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 846059 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.6.4
[junit4] 2> 846059 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 846059 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 846059 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-08T13:16:23.671Z
[junit4] 2> 846060 INFO (zkCallback-1295-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 846069 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 846069 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/shard-2-001/solr.xml
[junit4] 2> 846076 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 846077 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35511/solr
[junit4] 2> 846088 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [n:127.0.0.1:55160_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 846090 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [n:127.0.0.1:55160_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 846092 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [n:127.0.0.1:55160_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55160_
[junit4] 2> 846094 INFO (zkCallback-1289-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 846096 INFO (zkCallback-1285-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 846097 INFO (zkCallback-1295-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 846101 INFO (zkCallback-1301-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 846109 INFO (coreZkRegister-2435-thread-1-processing-n:127.0.0.1:51812_ x:collection1 c:collection1) [n:127.0.0.1:51812_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 846149 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [n:127.0.0.1:55160_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/shard-2-001/cores
[junit4] 2> 846149 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [n:127.0.0.1:55160_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 846151 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 846152 INFO (OverseerStateUpdate-99648469736620036-127.0.0.1:46394_-n_0000000000) [n:127.0.0.1:46394_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 846256 INFO (zkCallback-1295-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 846256 INFO (zkCallback-1301-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 847170 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.4
[junit4] 2> 847191 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 847321 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 847335 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
[junit4] 2> 847335 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:50475/solr_hdfs_home
[junit4] 2> 847335 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
[junit4] 2> 847335 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 847335 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/shard-2-001/cores/collection1], dataDir=[hdfs://localhost:50475/solr_hdfs_home/collection1/core_node2/data/]
[junit4] 2> 847336 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@33d2f39d
[junit4] 2> 847336 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:50475/solr_hdfs_home/collection1/core_node2/data/snapshot_metadata
[junit4] 2> 847345 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 847345 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 847345 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 847356 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 847357 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:50475/solr_hdfs_home/collection1/core_node2/data
[junit4] 2> 847380 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:50475/solr_hdfs_home/collection1/core_node2/data/index
[junit4] 2> 847398 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 847398 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 847398 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 847419 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 847419 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=29.3125, floorSegmentMB=0.25390625, forceMergeDeletesPctAllowed=0.9764929896714247, segmentsPerTier=50.0, maxCFSSegmentSizeMB=1.5185546875, noCFSRatio=0.30202736206383585
[junit4] 2> 847446 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43167 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b59e5c3f-1dff-4764-9bf2-322b33d05f46:NORMAL:127.0.0.1:36367|RBW], ReplicaUC[[DISK]DS-f41f3bb6-b5cd-4078-8203-8b333fac5fc3:NORMAL:127.0.0.1:43167|RBW]]} size 0
[junit4] 2> 847447 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:36367 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f41f3bb6-b5cd-4078-8203-8b333fac5fc3:NORMAL:127.0.0.1:43167|RBW], ReplicaUC[[DISK]DS-41fb63d0-80d6-461e-8463-0a2d5b3f9fb0:NORMAL:127.0.0.1:36367|FINALIZED]]} size 0
[junit4] 2> 847458 WARN (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 847506 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
[junit4] 2> 847506 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 847506 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
[junit4] 2> 847517 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 847517 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 847523 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=7, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=32.333984375, floorSegmentMB=1.826171875, forceMergeDeletesPctAllowed=21.043811099767726, segmentsPerTier=14.0, maxCFSSegmentSizeMB=0.912109375, noCFSRatio=1.0
[junit4] 2> 847532 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4dad7f83[collection1] main]
[junit4] 2> 847534 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 847535 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 847535 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 847538 INFO (searcherExecutor-2452-thread-1-processing-n:127.0.0.1:55160_ x:collection1 c:collection1) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@4dad7f83[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 847540 INFO (coreLoadExecutor-2451-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1594375521070743552
[junit4] 2> 847555 INFO (coreZkRegister-2446-thread-1-processing-n:127.0.0.1:55160_ x:collection1 c:collection1) [n:127.0.0.1:55160_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 847555 INFO (coreZkRegister-2446-thread-1-processing-n:127.0.0.1:55160_ x:collection1 c:collection1) [n:127.0.0.1:55160_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 847555 INFO (coreZkRegister-2446-thread-1-processing-n:127.0.0.1:55160_ x:collection1 c:collection1) [n:127.0.0.1:55160_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:55160/collection1/
[junit4] 2> 847556 INFO (coreZkRegister-2446-thread-1-processing-n:127.0.0.1:55160_ x:collection1 c:collection1) [n:127.0.0.1:55160_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 847556 INFO (coreZkRegister-2446-thread-1-processing-n:127.0.0.1:55160_ x:collection1 c:collection1) [n:127.0.0.1:55160_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:55160/collection1/ has no replicas
[junit4] 2> 847556 INFO (coreZkRegister-2446-thread-1-processing-n:127.0.0.1:55160_ x:collection1 c:collection1) [n:127.0.0.1:55160_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 847563 INFO (coreZkRegister-2446-thread-1-processing-n:127.0.0.1:55160_ x:collection1 c:collection1) [n:127.0.0.1:55160_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:55160/collection1/ shard1
[junit4] 2> 847660 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/shard-3-001/cores/collection1
[junit4] 2> 847661 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/shard-3-001
[junit4] 2> 847661 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 847663 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5827ebd3{/,null,AVAILABLE}
[junit4] 2> 847663 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@5b6011f{HTTP/1.1,[http/1.1]}{127.0.0.1:49401}
[junit4] 2> 847663 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.e.j.s.Server Started @850792ms
[junit4] 2> 847663 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:50475/hdfs__localhost_50475__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.6_checkout_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001_tempDir-002_jetty3, solrconfig=solrconfig.xml, hostContext=/, hostPort=49401, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/shard-3-001/cores}
[junit4] 2> 847663 ERROR (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 847671 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.6.4
[junit4] 2> 847671 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 847671 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 847671 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-08T13:16:25.283Z
[junit4] 2> 847672 INFO (zkCallback-1301-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 847672 INFO (zkCallback-1295-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 847682 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 847682 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/shard-3-001/solr.xml
[junit4] 2> 847690 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 847691 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35511/solr
[junit4] 2> 847703 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [n:127.0.0.1:49401_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 847706 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [n:127.0.0.1:49401_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 847708 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [n:127.0.0.1:49401_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49401_
[junit4] 2> 847711 INFO (zkCallback-1301-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 847711 INFO (zkCallback-1289-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 847712 INFO (zkCallback-1285-thread-1-processing-n:127.0.0.1:46394_) [n:127.0.0.1:46394_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 847712 INFO (zkCallback-1295-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 847720 INFO (zkCallback-1307-thread-1-processing-n:127.0.0.1:49401_) [n:127.0.0.1:49401_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 847721 INFO (coreZkRegister-2446-thread-1-processing-n:127.0.0.1:55160_ x:collection1 c:collection1) [n:127.0.0.1:55160_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 847810 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [n:127.0.0.1:49401_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_5FBDCDA8573442E0-001/shard-3-001/cores
[junit4] 2> 847810 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[5FBDCDA8573442E0]) [n:127.0.0.1:49401_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 847812 INFO (coreLoadExecutor-2462-thread-1-processing-n:127.0.0.1:49401_) [n:127.0.0.1:49401_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 847820 INFO (OverseerStateUpdate-99648469736620036-127.0.0.1:46394_-n_0000000000) [n:127.0.0.1:46394_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
[junit4] 2> 847923 INFO (zkCallback-1295-thread-1-processing-n:127.0.0.1:51812_) [n:127.0.0.1:51812_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 847923 INFO (zkCallback-1301-thread-1-processing-n:127.0.0.1:55160_) [n:127.0.0.1:55160_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 847923 INFO (zkCallback-1307-thread-1-processing-n:127.0.0.1:49401_) [n:127.0.0.1:49401_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 848840 INFO (coreLoadExecutor-2462-thread-1-processing-n:127.0.0.1:49401_) [n:127.0.0.1:49401_ c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.4
[junit4] 2> 848868 INFO (coreLoadExecutor-2462-thread-1-processing-n:127.0.0.1:49401_) [n:127.0.0.1:49401_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 848997 INFO (coreLoadExecutor-2462-thread-1-processing-n:127.0.0.1:49401_) [n:127.0.0.1:49401_ c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 849011 INFO (coreLoadExecutor-2462-thread-1-processing-n:127.0.0.1:49401_) [n:127.0.0.1:49401_ c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
[junit4] 2> 849012 INFO (coreLoadExecutor-2462-thread-1-processing-n:127.0.0.1:49401_) [n:127.0.0.1:49401_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:50475/solr_hdfs_home
[junit4] 2> 849012 INFO (coreLoadExecutor-2462-thread-1-processing-n:127.0.0.1:49401_) [n:127.0.0.1:49401_ c:collectio
[...truncated too long message...]
toryScanner: shutdown has been called
[junit4] 2> 3999434 INFO (SUITE-StressHdfsTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
[junit4] 2> 3999535 WARN (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_5FBDCDA8573442E0-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_5FBDCDA8573442E0-001/tempDir-001/hdfsBaseDir/data/data4/]] heartbeating to localhost/127.0.0.1:44368) [ ] o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager interrupted
[junit4] 2> 3999536 WARN (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_5FBDCDA8573442E0-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_5FBDCDA8573442E0-001/tempDir-001/hdfsBaseDir/data/data4/]] heartbeating to localhost/127.0.0.1:44368) [ ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-1812823391-10.41.0.5-1520517966047 (Datanode Uuid c856dec8-3b55-4fa1-b966-dd8c3f89f862) service to localhost/127.0.0.1:44368
[junit4] 2> 3999538 WARN (SUITE-StressHdfsTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
[junit4] 2> 3999561 INFO (SUITE-StressHdfsTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
[junit4] 2> 3999663 WARN (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_5FBDCDA8573442E0-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_5FBDCDA8573442E0-001/tempDir-001/hdfsBaseDir/data/data2/]] heartbeating to localhost/127.0.0.1:44368) [ ] o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager interrupted
[junit4] 2> 3999663 WARN (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_5FBDCDA8573442E0-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_5FBDCDA8573442E0-001/tempDir-001/hdfsBaseDir/data/data2/]] heartbeating to localhost/127.0.0.1:44368) [ ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-1812823391-10.41.0.5-1520517966047 (Datanode Uuid 1ea4f383-56b4-4097-9cc8-773136115633) service to localhost/127.0.0.1:44368
[junit4] 2> 3999674 INFO (SUITE-StressHdfsTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
[junit4] 2> 3999777 INFO (SUITE-StressHdfsTest-seed#[5FBDCDA8573442E0]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_5FBDCDA8573442E0-001
[junit4] 2> Mar 08, 2018 2:08:57 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 34 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene62): {range_facet_l_dv=BlockTreeOrds(blocksize=128), multiDefault=FST50, intDefault=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), id=BlockTreeOrds(blocksize=128), range_facet_i_dv=FST50, text=Lucene50(blocksize=128), range_facet_l=FST50, timestamp=FST50, txt_t=Lucene50(blocksize=128)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Memory), _version_=DocValuesFormat(name=Lucene54), range_facet_i_dv=DocValuesFormat(name=Lucene54), intDvoDefault=DocValuesFormat(name=Memory), timestamp=DocValuesFormat(name=Lucene54)}, maxPointsInLeafNode=1009, maxMBSortInHeap=5.078668139022737, sim=RandomSimilarity(queryNorm=false,coord=yes): {}, locale=pl, timezone=US/Arizona
[junit4] 2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=2,free=206427680,total=535298048
[junit4] 2> NOTE: All tests run in this JVM: [TestSchemaVersionResource, BitVectorTest, RequiredFieldsTest, ScriptEngineTest, CurrencyFieldOpenExchangeTest, TestQuerySenderNoQuery, TestDistribDocBasedVersion, TestUninvertingReader, QueryResultKeyTest, ResourceLoaderTest, NumericFieldsTest, UUIDFieldTest, LeaderElectionContextKeyTest, TestNamedUpdateProcessors, TestHighlightDedupGrouping, TestSchemaNameResource, UtilsToolTest, TestSolrCloudWithKerberosAlt, V2ApiIntegrationTest, DirectoryFactoryTest, TestFunctionQuery, TestCorePropertiesReload, TestValueSourceCache, TestManagedSynonymFilterFactory, TestStressRecovery, TestJettySolrRunner, TestSolrDeletionPolicy2, BlockDirectoryTest, TestSubQueryTransformer, TestSortByMinMaxFunction, CollectionReloadTest, SolrGangliaReporterTest, CollectionsAPIAsyncDistributedZkTest, TestPathTrie, ZkNodePropsTest, TestSolrQueryParserResource, SpatialRPTFieldTypeTest, TestHdfsBackupRestoreCore, TestConfigsApi, DocValuesNotIndexedTest, CdcrReplicationHandlerTest, TestCrossCoreJoin, TestExpandComponent, SimpleMLTQParserTest, TestFreeTextSuggestions, OverseerRolesTest, CdcrUpdateLogTest, DistributedFacetPivotLongTailTest, DistributedExpandComponentTest, SolrRequestParserTest, TestIBSimilarityFactory, CustomCollectionTest, HLLSerializationTest, TestCloudNestedDocsSort, SolrIndexMetricsTest, TestCloudPseudoReturnFields, TestLegacyFieldCache, TestDocSet, TestLazyCores, TestSolr4Spatial, DateFieldTest, PeerSyncReplicationTest, TestAnalyzeInfixSuggestions, PeerSyncTest, BaseCdcrDistributedZkTest, SynonymTokenizerTest, TestStressInPlaceUpdates, BlockJoinFacetRandomTest, DefaultValueUpdateProcessorTest, ChangedSchemaMergeTest, RegexBoostProcessorTest, TestFastLRUCache, BasicAuthIntegrationTest, TestConfigSetImmutable, ExternalFileFieldSortTest, TestSimpleTextCodec, TestReplicationHandler, TestDistributedSearch, TestFieldCacheSortRandom, TestOmitPositions, TestJsonFacets, TestBulkSchemaAPI, ShardSplitTest, DistributedFacetPivotWhiteBoxTest, TestXmlQParser, TestCollectionAPI, BasicZkTest, TestSolr4Spatial2, HdfsTlogReplayBufferedWhileIndexingTest, TestRandomDVFaceting, TestConfigSetsAPIZkFailure, VMParamsZkACLAndCredentialsProvidersTest, TestSha256AuthenticationProvider, TestMacroExpander, UpdateLogTest, HdfsNNFailoverTest, TestSolrXml, MultiThreadedOCPTest, TestQueryWrapperFilter, ConfigSetsAPITest, DeleteNodeTest, TestFieldResource, FileBasedSpellCheckerTest, SubstringBytesRefFilterTest, TestAtomicUpdateErrorCases, TestSchemalessBufferedUpdates, ClusterStateTest, TestRemoteStreaming, TestSolrCoreProperties, MetricUtilsTest, TestDocumentBuilder, PreAnalyzedUpdateProcessorTest, TestBadConfig, LeaderElectionIntegrationTest, RecoveryZkTest, TestRandomFaceting, TestZkChroot, AutoCommitTest, TestFaceting, TestRecovery, TestHashPartitioner, DistributedSpellCheckComponentTest, TermVectorComponentDistributedTest, TestRealTimeGet, TestReload, TestCoreContainer, SolrCoreTest, SolrCmdDistributorTest, QueryElevationComponentTest, BadIndexSchemaTest, TestFiltering, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest, SolrIndexSplitterTest, AnalysisAfterCoreReloadTest, SignatureUpdateProcessorFactoryTest, SpellCheckCollatorTest, TestFoldingMultitermQuery, DocValuesTest, TestStressLucene, TestTrie, TestCSVLoader, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest, TestPseudoReturnFields, QueryEqualityTest, SortByFunctionTest, XsltUpdateRequestHandlerTest, CacheHeaderTest, StandardRequestHandlerTest, DocumentAnalysisRequestHandlerTest, XmlUpdateRequestHandlerTest, TermVectorComponentTest, TestSolrQueryParser, LoggingHandlerTest, TestJmxIntegration, MBeansHandlerTest, TestAnalyzedSuggestions, JsonLoaderTest, TestPartialUpdateDeduplication, PingRequestHandlerTest, TestBinaryResponseWriter, HighlighterConfigTest, UpdateParamsTest, AlternateDirectoryTest, TestSolrIndexConfig, CopyFieldTest, BadComponentTest, TestBinaryField, TestElisionMultitermQuery, TestPostingsSolrHighlighter, DirectSolrConnectionTest, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestCodecSupport, TestLMJelinekMercerSimilarityFactory, TestFastWriter, TestJmxMonitoredMap, TestSystemIdResolver, PrimUtilsTest, TestSuggestSpellingConverter, RAMDirectoryFactoryTest, TestSolrJ, TestLRUCache, SliceStateTest, SystemInfoHandlerTest, CircularListTest, TestRTGBase, CursorPagingTest, SolrTestCaseJ4Test, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestSimpleTrackingShardHandler, TestTolerantSearch, AliasIntegrationTest, AsyncCallRequestStatusResponseTest, CdcrRequestHandlerTest, DeleteLastCustomShardedReplicaTest, DeleteShardTest, DistribCursorPagingTest, LeaderFailoverAfterPartitionTest, LeaderInitiatedRecoveryOnShardRestartTest, MigrateRouteKeyTest, NodeMutatorTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverseerCollectionConfigSetProcessorTest, RemoteQueryErrorTest, ReplicationFactorTest, RollingRestartTest, TestRebalanceLeaders, TestSegmentSorting, TestSolrCloudWithDelegationTokens, TestTolerantUpdateProcessorRandomCloud, TriLevelCompositeIdRoutingTest, HdfsThreadLeakTest, HdfsWriteToMultipleCollectionsTest, StressHdfsTest]
[junit4] Completed [564/714 (3!)] on J0 in 181.20s, 1 test, 1 error <<< FAILURES!
[...truncated 46772 lines...]
[JENKINS] Lucene-Solr-NightlyTests-6.6 - Build # 37 - Still
Unstable
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.6/37/
5 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsBasicDistributedZkTest
Error Message:
ObjectTracker found 1 object(s) that were not released!!! [NRTCachingDirectory] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.NRTCachingDirectory at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42) at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347) at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:353) at org.apache.solr.core.SolrCore.cleanupOldIndexDirectories(SolrCore.java:3044) at org.apache.solr.core.SolrCore.close(SolrCore.java:1575) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:969) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:830) at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:950) at org.apache.solr.core.CoreContainer.lambda$load$5(CoreContainer.java:565) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [NRTCachingDirectory]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.NRTCachingDirectory
at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:353)
at org.apache.solr.core.SolrCore.cleanupOldIndexDirectories(SolrCore.java:3044)
at org.apache.solr.core.SolrCore.close(SolrCore.java:1575)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:969)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:830)
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:950)
at org.apache.solr.core.CoreContainer.lambda$load$5(CoreContainer.java:565)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([DF5C6277AD21161E]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNull(Assert.java:551)
at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:302)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.hdfs.HdfsSyncSliceTest.test
Error Message:
Error from server at https://127.0.0.1:40361/_dp/e/collection1: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /_dp/e/collection1/update. Reason: <pre> {metadata={error-class=org.apache.solr.core.SolrCoreInitializationException,root-error-class=java.lang.OutOfMemoryError},msg=SolrCore 'collection1' is not available due to init failure: The max direct memory is likely too low. Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~8.388608 MB.,trace=org.apache.solr.core.SolrCoreInitializationException: SolrCore 'collection1' is not available due to init failure: The max direct memory is likely too low. Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~8.388608 MB. at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1361) at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:268) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:483) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:136) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:534) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.solr.common.SolrException: The max direct memory is likely too low. Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~8.388608 MB. at org.apache.solr.core.SolrCore.<init>(SolrCore.java:977) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:830) at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:950) at org.apache.solr.core.CoreContainer.lambda$load$5(CoreContainer.java:565) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more Caused by: java.lang.RuntimeException: The max direct memory is likely too low. Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~8.388608 MB. at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:312) at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:288) at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:228) at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:346) at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:482) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:917) ... 9 more Caused by: java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:694) at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) at org.apache.solr.store.blockcache.BlockCache.<init>(BlockCache.java:75) at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:310) ... 14 more ,code=500}</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.14.v20161028</a><hr/> </body> </html>
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:40361/_dp/e/collection1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /_dp/e/collection1/update. Reason:
<pre> {metadata={error-class=org.apache.solr.core.SolrCoreInitializationException,root-error-class=java.lang.OutOfMemoryError},msg=SolrCore 'collection1' is not available due to init failure: The max direct memory is likely too low. Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~8.388608 MB.,trace=org.apache.solr.core.SolrCoreInitializationException: SolrCore 'collection1' is not available due to init failure: The max direct memory is likely too low. Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~8.388608 MB.
at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1361)
at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:268)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:483)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:136)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.common.SolrException: The max direct memory is likely too low. Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~8.388608 MB.
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:977)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:830)
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:950)
at org.apache.solr.core.CoreContainer.lambda$load$5(CoreContainer.java:565)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: java.lang.RuntimeException: The max direct memory is likely too low. Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~8.388608 MB.
at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:312)
at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:288)
at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:228)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:346)
at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:482)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:917)
... 9 more
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:694)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at org.apache.solr.store.blockcache.BlockCache.<init>(BlockCache.java:75)
at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:310)
... 14 more
,code=500}</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.14.v20161028</a><hr/>
</body>
</html>
at __randomizedtesting.SeedInfo.seed([DF5C6277AD21161E:57085DAD03DD7BE6]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:580)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:895)
at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:858)
at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:873)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.del(AbstractFullDistribZkTestBase.java:835)
at org.apache.solr.cloud.SyncSliceTest.test(SyncSliceTest.java:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.core.TestDynamicLoading.testDynamicLoading
Error Message:
Could not get expected value 'org.apache.solr.core.BlobStoreTestRequestHandler' for path 'overlay/requestHandler/\/test1/class' full output: { "responseHeader":{ "status":0, "QTime":0}, "overlay":{ "znodeVersion":0, "runtimeLib":{"colltest":{ "name":"colltest", "version":1}}}}, from server: null
Stack Trace:
java.lang.AssertionError: Could not get expected value 'org.apache.solr.core.BlobStoreTestRequestHandler' for path 'overlay/requestHandler/\/test1/class' full output: {
"responseHeader":{
"status":0,
"QTime":0},
"overlay":{
"znodeVersion":0,
"runtimeLib":{"colltest":{
"name":"colltest",
"version":1}}}}, from server: null
at __randomizedtesting.SeedInfo.seed([DF5C6277AD21161E:7114F205AFCB3BE]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.core.TestSolrConfigHandler.testForResponseElement(TestSolrConfigHandler.java:556)
at org.apache.solr.core.TestDynamicLoading.testDynamicLoading(TestDynamicLoading.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload
Error Message:
expected:<[{indexVersion=1520441148100,generation=2,filelist=[_3my.cfe, _3my.cfs, _3my.si, _3nx.cfe, _3nx.cfs, _3nx.si, _3ny.fdt, _3ny.fdx, _3ny.fnm, _3ny.nvd, _3ny.nvm, _3ny.si, _3ny_Memory_0.ram, _3nz.fdt, _3nz.fdx, _3nz.fnm, _3nz.nvd, _3nz.nvm, _3nz.si, _3nz_Memory_0.ram, _3o1.fdt, _3o1.fdx, _3o1.fnm, _3o1.nvd, _3o1.nvm, _3o1.si, _3o1_Memory_0.ram, segments_2]}]> but was:<[{indexVersion=1520441148100,generation=2,filelist=[_3my.cfe, _3my.cfs, _3my.si, _3nx.cfe, _3nx.cfs, _3nx.si, _3ny.fdt, _3ny.fdx, _3ny.fnm, _3ny.nvd, _3ny.nvm, _3ny.si, _3ny_Memory_0.ram, _3nz.fdt, _3nz.fdx, _3nz.fnm, _3nz.nvd, _3nz.nvm, _3nz.si, _3nz_Memory_0.ram, _3o1.fdt, _3o1.fdx, _3o1.fnm, _3o1.nvd, _3o1.nvm, _3o1.si, _3o1_Memory_0.ram, segments_2]}, {indexVersion=1520441148100,generation=3,filelist=[_3o0.cfe, _3o0.cfs, _3o0.si, _3o1.fdt, _3o1.fdx, _3o1.fnm, _3o1.nvd, _3o1.nvm, _3o1.si, _3o1_Memory_0.ram, _3o2.fdt, _3o2.fdx, _3o2.fnm, _3o2.nvd, _3o2.nvm, _3o2.si, _3o2_Memory_0.ram, segments_3]}]>
Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1520441148100,generation=2,filelist=[_3my.cfe, _3my.cfs, _3my.si, _3nx.cfe, _3nx.cfs, _3nx.si, _3ny.fdt, _3ny.fdx, _3ny.fnm, _3ny.nvd, _3ny.nvm, _3ny.si, _3ny_Memory_0.ram, _3nz.fdt, _3nz.fdx, _3nz.fnm, _3nz.nvd, _3nz.nvm, _3nz.si, _3nz_Memory_0.ram, _3o1.fdt, _3o1.fdx, _3o1.fnm, _3o1.nvd, _3o1.nvm, _3o1.si, _3o1_Memory_0.ram, segments_2]}]> but was:<[{indexVersion=1520441148100,generation=2,filelist=[_3my.cfe, _3my.cfs, _3my.si, _3nx.cfe, _3nx.cfs, _3nx.si, _3ny.fdt, _3ny.fdx, _3ny.fnm, _3ny.nvd, _3ny.nvm, _3ny.si, _3ny_Memory_0.ram, _3nz.fdt, _3nz.fdx, _3nz.fnm, _3nz.nvd, _3nz.nvm, _3nz.si, _3nz_Memory_0.ram, _3o1.fdt, _3o1.fdx, _3o1.fnm, _3o1.nvd, _3o1.nvm, _3o1.si, _3o1_Memory_0.ram, segments_2]}, {indexVersion=1520441148100,generation=3,filelist=[_3o0.cfe, _3o0.cfs, _3o0.si, _3o1.fdt, _3o1.fdx, _3o1.fnm, _3o1.nvd, _3o1.nvm, _3o1.si, _3o1_Memory_0.ram, _3o2.fdt, _3o2.fdx, _3o2.fnm, _3o2.nvd, _3o2.nvm, _3o2.si, _3o2_Memory_0.ram, segments_3]}]>
at __randomizedtesting.SeedInfo.seed([DF5C6277AD21161E:FA8B7947DD69181D]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:147)
at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1281)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.test
Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:40687/r/ng
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:40687/r/ng
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:623)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at org.apache.solr.cloud.UnloadDistributedZkTest.testUnloadShardAndCollection(UnloadDistributedZkTest.java:134)
at org.apache.solr.cloud.UnloadDistributedZkTest.test(UnloadDistributedZkTest.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515)
... 46 more
Build Log:
[...truncated 12306 lines...]
[junit4] Suite: org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest
[junit4] 2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_DF5C6277AD21161E-001/init-core-data-001
[junit4] 2> 658009 WARN (SUITE-HdfsUnloadDistributedZkTest-seed#[DF5C6277AD21161E]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=8 numCloses=8
[junit4] 2> 658010 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[DF5C6277AD21161E]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields
[junit4] 2> 658011 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[DF5C6277AD21161E]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
[junit4] 2> 658011 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[DF5C6277AD21161E]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /r/ng
[junit4] 1> Formatting using clusterid: testClusterID
[junit4] 2> 658550 WARN (SUITE-HdfsUnloadDistributedZkTest-seed#[DF5C6277AD21161E]-worker) [ ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
[junit4] 2> 658630 WARN (SUITE-HdfsUnloadDistributedZkTest-seed#[DF5C6277AD21161E]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 658653 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[DF5C6277AD21161E]-worker) [ ] o.m.log jetty-6.1.26
[junit4] 2> 658764 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[DF5C6277AD21161E]-worker) [ ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_lucene2.us.west_apache_org_34818_hdfs____.p27vaw/webapp
[junit4] 2> 660656 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[DF5C6277AD21161E]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@lucene2-us-west.apache.org:34818
[junit4] 2> 661948 WARN (SUITE-HdfsUnloadDistributedZkTest-seed#[DF5C6277AD21161E]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 661949 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[DF5C6277AD21161E]-worker) [ ] o.m.log jetty-6.1.26
[junit4] 2> 662181 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[DF5C6277AD21161E]-worker) [ ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_43475_datanode____4qzb8a/webapp
[junit4] 2> 663520 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[DF5C6277AD21161E]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43475
[junit4] 2> 664594 WARN (SUITE-HdfsUnloadDistributedZkTest-seed#[DF5C6277AD21161E]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 664611 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[DF5C6277AD21161E]-worker) [ ] o.m.log jetty-6.1.26
[junit4] 2> 664793 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[DF5C6277AD21161E]-worker) [ ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_43791_datanode____b7aaaf/webapp
[junit4] 2> 669347 ERROR (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_DF5C6277AD21161E-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_DF5C6277AD21161E-001/tempDir-001/hdfsBaseDir/data/data2/]] heartbeating to lucene2-us-west.apache.org/127.0.0.1:41587) [ ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
[junit4] 2> 669607 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x9d9e77a5f62b8: from storage DS-b1adebb0-2962-402e-bba2-3aa07a7764fc node DatanodeRegistration(127.0.0.1:34423, datanodeUuid=8f2f064c-01f8-4041-a2ec-e3b933313ec7, infoPort=45142, infoSecurePort=0, ipcPort=38295, storageInfo=lv=-56;cid=testClusterID;nsid=1649085565;c=0), blocks: 0, hasStaleStorage: true, processing time: 126 msecs
[junit4] 2> 669608 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x9d9e77a5f62b8: from storage DS-53a096d0-87ee-4d67-8439-dae5169d8a99 node DatanodeRegistration(127.0.0.1:34423, datanodeUuid=8f2f064c-01f8-4041-a2ec-e3b933313ec7, infoPort=45142, infoSecurePort=0, ipcPort=38295, storageInfo=lv=-56;cid=testClusterID;nsid=1649085565;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
[junit4] 2> 670824 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[DF5C6277AD21161E]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43791
[junit4] 2> 672349 ERROR (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_DF5C6277AD21161E-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_DF5C6277AD21161E-001/tempDir-001/hdfsBaseDir/data/data4/]] heartbeating to lucene2-us-west.apache.org/127.0.0.1:41587) [ ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
[junit4] 2> 672418 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x9d9e82a0e2219: from storage DS-c9a73587-50f4-419c-8404-b3969f878909 node DatanodeRegistration(127.0.0.1:43167, datanodeUuid=33ad7626-df79-47a6-ba5f-d00f30ccf677, infoPort=34284, infoSecurePort=0, ipcPort=40891, storageInfo=lv=-56;cid=testClusterID;nsid=1649085565;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
[junit4] 2> 672419 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x9d9e82a0e2219: from storage DS-57f9dfe6-6066-4bf2-9121-7804eedc1ebd node DatanodeRegistration(127.0.0.1:43167, datanodeUuid=33ad7626-df79-47a6-ba5f-d00f30ccf677, infoPort=34284, infoSecurePort=0, ipcPort=40891, storageInfo=lv=-56;cid=testClusterID;nsid=1649085565;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
[junit4] 2> 673176 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 673265 INFO (Thread-8879) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 673265 INFO (Thread-8879) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 673290 ERROR (Thread-8879) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 673366 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.ZkTestServer start zk server on port:37866
[junit4] 2> 673524 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 673538 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 673539 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 673541 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 673554 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 673555 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 673556 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 673557 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 673574 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 673575 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 673591 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 674441 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_DF5C6277AD21161E-001/control-001/cores/collection1
[junit4] 2> 674455 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 674503 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5e6b2437{/r/ng,null,AVAILABLE}
[junit4] 2> 674503 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@281226c2{HTTP/1.1,[http/1.1]}{127.0.0.1:45189}
[junit4] 2> 674503 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.e.j.s.Server Started @686472ms
[junit4] 2> 674503 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:41587/hdfs__lucene2-us-west.apache.org_41587__home_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.6_checkout_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_DF5C6277AD21161E-001_tempDir-002_control_data, hostContext=/r/ng, hostPort=45189, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_DF5C6277AD21161E-001/control-001/cores}
[junit4] 2> 674503 ERROR (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 674503 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.6.4
[junit4] 2> 674503 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 674504 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 674504 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-07T15:11:07.101Z
[junit4] 2> 674631 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 674631 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_DF5C6277AD21161E-001/control-001/solr.xml
[junit4] 2> 674692 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 674695 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37866/solr
[junit4] 2> 674963 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [n:127.0.0.1:45189_r%2Fng ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 674980 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [n:127.0.0.1:45189_r%2Fng ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:45189_r%2Fng
[junit4] 2> 674981 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [n:127.0.0.1:45189_r%2Fng ] o.a.s.c.Overseer Overseer (id=99643258690797572-127.0.0.1:45189_r%2Fng-n_0000000000) starting
[junit4] 2> 675036 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [n:127.0.0.1:45189_r%2Fng ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45189_r%2Fng
[junit4] 2> 675050 INFO (zkCallback-158-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 675267 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [n:127.0.0.1:45189_r%2Fng ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_DF5C6277AD21161E-001/control-001/cores
[junit4] 2> 675267 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [n:127.0.0.1:45189_r%2Fng ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 675305 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 675343 INFO (OverseerStateUpdate-99643258690797572-127.0.0.1:45189_r%2Fng-n_0000000000) [n:127.0.0.1:45189_r%2Fng ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 676571 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.4
[junit4] 2> 676636 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 677409 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 677710 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection, trusted=true
[junit4] 2> 677711 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://lucene2-us-west.apache.org:41587/solr_hdfs_home
[junit4] 2> 677711 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
[junit4] 2> 677711 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 677711 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_DF5C6277AD21161E-001/control-001/cores/collection1], dataDir=[hdfs://lucene2-us-west.apache.org:41587/solr_hdfs_home/control_collection/core_node1/data/]
[junit4] 2> 677712 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1397dcb1
[junit4] 2> 677728 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:41587/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
[junit4] 2> 677796 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 677796 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 677796 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 679063 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 679090 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:41587/solr_hdfs_home/control_collection/core_node1/data
[junit4] 2> 679235 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:41587/solr_hdfs_home/control_collection/core_node1/data/index
[junit4] 2> 679310 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 679311 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 679311 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 680627 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 680628 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=43, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=63.90234375, floorSegmentMB=0.830078125, forceMergeDeletesPctAllowed=3.5752316798021857, segmentsPerTier=25.0, maxCFSSegmentSizeMB=1.09375, noCFSRatio=0.7147959364477267
[junit4] 2> 680884 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43167 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-53a096d0-87ee-4d67-8439-dae5169d8a99:NORMAL:127.0.0.1:34423|RBW], ReplicaUC[[DISK]DS-57f9dfe6-6066-4bf2-9121-7804eedc1ebd:NORMAL:127.0.0.1:43167|RBW]]} size 71
[junit4] 2> 680884 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:34423 is added to blk_1073741825_1001 size 71
[junit4] 2> 681455 WARN (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 682053 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
[junit4] 2> 682053 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 682053 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
[junit4] 2> 682143 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 682143 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 682179 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@77372943
[junit4] 2> 682451 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1affd7e9[collection1] main]
[junit4] 2> 682511 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 682547 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 682547 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 682630 INFO (searcherExecutor-307-thread-1-processing-n:127.0.0.1:45189_r%2Fng x:collection1 c:control_collection) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1affd7e9[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 682656 INFO (coreLoadExecutor-306-thread-1-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng c:control_collection x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1594292148898889728
[junit4] 2> 683052 INFO (coreZkRegister-299-thread-1-processing-n:127.0.0.1:45189_r%2Fng x:collection1 c:control_collection) [n:127.0.0.1:45189_r%2Fng c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 683052 INFO (coreZkRegister-299-thread-1-processing-n:127.0.0.1:45189_r%2Fng x:collection1 c:control_collection) [n:127.0.0.1:45189_r%2Fng c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 683052 INFO (coreZkRegister-299-thread-1-processing-n:127.0.0.1:45189_r%2Fng x:collection1 c:control_collection) [n:127.0.0.1:45189_r%2Fng c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:45189/r/ng/collection1/
[junit4] 2> 683052 INFO (coreZkRegister-299-thread-1-processing-n:127.0.0.1:45189_r%2Fng x:collection1 c:control_collection) [n:127.0.0.1:45189_r%2Fng c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 683052 INFO (coreZkRegister-299-thread-1-processing-n:127.0.0.1:45189_r%2Fng x:collection1 c:control_collection) [n:127.0.0.1:45189_r%2Fng c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:45189/r/ng/collection1/ has no replicas
[junit4] 2> 683053 INFO (coreZkRegister-299-thread-1-processing-n:127.0.0.1:45189_r%2Fng x:collection1 c:control_collection) [n:127.0.0.1:45189_r%2Fng c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 683106 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 683203 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37866/solr ready
[junit4] 2> 683203 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 683226 INFO (coreZkRegister-299-thread-1-processing-n:127.0.0.1:45189_r%2Fng x:collection1 c:control_collection) [n:127.0.0.1:45189_r%2Fng c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:45189/r/ng/collection1/ shard1
[junit4] 2> 683386 INFO (coreZkRegister-299-thread-1-processing-n:127.0.0.1:45189_r%2Fng x:collection1 c:control_collection) [n:127.0.0.1:45189_r%2Fng c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 684278 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_DF5C6277AD21161E-001/shard-1-001/cores/collection1
[junit4] 2> 684279 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_DF5C6277AD21161E-001/shard-1-001
[junit4] 2> 684306 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 684316 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6da0a504{/r/ng,null,AVAILABLE}
[junit4] 2> 684316 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@2014d012{HTTP/1.1,[http/1.1]}{127.0.0.1:40687}
[junit4] 2> 684316 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.e.j.s.Server Started @696285ms
[junit4] 2> 684316 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:41587/hdfs__lucene2-us-west.apache.org_41587__home_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.6_checkout_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_DF5C6277AD21161E-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/r/ng, hostPort=40687, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_DF5C6277AD21161E-001/shard-1-001/cores}
[junit4] 2> 684317 ERROR (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 684430 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.6.4
[junit4] 2> 684430 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 684430 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 684430 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-07T15:11:17.027Z
[junit4] 2> 684508 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 684508 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_DF5C6277AD21161E-001/shard-1-001/solr.xml
[junit4] 2> 684641 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 684719 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37866/solr
[junit4] 2> 685178 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [n:127.0.0.1:40687_r%2Fng ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 685180 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [n:127.0.0.1:40687_r%2Fng ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 685213 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [n:127.0.0.1:40687_r%2Fng ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40687_r%2Fng
[junit4] 2> 685417 INFO (zkCallback-158-thread-2-processing-n:127.0.0.1:45189_r%2Fng) [n:127.0.0.1:45189_r%2Fng ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 685559 INFO (zkCallback-167-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 685417 INFO (zkCallback-162-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 685919 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [n:127.0.0.1:40687_r%2Fng ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_DF5C6277AD21161E-001/shard-1-001/cores
[junit4] 2> 685919 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[DF5C6277AD21161E]) [n:127.0.0.1:40687_r%2Fng ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 685926 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 685955 INFO (OverseerStateUpdate-99643258690797572-127.0.0.1:45189_r%2Fng-n_0000000000) [n:127.0.0.1:45189_r%2Fng ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
[junit4] 2> 686047 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.4
[junit4] 2> 686142 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 688505 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 688580 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
[junit4] 2> 688580 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://lucene2-us-west.apache.org:41587/solr_hdfs_home
[junit4] 2> 688580 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
[junit4] 2> 688580 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 688580 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_DF5C6277AD21161E-001/shard-1-001/cores/collection1], dataDir=[hdfs://lucene2-us-west.apache.org:41587/solr_hdfs_home/collection1/core_node1/data/]
[junit4] 2> 688581 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1397dcb1
[junit4] 2> 688581 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:41587/solr_hdfs_home/collection1/core_node1/data/snapshot_metadata
[junit4] 2> 688737 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 688737 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 688737 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 689279 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 689279 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:41587/solr_hdfs_home/collection1/core_node1/data
[junit4] 2> 689516 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:41587/solr_hdfs_home/collection1/core_node1/data/index
[junit4] 2> 689617 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 689617 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 689617 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 689697 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 689697 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=43, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=63.90234375, floorSegmentMB=0.830078125, forceMergeDeletesPctAllowed=3.5752316798021857, segmentsPerTier=25.0, maxCFSSegmentSizeMB=1.09375, noCFSRatio=0.7147959364477267
[junit4] 2> 689986 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43167 is added to blk_1073741826_1002{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b1adebb0-2962-402e-bba2-3aa07a7764fc:NORMAL:127.0.0.1:34423|RBW], ReplicaUC[[DISK]DS-c9a73587-50f4-419c-8404-b3969f878909:NORMAL:127.0.0.1:43167|RBW]]} size 71
[junit4] 2> 690004 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:34423 is added to blk_1073741826_1002 size 71
[junit4] 2> 690477 WARN (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 690802 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
[junit4] 2> 690802 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 690822 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
[junit4] 2> 690930 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 690930 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 690966 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@417990e6
[junit4] 2> 691160 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@fe2aa9a[collection1] main]
[junit4] 2> 691161 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 691161 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 691161 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 691208 INFO (searcherExecutor-318-thread-1-processing-n:127.0.0.1:40687_r%2Fng x:collection1 c:collection1) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@fe2aa9a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 691283 INFO (coreLoadExecutor-317-thread-1-processing-n:127.0.0.1:40687_r%2Fng) [n:127.0.0.1:40687_r%2Fng c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1594292157944954880
[junit4] 2> 691436 INFO (coreZkRegister-312-thread-1-processing-n:127.0.0.1:40687_r%2Fng x:collection1 c:collection1) [n:127.0.0.1:40687_r%2Fng c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 691436 INFO (coreZkRegister-312-thread-1-processing-n:127.0.0.1:40687_r%2Fng x:collection1 c:collection1) [n:127.0.0.1:40687_r%2Fng c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 691436 INFO (coreZkRegister-312-thread-1-processing-n:127.0.0.1:40687_r%2Fng x:collection1 c:collection1) [n:127.0.0.1:40687_r%2Fng c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:40687/r/ng/collection1/
[junit4] 2> 691436 INFO (coreZkRegister-312-thread-1-processing-n:127.0.0.1:40687_r%2Fng x:collection1 c:collection1) [n:127.0.0.1:40687_r%2Fng c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 691436 INFO (coreZkRegister-312-thread-1-processing-n:127.0.0.1:40687_r%2Fng x:collection1 c:collection1) [n:127.0.0.1:40687_r%2Fng c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:40687/r/ng/collection1/ has no replicas
[junit4] 2> 691436 INFO (coreZkRegister-312-thread-1-processing-n:127.0.0.1:40687_r%2Fng x:collection1 c:collection1) [n:127.0.0.1:40687_r%2Fng c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 691439 INFO (coreZkRegister-312-thread-1-processing-n:127.0.0.1:40687_r%2Fng x:collection1 c:collection1) [n:127.0.0.1:40687_r%2Fng c:collection1 s:shard2 r:core_node1 x:co
[...truncated too long message...]
n.java:244)
[junit4] 2> com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
[junit4] 2> com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
[junit4] 2>
[junit4] 2> LeaseRenewer:jenkins@lucene2-us-west.apache.org:41032:
[junit4] 2> java.lang.Thread.sleep(Native Method)
[junit4] 2> org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444)
[junit4] 2> org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
[junit4] 2> org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304)
[junit4] 2> java.lang.Thread.run(Thread.java:748)
[junit4] 2>
[junit4] 2> process reaper:
[junit4] 2> [...sun.*]
[junit4] 2> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] 2> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] 2> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] 2> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] 2> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
[junit4] 2> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] 2> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] 2> java.lang.Thread.run(Thread.java:748)
[junit4] 2>
[junit4] 2> TimeLimitedCollector timer thread:
[junit4] 2> java.lang.Thread.sleep(Native Method)
[junit4] 2> org.apache.lucene.search.TimeLimitingCollector$TimerThread.run(TimeLimitingCollector.java:261)
[junit4] 2>
[junit4] 2> IPC Parameter Sending Thread #7:
[junit4] 2> [...sun.*]
[junit4] 2> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] 2> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] 2> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] 2> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] 2> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
[junit4] 2> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] 2> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] 2> java.lang.Thread.run(Thread.java:748)
[junit4] 2>
[junit4] 2> IPC Parameter Sending Thread #8:
[junit4] 2> [...sun.*]
[junit4] 2> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] 2> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] 2> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] 2> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] 2> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
[junit4] 2> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] 2> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] 2> java.lang.Thread.run(Thread.java:748)
[junit4] 2>
[junit4] 2> 7847832 INFO (SUITE-HdfsBasicDistributedZkTest-seed#[DF5C6277AD21161E]-worker) [ ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
[junit4] 2> NOTE: test params are: codec=Lucene62, sim=RandomSimilarity(queryNorm=false,coord=yes): {}, locale=sv-SE, timezone=Europe/Rome
[junit4] 2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=2,free=208046768,total=534249472
[junit4] 2> NOTE: All tests run in this JVM: [TestComplexPhraseLeadingWildcard, TestScoreJoinQPScore, MultiThreadedOCPTest, TestDynamicFieldCollectionResource, TestCloudPseudoReturnFields, ChaosMonkeyNothingIsSafeTest, VMParamsZkACLAndCredentialsProvidersTest, SuggestComponentContextFilterQueryTest, TestStressRecovery, DocumentAnalysisRequestHandlerTest, QueryElevationComponentTest, TestOmitPositions, BasicZkTest, SynonymTokenizerTest, DirectoryFactoryTest, SolrRequestParserTest, PreAnalyzedUpdateProcessorTest, TestBulkSchemaAPI, TestSha256AuthenticationProvider, DistributedFacetPivotWhiteBoxTest, TestExactSharedStatsCache, AnalyticsMergeStrategyTest, TestAuthorizationFramework, TestSolr4Spatial, SolrCoreTest, TestLazyCores, TestXmlQParser, RollingRestartTest, TestFieldCacheSortRandom, TestJsonFacets, RegexBoostProcessorTest, BlockDirectoryTest, TestTolerantUpdateProcessorRandomCloud, TestSortByMinMaxFunction, ScriptEngineTest, ReturnFieldsTest, AliasIntegrationTest, TestStressInPlaceUpdates, TestSchemaVersionResource, TestSolrXml, PeerSyncTest, TestRandomDVFaceting, DeleteNodeTest, TestSolrCloudWithKerberosAlt, TestPathTrie, SolrXmlInZkTest, DistribJoinFromCollectionTest, TestNonDefinedSimilarityFactory, SimpleCollectionCreateDeleteTest, TestSubQueryTransformer, TestExpandComponent, TestSolr4Spatial2, TestSolrQueryParserResource, V2ApiIntegrationTest, TestCSVLoader, NotRequiredUniqueKeyTest, DistributedSpellCheckComponentTest, ExternalFileFieldSortTest, TestSlowCompositeReaderWrapper, DistributedQueryElevationComponentTest, TestBadConfig, SolrGangliaReporterTest, DeleteStatusTest, TestCloudInspectUtil, SortSpecParsingTest, TestReRankQParserPlugin, DistributedExpandComponentTest, HdfsChaosMonkeyNothingIsSafeTest, TestJavabinTupleStreamParser, ChangedSchemaMergeTest, ClusterStateTest, MigrateRouteKeyTest, TestHalfAndHalfDocValues, TestRandomFaceting, TestCloudNestedDocsSort, EnumFieldTest, TestRequestStatusCollectionAPI, SignatureUpdateProcessorFactoryTest, LargeFieldTest, TestDocSet, TestSchemalessBufferedUpdates, DocValuesNotIndexedTest, TestRealTimeGet, TestJmxIntegration, CoreMergeIndexesAdminHandlerTest, TestStressLucene, CustomCollectionTest, SpatialRPTFieldTypeTest, TestFieldResource, TestDistributedSearch, HdfsTlogReplayBufferedWhileIndexingTest, TestFaceting, TestBinaryResponseWriter, TestFieldSortValues, TestMacroExpander, SyncSliceTest, SubstringBytesRefFilterTest, HdfsNNFailoverTest, QueryEqualityTest, TestFreeTextSuggestions, CacheHeaderTest, PropertiesRequestHandlerTest, TestUninvertingReader, TestSearcherReuse, TestQueryWrapperFilter, DefaultValueUpdateProcessorTest, PolyFieldTest, JavabinLoaderTest, TestManagedSynonymFilterFactory, BlockJoinFacetRandomTest, TestCodecSupport, UniqFieldsUpdateProcessorFactoryTest, TestFoldingMultitermQuery, TestStandardQParsers, UUIDFieldTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, LeaderElectionIntegrationTest, TestReplicationHandler, SolrCLIZkUtilsTest, TestCloudJSONFacetJoinDomain, TestConfigSetsAPIZkFailure, TestDistribDocBasedVersion, TestDownShardTolerantSearch, TestLeaderInitiatedRecoveryThread, TestLockTree, TestMiniSolrCloudCluster, TestRebalanceLeaders, TestRequestForwarding, TestSizeLimitedDistributedMap, HdfsBasicDistributedZkTest]
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=HdfsBasicDistributedZkTest -Dtests.seed=DF5C6277AD21161E -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/test-data/enwiki.random.lines.txt -Dtests.locale=sv-SE -Dtests.timezone=Europe/Rome -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 0.00s J0 | HdfsBasicDistributedZkTest (suite) <<<
[junit4] > Throwable #1: java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [NRTCachingDirectory]
[junit4] > org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.NRTCachingDirectory
[junit4] > at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
[junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
[junit4] > at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:353)
[junit4] > at org.apache.solr.core.SolrCore.cleanupOldIndexDirectories(SolrCore.java:3044)
[junit4] > at org.apache.solr.core.SolrCore.close(SolrCore.java:1575)
[junit4] > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:969)
[junit4] > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:830)
[junit4] > at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:950)
[junit4] > at org.apache.solr.core.CoreContainer.lambda$load$5(CoreContainer.java:565)
[junit4] > at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
[junit4] > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > at __randomizedtesting.SeedInfo.seed([DF5C6277AD21161E]:0)
[junit4] > at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:302)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] Completed [689/714 (5!)] on J0 in 939.02s, 1 test, 1 failure <<< FAILURES!
[...truncated 46399 lines...]
[JENKINS] Lucene-Solr-NightlyTests-6.6 - Build # 36 - Still
Unstable
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.6/36/
5 tests failed.
FAILED: org.apache.lucene.search.join.TestCheckJoinIndex.testOrphans
Error Message:
Invalid index
Stack Trace:
java.lang.AssertionError: Invalid index
at __randomizedtesting.SeedInfo.seed([F6A8C9D5BA982C0A:993789BFDA55BC27]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.lucene.search.join.TestCheckJoinIndex.testOrphans(TestCheckJoinIndex.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.test
Error Message:
Captured an uncaught exception in thread: Thread[id=3186, name=testExecutor-1568-thread-5, state=RUNNABLE, group=TGRP-HdfsUnloadDistributedZkTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3186, name=testExecutor-1568-thread-5, state=RUNNABLE, group=TGRP-HdfsUnloadDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:40347: Error CREATEing SolrCore 'multiunload4': Unable to create core [multiunload4] Caused by: Direct buffer memory
at __randomizedtesting.SeedInfo.seed([D4AAAF8E12BF4EE2]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:612)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at org.apache.solr.cloud.BasicDistributedZkTest.lambda$createCollectionInOneInstance$0(BasicDistributedZkTest.java:589)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.core.TestCustomStream.testDynamicLoadingCustomStream
Error Message:
KeeperErrorCode = Session expired for /clusterstate.json
Stack Trace:
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
at __randomizedtesting.SeedInfo.seed([D4AAAF8E12BF4EE2:A41FA6ACE8450555]:0)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
at org.apache.solr.common.cloud.ZkStateReader.refreshLegacyClusterState(ZkStateReader.java:532)
at org.apache.solr.common.cloud.ZkStateReader.forceUpdateCollection(ZkStateReader.java:300)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.updateMappingsFromZk(AbstractFullDistribZkTestBase.java:622)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.updateMappingsFromZk(AbstractFullDistribZkTestBase.java:617)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:430)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:331)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:990)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.client.solrj.TestSolrJErrorHandling.testWithXml
Error Message:
expected:<568> but was:<558>
Stack Trace:
java.lang.AssertionError: expected:<568> but was:<558>
at __randomizedtesting.SeedInfo.seed([EA4D8BD788B670D5:41AF32210AB069B8]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.client.solrj.TestSolrJErrorHandling.doThreads(TestSolrJErrorHandling.java:185)
at org.apache.solr.client.solrj.TestSolrJErrorHandling.doIt(TestSolrJErrorHandling.java:200)
at org.apache.solr.client.solrj.TestSolrJErrorHandling.testWithXml(TestSolrJErrorHandling.java:110)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.hdfs.StressHdfsTest.test
Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:36507/_jej
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:36507/_jej
at __randomizedtesting.SeedInfo.seed([D4AAAF8E12BF4EE2:5CFE9054BC43231A]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:623)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:447)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:388)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1383)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1134)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at org.apache.solr.cloud.hdfs.StressHdfsTest.createAndDeleteCollection(StressHdfsTest.java:220)
at org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:103)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515)
... 51 more
Build Log:
[...truncated 8351 lines...]
[junit4] Suite: org.apache.lucene.search.join.TestCheckJoinIndex
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestCheckJoinIndex -Dtests.method=testOrphans -Dtests.seed=F6A8C9D5BA982C0A -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/test-data/enwiki.random.lines.txt -Dtests.locale=sr-ME -Dtests.timezone=America/Winnipeg -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] FAILURE 1.03s J0 | TestCheckJoinIndex.testOrphans <<<
[junit4] > Throwable #1: java.lang.AssertionError: Invalid index
[junit4] > at __randomizedtesting.SeedInfo.seed([F6A8C9D5BA982C0A:993789BFDA55BC27]:0)
[junit4] > at org.apache.lucene.search.join.TestCheckJoinIndex.testOrphans(TestCheckJoinIndex.java:92)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] 2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=5, maxDocsPerChunk=7, blockSize=9), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=5, blockSize=9)), sim=RandomSimilarity(queryNorm=true,coord=yes): {}, locale=sr-ME, timezone=America/Winnipeg
[junit4] 2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=208828144,total=253231104
[junit4] 2> NOTE: All tests run in this JVM: [TestCheckJoinIndex]
[junit4] Completed [1/7 (1!)] on J0 in 5.25s, 3 tests, 1 failure <<< FAILURES!
[...truncated 4009 lines...]
[junit4] Suite: org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest
[junit4] 2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/init-core-data-001
[junit4] 2> 580509 WARN (SUITE-HdfsUnloadDistributedZkTest-seed#[D4AAAF8E12BF4EE2]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
[junit4] 2> 580509 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[D4AAAF8E12BF4EE2]-worker) [ ] o.a.s.SolrTestCaseJ4 Using TrieFields
[junit4] 2> 580510 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[D4AAAF8E12BF4EE2]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
[junit4] 2> 580543 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[D4AAAF8E12BF4EE2]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
[junit4] 2> 583294 WARN (SUITE-HdfsUnloadDistributedZkTest-seed#[D4AAAF8E12BF4EE2]-worker) [ ] o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[junit4] 1> Formatting using clusterid: testClusterID
[junit4] 2> 586914 WARN (SUITE-HdfsUnloadDistributedZkTest-seed#[D4AAAF8E12BF4EE2]-worker) [ ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
[junit4] 2> 587676 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[D4AAAF8E12BF4EE2]-worker) [ ] o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
[junit4] 2> 587776 WARN (SUITE-HdfsUnloadDistributedZkTest-seed#[D4AAAF8E12BF4EE2]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 589261 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[D4AAAF8E12BF4EE2]-worker) [ ] o.m.log jetty-6.1.26
[junit4] 2> 589502 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[D4AAAF8E12BF4EE2]-worker) [ ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_lucene2.us.west_apache_org_38587_hdfs____8qrt21/webapp
[junit4] 2> 591664 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[D4AAAF8E12BF4EE2]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@lucene2-us-west.apache.org:38587
[junit4] 2> 596727 WARN (SUITE-HdfsUnloadDistributedZkTest-seed#[D4AAAF8E12BF4EE2]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 596749 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[D4AAAF8E12BF4EE2]-worker) [ ] o.m.log jetty-6.1.26
[junit4] 2> 596815 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[D4AAAF8E12BF4EE2]-worker) [ ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_43962_datanode____a9lcjh/webapp
[junit4] 2> 597815 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[D4AAAF8E12BF4EE2]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43962
[junit4] 2> 600735 WARN (SUITE-HdfsUnloadDistributedZkTest-seed#[D4AAAF8E12BF4EE2]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 600736 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[D4AAAF8E12BF4EE2]-worker) [ ] o.m.log jetty-6.1.26
[junit4] 2> 601013 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[D4AAAF8E12BF4EE2]-worker) [ ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_45890_datanode____o3lyj9/webapp
[junit4] 2> 606338 INFO (SUITE-HdfsUnloadDistributedZkTest-seed#[D4AAAF8E12BF4EE2]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45890
[junit4] 2> 611052 ERROR (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/tempDir-001/hdfsBaseDir/data/data2/]] heartbeating to lucene2-us-west.apache.org/127.0.0.1:46485) [ ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
[junit4] 2> 611035 ERROR (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/tempDir-001/hdfsBaseDir/data/data4/]] heartbeating to lucene2-us-west.apache.org/127.0.0.1:46485) [ ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
[junit4] 2> 612621 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x97b8f8deaae49: from storage DS-eb10a6dd-9a00-462c-adf9-f20dd959ebae node DatanodeRegistration(127.0.0.1:33171, datanodeUuid=269e155a-c845-48b9-8fe3-a47c91de5130, infoPort=44041, infoSecurePort=0, ipcPort=43728, storageInfo=lv=-56;cid=testClusterID;nsid=393357738;c=0), blocks: 0, hasStaleStorage: true, processing time: 27 msecs
[junit4] 2> 612621 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x97b8f8ceec238: from storage DS-de8360fd-e272-4456-9a38-31229d37fb24 node DatanodeRegistration(127.0.0.1:39183, datanodeUuid=f345b3d7-26c3-4e93-b54c-8bf7541725cb, infoPort=41505, infoSecurePort=0, ipcPort=37452, storageInfo=lv=-56;cid=testClusterID;nsid=393357738;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
[junit4] 2> 612621 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x97b8f8deaae49: from storage DS-5f88f666-4108-4360-9878-fd87a63591d1 node DatanodeRegistration(127.0.0.1:33171, datanodeUuid=269e155a-c845-48b9-8fe3-a47c91de5130, infoPort=44041, infoSecurePort=0, ipcPort=43728, storageInfo=lv=-56;cid=testClusterID;nsid=393357738;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
[junit4] 2> 612645 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x97b8f8ceec238: from storage DS-27955c67-62e1-456f-9abe-a1b0847619be node DatanodeRegistration(127.0.0.1:39183, datanodeUuid=f345b3d7-26c3-4e93-b54c-8bf7541725cb, infoPort=41505, infoSecurePort=0, ipcPort=37452, storageInfo=lv=-56;cid=testClusterID;nsid=393357738;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
[junit4] 2> 614264 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 614274 INFO (Thread-790) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 614274 INFO (Thread-790) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 614276 ERROR (Thread-790) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 614374 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.ZkTestServer start zk server on port:35651
[junit4] 2> 614512 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 614523 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 614525 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 614526 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 614544 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 614545 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 614558 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 614560 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 614571 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 614572 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 614573 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 615051 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/control-001/cores/collection1
[junit4] 2> 615069 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 615140 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@574e666a{/,null,AVAILABLE}
[junit4] 2> 615141 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@466a71d1{HTTP/1.1,[http/1.1]}{127.0.0.1:33299}
[junit4] 2> 615141 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.e.j.s.Server Started @626698ms
[junit4] 2> 615141 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:46485/hdfs__lucene2-us-west.apache.org_46485__home_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.6_checkout_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001_tempDir-002_control_data, hostContext=/, hostPort=33299, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/control-001/cores}
[junit4] 2> 615141 ERROR (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 615141 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.6.3
[junit4] 2> 615141 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 615141 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 615141 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-06T10:22:13.011Z
[junit4] 2> 615286 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 615286 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/control-001/solr.xml
[junit4] 2> 615323 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 615324 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35651/solr
[junit4] 2> 615799 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [n:127.0.0.1:33299_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 615799 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [n:127.0.0.1:33299_ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33299_
[junit4] 2> 615800 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [n:127.0.0.1:33299_ ] o.a.s.c.Overseer Overseer (id=99636460397854724-127.0.0.1:33299_-n_0000000000) starting
[junit4] 2> 615957 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [n:127.0.0.1:33299_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33299_
[junit4] 2> 616021 INFO (zkCallback-423-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 616249 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [n:127.0.0.1:33299_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/control-001/cores
[junit4] 2> 616249 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [n:127.0.0.1:33299_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 616309 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 616395 INFO (OverseerStateUpdate-99636460397854724-127.0.0.1:33299_-n_0000000000) [n:127.0.0.1:33299_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 617991 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.3
[junit4] 2> 618226 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 619007 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 619472 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection, trusted=true
[junit4] 2> 619561 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://lucene2-us-west.apache.org:46485/solr_hdfs_home
[junit4] 2> 619562 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
[junit4] 2> 619562 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 619562 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/control-001/cores/collection1], dataDir=[hdfs://lucene2-us-west.apache.org:46485/solr_hdfs_home/control_collection/core_node1/data/]
[junit4] 2> 619562 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@62b17bb9
[junit4] 2> 619563 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46485/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
[junit4] 2> 619710 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 619746 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 619746 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 623193 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 623205 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46485/solr_hdfs_home/control_collection/core_node1/data
[junit4] 2> 623447 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46485/solr_hdfs_home/control_collection/core_node1/data/index
[junit4] 2> 623503 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 623503 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 623503 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 623555 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 623556 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=7, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 625636 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39183 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-de8360fd-e272-4456-9a38-31229d37fb24:NORMAL:127.0.0.1:39183|RBW], ReplicaUC[[DISK]DS-eb10a6dd-9a00-462c-adf9-f20dd959ebae:NORMAL:127.0.0.1:33171|RBW]]} size 71
[junit4] 2> 625689 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33171 is added to blk_1073741825_1001 size 71
[junit4] 2> 626167 WARN (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 626566 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
[junit4] 2> 626566 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 626566 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
[junit4] 2> 626639 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 626639 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 626656 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1593055026, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 626944 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@e032f9b[collection1] main]
[junit4] 2> 626977 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 626977 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 626992 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 626994 INFO (searcherExecutor-1492-thread-1-processing-n:127.0.0.1:33299_ x:collection1 c:control_collection) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@e032f9b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 627015 INFO (coreLoadExecutor-1491-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ c:control_collection x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1594183379724533760
[junit4] 2> 627036 INFO (coreZkRegister-1484-thread-1-processing-n:127.0.0.1:33299_ x:collection1 c:control_collection) [n:127.0.0.1:33299_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 627036 INFO (coreZkRegister-1484-thread-1-processing-n:127.0.0.1:33299_ x:collection1 c:control_collection) [n:127.0.0.1:33299_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 627036 INFO (coreZkRegister-1484-thread-1-processing-n:127.0.0.1:33299_ x:collection1 c:control_collection) [n:127.0.0.1:33299_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33299/collection1/
[junit4] 2> 627036 INFO (coreZkRegister-1484-thread-1-processing-n:127.0.0.1:33299_ x:collection1 c:control_collection) [n:127.0.0.1:33299_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 627036 INFO (coreZkRegister-1484-thread-1-processing-n:127.0.0.1:33299_ x:collection1 c:control_collection) [n:127.0.0.1:33299_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:33299/collection1/ has no replicas
[junit4] 2> 627036 INFO (coreZkRegister-1484-thread-1-processing-n:127.0.0.1:33299_ x:collection1 c:control_collection) [n:127.0.0.1:33299_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 627161 INFO (coreZkRegister-1484-thread-1-processing-n:127.0.0.1:33299_ x:collection1 c:control_collection) [n:127.0.0.1:33299_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33299/collection1/ shard1
[junit4] 2> 627162 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 627189 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35651/solr ready
[junit4] 2> 627189 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 627322 INFO (coreZkRegister-1484-thread-1-processing-n:127.0.0.1:33299_ x:collection1 c:control_collection) [n:127.0.0.1:33299_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 627748 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/shard-1-001/cores/collection1
[junit4] 2> 627748 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/shard-1-001
[junit4] 2> 627750 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 627751 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2ff4b313{/,null,AVAILABLE}
[junit4] 2> 627751 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@eafc2ed{HTTP/1.1,[http/1.1]}{127.0.0.1:40347}
[junit4] 2> 627760 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.e.j.s.Server Started @639310ms
[junit4] 2> 627760 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:46485/hdfs__lucene2-us-west.apache.org_46485__home_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.6_checkout_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=40347, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/shard-1-001/cores}
[junit4] 2> 627760 ERROR (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 627760 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.6.3
[junit4] 2> 627760 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 627760 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 627760 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-06T10:22:25.630Z
[junit4] 2> 627779 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 627779 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/shard-1-001/solr.xml
[junit4] 2> 627805 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 627806 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35651/solr
[junit4] 2> 627992 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [n:127.0.0.1:40347_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 627998 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [n:127.0.0.1:40347_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 628041 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [n:127.0.0.1:40347_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40347_
[junit4] 2> 628162 INFO (zkCallback-432-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 628163 INFO (zkCallback-423-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 628163 INFO (zkCallback-427-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 628364 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [n:127.0.0.1:40347_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/shard-1-001/cores
[junit4] 2> 628364 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [n:127.0.0.1:40347_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 628411 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 628435 INFO (OverseerStateUpdate-99636460397854724-127.0.0.1:33299_-n_0000000000) [n:127.0.0.1:33299_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
[junit4] 2> 629503 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.3
[junit4] 2> 629561 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 630186 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 630397 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
[junit4] 2> 630398 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://lucene2-us-west.apache.org:46485/solr_hdfs_home
[junit4] 2> 630435 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
[junit4] 2> 630435 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 630435 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/shard-1-001/cores/collection1], dataDir=[hdfs://lucene2-us-west.apache.org:46485/solr_hdfs_home/collection1/core_node1/data/]
[junit4] 2> 630435 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@62b17bb9
[junit4] 2> 630436 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46485/solr_hdfs_home/collection1/core_node1/data/snapshot_metadata
[junit4] 2> 630493 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 630493 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 630493 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 632296 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 632297 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46485/solr_hdfs_home/collection1/core_node1/data
[junit4] 2> 632403 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46485/solr_hdfs_home/collection1/core_node1/data/index
[junit4] 2> 632495 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 632495 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 632495 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 633394 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 633395 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=7, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 633469 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33171 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-de8360fd-e272-4456-9a38-31229d37fb24:NORMAL:127.0.0.1:39183|RBW], ReplicaUC[[DISK]DS-5f88f666-4108-4360-9878-fd87a63591d1:NORMAL:127.0.0.1:33171|RBW]]} size 0
[junit4] 2> 633482 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39183 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-de8360fd-e272-4456-9a38-31229d37fb24:NORMAL:127.0.0.1:39183|RBW], ReplicaUC[[DISK]DS-5f88f666-4108-4360-9878-fd87a63591d1:NORMAL:127.0.0.1:33171|RBW]]} size 0
[junit4] 2> 633549 WARN (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 633736 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
[junit4] 2> 633736 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 633736 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
[junit4] 2> 633790 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 633790 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 633885 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1593055026, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 634197 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3fabac7e[collection1] main]
[junit4] 2> 634283 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 634291 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 634291 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 634304 INFO (searcherExecutor-1503-thread-1-processing-n:127.0.0.1:40347_ x:collection1 c:collection1) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3fabac7e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 634420 INFO (coreLoadExecutor-1502-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1594183387489239040
[junit4] 2> 634457 INFO (coreZkRegister-1497-thread-1-processing-n:127.0.0.1:40347_ x:collection1 c:collection1) [n:127.0.0.1:40347_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 634457 INFO (coreZkRegister-1497-thread-1-processing-n:127.0.0.1:40347_ x:collection1 c:collection1) [n:127.0.0.1:40347_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 634457 INFO (coreZkRegister-1497-thread-1-processing-n:127.0.0.1:40347_ x:collection1 c:collection1) [n:127.0.0.1:40347_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:40347/collection1/
[junit4] 2> 634457 INFO (coreZkRegister-1497-thread-1-processing-n:127.0.0.1:40347_ x:collection1 c:collection1) [n:127.0.0.1:40347_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 634457 INFO (coreZkRegister-1497-thread-1-processing-n:127.0.0.1:40347_ x:collection1 c:collection1) [n:127.0.0.1:40347_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:40347/collection1/ has no replicas
[junit4] 2> 634457 INFO (coreZkRegister-1497-thread-1-processing-n:127.0.0.1:40347_ x:collection1 c:collection1) [n:127.0.0.1:40347_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 634501 INFO (coreZkRegister-1497-thread-1-processing-n:127.0.0.1:40347_ x:collection1 c:collection1) [n:127.0.0.1:40347_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:40347/collection1/ shard2
[junit4] 2> 634669 INFO (coreZkRegister-1497-thread-1-processing-n:127.0.0.1:40347_ x:collection1 c:collection1) [n:127.0.0.1:40347_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 635595 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/shard-2-001/cores/collection1
[junit4] 2> 635595 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/shard-2-001
[junit4] 2> 635614 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 635635 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6ed800e0{/,null,AVAILABLE}
[junit4] 2> 635635 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@71a74b2{HTTP/1.1,[http/1.1]}{127.0.0.1:33173}
[junit4] 2> 635635 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.e.j.s.Server Started @647193ms
[junit4] 2> 635635 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:46485/hdfs__lucene2-us-west.apache.org_46485__home_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.6_checkout_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001_tempDir-002_jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=33173, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/shard-2-001/cores}
[junit4] 2> 635635 ERROR (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 635635 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.6.3
[junit4] 2> 635635 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 635636 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 635636 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-06T10:22:33.506Z
[junit4] 2> 635696 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 635696 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/shard-2-001/solr.xml
[junit4] 2> 635736 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 635736 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35651/solr
[junit4] 2> 636041 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [n:127.0.0.1:33173_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 636043 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [n:127.0.0.1:33173_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 636077 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [n:127.0.0.1:33173_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33173_
[junit4] 2> 636078 INFO (zkCallback-427-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 636079 INFO (zkCallback-438-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 636096 INFO (zkCallback-432-thread-1-processing-n:127.0.0.1:40347_) [n:127.0.0.1:40347_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 636112 INFO (zkCallback-423-thread-1-processing-n:127.0.0.1:33299_) [n:127.0.0.1:33299_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 636880 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [n:127.0.0.1:33173_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/shard-2-001/cores
[junit4] 2> 636881 INFO (TEST-HdfsUnloadDistributedZkTest.test-seed#[D4AAAF8E12BF4EE2]) [n:127.0.0.1:33173_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 636881 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 636966 INFO (OverseerStateUpdate-99636460397854724-127.0.0.1:33299_-n_0000000000) [n:127.0.0.1:33299_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 637969 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.3
[junit4] 2> 638018 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 638464 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 638512 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
[junit4] 2> 638513 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://lucene2-us-west.apache.org:46485/solr_hdfs_home
[junit4] 2> 638514 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
[junit4] 2> 638514 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 638522 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_D4AAAF8E12BF4EE2-001/shard-2-001/cores/collection1], dataDir=[hdfs://lucene2-us-west.apache.org:46485/solr_hdfs_home/collection1/core_node2/data/]
[junit4] 2> 638523 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@62b17bb9
[junit4] 2> 638523 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46485/solr_hdfs_home/collection1/core_node2/data/snapshot_metadata
[junit4] 2> 638559 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 638559 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 638559 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 638705 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 638731 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46485/solr_hdfs_home/collection1/core_node2/data
[junit4] 2> 638790 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46485/solr_hdfs_home/collection1/core_node2/data/index
[junit4] 2> 638796 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 638796 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 638796 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 638862 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 638863 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=7, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 638941 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33171 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-27955c67-62e1-456f-9abe-a1b0847619be:NORMAL:127.0.0.1:39183|RBW], ReplicaUC[[DISK]DS-eb10a6dd-9a00-462c-adf9-f20dd959ebae:NORMAL:127.0.0.1:33171|FINALIZED]]} size 0
[junit4] 2> 638942 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39183 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-27955c67-62e1-456f-9abe-a1b0847619be:NORMAL:127.0.0.1:39183|RBW], ReplicaUC[[DISK]DS-eb10a6dd-9a00-462c-adf9-f20dd959ebae:NORMAL:127.0.0.1:33171|FINALIZED]]} size 0
[junit4] 2> 638978 WARN (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 639167 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
[junit4] 2> 639167 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 639167 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
[junit4] 2> 639269 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 639269 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 639343 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1593055026, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 639434 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@70155ef8[collection1] main]
[junit4] 2> 639458 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 639458 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 639458 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 639460 INFO (searcherExecutor-1514-thread-1-processing-n:127.0.0.1:33173_ x:collection1 c:collection1) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@70155ef8[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 639485 INFO (coreLoadExecutor-1513-thread-1-processing-n:127.0.0.1:33173_) [n:127.0.0.1:33173_ c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1594183392800276480
[junit4] 2> 639614 INFO (coreZkRegister-1508-thread-1-processing-n:127.0.0.1:33173_ x:collection1 c:collection1) [n:127.0.0.1:33173_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 639628 INFO (coreZkRegister-1508-thread-1-processing-n:127.0.0.1:33173_ x:collection1 c:collection1) [n:127.0.0.1:33173_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 639628 INFO (coreZkRegister-1508-thread-1-processing-n:127.0.0.1:33173_ x:collection1 c:collection1) [n:127.0.0.1:33173_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33173/collection1/
[junit4] 2> 639628 INFO (coreZkRegister-1508-thread-1-processing-n:127.0.0.1:33173_ x:collection1 c:collection1) [n:127.0.0.1:33173_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 639628 INFO (coreZkRegister-1508-thread-1-processing-n:127.0.0.1:33173_ x:collection1 c:collection1) [n:127.0.0.1:33173_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:33173/collection1/ has no replicas
[junit4] 2> 639628 INFO (coreZkRegister-1508-thread-1-processing-n:127.0.0.1:33173_ x:collection1 c:collection1) [n:127.0.0.1:33173_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4]
[...truncated too long message...]
llection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: start flush: applyAllDeletes=true
[junit4] 2> 221094 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: index before flush _6(6.6.3):C392/391:delGen=1
[junit4] 2> 221094 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [DW][coreCloseExecutor-203-thread-1]: startFullFlush
[junit4] 2> 221094 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [DW][coreCloseExecutor-203-thread-1]: coreCloseExecutor-203-thread-1 finishFullFlush success=true
[junit4] 2> 221094 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: apply all deletes during flush
[junit4] 2> 221094 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: now apply all deletes for all segments maxDoc=392
[junit4] 2> 221094 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [BD][coreCloseExecutor-203-thread-1]: applyDeletes: open segment readers took 0 msec
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [BD][coreCloseExecutor-203-thread-1]: applyDeletes: no segments; skipping
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [BD][coreCloseExecutor-203-thread-1]: prune sis=segments_6: _6(6.6.3):C392/391:delGen=1 minGen=31 packetCount=0
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [MS][coreCloseExecutor-203-thread-1]: updateMergeThreads ioThrottle=true targetMBPerSec=10240.0 MB/sec
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [MS][coreCloseExecutor-203-thread-1]: now merge
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [MS][coreCloseExecutor-203-thread-1]: index: _6(6.6.3):C392/391:delGen=1
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [MS][coreCloseExecutor-203-thread-1]: no more merges pending; now return
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: waitForMerges
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: waitForMerges done
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: commit: start
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: commit: enter lock
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: commit: now prepare
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: prepareCommit: flush
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: index before flush _6(6.6.3):C392/391:delGen=1
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [DW][coreCloseExecutor-203-thread-1]: startFullFlush
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: apply all deletes during flush
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: now apply all deletes for all segments maxDoc=392
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [BD][coreCloseExecutor-203-thread-1]: applyDeletes: open segment readers took 0 msec
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [BD][coreCloseExecutor-203-thread-1]: applyDeletes: no segments; skipping
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [BD][coreCloseExecutor-203-thread-1]: prune sis=segments_6: _6(6.6.3):C392/391:delGen=1 minGen=31 packetCount=0
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [DW][coreCloseExecutor-203-thread-1]: coreCloseExecutor-203-thread-1 finishFullFlush success=true
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: startCommit(): start
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: skip startCommit(): no changes pending
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreCloseExecutor-203-thread-1]: delete []
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: commit: pendingCommit == null; skip
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: commit: took 0.3 msec
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: commit: done
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: rollback
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: all running merges have aborted
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: rollback: done finish merges
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [DW][coreCloseExecutor-203-thread-1]: abort
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [DW][coreCloseExecutor-203-thread-1]: done abort success=true
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-203-thread-1]: rollback: infos=_6(6.6.3):C392/391:delGen=1
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreCloseExecutor-203-thread-1]: now checkpoint "_6(6.6.3):C392/391:delGen=1" [1 segments ; isCommit = false]
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreCloseExecutor-203-thread-1]: delete []
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreCloseExecutor-203-thread-1]: 0 msec to checkpoint
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreCloseExecutor-203-thread-1]: delete []
[junit4] 2> 221095 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreCloseExecutor-203-thread-1]: delete []
[junit4] 2> 221279 INFO (coreCloseExecutor-203-thread-1) [ x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
[junit4] 2> 221293 INFO (SUITE-TestSolrJErrorHandling-seed#[EA4D8BD788B670D5]-worker) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@3c8ec091{/solr,null,UNAVAILABLE}
[junit4] 2> 221330 INFO (SUITE-TestSolrJErrorHandling-seed#[EA4D8BD788B670D5]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestSolrJErrorHandling_EA4D8BD788B670D5-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene62): {}, docValues:{}, maxPointsInLeafNode=1255, maxMBSortInHeap=5.055302569082642, sim=RandomSimilarity(queryNorm=true,coord=no): {}, locale=sr, timezone=Asia/Baghdad
[junit4] 2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=117913912,total=345505792
[junit4] 2> NOTE: All tests run in this JVM: [TestDelegationTokenResponse, ConcurrentUpdateSolrClientBuilderTest, LessThanEqualToEvaluatorTest, TestSolrProperties, SelectWithEvaluatorsTest, HyperbolicCosineEvaluatorTest, TestFastInputStream, CommonAdminParamsTest, RoundEvaluatorTest, GetByIdTest, SolrDocumentTest, DocumentAnalysisResponseTest, TestCollectionAdminRequest, SubtractEvaluatorTest, MergeIndexesEmbeddedTest, CoalesceEvaluatorTest, UuidEvaluatorTest, ArcSineEvaluatorTest, SolrExampleStreamingBinaryTest, LBHttpSolrClientBuilderTest, SolrExampleJettyTest, SolrExampleEmbeddedTest, LargeVolumeBinaryJettyTest, SolrParamTest, QueryResponseTest, TestNamedListCodec, TestDocumentObjectBinder, SolrQueryTest, ModifiableSolrParamsTest, TestXMLEscaping, HttpClientUtilTest, TestUpdateRequestCodec, FieldAnalysisResponseTest, ClientUtilsTest, NamedListTest, AnlysisResponseBaseTest, CollectionAdminRequestRequiredParamsTest, TestSolrJErrorHandling]
[junit4] Completed [60/130 (1!)] on J1 in 78.30s, 4 tests, 1 failure <<< FAILURES!
[...truncated 45656 lines...]
[JENKINS] Lucene-Solr-NightlyTests-6.6 - Build # 35 - Still
unstable
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.6/35/
5 tests failed.
FAILED: org.apache.solr.cloud.CdcrReplicationHandlerTest.testReplicationWithBufferedUpdates
Error Message:
Timeout while trying to assert number of documents @ source_collection
Stack Trace:
java.lang.AssertionError: Timeout while trying to assert number of documents @ source_collection
at __randomizedtesting.SeedInfo.seed([D1CF6CAB31D9C539:2C63CB5744A59AE]:0)
at org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertNumDocs(BaseCdcrDistributedZkTest.java:271)
at org.apache.solr.cloud.CdcrReplicationHandlerTest.testReplicationWithBufferedUpdates(CdcrReplicationHandlerTest.java:233)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.AssertionError: expected:<50> but was:<60>
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertNumDocs(BaseCdcrDistributedZkTest.java:262)
... 42 more
FAILED: org.apache.solr.cloud.ShardSplitTest.testSplitAfterFailedSplit
Error Message:
expected:<1> but was:<2>
Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
at __randomizedtesting.SeedInfo.seed([D1CF6CAB31D9C539:2882FF040DAC88B3]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.ShardSplitTest.testSplitAfterFailedSplit(ShardSplitTest.java:280)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.ZkSolrClientTest.testMultipleWatchesAsync
Error Message:
Stack Trace:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([D1CF6CAB31D9C539:B979BF09A43DC4A7]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at org.apache.solr.cloud.ZkSolrClientTest.testMultipleWatchesAsync(ZkSolrClientTest.java:257)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.hdfs.StressHdfsTest.test
Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:35670
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:35670
at __randomizedtesting.SeedInfo.seed([D1CF6CAB31D9C539:599B53719F25A8C1]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:623)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:447)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:388)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1383)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1134)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at org.apache.solr.cloud.hdfs.StressHdfsTest.createAndDeleteCollection(StressHdfsTest.java:220)
at org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:103)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515)
... 51 more
FAILED: org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig
Error Message:
The max direct memory is likely too low. Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.
Stack Trace:
java.lang.RuntimeException: The max direct memory is likely too low. Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.
at __randomizedtesting.SeedInfo.seed([D1CF6CAB31D9C539:2660A580CC502F12]:0)
at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:312)
at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:288)
at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:228)
at org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig(HdfsDirectoryFactoryTest.java:108)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:694)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at org.apache.solr.store.blockcache.BlockCache.<init>(BlockCache.java:75)
at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:310)
... 42 more
Build Log:
[...truncated 12242 lines...]
[junit4] Suite: org.apache.solr.cloud.ZkSolrClientTest
[junit4] 2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ZkSolrClientTest_D1CF6CAB31D9C539-001/init-core-data-001
[junit4] 2> 668539 WARN (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=46 numCloses=46
[junit4] 2> 668539 INFO (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields
[junit4] 2> 668557 INFO (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 668573 INFO (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.SolrTestCaseJ4 ####initCore
[junit4] 2> 668574 INFO (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/lib, /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
[junit4] 2> 668793 WARN (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 668794 INFO (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.3
[junit4] 2> 668886 INFO (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.s.IndexSchema [null] Schema name=test
[junit4] 2> 669437 INFO (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 669508 INFO (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=true
[junit4] 2> 669689 INFO (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@70a36c78
[junit4] 2> 669707 INFO (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@70a36c78
[junit4] 2> 669707 INFO (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@70a36c78
[junit4] 2> 669736 INFO (coreLoadExecutor-663-thread-1) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 669736 INFO (coreLoadExecutor-663-thread-1) [ x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/lib, /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
[junit4] 2> 669819 WARN (coreLoadExecutor-663-thread-1) [ x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 669841 INFO (coreLoadExecutor-663-thread-1) [ x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.3
[junit4] 2> 669916 INFO (coreLoadExecutor-663-thread-1) [ x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 670333 INFO (coreLoadExecutor-663-thread-1) [ x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 670356 INFO (coreLoadExecutor-663-thread-1) [ x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1, trusted=true
[junit4] 2> 670356 INFO (coreLoadExecutor-663-thread-1) [ x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@70a36c78
[junit4] 2> 670356 INFO (coreLoadExecutor-663-thread-1) [ x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 670356 INFO (coreLoadExecutor-663-thread-1) [ x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ZkSolrClientTest_D1CF6CAB31D9C539-001/init-core-data-001/]
[junit4] 2> 670357 INFO (coreLoadExecutor-663-thread-1) [ x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@70a36c78
[junit4] 2> 670371 INFO (coreLoadExecutor-663-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@e931a6a
[junit4] 2> 670642 INFO (coreLoadExecutor-663-thread-1) [ x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 670642 INFO (coreLoadExecutor-663-thread-1) [ x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 670643 INFO (coreLoadExecutor-663-thread-1) [ x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 670643 INFO (coreLoadExecutor-663-thread-1) [ x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 670644 INFO (coreLoadExecutor-663-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1713991106, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 670645 INFO (coreLoadExecutor-663-thread-1) [ x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@79dffdc0[collection1] main]
[junit4] 2> 670645 INFO (coreLoadExecutor-663-thread-1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
[junit4] 2> 670715 INFO (coreLoadExecutor-663-thread-1) [ x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4] 2> 670852 INFO (coreLoadExecutor-663-thread-1) [ x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 670852 INFO (searcherExecutor-664-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
[junit4] 2> 670853 INFO (searcherExecutor-664-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
[junit4] 2> 670853 INFO (searcherExecutor-664-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
[junit4] 2> 670853 INFO (searcherExecutor-664-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
[junit4] 2> 670853 INFO (searcherExecutor-664-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
[junit4] 2> 670854 INFO (searcherExecutor-664-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
[junit4] 2> 670854 INFO (searcherExecutor-664-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
[junit4] 2> 670854 INFO (searcherExecutor-664-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
[junit4] 2> 670854 INFO (searcherExecutor-664-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
[junit4] 2> 670856 INFO (searcherExecutor-664-thread-1-processing-x:collection1) [ x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@79dffdc0[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 670874 INFO (coreLoadExecutor-663-thread-1) [ x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1594091586460844032
[junit4] 2> 670877 INFO (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.SolrTestCaseJ4 ####initCore end
[junit4] 2> 670897 INFO (TEST-ZkSolrClientTest.testReconnect-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testReconnect
[junit4] 2> 670897 INFO (TEST-ZkSolrClientTest.testReconnect-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 670898 INFO (Thread-3811) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 670910 INFO (Thread-3811) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 670956 ERROR (Thread-3811) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 671000 INFO (TEST-ZkSolrClientTest.testReconnect-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer start zk server on port:42353
[junit4] 2> 671092 ERROR (TEST-ZkSolrClientTest.testReconnect-seed#[D1CF6CAB31D9C539]) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 671093 INFO (TEST-ZkSolrClientTest.testReconnect-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42353 42353
[junit4] 2> 671397 INFO (Thread-3811) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42353 42353
[junit4] 2> 671497 INFO (TEST-ZkSolrClientTest.testReconnect-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 671497 WARN (zkCallback-933-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5ee3c059 name: ZooKeeperConnection Watcher:127.0.0.1:42353/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
[junit4] 2> 671497 WARN (zkCallback-933-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
[junit4] 2> 671497 INFO (Thread-3812) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:42353
[junit4] 2> 671497 INFO (Thread-3812) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 671499 ERROR (Thread-3812) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 671597 INFO (TEST-ZkSolrClientTest.testReconnect-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer start zk server on port:42353
[junit4] 2> 673058 WARN (zkCallback-933-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5ee3c059 name: ZooKeeperConnection Watcher:127.0.0.1:42353/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
[junit4] 2> 673059 WARN (zkCallback-933-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
[junit4] 2> 674874 WARN (TEST-ZkSolrClientTest.testReconnect-seed#[D1CF6CAB31D9C539]-SendThread(127.0.0.1:42353)) [ ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x161f59da21c0002 has expired
[junit4] 2> 674882 WARN (zkCallback-933-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5ee3c059 name: ZooKeeperConnection Watcher:127.0.0.1:42353/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
[junit4] 2> 674882 WARN (zkCallback-933-thread-1) [ ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
[junit4] 2> 674882 WARN (zkCallback-933-thread-1) [ ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
[junit4] 2> 674935 INFO (zkCallback-933-thread-1) [ ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
[junit4] 2> 674935 INFO (zkCallback-933-thread-1) [ ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
[junit4] 2> 674935 INFO (zkCallback-933-thread-1) [ ] o.a.s.c.c.ConnectionManager Connected:true
[junit4] 2> 675050 ERROR (TEST-ZkSolrClientTest.testReconnect-seed#[D1CF6CAB31D9C539]) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 675054 INFO (TEST-ZkSolrClientTest.testReconnect-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42353 42353
[junit4] 2> 675054 INFO (Thread-3812) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42353 42353
[junit4] 2> 675055 INFO (TEST-ZkSolrClientTest.testReconnect-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testReconnect
[junit4] 2> 675089 INFO (TEST-ZkSolrClientTest.testConnect-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testConnect
[junit4] 2> 675091 INFO (TEST-ZkSolrClientTest.testConnect-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 675091 INFO (Thread-3813) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 675091 INFO (Thread-3813) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 675092 ERROR (Thread-3813) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 675191 INFO (TEST-ZkSolrClientTest.testConnect-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer start zk server on port:40167
[junit4] 2> 675416 ERROR (TEST-ZkSolrClientTest.testConnect-seed#[D1CF6CAB31D9C539]) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 675417 INFO (TEST-ZkSolrClientTest.testConnect-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40167 40167
[junit4] 2> 675703 INFO (Thread-3813) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40167 40167
[junit4] 2> 675714 INFO (TEST-ZkSolrClientTest.testConnect-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testConnect
[junit4] 2> 675717 INFO (TEST-ZkSolrClientTest.testMultipleWatchesAsync-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testMultipleWatchesAsync
[junit4] 2> 675717 INFO (TEST-ZkSolrClientTest.testMultipleWatchesAsync-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 675717 INFO (Thread-3814) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 675717 INFO (Thread-3814) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 675730 ERROR (Thread-3814) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 675821 INFO (TEST-ZkSolrClientTest.testMultipleWatchesAsync-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer start zk server on port:43786
[junit4] 2> 679951 ERROR (TEST-ZkSolrClientTest.testMultipleWatchesAsync-seed#[D1CF6CAB31D9C539]) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 679952 INFO (TEST-ZkSolrClientTest.testMultipleWatchesAsync-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43786 43786
[junit4] 2> 680153 INFO (Thread-3814) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43786 43786
[junit4] 2> 680154 INFO (TEST-ZkSolrClientTest.testMultipleWatchesAsync-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testMultipleWatchesAsync
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ZkSolrClientTest -Dtests.method=testMultipleWatchesAsync -Dtests.seed=D1CF6CAB31D9C539 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/test-data/enwiki.random.lines.txt -Dtests.locale=hu-HU -Dtests.timezone=Etc/GMT-12 -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] FAILURE 4.46s J1 | ZkSolrClientTest.testMultipleWatchesAsync <<<
[junit4] > Throwable #1: java.lang.AssertionError
[junit4] > at __randomizedtesting.SeedInfo.seed([D1CF6CAB31D9C539:B979BF09A43DC4A7]:0)
[junit4] > at org.apache.solr.cloud.ZkSolrClientTest.testMultipleWatchesAsync(ZkSolrClientTest.java:257)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 680173 INFO (TEST-ZkSolrClientTest.testZkCmdExectutor-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testZkCmdExectutor
[junit4] 2> 680173 INFO (TEST-ZkSolrClientTest.testZkCmdExectutor-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 680174 INFO (Thread-3815) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 680174 INFO (Thread-3815) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 680175 ERROR (Thread-3815) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 680274 INFO (TEST-ZkSolrClientTest.testZkCmdExectutor-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer start zk server on port:43309
[junit4] 2> 695293 ERROR (TEST-ZkSolrClientTest.testZkCmdExectutor-seed#[D1CF6CAB31D9C539]) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 695293 INFO (TEST-ZkSolrClientTest.testZkCmdExectutor-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43309 43309
[junit4] 2> 695395 INFO (Thread-3815) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43309 43309
[junit4] 2> 695396 INFO (TEST-ZkSolrClientTest.testZkCmdExectutor-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testZkCmdExectutor
[junit4] 2> 695399 INFO (TEST-ZkSolrClientTest.testWatchChildren-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testWatchChildren
[junit4] 2> 695399 INFO (TEST-ZkSolrClientTest.testWatchChildren-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 695399 INFO (Thread-3816) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 695399 INFO (Thread-3816) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 695401 ERROR (Thread-3816) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 695499 INFO (TEST-ZkSolrClientTest.testWatchChildren-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer start zk server on port:45455
[junit4] 2> 696243 ERROR (TEST-ZkSolrClientTest.testWatchChildren-seed#[D1CF6CAB31D9C539]) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 696244 INFO (TEST-ZkSolrClientTest.testWatchChildren-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:45455 45455
[junit4] 2> 696308 INFO (Thread-3816) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:45455 45455
[junit4] 2> 696309 INFO (TEST-ZkSolrClientTest.testWatchChildren-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testWatchChildren
[junit4] 2> 696314 INFO (TEST-ZkSolrClientTest.testMakeRootNode-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testMakeRootNode
[junit4] 2> 696315 INFO (TEST-ZkSolrClientTest.testMakeRootNode-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 696315 INFO (Thread-3817) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 696315 INFO (Thread-3817) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 696328 ERROR (Thread-3817) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 696415 INFO (TEST-ZkSolrClientTest.testMakeRootNode-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer start zk server on port:41630
[junit4] 2> 696681 ERROR (TEST-ZkSolrClientTest.testMakeRootNode-seed#[D1CF6CAB31D9C539]) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 696714 INFO (TEST-ZkSolrClientTest.testMakeRootNode-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41630 41630
[junit4] 2> 697043 INFO (Thread-3817) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41630 41630
[junit4] 2> 697076 INFO (TEST-ZkSolrClientTest.testMakeRootNode-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testMakeRootNode
[junit4] 2> 697079 INFO (TEST-ZkSolrClientTest.testClean-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testClean
[junit4] 2> 697080 INFO (TEST-ZkSolrClientTest.testClean-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 697096 INFO (Thread-3818) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 697096 INFO (Thread-3818) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 697099 ERROR (Thread-3818) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 697196 INFO (TEST-ZkSolrClientTest.testClean-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer start zk server on port:37246
[junit4] 2> 697415 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x161f59e08560002, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 697435 ERROR (TEST-ZkSolrClientTest.testClean-seed#[D1CF6CAB31D9C539]) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 697435 INFO (TEST-ZkSolrClientTest.testClean-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37246 37246
[junit4] 2> 697776 INFO (Thread-3818) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37246 37246
[junit4] 2> 697777 INFO (TEST-ZkSolrClientTest.testClean-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testClean
[junit4] 2> 697792 INFO (TEST-ZkSolrClientTest.testSkipPathPartsOnMakePath-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testSkipPathPartsOnMakePath
[junit4] 2> 697792 INFO (TEST-ZkSolrClientTest.testSkipPathPartsOnMakePath-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 697794 INFO (Thread-3819) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 697794 INFO (Thread-3819) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 697800 ERROR (Thread-3819) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 697894 INFO (TEST-ZkSolrClientTest.testSkipPathPartsOnMakePath-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer start zk server on port:41345
[junit4] 2> 697962 ERROR (TEST-ZkSolrClientTest.testSkipPathPartsOnMakePath-seed#[D1CF6CAB31D9C539]) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 697963 INFO (TEST-ZkSolrClientTest.testSkipPathPartsOnMakePath-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41345 41345
[junit4] 2> 698198 INFO (Thread-3819) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41345 41345
[junit4] 2> 698199 INFO (TEST-ZkSolrClientTest.testSkipPathPartsOnMakePath-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testSkipPathPartsOnMakePath
[junit4] 2> 700201 INFO (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> 700201 INFO (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1723966582
[junit4] 2> 700201 INFO (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
[junit4] 2> 700201 INFO (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@648c1214: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@7d846b85
[junit4] 2> 700203 INFO (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jvm
[junit4] 2> 700203 INFO (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5ccad9fb: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@2b283596
[junit4] 2> 700205 INFO (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jetty
[junit4] 2> 700205 INFO (SUITE-ZkSolrClientTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5b581933: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@68a6bd3d
[junit4] 2> 700230 INFO (coreCloseExecutor-669-thread-1) [ x:collection1] o.a.s.c.SolrCore [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@5bb70944
[junit4] 2> 700324 INFO (coreCloseExecutor-669-thread-1) [ x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
[junit4] 2> 700324 INFO (coreCloseExecutor-669-thread-1) [ x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@92a4ee0: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@578cc24e
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ZkSolrClientTest_D1CF6CAB31D9C539-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene62): {}, docValues:{}, maxPointsInLeafNode=991, maxMBSortInHeap=7.97808786154156, sim=RandomSimilarity(queryNorm=true,coord=crazy): {}, locale=hu-HU, timezone=Etc/GMT-12
[junit4] 2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=309858200,total=504365056
[junit4] 2> NOTE: All tests run in this JVM: [EchoParamsTest, TestHighFrequencyDictionaryFactory, TestCryptoKeys, DocValuesTest, MultiTermTest, BlobRepositoryCloudTest, TestUpdate, HdfsChaosMonkeySafeLeaderTest, TestNestedDocsSort, HighlighterTest, TestSolrCLIRunExample, HighlighterMaxOffsetTest, AddBlockUpdateTest, SolrSlf4jReporterTest, SuggestComponentTest, ZkSolrClientTest]
[junit4] Completed [58/714 (1!)] on J1 in 31.94s, 8 tests, 1 failure <<< FAILURES!
[...truncated 48 lines...]
[junit4] Suite: org.apache.solr.cloud.ShardSplitTest
[junit4] 2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_D1CF6CAB31D9C539-001/init-core-data-001
[junit4] 2> 71494 WARN (SUITE-ShardSplitTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=4 numCloses=4
[junit4] 2> 71494 INFO (SUITE-ShardSplitTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields
[junit4] 2> 71496 INFO (SUITE-ShardSplitTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
[junit4] 2> 71496 INFO (SUITE-ShardSplitTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
[junit4] 2> 71639 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 71698 INFO (Thread-13) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 71700 INFO (Thread-13) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 72347 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkTestServer start zk server on port:38692
[junit4] 2> 73476 ERROR (Thread-13) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 74912 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 75059 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 75064 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 75118 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 75155 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 75191 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 75265 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 75329 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 75410 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 75413 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 75548 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 77134 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_D1CF6CAB31D9C539-001/control-001/cores/collection1
[junit4] 2> 78501 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 78750 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@255ae242{/,null,AVAILABLE}
[junit4] 2> 78806 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@453c5d5f{HTTP/1.1,[http/1.1]}{127.0.0.1:34701}
[junit4] 2> 78806 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.e.j.s.Server Started @89774ms
[junit4] 2> 78807 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_D1CF6CAB31D9C539-001/tempDir-001/control/data, hostContext=/, hostPort=34701, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_D1CF6CAB31D9C539-001/control-001/cores}
[junit4] 2> 78896 ERROR (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 78896 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solrâ„¢ version 6.6.3
[junit4] 2> 78896 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 78897 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 78930 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-05T09:53:31.699Z
[junit4] 2> 78931 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
[junit4] 2> 79115 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x161f5947e930002, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 79141 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 79156 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_D1CF6CAB31D9C539-001/control-001/solr.xml
[junit4] 2> 79403 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 79423 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38692/solr
[junit4] 2> 80494 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [n:127.0.0.1:34701_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 80529 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [n:127.0.0.1:34701_ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34701_
[junit4] 2> 80532 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [n:127.0.0.1:34701_ ] o.a.s.c.Overseer Overseer (id=99630684906389508-127.0.0.1:34701_-n_0000000000) starting
[junit4] 2> 81074 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [n:127.0.0.1:34701_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34701_
[junit4] 2> 81223 INFO (zkCallback-31-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 82521 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [n:127.0.0.1:34701_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_D1CF6CAB31D9C539-001/control-001/cores
[junit4] 2> 82522 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [n:127.0.0.1:34701_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 82698 INFO (coreLoadExecutor-154-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 83167 INFO (OverseerStateUpdate-99630684906389508-127.0.0.1:34701_-n_0000000000) [n:127.0.0.1:34701_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 84173 WARN (coreLoadExecutor-154-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ c:control_collection x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 84173 INFO (coreLoadExecutor-154-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ c:control_collection x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.3
[junit4] 2> 84260 INFO (coreLoadExecutor-154-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 84605 INFO (coreLoadExecutor-154-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ c:control_collection x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 84874 INFO (coreLoadExecutor-154-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ c:control_collection x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection, trusted=true
[junit4] 2> 84910 INFO (coreLoadExecutor-154-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ c:control_collection x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 84910 INFO (coreLoadExecutor-154-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ c:control_collection x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_D1CF6CAB31D9C539-001/control-001/cores/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_D1CF6CAB31D9C539-001/control-001/cores/collection1/data/]
[junit4] 2> 84946 INFO (coreLoadExecutor-154-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ c:control_collection x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5105c032
[junit4] 2> 85022 INFO (coreLoadExecutor-154-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ c:control_collection x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=588641108, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 85326 WARN (coreLoadExecutor-154-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ c:control_collection x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 85982 INFO (coreLoadExecutor-154-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ c:control_collection x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 85982 INFO (coreLoadExecutor-154-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ c:control_collection x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 85983 INFO (coreLoadExecutor-154-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ c:control_collection x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 85983 INFO (coreLoadExecutor-154-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ c:control_collection x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 85984 INFO (coreLoadExecutor-154-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ c:control_collection x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=33, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 86036 INFO (coreLoadExecutor-154-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ c:control_collection x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@56b0aba[collection1] main]
[junit4] 2> 86116 INFO (coreLoadExecutor-154-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 86124 INFO (coreLoadExecutor-154-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 86124 INFO (coreLoadExecutor-154-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ c:control_collection x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 86230 INFO (searcherExecutor-155-thread-1-processing-n:127.0.0.1:34701_ x:collection1 c:control_collection) [n:127.0.0.1:34701_ c:control_collection x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@56b0aba[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 86231 INFO (coreLoadExecutor-154-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ c:control_collection x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1594090973070098432
[junit4] 2> 86380 INFO (coreZkRegister-147-thread-1-processing-n:127.0.0.1:34701_ x:collection1 c:control_collection) [n:127.0.0.1:34701_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 86380 INFO (coreZkRegister-147-thread-1-processing-n:127.0.0.1:34701_ x:collection1 c:control_collection) [n:127.0.0.1:34701_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 86393 INFO (coreZkRegister-147-thread-1-processing-n:127.0.0.1:34701_ x:collection1 c:control_collection) [n:127.0.0.1:34701_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:34701/collection1/
[junit4] 2> 86393 INFO (coreZkRegister-147-thread-1-processing-n:127.0.0.1:34701_ x:collection1 c:control_collection) [n:127.0.0.1:34701_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 86393 INFO (coreZkRegister-147-thread-1-processing-n:127.0.0.1:34701_ x:collection1 c:control_collection) [n:127.0.0.1:34701_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:34701/collection1/ has no replicas
[junit4] 2> 86393 INFO (coreZkRegister-147-thread-1-processing-n:127.0.0.1:34701_ x:collection1 c:control_collection) [n:127.0.0.1:34701_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 86672 INFO (coreZkRegister-147-thread-1-processing-n:127.0.0.1:34701_ x:collection1 c:control_collection) [n:127.0.0.1:34701_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:34701/collection1/ shard1
[junit4] 2> 86853 INFO (coreZkRegister-147-thread-1-processing-n:127.0.0.1:34701_ x:collection1 c:control_collection) [n:127.0.0.1:34701_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 87211 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 87226 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38692/solr ready
[junit4] 2> 87227 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 87227 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
[junit4] 2> 88226 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_D1CF6CAB31D9C539-001/shard-1-001/cores/collection1
[junit4] 2> 88227 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_D1CF6CAB31D9C539-001/shard-1-001
[junit4] 2> 88262 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 88343 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4ddb013b{/,null,AVAILABLE}
[junit4] 2> 88344 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@7a5f79ba{HTTP/1.1,[http/1.1]}{127.0.0.1:36188}
[junit4] 2> 88344 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.e.j.s.Server Started @99311ms
[junit4] 2> 88344 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_D1CF6CAB31D9C539-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=36188, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_D1CF6CAB31D9C539-001/shard-1-001/cores}
[junit4] 2> 88344 ERROR (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 88379 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solrâ„¢ version 6.6.3
[junit4] 2> 88379 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 88379 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 88379 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-05T09:53:41.180Z
[junit4] 2> 88460 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 88460 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_D1CF6CAB31D9C539-001/shard-1-001/solr.xml
[junit4] 2> 88570 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 88586 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38692/solr
[junit4] 2> 88807 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [n:127.0.0.1:36188_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 88829 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [n:127.0.0.1:36188_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 88969 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [n:127.0.0.1:36188_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36188_
[junit4] 2> 88992 INFO (zkCallback-31-thread-1-processing-n:127.0.0.1:34701_) [n:127.0.0.1:34701_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 89041 INFO (zkCallback-41-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 89114 INFO (zkCallback-35-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 89434 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [n:127.0.0.1:36188_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_D1CF6CAB31D9C539-001/shard-1-001/cores
[junit4] 2> 89434 INFO (TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D1CF6CAB31D9C539]) [n:127.0.0.1:36188_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 89503 INFO (coreLoadExecutor-165-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 89602 INFO (OverseerStateUpdate-99630684906389508-127.0.0.1:34701_-n_0000000000) [n:127.0.0.1:34701_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
[junit4] 2> 89785 INFO (zkCallback-41-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 90694 WARN (coreLoadExecutor-165-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ c:collection1 x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 90747 INFO (coreLoadExecutor-165-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.3
[junit4] 2> 90857 INFO (coreLoadExecutor-165-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 91506 INFO (coreLoadExecutor-165-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 91788 INFO (coreLoadExecutor-165-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
[junit4] 2> 91788 INFO (coreLoadExecutor-165-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ c:collection1 x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 91788 INFO (coreLoadExecutor-165-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_D1CF6CAB31D9C539-001/shard-1-001/cores/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_D1CF6CAB31D9C539-001/shard-1-001/cores/collection1/data/]
[junit4] 2> 91789 INFO (coreLoadExecutor-165-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5105c032
[junit4] 2> 91803 INFO (coreLoadExecutor-165-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=588641108, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 91994 WARN (coreLoadExecutor-165-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 92913 INFO (coreLoadExecutor-165-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 92913 INFO (coreLoadExecutor-165-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 92914 INFO (coreLoadExecutor-165-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 92914 INFO (coreLoadExecutor-165-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 93040 INFO (coreLoadExecutor-165-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=33, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 93148 INFO (coreLoadExecutor-165-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@de20b51[collection1] main]
[junit4] 2> 93150 INFO (coreLoadExecutor-165-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 93150 INFO (coreLoadExecutor-165-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 93151 INFO (coreLoadExecutor-165-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 93313 INFO (searcherExecutor-166-thread-1-processing-n:127.0.0.1:36188_ x:collection1 c:collection1) [n:127.0.0.1:36188_ c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@de20b51[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 93314 INFO (coreLoadExecutor-165-thread-1-processing-n:127.0.0.1:36188_) [n:127.0.0.1:36188_ c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1594090980497162240
[junit4] 2> 93380 INFO (coreZkRegister-160-thread-1-processing-n:127.0.0.1:36188_ x:collection1 c:collection1) [n:127.0.0.1:36188_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to co
[...truncated too long message...]
... 51 more
[junit4] 2> 7744673 WARN (SUITE-StressHdfsTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
[junit4] 2> 7744781 INFO (SUITE-StressHdfsTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
[junit4] 2> 7744819 WARN (1806785745@qtp-1893282705-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41563) [ ] o.a.h.h.HttpServer2 HttpServer Acceptor: isRunning is false. Rechecking.
[junit4] 2> 7744819 WARN (1806785745@qtp-1893282705-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41563) [ ] o.a.h.h.HttpServer2 HttpServer Acceptor: isRunning is false
[junit4] 2> 7744967 WARN (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D1CF6CAB31D9C539-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D1CF6CAB31D9C539-001/tempDir-001/hdfsBaseDir/data/data4/]] heartbeating to lucene2-us-west.apache.org/127.0.0.1:33234) [ ] o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager interrupted
[junit4] 2> 7744967 WARN (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D1CF6CAB31D9C539-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D1CF6CAB31D9C539-001/tempDir-001/hdfsBaseDir/data/data4/]] heartbeating to lucene2-us-west.apache.org/127.0.0.1:33234) [ ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-668678065-127.0.0.1-1520250732384 (Datanode Uuid 15977575-7969-4b1b-a518-5c3b332a4218) service to lucene2-us-west.apache.org/127.0.0.1:33234
[junit4] 2> 7745042 WARN (SUITE-StressHdfsTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
[junit4] 2> 7745374 INFO (SUITE-StressHdfsTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
[junit4] 2> 7745374 WARN (961297972@qtp-1371905720-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33390) [ ] o.a.h.h.HttpServer2 HttpServer Acceptor: isRunning is false. Rechecking.
[junit4] 2> 7745441 WARN (961297972@qtp-1371905720-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33390) [ ] o.a.h.h.HttpServer2 HttpServer Acceptor: isRunning is false
[junit4] 2> 7745705 WARN (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D1CF6CAB31D9C539-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D1CF6CAB31D9C539-001/tempDir-001/hdfsBaseDir/data/data2/]] heartbeating to lucene2-us-west.apache.org/127.0.0.1:33234) [ ] o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager interrupted
[junit4] 2> 7745705 WARN (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D1CF6CAB31D9C539-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D1CF6CAB31D9C539-001/tempDir-001/hdfsBaseDir/data/data2/]] heartbeating to lucene2-us-west.apache.org/127.0.0.1:33234) [ ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-668678065-127.0.0.1-1520250732384 (Datanode Uuid dcf412c7-0bbc-427c-931b-294d1a7836eb) service to lucene2-us-west.apache.org/127.0.0.1:33234
[junit4] 2> 7746040 INFO (SUITE-StressHdfsTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@lucene2-us-west.apache.org:0
[junit4] 2> 7746040 WARN (800265047@qtp-235526813-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@lucene2-us-west.apache.org:46096) [ ] o.a.h.h.HttpServer2 HttpServer Acceptor: isRunning is false. Rechecking.
[junit4] 2> 7746040 WARN (800265047@qtp-235526813-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@lucene2-us-west.apache.org:46096) [ ] o.a.h.h.HttpServer2 HttpServer Acceptor: isRunning is false
[junit4] 2> 7746142 INFO (SUITE-StressHdfsTest-seed#[D1CF6CAB31D9C539]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D1CF6CAB31D9C539-001
[junit4] 2> Mar 05, 2018 12:01:19 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 34 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene62): {range_facet_l_dv=Lucene50(blocksize=128), multiDefault=Lucene50(blocksize=128), intDefault=PostingsFormat(name=MockRandom), id=Lucene50(blocksize=128), range_facet_i_dv=Lucene50(blocksize=128), text=PostingsFormat(name=LuceneVarGapFixedInterval), range_facet_l=Lucene50(blocksize=128), timestamp=Lucene50(blocksize=128), txt_t=PostingsFormat(name=LuceneVarGapFixedInterval)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Memory), _version_=DocValuesFormat(name=Lucene54), range_facet_i_dv=DocValuesFormat(name=Lucene54), intDvoDefault=DocValuesFormat(name=Memory), timestamp=DocValuesFormat(name=Lucene54)}, maxPointsInLeafNode=812, maxMBSortInHeap=6.408973382775418, sim=RandomSimilarity(queryNorm=true,coord=crazy): {}, locale=zh-HK, timezone=Africa/Johannesburg
[junit4] 2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=115619736,total=534773760
[junit4] 2> NOTE: All tests run in this JVM: [EchoParamsTest, TestHighFrequencyDictionaryFactory, TestCryptoKeys, DocValuesTest, MultiTermTest, BlobRepositoryCloudTest, TestUpdate, HdfsChaosMonkeySafeLeaderTest, TestNestedDocsSort, HighlighterTest, TestSolrCLIRunExample, HighlighterMaxOffsetTest, AddBlockUpdateTest, SolrSlf4jReporterTest, SuggestComponentTest, ZkSolrClientTest, ConjunctionSolrSpellCheckerTest, SolrInfoMBeanTest, TestBlendedInfixSuggestions, TermsComponentTest, DeleteReplicaTest, TestLuceneMatchVersion, HLLUtilTest, InfoHandlerTest, TestQueryUtils, TestReloadAndDeleteDocs, TestStressVersions, ShufflingReplicaListTransformerTest, SystemInfoHandlerTest, TestPointFields, WrapperMergePolicyFactoryTest, TestFieldCacheVsDocValues, TestDistributedGrouping, TestExactStatsCache, DistribCursorPagingTest, SecurityConfHandlerTest, DistributedIntervalFacetingTest, SolrMetricReporterTest, TestSurroundQueryParser, TestSolrCoreParser, DistributedFacetPivotSmallAdvancedTest, BasicAuthStandaloneTest, TestDocBasedVersionConstraints, FileUtilsTest, DateRangeFieldTest, LeaderInitiatedRecoveryOnShardRestartTest, TermVectorComponentTest, SpellingQueryConverterTest, AssignTest, CdcrBootstrapTest, DateMathParserTest, OverseerStatusTest, TestDownShardTolerantSearch, TestComplexPhraseQParserPlugin, TestCoreAdminApis, SuggesterFSTTest, ShardRoutingTest, ExplicitHLLTest, TestCoreContainer, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestInPlaceUpdatesDistrib, TestFilteredDocIdSet, TestRecoveryHdfs, SolrGraphiteReporterTest, TestEmbeddedSolrServerConstructors, TestSolrQueryResponse, TestPayloadCheckQParserPlugin, TestNumericTerms32, HdfsRecoverLeaseTest, DistributedMLTComponentTest, JsonValidatorTest, TestManagedResourceStorage, DistributedFacetExistsSmallTest, BlockJoinFacetSimpleTest, DirectSolrSpellCheckerTest, TestRTGBase, SharedFSAutoReplicaFailoverTest, DeleteInactiveReplicaTest, MissingSegmentRecoveryTest, TestPivotHelperCode, TestDocTermOrds, CollectionStateFormat2Test, ZkCLITest, TestSolrCloudWithSecureImpersonation, LoggingHandlerTest, TestSQLHandler, TestQueryTypes, SoftAutoCommitTest, TestLMDirichletSimilarityFactory, SSLMigrationTest, TestFieldTypeCollectionResource, BigEndianAscendingWordDeserializerTest, TestJsonRequest, TestSolrConfigHandler, TestPartialUpdateDeduplication, MoreLikeThisHandlerTest, DocumentBuilderTest, OverseerCollectionConfigSetProcessorTest, TestApiFramework, TestReplicationHandlerBackup, TestCSVResponseWriter, TestReplicaProperties, SliceStateTest, TestLRUCache, SimplePostToolTest, CoreAdminCreateDiscoverTest, TolerantUpdateProcessorTest, TestBlobHandler, TestPerFieldSimilarityClassic, ConnectionReuseTest, TestMinMaxOnMultiValuedField, SolrCLIZkUtilsTest, TestLeaderElectionWithEmptyReplica, SchemaVersionSpecificBehaviorTest, ExitableDirectoryReaderTest, IndexSchemaTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, SyncSliceTest, OpenCloseCoreStressTest, OverseerTest, LeaderElectionIntegrationTest, TestReplicationHandler, TestStressReorder, TestRangeQuery, HttpPartitionTest, TestCloudRecovery, TestExclusionRuleCollectionAccess, TestHdfsCloudBackupRestore, TestLocalFSCloudBackupRestore, TestOnReconnectListenerSupport, TestRebalanceLeaders, TestSegmentSorting, TestStressLiveNodes, TestTolerantUpdateProcessorRandomCloud, TriLevelCompositeIdRoutingTest, HdfsWriteToMultipleCollectionsTest, StressHdfsTest]
[junit4] Completed [593/714 (5!)] on J1 in 555.92s, 1 test, 1 error <<< FAILURES!
[...truncated 46679 lines...]
[JENKINS] Lucene-Solr-NightlyTests-6.6 - Build # 34 - Failure
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.6/34/
5 tests failed.
FAILED: org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.testCollectionsAPI
Error Message:
Something is broken in the assert for no shards using the same indexDir - probably something was changed in the attributes published in the MBean of SolrCore : {}
Stack Trace:
java.lang.AssertionError: Something is broken in the assert for no shards using the same indexDir - probably something was changed in the attributes published in the MBean of SolrCore : {}
at __randomizedtesting.SeedInfo.seed([78CE862CA70D20DD:30BBF298A13E0F48]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.checkNoTwoShardsUseTheSameIndexDir(CollectionsAPIDistributedZkTest.java:646)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:524)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.handler.admin.MetricsHandlerTest.test
Error Message:
Stack Trace:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([78CE862CA70D20DD:F09AB9F609F14D25]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at org.junit.Assert.assertNotNull(Assert.java:537)
at org.apache.solr.handler.admin.MetricsHandlerTest.test(MetricsHandlerTest.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.update.TestInPlaceUpdatesDistrib.test
Error Message:
Error from server at https://127.0.0.1:46742/t_r/y/collection1: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /t_r/y/collection1/update. Reason: <pre> java.lang.OutOfMemoryError: GC overhead limit exceeded</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.14.v20161028</a><hr/> </body> </html>
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:46742/t_r/y/collection1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /t_r/y/collection1/update. Reason:
<pre> java.lang.OutOfMemoryError: GC overhead limit exceeded</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.14.v20161028</a><hr/>
</body>
</html>
at __randomizedtesting.SeedInfo.seed([78CE862CA70D20DD:F09AB9F609F14D25]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:580)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:177)
at org.apache.solr.update.TestInPlaceUpdatesDistrib.addDocAndGetVersion(TestInPlaceUpdatesDistrib.java:1078)
at org.apache.solr.update.TestInPlaceUpdatesDistrib.buildRandomIndex(TestInPlaceUpdatesDistrib.java:1123)
at org.apache.solr.update.TestInPlaceUpdatesDistrib.docValuesUpdateTest(TestInPlaceUpdatesDistrib.java:327)
at org.apache.solr.update.TestInPlaceUpdatesDistrib.test(TestInPlaceUpdatesDistrib.java:154)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: junit.framework.TestSuite.org.apache.solr.update.TestInPlaceUpdatesDistrib
Error Message:
2 threads leaked from SUITE scope at org.apache.solr.update.TestInPlaceUpdatesDistrib: 1) Thread[id=63443, name=searcherExecutor-10694-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2) Thread[id=63409, name=searcherExecutor-10683-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.update.TestInPlaceUpdatesDistrib:
1) Thread[id=63443, name=searcherExecutor-10694-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2) Thread[id=63409, name=searcherExecutor-10683-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([78CE862CA70D20DD]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.update.TestInPlaceUpdatesDistrib
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=63443, name=searcherExecutor-10694-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2) Thread[id=63409, name=searcherExecutor-10683-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=63443, name=searcherExecutor-10694-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2) Thread[id=63409, name=searcherExecutor-10683-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([78CE862CA70D20DD]:0)
Build Log:
[...truncated 12004 lines...]
[junit4] Suite: org.apache.solr.handler.admin.MetricsHandlerTest
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.handler.admin.MetricsHandlerTest_78CE862CA70D20DD-001/init-core-data-001
[junit4] 2> 242026 WARN (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=6 numCloses=6
[junit4] 2> 242026 INFO (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.SolrTestCaseJ4 Using TrieFields
[junit4] 2> 242028 INFO (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 242029 INFO (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.SolrTestCaseJ4 ####initCore
[junit4] 2> 242030 INFO (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
[junit4] 2> 242081 WARN (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 242083 INFO (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.3
[junit4] 2> 242114 INFO (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.s.IndexSchema [null] Schema name=test
[junit4] 2> 243050 INFO (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 243062 INFO (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=true
[junit4] 2> 243093 WARN (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.m.r.SolrJmxReporter No JMX server found. Not exposing Solr metrics via JMX.
[junit4] 2> 243093 WARN (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.m.r.SolrJmxReporter No JMX server found. Not exposing Solr metrics via JMX.
[junit4] 2> 243093 WARN (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.m.r.SolrJmxReporter No JMX server found. Not exposing Solr metrics via JMX.
[junit4] 2> 243094 INFO (coreLoadExecutor-1476-thread-1) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 243096 INFO (coreLoadExecutor-1476-thread-1) [ x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
[junit4] 2> 243133 WARN (coreLoadExecutor-1476-thread-1) [ x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 243135 INFO (coreLoadExecutor-1476-thread-1) [ x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.3
[junit4] 2> 243165 INFO (coreLoadExecutor-1476-thread-1) [ x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 243306 INFO (coreLoadExecutor-1476-thread-1) [ x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 243314 INFO (coreLoadExecutor-1476-thread-1) [ x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1, trusted=true
[junit4] 2> 243314 WARN (coreLoadExecutor-1476-thread-1) [ x:collection1] o.a.s.m.r.SolrJmxReporter No JMX server found. Not exposing Solr metrics via JMX.
[junit4] 2> 243314 INFO (coreLoadExecutor-1476-thread-1) [ x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 243315 INFO (coreLoadExecutor-1476-thread-1) [ x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.handler.admin.MetricsHandlerTest_78CE862CA70D20DD-001/init-core-data-001/]
[junit4] 2> 243317 INFO (coreLoadExecutor-1476-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=35, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.41176649114580666]
[junit4] 2> 243383 INFO (coreLoadExecutor-1476-thread-1) [ x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 243383 INFO (coreLoadExecutor-1476-thread-1) [ x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 243384 INFO (coreLoadExecutor-1476-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=6, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.0146484375, noCFSRatio=0.4752696713000235]
[junit4] 2> 243396 INFO (coreLoadExecutor-1476-thread-1) [ x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@75a7cdad[collection1] main]
[junit4] 2> 243397 INFO (coreLoadExecutor-1476-thread-1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
[junit4] 2> 243419 INFO (coreLoadExecutor-1476-thread-1) [ x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4] 2> 243456 INFO (coreLoadExecutor-1476-thread-1) [ x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 243456 INFO (searcherExecutor-1477-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
[junit4] 2> 243457 INFO (searcherExecutor-1477-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
[junit4] 2> 243457 INFO (searcherExecutor-1477-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
[junit4] 2> 243457 INFO (searcherExecutor-1477-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
[junit4] 2> 243457 INFO (searcherExecutor-1477-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
[junit4] 2> 243460 INFO (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.SolrTestCaseJ4 ####initCore end
[junit4] 2> 243461 INFO (searcherExecutor-1477-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
[junit4] 2> 243462 INFO (searcherExecutor-1477-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
[junit4] 2> 243463 INFO (searcherExecutor-1477-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
[junit4] 2> 243463 INFO (searcherExecutor-1477-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
[junit4] 2> 243464 INFO (searcherExecutor-1477-thread-1-processing-x:collection1) [ x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@75a7cdad[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 243464 INFO (TEST-MetricsHandlerTest.testCompact-seed#[78CE862CA70D20DD]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testCompact
[junit4] 2> 243553 INFO (TEST-MetricsHandlerTest.testCompact-seed#[78CE862CA70D20DD]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testCompact
[junit4] 2> 243555 INFO (TEST-MetricsHandlerTest.test-seed#[78CE862CA70D20DD]) [ ] o.a.s.SolrTestCaseJ4 ###Starting test
[junit4] 2> 243581 INFO (TEST-MetricsHandlerTest.test-seed#[78CE862CA70D20DD]) [ ] o.a.s.SolrTestCaseJ4 ###Ending test
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=MetricsHandlerTest -Dtests.method=test -Dtests.seed=78CE862CA70D20DD -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/test-data/enwiki.random.lines.txt -Dtests.locale=es-VE -Dtests.timezone=Pacific/Kiritimati -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] FAILURE 0.03s J2 | MetricsHandlerTest.test <<<
[junit4] > Throwable #1: java.lang.AssertionError
[junit4] > at __randomizedtesting.SeedInfo.seed([78CE862CA70D20DD:F09AB9F609F14D25]:0)
[junit4] > at org.apache.solr.handler.admin.MetricsHandlerTest.test(MetricsHandlerTest.java:54)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 243584 INFO (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> 243584 INFO (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=178690372
[junit4] 2> 243584 INFO (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
[junit4] 2> 243584 INFO (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@25f00628: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / null
[junit4] 2> 243584 INFO (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jvm
[junit4] 2> 243584 INFO (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2b4472db: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / null
[junit4] 2> 243584 INFO (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jetty
[junit4] 2> 243584 INFO (SUITE-MetricsHandlerTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4ff1b81b: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / null
[junit4] 2> 243585 INFO (coreCloseExecutor-1481-thread-1) [ x:collection1] o.a.s.c.SolrCore [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@1c8dd5cd
[junit4] 2> 243592 INFO (coreCloseExecutor-1481-thread-1) [ x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
[junit4] 2> 243592 INFO (coreCloseExecutor-1481-thread-1) [ x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@30a1bdef: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / null
[junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.handler.admin.MetricsHandlerTest_78CE862CA70D20DD-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene62): {}, docValues:{}, maxPointsInLeafNode=1765, maxMBSortInHeap=6.713281055499272, sim=RandomSimilarity(queryNorm=false,coord=no): {}, locale=es-VE, timezone=Pacific/Kiritimati
[junit4] 2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=205503176,total=516947968
[junit4] 2> NOTE: All tests run in this JVM: [SolrJmxReporterCloudTest, RecoveryZkTest, TestCollationFieldDocValues, TestUnifiedSolrHighlighter, PluginInfoTest, TestLMJelinekMercerSimilarityFactory, DeleteLastCustomShardedReplicaTest, TestTestInjection, FieldMutatingUpdateProcessorTest, TestExceedMaxTermLength, TestAnalyzedSuggestions, TestRestManager, TestClusterProperties, TestMiniSolrCloudClusterSSL, TestPostingsSolrHighlighter, SolrCoreCheckLockOnStartupTest, OutOfBoxZkACLAndCredentialsProvidersTest, LeaderElectionTest, TestImpersonationWithHadoopAuth, TestSolrJ, TestSolrDynamicMBean, MoveReplicaTest, TestTrackingShardHandlerFactory, OverseerModifyCollectionTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestCloudDeleteByQuery, TestRecovery, TestCopyFieldCollectionResource, QueryParsingTest, PrimUtilsTest, ClassificationUpdateProcessorFactoryTest, TestRawResponseWriter, SharedFSAutoReplicaFailoverUtilsTest, TestDynamicLoading, TestStressReorder, PeerSyncWithIndexFingerprintCachingTest, CloudExitableDirectoryReaderTest, HttpSolrCallGetCoreTest, MetricsHandlerTest]
[junit4] Completed [65/714 (1!)] on J2 in 1.59s, 2 tests, 1 failure <<< FAILURES!
[...truncated 1579 lines...]
[junit4] Suite: org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_78CE862CA70D20DD-001/init-core-data-001
[junit4] 2> 3676714 WARN (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=48 numCloses=48
[junit4] 2> 3676715 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.SolrTestCaseJ4 Using TrieFields
[junit4] 2> 3676716 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 3676717 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_78CE862CA70D20DD-001/tempDir-001
[junit4] 2> 3676717 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 3676717 INFO (Thread-38667) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 3676717 INFO (Thread-38667) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 3676724 ERROR (Thread-38667) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 3676817 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:51101
[junit4] 2> 3676824 INFO (jetty-launcher-4480-thread-1) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 3676829 INFO (jetty-launcher-4480-thread-3) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 3676829 INFO (jetty-launcher-4480-thread-2) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 3676829 INFO (jetty-launcher-4480-thread-4) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 3676837 INFO (jetty-launcher-4480-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@32e9e77b{/solr,null,AVAILABLE}
[junit4] 2> 3676843 INFO (jetty-launcher-4480-thread-4) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1eac8a36{/solr,null,AVAILABLE}
[junit4] 2> 3676845 INFO (jetty-launcher-4480-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@164b0e94{/solr,null,AVAILABLE}
[junit4] 2> 3676849 INFO (jetty-launcher-4480-thread-4) [ ] o.e.j.s.AbstractConnector Started ServerConnector@13679d23{HTTP/1.1,[http/1.1]}{127.0.0.1:53244}
[junit4] 2> 3676849 INFO (jetty-launcher-4480-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@7e8b00b0{HTTP/1.1,[http/1.1]}{127.0.0.1:35293}
[junit4] 2> 3676849 INFO (jetty-launcher-4480-thread-4) [ ] o.e.j.s.Server Started @3679779ms
[junit4] 2> 3676849 INFO (jetty-launcher-4480-thread-1) [ ] o.e.j.s.Server Started @3679779ms
[junit4] 2> 3676849 INFO (jetty-launcher-4480-thread-4) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53244}
[junit4] 2> 3676850 INFO (jetty-launcher-4480-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35293}
[junit4] 2> 3676850 ERROR (jetty-launcher-4480-thread-4) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3676850 INFO (jetty-launcher-4480-thread-4) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solrâ„¢ version 6.6.3
[junit4] 2> 3676850 INFO (jetty-launcher-4480-thread-4) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3676850 INFO (jetty-launcher-4480-thread-4) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 3676850 INFO (jetty-launcher-4480-thread-4) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-04T09:41:37Z
[junit4] 2> 3676850 ERROR (jetty-launcher-4480-thread-1) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3676851 INFO (jetty-launcher-4480-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solrâ„¢ version 6.6.3
[junit4] 2> 3676851 INFO (jetty-launcher-4480-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3676851 INFO (jetty-launcher-4480-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 3676851 INFO (jetty-launcher-4480-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-04T09:41:37.001Z
[junit4] 2> 3676850 INFO (jetty-launcher-4480-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@19072b2f{HTTP/1.1,[http/1.1]}{127.0.0.1:38931}
[junit4] 2> 3676852 INFO (jetty-launcher-4480-thread-2) [ ] o.e.j.s.Server Started @3679782ms
[junit4] 2> 3676852 INFO (jetty-launcher-4480-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38931}
[junit4] 2> 3676852 ERROR (jetty-launcher-4480-thread-2) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3676852 INFO (jetty-launcher-4480-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solrâ„¢ version 6.6.3
[junit4] 2> 3676852 INFO (jetty-launcher-4480-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3676852 INFO (jetty-launcher-4480-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 3676852 INFO (jetty-launcher-4480-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-04T09:41:37.002Z
[junit4] 2> 3676853 INFO (jetty-launcher-4480-thread-3) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@793c289{/solr,null,AVAILABLE}
[junit4] 2> 3676855 INFO (jetty-launcher-4480-thread-3) [ ] o.e.j.s.AbstractConnector Started ServerConnector@65cc7b8c{HTTP/1.1,[http/1.1]}{127.0.0.1:57105}
[junit4] 2> 3676859 INFO (jetty-launcher-4480-thread-3) [ ] o.e.j.s.Server Started @3679789ms
[junit4] 2> 3676859 INFO (jetty-launcher-4480-thread-3) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=57105}
[junit4] 2> 3676865 ERROR (jetty-launcher-4480-thread-3) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3676865 INFO (jetty-launcher-4480-thread-3) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solrâ„¢ version 6.6.3
[junit4] 2> 3676865 INFO (jetty-launcher-4480-thread-3) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3676865 INFO (jetty-launcher-4480-thread-3) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 3676865 INFO (jetty-launcher-4480-thread-3) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-04T09:41:37.015Z
[junit4] 2> 3676867 INFO (jetty-launcher-4480-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3676867 INFO (jetty-launcher-4480-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3676873 INFO (jetty-launcher-4480-thread-4) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3676873 INFO (jetty-launcher-4480-thread-3) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3676878 INFO (jetty-launcher-4480-thread-1) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 3676879 INFO (jetty-launcher-4480-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51101/solr
[junit4] 2> 3676882 INFO (jetty-launcher-4480-thread-2) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 3676883 INFO (jetty-launcher-4480-thread-4) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 3676884 INFO (jetty-launcher-4480-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51101/solr
[junit4] 2> 3676885 INFO (jetty-launcher-4480-thread-4) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51101/solr
[junit4] 2> 3676886 INFO (jetty-launcher-4480-thread-3) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 3676886 INFO (jetty-launcher-4480-thread-3) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51101/solr
[junit4] 2> 3676887 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x161f06353e50006, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 3676910 INFO (jetty-launcher-4480-thread-1) [n:127.0.0.1:35293_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3676910 INFO (jetty-launcher-4480-thread-4) [n:127.0.0.1:53244_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3676910 INFO (jetty-launcher-4480-thread-3) [n:127.0.0.1:57105_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3676910 INFO (jetty-launcher-4480-thread-2) [n:127.0.0.1:38931_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3676911 INFO (jetty-launcher-4480-thread-3) [n:127.0.0.1:57105_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:57105_solr
[junit4] 2> 3676912 INFO (jetty-launcher-4480-thread-4) [n:127.0.0.1:53244_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53244_solr
[junit4] 2> 3676912 INFO (jetty-launcher-4480-thread-2) [n:127.0.0.1:38931_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38931_solr
[junit4] 2> 3676912 INFO (jetty-launcher-4480-thread-1) [n:127.0.0.1:35293_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35293_solr
[junit4] 2> 3676912 INFO (jetty-launcher-4480-thread-3) [n:127.0.0.1:57105_solr ] o.a.s.c.Overseer Overseer (id=99624976178806796-127.0.0.1:57105_solr-n_0000000000) starting
[junit4] 2> 3676914 INFO (zkCallback-4500-thread-1-processing-n:127.0.0.1:53244_solr) [n:127.0.0.1:53244_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 3676914 INFO (zkCallback-4496-thread-1-processing-n:127.0.0.1:35293_solr) [n:127.0.0.1:35293_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 3676914 INFO (zkCallback-4502-thread-1-processing-n:127.0.0.1:57105_solr) [n:127.0.0.1:57105_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 3676922 INFO (zkCallback-4501-thread-1-processing-n:127.0.0.1:38931_solr) [n:127.0.0.1:38931_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 3676935 INFO (jetty-launcher-4480-thread-3) [n:127.0.0.1:57105_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57105_solr
[junit4] 2> 3676937 INFO (zkCallback-4501-thread-1-processing-n:127.0.0.1:38931_solr) [n:127.0.0.1:38931_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3676937 INFO (zkCallback-4496-thread-1-processing-n:127.0.0.1:35293_solr) [n:127.0.0.1:35293_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3676937 INFO (zkCallback-4500-thread-1-processing-n:127.0.0.1:53244_solr) [n:127.0.0.1:53244_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3676937 INFO (zkCallback-4502-thread-1-processing-n:127.0.0.1:57105_solr) [n:127.0.0.1:57105_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3676965 INFO (jetty-launcher-4480-thread-2) [n:127.0.0.1:38931_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_78CE862CA70D20DD-001/tempDir-001/node3/.
[junit4] 2> 3676965 INFO (jetty-launcher-4480-thread-4) [n:127.0.0.1:53244_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_78CE862CA70D20DD-001/tempDir-001/node4/.
[junit4] 2> 3676989 INFO (jetty-launcher-4480-thread-1) [n:127.0.0.1:35293_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_78CE862CA70D20DD-001/tempDir-001/node1/.
[junit4] 2> 3676995 INFO (jetty-launcher-4480-thread-3) [n:127.0.0.1:57105_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_78CE862CA70D20DD-001/tempDir-001/node2/.
[junit4] 2> 3677005 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 3677006 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:51101/solr ready
[junit4] 1> Formatting using clusterid: testClusterID
[junit4] 2> 3677057 WARN (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
[junit4] 2> 3677065 WARN (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 3677077 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[78CE862CA70D20DD]-worker) [ ] o.m.log jetty-6.1.26
[junit4] 2> 3677099 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[78CE862CA70D20DD]-worker) [ ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_41817_hdfs____mmu3xn/webapp
[junit4] 2> 3677502 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[78CE862CA70D20DD]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41817
[junit4] 2> 3677560 WARN (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[78CE862CA70D20DD]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 3677561 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[78CE862CA70D20DD]-worker) [ ] o.m.log jetty-6.1.26
[junit4] 2> 3677575 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[78CE862CA70D20DD]-worker) [ ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_45588_datanode____.a4fgw5/webapp
[junit4] 2> 3678003 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[78CE862CA70D20DD]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45588
[junit4] 2> 3678088 ERROR (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_78CE862CA70D20DD-001/tempDir-002/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_78CE862CA70D20DD-001/tempDir-002/hdfsBaseDir/data/data2/]] heartbeating to localhost/127.0.0.1:52843) [ ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
[junit4] 2> 3678094 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x48815576400c08: from storage DS-ff004887-7b11-462a-bc33-2e49b7d936dc node DatanodeRegistration(127.0.0.1:50209, datanodeUuid=61a0fe6e-62b9-4c1f-a6ae-3e29a3811d18, infoPort=51077, infoSecurePort=0, ipcPort=49138, storageInfo=lv=-56;cid=testClusterID;nsid=1545822465;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
[junit4] 2> 3678094 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x48815576400c08: from storage DS-31e16253-85d5-470b-8675-76bdaa74acd0 node DatanodeRegistration(127.0.0.1:50209, datanodeUuid=61a0fe6e-62b9-4c1f-a6ae-3e29a3811d18, infoPort=51077, infoSecurePort=0, ipcPort=49138, storageInfo=lv=-56;cid=testClusterID;nsid=1545822465;c=0), blocks: 0, hasStaleStorage: false, processing time: 1 msecs
[junit4] 2> 3678208 INFO (TEST-HdfsCollectionsAPIDistributedZkTest.addReplicaTest-seed#[78CE862CA70D20DD]) [ ] o.a.s.SolrTestCaseJ4 ###Starting addReplicaTest
[junit4] 2> 3678210 INFO (TEST-HdfsCollectionsAPIDistributedZkTest.addReplicaTest-seed#[78CE862CA70D20DD]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 3678214 INFO (qtp1613196478-56381) [n:127.0.0.1:53244_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf&maxShardsPerNode=4&name=addReplicaColl&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 3678217 INFO (OverseerThreadFactory-8695-thread-1-processing-n:127.0.0.1:57105_solr) [n:127.0.0.1:57105_solr ] o.a.s.c.CreateCollectionCmd Create collection addReplicaColl
[junit4] 2> 3678323 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=addReplicaColl_shard1_replica1&action=CREATE&numShards=2&collection=addReplicaColl&shard=shard1&wt=javabin&version=2
[junit4] 2> 3678323 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=addReplicaColl_shard2_replica1&action=CREATE&numShards=2&collection=addReplicaColl&shard=shard2&wt=javabin&version=2
[junit4] 2> 3678323 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 3678324 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 3678325 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=addReplicaColl_shard1_replica2&action=CREATE&numShards=2&collection=addReplicaColl&shard=shard1&wt=javabin&version=2
[junit4] 2> 3678325 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 3678326 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=addReplicaColl_shard2_replica2&action=CREATE&numShards=2&collection=addReplicaColl&shard=shard2&wt=javabin&version=2
[junit4] 2> 3678326 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 3678433 INFO (zkCallback-4500-thread-1-processing-n:127.0.0.1:53244_solr) [n:127.0.0.1:53244_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4])
[junit4] 2> 3678433 INFO (zkCallback-4501-thread-1-processing-n:127.0.0.1:38931_solr) [n:127.0.0.1:38931_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4])
[junit4] 2> 3678433 INFO (zkCallback-4496-thread-1-processing-n:127.0.0.1:35293_solr) [n:127.0.0.1:35293_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4])
[junit4] 2> 3678433 INFO (zkCallback-4502-thread-1-processing-n:127.0.0.1:57105_solr) [n:127.0.0.1:57105_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4])
[junit4] 2> 3679343 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.3
[junit4] 2> 3679344 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.3
[junit4] 2> 3679344 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.3
[junit4] 2> 3679371 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.3
[junit4] 2> 3679376 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.s.IndexSchema [addReplicaColl_shard1_replica1] Schema name=minimal
[junit4] 2> 3679379 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 3679379 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'addReplicaColl_shard1_replica1' using configuration from collection addReplicaColl, trusted=true
[junit4] 2> 3679380 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.s.IndexSchema [addReplicaColl_shard1_replica2] Schema name=minimal
[junit4] 2> 3679380 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.s.IndexSchema [addReplicaColl_shard2_replica2] Schema name=minimal
[junit4] 2> 3679382 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 3679382 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.c.CoreContainer Creating SolrCore 'addReplicaColl_shard2_replica2' using configuration from collection addReplicaColl, trusted=true
[junit4] 2> 3679384 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 3679384 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.c.CoreContainer Creating SolrCore 'addReplicaColl_shard1_replica2' using configuration from collection addReplicaColl, trusted=true
[junit4] 2> 3679388 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:52843/data
[junit4] 2> 3679388 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
[junit4] 2> 3679388 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 3679388 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:52843/data
[junit4] 2> 3679388 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.c.SolrCore [[addReplicaColl_shard1_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_78CE862CA70D20DD-001/tempDir-001/node3/addReplicaColl_shard1_replica1], dataDir=[hdfs://localhost:52843/data/addReplicaColl/core_node1/data/]
[junit4] 2> 3679388 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
[junit4] 2> 3679388 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 3679388 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.c.SolrCore [[addReplicaColl_shard1_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_78CE862CA70D20DD-001/tempDir-001/node4/addReplicaColl_shard1_replica2], dataDir=[hdfs://localhost:52843/data/addReplicaColl/core_node3/data/]
[junit4] 2> 3679389 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:52843/data
[junit4] 2> 3679389 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
[junit4] 2> 3679389 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 3679389 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.c.SolrCore [[addReplicaColl_shard2_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_78CE862CA70D20DD-001/tempDir-001/node2/addReplicaColl_shard2_replica2], dataDir=[hdfs://localhost:52843/data/addReplicaColl/core_node4/data/]
[junit4] 2> 3679389 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:52843/data/addReplicaColl/core_node1/data/snapshot_metadata
[junit4] 2> 3679389 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:52843/data/addReplicaColl/core_node3/data/snapshot_metadata
[junit4] 2> 3679389 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:52843/data/addReplicaColl/core_node4/data/snapshot_metadata
[junit4] 2> 3679393 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.s.IndexSchema [addReplicaColl_shard2_replica1] Schema name=minimal
[junit4] 2> 3679396 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 3679396 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.c.CoreContainer Creating SolrCore 'addReplicaColl_shard2_replica1' using configuration from collection addReplicaColl, trusted=true
[junit4] 2> 3679396 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:52843/data
[junit4] 2> 3679396 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
[junit4] 2> 3679397 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 3679397 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.c.SolrCore [[addReplicaColl_shard2_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_78CE862CA70D20DD-001/tempDir-001/node1/addReplicaColl_shard2_replica1], dataDir=[hdfs://localhost:52843/data/addReplicaColl/core_node2/data/]
[junit4] 2> 3679397 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:52843/data/addReplicaColl/core_node2/data/snapshot_metadata
[junit4] 2> 3679413 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 3679413 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
[junit4] 2> 3679413 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 3679415 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 3679415 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
[junit4] 2> 3679415 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 3679415 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 3679415 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 3679415 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
[junit4] 2> 3679416 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
[junit4] 2> 3679416 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 3679416 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 3679424 WARN (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
[junit4] 2> 3679424 WARN (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
[junit4] 2> 3679425 WARN (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
[junit4] 2> 3679426 WARN (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
[junit4] 2> 3684431 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 3684433 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 3684436 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 3684436 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:52843/data/addReplicaColl/core_node3/data
[junit4] 2> 3684443 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 3684443 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:52843/data/addReplicaColl/core_node4/data
[junit4] 2> 3684446 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:52843/data/addReplicaColl/core_node2/data
[junit4] 2> 3684447 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:52843/data/addReplicaColl/core_node1/data
[junit4] 2> 3684468 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:52843/data/addReplicaColl/core_node3/data/index
[junit4] 2> 3684476 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 3684477 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
[junit4] 2> 3684477 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 3684477 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:52843/data/addReplicaColl/core_node4/data/index
[junit4] 2> 3684481 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 3684485 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:52843/data/addReplicaColl/core_node1/data/index
[junit4] 2> 3684486 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 3684486 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
[junit4] 2> 3684486 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 3684493 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 3684495 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:52843/data/addReplicaColl/core_node2/data/index
[junit4] 2> 3684497 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 3684497 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
[junit4] 2> 3684497 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 3684503 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 3684509 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50209 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-31e16253-85d5-470b-8675-76bdaa74acd0:NORMAL:127.0.0.1:50209|RBW]]} size 0
[junit4] 2> 3684516 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 3684516 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
[junit4] 2> 3684516 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
[junit4] 2> 3684523 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 3684536 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50209 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ff004887-7b11-462a-bc33-2e49b7d936dc:NORMAL:127.0.0.1:50209|FINALIZED]]} size 0
[junit4] 2> 3684537 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50209 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-31e16253-85d5-470b-8675-76bdaa74acd0:NORMAL:127.0.0.1:50209|RBW]]} size 0
[junit4] 2> 3684598 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50209 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ff004887-7b11-462a-bc33-2e49b7d936dc:NORMAL:127.0.0.1:50209|FINALIZED]]} size 0
[junit4] 2> 3684599 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
[junit4] 2> 3684599 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 3684599 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
[junit4] 2> 3684626 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 3684626 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 3684635 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
[junit4] 2> 3684635 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 3684635 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
[junit4] 2> 3684638 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
[junit4] 2> 3684638 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 3684638 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
[junit4] 2> 3684652 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 3684652 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@107e8137[addReplicaColl_shard1_replica2] main]
[junit4] 2> 3684652 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 3684654 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 3684654 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 3684654 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 3684655 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1593999627182407680
[junit4] 2> 3684657 INFO (searcherExecutor-8706-thread-1-processing-n:127.0.0.1:53244_solr x:addReplicaColl_shard1_replica2 s:shard1 c:addReplicaColl) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.c.SolrCore [addReplicaColl_shard1_replica2] Registered new searcher Searcher@107e8137[addReplicaColl_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 3684661 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 3684661 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 3684663 INFO (qtp1613196478-56379) [n:127.0.0.1:53244_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
[junit4] 2> 3684665 INFO (zkCallback-4501-thread-1-processing-n:127.0.0.1:38931_solr) [n:127.0.0.1:38931_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4])
[junit4] 2> 3684665 INFO (zkCallback-4496-thread-1-processing-n:127.0.0.1:35293_solr) [n:127.0.0.1:35293_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4])
[junit4] 2> 3684665 INFO (zkCallback-4502-thread-1-processing-n:127.0.0.1:57105_solr) [n:127.0.0.1:57105_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4])
[junit4] 2> 3684665 INFO (zkCallback-4500-thread-1-processing-n:127.0.0.1:53244_solr) [n:127.0.0.1:53244_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4])
[junit4] 2> 3684672 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
[junit4] 2> 3684672 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 3684672 INFO (qtp167393648-56371) [n:127.0.0.1:35293_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
[junit4] 2> 3684678 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@446c4372[addReplicaColl_shard1_replica1] main]
[junit4] 2> 3684680 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 3684681 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 3684681 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@655b13ba[addReplicaColl_shard2_replica2] main]
[junit4] 2> 3684681 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 3684681 INFO (qtp990896924-56391) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1593999627209670656
[junit4] 2> 3684682 INFO (searcherExecutor-8704-thread-1-processing-n:127.0.0.1:38931_solr x:addReplicaColl_shard1_replica1 s:shard1 c:addReplicaColl) [n:127.0.0.1:38931_solr c:addReplicaColl s:shard1 x:addReplicaColl_shard1_replica1] o.a.s.c.SolrCore [addReplicaColl_shard1_replica1] Registered new searcher Searcher@446c4372[addReplicaColl_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 3684683 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 3684683 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 3684683 INFO (qtp1299584871-56401) [n:127.0.0.1:57105_solr c:addReplicaColl s:shard2 x:addReplicaColl_shard2_replica2] o.a.s.h.Replicati
[...truncated too long message...]
entDeleteAndCreateCollectionTest, OverseerTaskQueueTest, TestSort, CloudMLTQParserTest, CoreSorterTest, TestShardHandlerFactory, TestConfigSets, AsyncCallRequestStatusResponseTest, CoreAdminRequestStatusTest, RuleEngineTest, PKIAuthenticationIntegrationTest, TestTrieFacet, TestSolrQueryParserDefaultOperatorResource, LeaderFailureAfterFreshStartTest, TestPerFieldSimilarityWithDefaultOverride, TestFieldCacheSort, OpenExchangeRatesOrgProviderTest, TestObjectReleaseTracker, ShowFileRequestHandlerTest, HdfsBasicDistributedZkTest, PreAnalyzedFieldManagedSchemaCloudTest, TestSegmentSorting, PreAnalyzedFieldTest, ParsingFieldUpdateProcessorsTest, NoCacheHeaderTest, TestChildDocTransformer, BlockJoinFacetDistribTest, SolrMetricsIntegrationTest, TestMergePolicyConfig, StatelessScriptUpdateProcessorFactoryTest, ZkStateReaderTest, TestNumericTerms64, TestConfigOverlay, CursorMarkTest, TestFieldCollectionResource, LegacyCloudClusterPropTest, SolrCoreMetricManagerTest, DistributedVersionInfoTest, TestJsonFacetRefinement, DebugComponentTest, UnloadDistributedZkTest, OverseerTest, LeaderElectionIntegrationTest, BasicZkTest, FullSolrCloudDistribCmdsTest, TestHashPartitioner, TermVectorComponentDistributedTest, TestReload, TestRangeQuery, HighlighterConfigTest, TestQuerySenderListener, ResponseLogComponentTest, SolrIndexConfigTest, BadComponentTest, TestStressRecovery, SynonymTokenizerTest, TestXIncludeConfig, EchoParamsTest, TestIBSimilarityFactory, TestFastWriter, TimeZoneUtilsTest, ScriptEngineTest, ChaosMonkeyNothingIsSafeTest, CdcrReplicationDistributedZkTest, CdcrReplicationHandlerTest, ConfigSetsAPITest, CreateCollectionCleanupTest, DistribCursorPagingTest, ForceLeaderTest, SSLMigrationTest, SaslZkACLProviderTest, ShardSplitTest, TestCloudPivotFacet, TestCloudPseudoReturnFields, TestCloudRecovery, TestCollectionAPI, TestCryptoKeys, TestLeaderElectionWithEmptyReplica, TestLeaderElectionZkExpiry, TestLockTree, TestMiniSolrCloudCluster, TestRandomRequestDistribution, TestRebalanceLeaders, TestReplicaProperties, TestRequestForwarding, TestSSLRandomization, TestSizeLimitedDistributedMap, TestTolerantUpdateProcessorCloud, TestTolerantUpdateProcessorRandomCloud, TriLevelCompositeIdRoutingTest, VMParamsZkACLAndCredentialsProvidersTest, HdfsBasicDistributedZk2Test, HdfsCollectionsAPIDistributedZkTest, TestClusterStateMutator, RulesTest, HdfsDirectoryFactoryTest, TestConfigSetImmutable, TestConfigSetProperties, TestCustomStream, TestImplicitCoreProperties, TestNRTOpen, TestSolrConfigHandler, TestHdfsBackupRestoreCore, CoreAdminCreateDiscoverTest, CoreMergeIndexesAdminHandlerTest, InfoHandlerTest, SecurityConfHandlerTest, StatsReloadRaceTest, TestCollectionAPIs, TestCoreAdminApis, DistributedExpandComponentTest, DistributedFacetPivotSmallTest, DistributedSuggestComponentTest, JavabinLoaderTest, HighlighterMaxOffsetTest, TestSlowCompositeReaderWrapper, WrapperMergePolicyFactoryTest, CheckHdfsIndexTest, TestCloudSchemaless, TestUseDocValuesAsStored2, RankQueryTest, SortSpecParsingTest, TestCollapseQParserPlugin, TestComplexPhraseLeadingWildcard, TestFieldSortValues, TestFilteredDocIdSet, TestHashQParserPlugin, TestNoOpRegenerator, TestRandomCollapseQParserPlugin, TestSmileRequest, TestStandardQParsers, TestXmlQParser, TestXmlQParserPlugin, TestJsonFacets, TestAuthorizationFramework, TestFieldCacheReopen, TestFieldCacheSortRandom, TestLegacyFieldCache, TestNumericTerms32, TestUninvertingReader, AddBlockUpdateTest, AnalysisErrorHandlingTest, CdcrUpdateLogTest, TestInPlaceUpdatesDistrib]
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestInPlaceUpdatesDistrib -Dtests.seed=78CE862CA70D20DD -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/test-data/enwiki.random.lines.txt -Dtests.locale=es-CU -Dtests.timezone=Etc/Zulu -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J2 | TestInPlaceUpdatesDistrib (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.update.TestInPlaceUpdatesDistrib:
[junit4] > 1) Thread[id=63443, name=searcherExecutor-10694-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 2) Thread[id=63409, name=searcherExecutor-10683-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > at __randomizedtesting.SeedInfo.seed([78CE862CA70D20DD]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=63443, name=searcherExecutor-10694-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 2) Thread[id=63409, name=searcherExecutor-10683-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > at __randomizedtesting.SeedInfo.seed([78CE862CA70D20DD]:0)
[junit4] Completed [714/714 (3!)] on J2 in 1465.25s, 1 test, 3 errors <<< FAILURES!
[...truncated 1 lines...]
[junit4] JVM J2: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/temp/junit4-J2-20180304_084016_6928344625705321710958.sysout
[junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
[junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
[junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/heapdumps/java_pid7833.hprof ...
[junit4] Heap dump file created [660867319 bytes in 3.026 secs]
[junit4] <<< JVM J2: EOF ----
[junit4] JVM J2: stderr was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/temp/junit4-J2-20180304_084016_692805811283229828314.syserr
[junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
[junit4]
[junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "SessionTracker"
[junit4] <<< JVM J2: EOF ----
[...truncated 8104 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/build.xml:783: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/build.xml:727: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid7833.hprof
Total time: 277 minutes 55 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-NightlyTests-6.6 - Build # 33 - Still
Unstable
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.6/33/
2 tests failed.
FAILED: org.apache.solr.cloud.MissingSegmentRecoveryTest.testLeaderRecovery
Error Message:
Expected a collection with one shard and two replicas null Last available state: DocCollection(MissingSegmentRecoveryTest//collections/MissingSegmentRecoveryTest/state.json/7)={ "replicationFactor":"2", "shards":{"shard1":{ "range":"80000000-7fffffff", "state":"active", "replicas":{ "core_node1":{ "core":"MissingSegmentRecoveryTest_shard1_replica1", "base_url":"https://127.0.0.1:42561/solr", "node_name":"127.0.0.1:42561_solr", "state":"active", "leader":"true"}, "core_node2":{ "core":"MissingSegmentRecoveryTest_shard1_replica2", "base_url":"https://127.0.0.1:35061/solr", "node_name":"127.0.0.1:35061_solr", "state":"down"}}}}, "router":{"name":"compositeId"}, "maxShardsPerNode":"1", "autoAddReplicas":"false"}
Stack Trace:
java.lang.AssertionError: Expected a collection with one shard and two replicas
null
Last available state: DocCollection(MissingSegmentRecoveryTest//collections/MissingSegmentRecoveryTest/state.json/7)={
"replicationFactor":"2",
"shards":{"shard1":{
"range":"80000000-7fffffff",
"state":"active",
"replicas":{
"core_node1":{
"core":"MissingSegmentRecoveryTest_shard1_replica1",
"base_url":"https://127.0.0.1:42561/solr",
"node_name":"127.0.0.1:42561_solr",
"state":"active",
"leader":"true"},
"core_node2":{
"core":"MissingSegmentRecoveryTest_shard1_replica2",
"base_url":"https://127.0.0.1:35061/solr",
"node_name":"127.0.0.1:35061_solr",
"state":"down"}}}},
"router":{"name":"compositeId"},
"maxShardsPerNode":"1",
"autoAddReplicas":"false"}
at __randomizedtesting.SeedInfo.seed([9B90977827D05434:CBC50F7B7EF1E229]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:265)
at org.apache.solr.cloud.MissingSegmentRecoveryTest.testLeaderRecovery(MissingSegmentRecoveryTest.java:105)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test
Error Message:
Stack Trace:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([9B90977827D05434:13C4A8A2892C39CC]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:307)
at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 12210 lines...]
[junit4] Suite: org.apache.solr.cloud.MissingSegmentRecoveryTest
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001/init-core-data-001
[junit4] 2> 640887 INFO (SUITE-MissingSegmentRecoveryTest-seed#[9B90977827D05434]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields
[junit4] 2> 640887 INFO (SUITE-MissingSegmentRecoveryTest-seed#[9B90977827D05434]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 640892 INFO (SUITE-MissingSegmentRecoveryTest-seed#[9B90977827D05434]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001/tempDir-001
[junit4] 2> 640892 INFO (SUITE-MissingSegmentRecoveryTest-seed#[9B90977827D05434]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 640899 INFO (Thread-787) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 640899 INFO (Thread-787) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 640939 ERROR (Thread-787) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 640999 INFO (SUITE-MissingSegmentRecoveryTest-seed#[9B90977827D05434]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:54884
[junit4] 2> 641045 INFO (jetty-launcher-566-thread-1) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 641061 INFO (jetty-launcher-566-thread-2) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 641071 INFO (jetty-launcher-566-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@21f9f662{/solr,null,AVAILABLE}
[junit4] 2> 641073 INFO (jetty-launcher-566-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@33a91394{/solr,null,AVAILABLE}
[junit4] 2> 641074 INFO (jetty-launcher-566-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@766c7140{SSL,[ssl, http/1.1]}{127.0.0.1:35061}
[junit4] 2> 641074 INFO (jetty-launcher-566-thread-2) [ ] o.e.j.s.Server Started @643344ms
[junit4] 2> 641074 INFO (jetty-launcher-566-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35061}
[junit4] 2> 641074 ERROR (jetty-launcher-566-thread-2) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 641075 INFO (jetty-launcher-566-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.6.3
[junit4] 2> 641075 INFO (jetty-launcher-566-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 641075 INFO (jetty-launcher-566-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 641075 INFO (jetty-launcher-566-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-03T04:33:23.102Z
[junit4] 2> 641076 INFO (jetty-launcher-566-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@13ffe15b{SSL,[ssl, http/1.1]}{127.0.0.1:42561}
[junit4] 2> 641076 INFO (jetty-launcher-566-thread-1) [ ] o.e.j.s.Server Started @643345ms
[junit4] 2> 641076 INFO (jetty-launcher-566-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42561}
[junit4] 2> 641076 ERROR (jetty-launcher-566-thread-1) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 641076 INFO (jetty-launcher-566-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.6.3
[junit4] 2> 641076 INFO (jetty-launcher-566-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 641076 INFO (jetty-launcher-566-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 641076 INFO (jetty-launcher-566-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-03T04:33:23.103Z
[junit4] 2> 641078 INFO (jetty-launcher-566-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 641087 INFO (jetty-launcher-566-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 641095 INFO (jetty-launcher-566-thread-2) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 641096 INFO (jetty-launcher-566-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54884/solr
[junit4] 2> 641097 INFO (jetty-launcher-566-thread-1) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 641098 INFO (jetty-launcher-566-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54884/solr
[junit4] 2> 641110 INFO (jetty-launcher-566-thread-2) [n:127.0.0.1:35061_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 641110 INFO (jetty-launcher-566-thread-2) [n:127.0.0.1:35061_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:35061_solr
[junit4] 2> 641111 INFO (jetty-launcher-566-thread-2) [n:127.0.0.1:35061_solr ] o.a.s.c.Overseer Overseer (id=99618101850275845-127.0.0.1:35061_solr-n_0000000000) starting
[junit4] 2> 641117 INFO (jetty-launcher-566-thread-1) [n:127.0.0.1:42561_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 641117 INFO (jetty-launcher-566-thread-1) [n:127.0.0.1:42561_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42561_solr
[junit4] 2> 641362 INFO (jetty-launcher-566-thread-2) [n:127.0.0.1:35061_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35061_solr
[junit4] 2> 641367 INFO (zkCallback-578-thread-1-processing-n:127.0.0.1:42561_solr) [n:127.0.0.1:42561_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 641396 INFO (zkCallback-577-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 641397 INFO (OverseerStateUpdate-99618101850275845-127.0.0.1:35061_solr-n_0000000000) [n:127.0.0.1:35061_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 641408 INFO (zkCallback-578-thread-2-processing-n:127.0.0.1:42561_solr) [n:127.0.0.1:42561_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 641421 INFO (jetty-launcher-566-thread-2) [n:127.0.0.1:35061_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001/tempDir-001/node2/.
[junit4] 2> 641443 INFO (jetty-launcher-566-thread-1) [n:127.0.0.1:42561_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001/tempDir-001/node1/.
[junit4] 2> 641497 INFO (SUITE-MissingSegmentRecoveryTest-seed#[9B90977827D05434]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 641497 INFO (SUITE-MissingSegmentRecoveryTest-seed#[9B90977827D05434]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:54884/solr ready
[junit4] 2> 641518 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testLeaderRecovery
[junit4] 2> 641593 INFO (qtp968564639-3268) [n:127.0.0.1:42561_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf&maxShardsPerNode=1&name=MissingSegmentRecoveryTest&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 641603 INFO (OverseerThreadFactory-1671-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr ] o.a.s.c.CreateCollectionCmd Create collection MissingSegmentRecoveryTest
[junit4] 2> 641753 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=MissingSegmentRecoveryTest_shard1_replica1&action=CREATE&numShards=1&collection=MissingSegmentRecoveryTest&shard=shard1&wt=javabin&version=2
[junit4] 2> 641754 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 641806 INFO (qtp228215942-3279) [n:127.0.0.1:35061_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=MissingSegmentRecoveryTest_shard1_replica2&action=CREATE&numShards=1&collection=MissingSegmentRecoveryTest&shard=shard1&wt=javabin&version=2
[junit4] 2> 641806 INFO (qtp228215942-3279) [n:127.0.0.1:35061_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 641909 INFO (zkCallback-577-thread-2-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 641909 INFO (zkCallback-578-thread-2-processing-n:127.0.0.1:42561_solr) [n:127.0.0.1:42561_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 642770 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.3
[junit4] 2> 642777 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.s.IndexSchema [MissingSegmentRecoveryTest_shard1_replica1] Schema name=minimal
[junit4] 2> 642783 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 642783 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'MissingSegmentRecoveryTest_shard1_replica1' using configuration from collection MissingSegmentRecoveryTest, trusted=true
[junit4] 2> 642784 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 642784 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.SolrCore [[MissingSegmentRecoveryTest_shard1_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001/tempDir-001/node1/MissingSegmentRecoveryTest_shard1_replica1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001/tempDir-001/node1/./MissingSegmentRecoveryTest_shard1_replica1/data/]
[junit4] 2> 642837 INFO (qtp228215942-3279) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.3
[junit4] 2> 642850 INFO (qtp228215942-3279) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.s.IndexSchema [MissingSegmentRecoveryTest_shard1_replica2] Schema name=minimal
[junit4] 2> 642852 INFO (qtp228215942-3279) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 642852 INFO (qtp228215942-3279) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.CoreContainer Creating SolrCore 'MissingSegmentRecoveryTest_shard1_replica2' using configuration from collection MissingSegmentRecoveryTest, trusted=true
[junit4] 2> 642852 INFO (qtp228215942-3279) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 642852 INFO (qtp228215942-3279) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.SolrCore [[MissingSegmentRecoveryTest_shard1_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001/tempDir-001/node2/MissingSegmentRecoveryTest_shard1_replica2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001/tempDir-001/node2/./MissingSegmentRecoveryTest_shard1_replica2/data/]
[junit4] 2> 642864 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 642864 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 642865 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 642865 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 642866 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@49c30950[MissingSegmentRecoveryTest_shard1_replica1] main]
[junit4] 2> 642867 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 642867 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 642868 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 643059 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1593889631854657536
[junit4] 2> 643063 INFO (searcherExecutor-1676-thread-1-processing-n:127.0.0.1:42561_solr x:MissingSegmentRecoveryTest_shard1_replica1 s:shard1 c:MissingSegmentRecoveryTest) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.SolrCore [MissingSegmentRecoveryTest_shard1_replica1] Registered new searcher Searcher@49c30950[MissingSegmentRecoveryTest_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 643069 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
[junit4] 2> 643115 INFO (qtp228215942-3279) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 643115 INFO (qtp228215942-3279) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 643129 INFO (qtp228215942-3279) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 643130 INFO (qtp228215942-3279) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 643131 INFO (qtp228215942-3279) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@7f46291b[MissingSegmentRecoveryTest_shard1_replica2] main]
[junit4] 2> 643132 INFO (qtp228215942-3279) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 643132 INFO (qtp228215942-3279) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 643132 INFO (qtp228215942-3279) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 643132 INFO (qtp228215942-3279) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1593889631931203584
[junit4] 2> 643136 INFO (searcherExecutor-1677-thread-1-processing-n:127.0.0.1:35061_solr x:MissingSegmentRecoveryTest_shard1_replica2 s:shard1 c:MissingSegmentRecoveryTest) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.SolrCore [MissingSegmentRecoveryTest_shard1_replica2] Registered new searcher Searcher@7f46291b[MissingSegmentRecoveryTest_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 643170 INFO (zkCallback-577-thread-2-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 643170 INFO (zkCallback-578-thread-2-processing-n:127.0.0.1:42561_solr) [n:127.0.0.1:42561_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 643571 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 643571 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 643571 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:42561/solr/MissingSegmentRecoveryTest_shard1_replica1/
[junit4] 2> 643572 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.PeerSync PeerSync: core=MissingSegmentRecoveryTest_shard1_replica1 url=https://127.0.0.1:42561/solr START replicas=[https://127.0.0.1:35061/solr/MissingSegmentRecoveryTest_shard1_replica2/] nUpdates=100
[junit4] 2> 643575 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.PeerSync PeerSync: core=MissingSegmentRecoveryTest_shard1_replica1 url=https://127.0.0.1:42561/solr DONE. We have no versions. sync failed.
[junit4] 2> 643585 INFO (qtp228215942-3277) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.S.Request [MissingSegmentRecoveryTest_shard1_replica2] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 643586 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 643586 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 643586 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 643600 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:42561/solr/MissingSegmentRecoveryTest_shard1_replica1/ shard1
[junit4] 2> 643702 INFO (zkCallback-577-thread-2-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 643702 INFO (zkCallback-578-thread-2-processing-n:127.0.0.1:42561_solr) [n:127.0.0.1:42561_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 643751 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 643753 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=MissingSegmentRecoveryTest_shard1_replica1&action=CREATE&numShards=1&collection=MissingSegmentRecoveryTest&shard=shard1&wt=javabin&version=2} status=0 QTime=1999
[junit4] 2> 643854 INFO (zkCallback-578-thread-2-processing-n:127.0.0.1:42561_solr) [n:127.0.0.1:42561_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 643854 INFO (zkCallback-577-thread-2-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 644138 INFO (qtp228215942-3279) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=MissingSegmentRecoveryTest_shard1_replica2&action=CREATE&numShards=1&collection=MissingSegmentRecoveryTest&shard=shard1&wt=javabin&version=2} status=0 QTime=2332
[junit4] 2> 644142 INFO (qtp968564639-3268) [n:127.0.0.1:42561_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 644244 INFO (zkCallback-578-thread-2-processing-n:127.0.0.1:42561_solr) [n:127.0.0.1:42561_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 644244 INFO (zkCallback-577-thread-2-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 645142 INFO (qtp968564639-3268) [n:127.0.0.1:42561_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=2&collection.configName=conf&maxShardsPerNode=1&name=MissingSegmentRecoveryTest&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=3549
[junit4] 2> 645166 INFO (qtp228215942-3273) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [MissingSegmentRecoveryTest_shard1_replica2] webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:42561/solr/MissingSegmentRecoveryTest_shard1_replica1/&wt=javabin&version=2}{add=[0 (1593889634056667136), 1 (1593889634058764288), 2 (1593889634058764289), 3 (1593889634058764290), 4 (1593889634058764291), 5 (1593889634058764292), 6 (1593889634058764293), 7 (1593889634058764294), 8 (1593889634058764295), 9 (1593889634058764296)]} 0 1
[junit4] 2> 645166 INFO (qtp968564639-3269) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [MissingSegmentRecoveryTest_shard1_replica1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[0 (1593889634056667136), 1 (1593889634058764288), 2 (1593889634058764289), 3 (1593889634058764290), 4 (1593889634058764291), 5 (1593889634058764292), 6 (1593889634058764293), 7 (1593889634058764294), 8 (1593889634058764295), 9 (1593889634058764296)]} 0 7
[junit4] 2> 645180 INFO (qtp228215942-3276) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 645180 INFO (qtp228215942-3276) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@250c1752
[junit4] 2> 645196 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 645196 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@257d07d2
[junit4] 2> 645231 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4aed6040[MissingSegmentRecoveryTest_shard1_replica1] main]
[junit4] 2> 645232 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 645232 INFO (searcherExecutor-1676-thread-1-processing-n:127.0.0.1:42561_solr x:MissingSegmentRecoveryTest_shard1_replica1 s:shard1 c:MissingSegmentRecoveryTest r:core_node1) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.SolrCore [MissingSegmentRecoveryTest_shard1_replica1] Registered new searcher Searcher@4aed6040[MissingSegmentRecoveryTest_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.6.3):C10)))}
[junit4] 2> 645232 INFO (qtp228215942-3276) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@e97669f[MissingSegmentRecoveryTest_shard1_replica2] main]
[junit4] 2> 645233 INFO (qtp228215942-3276) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 645233 INFO (searcherExecutor-1677-thread-1-processing-n:127.0.0.1:35061_solr x:MissingSegmentRecoveryTest_shard1_replica2 s:shard1 c:MissingSegmentRecoveryTest r:core_node2) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.SolrCore [MissingSegmentRecoveryTest_shard1_replica2] Registered new searcher Searcher@e97669f[MissingSegmentRecoveryTest_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.6.3):C10)))}
[junit4] 2> 645233 INFO (qtp228215942-3276) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [MissingSegmentRecoveryTest_shard1_replica2] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:42561/solr/MissingSegmentRecoveryTest_shard1_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 53
[junit4] 2> 645233 INFO (qtp968564639-3263) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [MissingSegmentRecoveryTest_shard1_replica1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:42561/solr/MissingSegmentRecoveryTest_shard1_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 36
[junit4] 2> 645234 INFO (qtp968564639-3267) [n:127.0.0.1:42561_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [MissingSegmentRecoveryTest_shard1_replica1] webapp=/solr path=/update params={_stateVer_=MissingSegmentRecoveryTest:5&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 66
[junit4] 2> 645247 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@766c7140{SSL,[ssl, http/1.1]}{127.0.0.1:0}
[junit4] 2> 645263 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=934350477
[junit4] 2> 645263 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
[junit4] 2> 645263 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jvm
[junit4] 2> 645263 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jetty
[junit4] 2> 645275 INFO (coreCloseExecutor-1686-thread-1) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.SolrCore [MissingSegmentRecoveryTest_shard1_replica2] CLOSING SolrCore org.apache.solr.core.SolrCore@694ba92f
[junit4] 2> 645292 INFO (coreCloseExecutor-1686-thread-1) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.MissingSegmentRecoveryTest.shard1.replica2
[junit4] 2> 645294 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.a.s.c.Overseer Overseer (id=99618101850275845-127.0.0.1:35061_solr-n_0000000000) closing
[junit4] 2> 645294 INFO (OverseerStateUpdate-99618101850275845-127.0.0.1:35061_solr-n_0000000000) [n:127.0.0.1:35061_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:35061_solr
[junit4] 2> 645297 INFO (zkCallback-584-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 645297 INFO (zkCallback-578-thread-1-processing-n:127.0.0.1:42561_solr) [n:127.0.0.1:42561_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 645297 INFO (zkCallback-578-thread-2-processing-n:127.0.0.1:42561_solr) [n:127.0.0.1:42561_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42561_solr
[junit4] 2> 645297 INFO (zkCallback-578-thread-2-processing-n:127.0.0.1:42561_solr) [n:127.0.0.1:42561_solr ] o.a.s.c.Overseer Overseer (id=99618101850275846-127.0.0.1:42561_solr-n_0000000001) starting
[junit4] 2> 645311 INFO (zkCallback-578-thread-2-processing-n:127.0.0.1:42561_solr) [n:127.0.0.1:42561_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [1])
[junit4] 2> 646797 WARN (zkCallback-577-thread-2-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 646798 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@33a91394{/solr,null,UNAVAILABLE}
[junit4] 2> 646801 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 646802 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3a008140{/solr,null,AVAILABLE}
[junit4] 2> 646804 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@304ee53e{SSL,[ssl, http/1.1]}{127.0.0.1:35061}
[junit4] 2> 646804 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.e.j.s.Server Started @649073ms
[junit4] 2> 646804 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35061}
[junit4] 2> 646804 ERROR (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 646805 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.6.3
[junit4] 2> 646805 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 646805 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 646805 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-03T04:33:28.832Z
[junit4] 2> 646814 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 646820 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 646828 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54884/solr
[junit4] 2> 646835 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [n:127.0.0.1:35061_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 646836 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [n:127.0.0.1:35061_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 646837 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [n:127.0.0.1:35061_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35061_solr
[junit4] 2> 646838 INFO (zkCallback-584-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 646838 INFO (zkCallback-578-thread-2-processing-n:127.0.0.1:42561_solr) [n:127.0.0.1:42561_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 646845 INFO (zkCallback-589-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 646903 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [n:127.0.0.1:35061_solr ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001/tempDir-001/node2/.
[junit4] 2> 646904 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [n:127.0.0.1:35061_solr ] o.a.s.c.CorePropertiesLocator Cores are: [MissingSegmentRecoveryTest_shard1_replica2]
[junit4] 2> 646905 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 646924 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.3
[junit4] 2> 646932 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.s.IndexSchema [MissingSegmentRecoveryTest_shard1_replica2] Schema name=minimal
[junit4] 2> 646934 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 646934 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.CoreContainer Creating SolrCore 'MissingSegmentRecoveryTest_shard1_replica2' using configuration from collection MissingSegmentRecoveryTest, trusted=true
[junit4] 2> 646934 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 646934 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.SolrCore [[MissingSegmentRecoveryTest_shard1_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001/tempDir-001/node2/MissingSegmentRecoveryTest_shard1_replica2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001/tempDir-001/node2/./MissingSegmentRecoveryTest_shard1_replica2/data/]
[junit4] 2> 646961 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 646961 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 646962 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 646962 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 646964 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.SolrCore [MissingSegmentRecoveryTest_shard1_replica2] CLOSING SolrCore org.apache.solr.core.SolrCore@7bd72a33
[junit4] 2> 646964 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.MissingSegmentRecoveryTest.shard1.replica2
[junit4] 2> 646965 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.CoreContainer Found active leader, will attempt to create fresh core and recover.
[junit4] 2> 646966 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.SolrCore Updating index properties... index=index.20180303013328993
[junit4] 2> 646970 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 646970 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.SolrCore [[MissingSegmentRecoveryTest_shard1_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001/tempDir-001/node2/MissingSegmentRecoveryTest_shard1_replica2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001/tempDir-001/node2/./MissingSegmentRecoveryTest_shard1_replica2/data/]
[junit4] 2> 646975 INFO (OldIndexDirectoryCleanupThreadForCore-MissingSegmentRecoveryTest_shard1_replica2) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.DirectoryFactory Found 1 old index directories to clean-up under /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001/tempDir-001/node2/./MissingSegmentRecoveryTest_shard1_replica2/data/ afterReload=false
[junit4] 2> 646979 INFO (OldIndexDirectoryCleanupThreadForCore-MissingSegmentRecoveryTest_shard1_replica2) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.DirectoryFactory Deleted old index directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001/tempDir-001/node2/./MissingSegmentRecoveryTest_shard1_replica2/data/index.20180303013328993
[junit4] 2> 646980 ERROR (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.SolrIndexWriter Error closing IndexWriter
[junit4] 2> java.nio.file.NoSuchFileException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001/tempDir-001/node2/MissingSegmentRecoveryTest_shard1_replica2/data/index.20180303013328993
[junit4] 2> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
[junit4] 2> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
[junit4] 2> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
[junit4] 2> at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
[junit4] 2> at java.nio.file.Files.newDirectoryStream(Files.java:457)
[junit4] 2> at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:215)
[junit4] 2> at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:234)
[junit4] 2> at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57)
[junit4] 2> at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:429)
[junit4] 2> at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2249)
[junit4] 2> at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2193)
[junit4] 2> at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1129)
[junit4] 2> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1171)
[junit4] 2> at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:278)
[junit4] 2> at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:730)
[junit4] 2> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:923)
[junit4] 2> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:830)
[junit4] 2> at org.apache.solr.core.CoreContainer.processCoreCreateException(CoreContainer.java:1026)
[junit4] 2> at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:952)
[junit4] 2> at org.apache.solr.core.CoreContainer.lambda$load$5(CoreContainer.java:565)
[junit4] 2> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 647003 INFO (OldIndexDirectoryCleanupThreadForCore-MissingSegmentRecoveryTest_shard1_replica2) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.DirectoryFactory Found 1 old index directories to clean-up under /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001/tempDir-001/node2/./MissingSegmentRecoveryTest_shard1_replica2/data/ afterReload=false
[junit4] 2> 647022 INFO (zkCallback-578-thread-2-processing-n:127.0.0.1:42561_solr) [n:127.0.0.1:42561_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 647022 INFO (OldIndexDirectoryCleanupThreadForCore-MissingSegmentRecoveryTest_shard1_replica2) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.DirectoryFactory Deleted old index directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001/tempDir-001/node2/./MissingSegmentRecoveryTest_shard1_replica2/data/index
[junit4] 2> 647023 INFO (zkCallback-589-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 647033 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 647033 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 647034 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 647034 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 647035 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.SolrCore [MissingSegmentRecoveryTest_shard1_replica2] CLOSING SolrCore org.apache.solr.core.SolrCore@5b100ec
[junit4] 2> 647036 INFO (coreLoadExecutor-1695-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.MissingSegmentRecoveryTest.shard1.replica2
[junit4] 2> 647036 ERROR (coreContainerWorkExecutor-1691-thread-1-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr ] o.a.s.c.CoreContainer Error waiting for SolrCore to be created
[junit4] 2> java.util.concurrent.ExecutionException: org.apache.solr.common.SolrException: Unable to create core [MissingSegmentRecoveryTest_shard1_replica2]
[junit4] 2> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
[junit4] 2> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
[junit4] 2> at org.apache.solr.core.CoreContainer.lambda$load$6(CoreContainer.java:594)
[junit4] 2> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> Caused by: org.apache.solr.common.SolrException: Unable to create core [MissingSegmentRecoveryTest_shard1_replica2]
[junit4] 2> at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:966)
[junit4] 2> at org.apache.solr.core.CoreContainer.lambda$load$5(CoreContainer.java:565)
[junit4] 2> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
[junit4] 2> ... 5 more
[junit4] 2> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
[junit4] 2> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:977)
[junit4] 2> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:830)
[junit4] 2> at org.apache.solr.core.CoreContainer.processCoreCreateException(CoreContainer.java:1026)
[junit4] 2> at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:952)
[junit4] 2> ... 7 more
[junit4] 2> Suppressed: org.apache.solr.common.SolrException: Error opening new searcher
[junit4] 2> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:977)
[junit4] 2> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:830)
[junit4] 2> at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:950)
[junit4] 2> ... 7 more
[junit4] 2> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
[junit4] 2> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2069)
[junit4] 2> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2189)
[junit4] 2> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1071)
[junit4] 2> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:949)
[junit4] 2> ... 9 more
[junit4] 2> Caused by: org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001/tempDir-001/node2/MissingSegmentRecoveryTest_shard1_replica2/data/index/segments_2")))
[junit4] 2> at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:290)
[junit4] 2> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:930)
[junit4] 2> at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:118)
[junit4] 2> at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:93)
[junit4] 2> at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:257)
[junit4] 2> at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:131)
[junit4] 2> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2030)
[junit4] 2> ... 12 more
[junit4] 2> Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001/tempDir-001/node2/MissingSegmentRecoveryTest_shard1_replica2/data/index/segments_2")
[junit4] 2> at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75)
[junit4] 2> at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41)
[junit4] 2> at org.apache.lucene.store.DataInput.readInt(DataInput.java:101)
[junit4] 2> at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:300)
[junit4] 2> at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:288)
[junit4] 2> ... 18 more
[junit4] 2> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
[junit4] 2> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2069)
[junit4] 2> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2189)
[junit4] 2> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1071)
[junit4] 2> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:949)
[junit4] 2> ... 10 more
[junit4] 2> Caused by: org.apache.lucene.index.IndexNotFoundException: no segments* file found in LockValidatingDirectoryWrapper(MMapDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001/tempDir-001/node2/MissingSegmentRecoveryTest_shard1_replica2/data/index.20180303013328993 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e4c7412): files: [write.lock]
[junit4] 2> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:925)
[junit4] 2> at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:118)
[junit4] 2> at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:93)
[junit4] 2> at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:257)
[junit4] 2> at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:131)
[junit4] 2> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2030)
[junit4] 2> ... 13 more
[junit4] 2> 737053 INFO (qtp968564639-3268) [n:127.0.0.1:42561_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=MissingSegmentRecoveryTest&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 737057 INFO (OverseerThreadFactory-1689-thread-1-processing-n:127.0.0.1:42561_solr) [n:127.0.0.1:42561_solr ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
[junit4] 2> 737061 INFO (qtp968564639-3267) [n:127.0.0.1:42561_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.MissingSegmentRecoveryTest.shard1.replica1
[junit4] 2> 737061 INFO (qtp968564639-3267) [n:127.0.0.1:42561_solr ] o.a.s.c.SolrCore [MissingSegmentRecoveryTest_shard1_replica1] CLOSING SolrCore org.apache.solr.core.SolrCore@2e4857ee
[junit4] 2> 737068 INFO (qtp968564639-3267) [n:127.0.0.1:42561_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.MissingSegmentRecoveryTest.shard1.replica1
[junit4] 2> 737084 INFO (qtp968564639-3267) [n:127.0.0.1:42561_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=MissingSegmentRecoveryTest_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=22
[junit4] 2> 737088 INFO (zkCallback-589-thread-2-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 737126 INFO (qtp2111350895-3350) [n:127.0.0.1:35061_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=MissingSegmentRecoveryTest_shard1_replica2&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 737233 INFO (zkCallback-589-thread-2-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 737834 INFO (qtp968564639-3268) [n:127.0.0.1:42561_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=MissingSegmentRecoveryTest&action=DELETE&wt=javabin&version=2} status=0 QTime=780
[junit4] 2> 737834 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[9B90977827D05434]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testLeaderRecovery
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=MissingSegmentRecoveryTest -Dtests.method=testLeaderRecovery -Dtests.seed=9B90977827D05434 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/test-data/enwiki.random.lines.txt -Dtests.locale=el-CY -Dtests.timezone=America/Argentina/San_Luis -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
[junit4] FAILURE 96.3s J1 | MissingSegmentRecoveryTest.testLeaderRecovery <<<
[junit4] > Throwable #1: java.lang.AssertionError: Expected a collection with one shard and two replicas
[junit4] > null
[junit4] > Last available state: DocCollection(MissingSegmentRecoveryTest//collections/MissingSegmentRecoveryTest/state.json/7)={
[junit4] > "replicationFactor":"2",
[junit4] > "shards":{"shard1":{
[junit4] > "range":"80000000-7fffffff",
[junit4] > "state":"active",
[junit4] > "replicas":{
[junit4] > "core_node1":{
[junit4] > "core":"MissingSegmentRecoveryTest_shard1_replica1",
[junit4] > "base_url":"https://127.0.0.1:42561/solr",
[junit4] > "node_name":"127.0.0.1:42561_solr",
[junit4] > "state":"active",
[junit4] > "leader":"true"},
[junit4] > "core_node2":{
[junit4] > "core":"MissingSegmentRecoveryTest_shard1_replica2",
[junit4] > "base_url":"https://127.0.0.1:35061/solr",
[junit4] > "node_name":"127.0.0.1:35061_solr",
[junit4] > "state":"down"}}}},
[junit4] > "router":{"name":"compositeId"},
[junit4] > "maxShardsPerNode":"1",
[junit4] > "autoAddReplicas":"false"}
[junit4] > at __randomizedtesting.SeedInfo.seed([9B90977827D05434:CBC50F7B7EF1E229]:0)
[junit4] > at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:265)
[junit4] > at org.apache.solr.cloud.MissingSegmentRecoveryTest.testLeaderRecovery(MissingSegmentRecoveryTest.java:105)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 737840 INFO (jetty-closer-567-thread-1) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@304ee53e{SSL,[ssl, http/1.1]}{127.0.0.1:35061}
[junit4] 2> 737840 INFO (jetty-closer-567-thread-1) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1792811725
[junit4] 2> 737841 INFO (jetty-closer-567-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
[junit4] 2> 737841 INFO (jetty-closer-567-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jvm
[junit4] 2> 737841 INFO (jetty-closer-567-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jetty
[junit4] 2> 737841 INFO (jetty-closer-567-thread-2) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@13ffe15b{SSL,[ssl, http/1.1]}{127.0.0.1:0}
[junit4] 2> 737841 INFO (jetty-closer-567-thread-2) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=858018595
[junit4] 2> 737841 INFO (jetty-closer-567-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
[junit4] 2> 737841 INFO (jetty-closer-567-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jvm
[junit4] 2> 737842 INFO (jetty-closer-567-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jetty
[junit4] 2> 737848 INFO (zkCallback-578-thread-3-processing-n:127.0.0.1:42561_solr) [n:127.0.0.1:42561_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 737849 INFO (jetty-closer-567-thread-2) [ ] o.a.s.c.Overseer Overseer (id=99618101850275846-127.0.0.1:42561_solr-n_0000000001) closing
[junit4] 2> 737850 INFO (OverseerStateUpdate-99618101850275846-127.0.0.1:42561_solr-n_0000000001) [n:127.0.0.1:42561_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:42561_solr
[junit4] 2> 737851 WARN (zkCallback-578-thread-4-processing-n:127.0.0.1:42561_solr) [n:127.0.0.1:42561_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 737852 INFO (jetty-closer-567-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@21f9f662{/solr,null,UNAVAILABLE}
[junit4] 2> 739348 WARN (zkCallback-589-thread-2-processing-n:127.0.0.1:35061_solr) [n:127.0.0.1:35061_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 739349 INFO (jetty-closer-567-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@3a008140{/solr,null,UNAVAILABLE}
[junit4] 2> 739350 ERROR (SUITE-MissingSegmentRecoveryTest-seed#[9B90977827D05434]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 739351 INFO (SUITE-MissingSegmentRecoveryTest-seed#[9B90977827D05434]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:54884 54884
[junit4] 2> 739530 INFO (Thread-787) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:54884 54884
[junit4] 2> 739531 WARN (Thread-787) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/aliases.json
[junit4] 2> 3 /solr/security.json
[junit4] 2> 2 /solr/configs/conf
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/clusterstate.json
[junit4] 2> 4 /solr/collections/MissingSegmentRecoveryTest/state.json
[junit4] 2> 4 /solr/clusterprops.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/collections
[junit4] 2> 3 /solr/live_nodes
[junit4] 2> 2 /solr/overseer/queue
[junit4] 2> 2 /solr/overseer/collection-queue-work
[junit4] 2>
[junit4] 2> 739531 INFO (SUITE-MissingSegmentRecoveryTest-seed#[9B90977827D05434]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_9B90977827D05434-001
[junit4] 2> Mar 03, 2018 4:35:01 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene62): {_version_=BlockTreeOrds(blocksize=128), id=PostingsFormat(name=Memory doPackFST= true)}, docValues:{}, maxPointsInLeafNode=816, maxMBSortInHeap=5.423348067235001, sim=RandomSimilarity(queryNorm=true,coord=yes): {}, locale=el-CY, timezone=America/Argentina/San_Luis
[junit4] 2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=67306928,total=531628032
[junit4] 2> NOTE: All tests run in this JVM: [TestSolrCloudSnapshots, MoreLikeThisHandlerTest, TestRandomFlRTGCloud, TestSolrCloudWithSecureImpersonation, TestFastOutputStream, BasicFunctionalityTest, TestReloadAndDeleteDocs, IndexBasedSpellCheckerTest, FileUtilsTest, TestSizeLimitedDistributedMap, TestCoreContainer, TestSearchPerf, ChaosMonkeySafeLeaderTest, TestBlendedInfixSuggestions, TestPointFields, CollectionsAPISolrJTest, TestMinMaxOnMultiValuedField, TestManagedSchema, TestMacros, DocValuesTest, BadCopyFieldTest, MissingSegmentRecoveryTest]
[junit4] Completed [116/714 (1!)] on J1 in 99.19s, 1 test, 1 failure <<< FAILURES!
[...truncated 93 lines...]
[junit4] Suite: org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9B90977827D05434-001/init-core-data-001
[junit4] 2> 741039 WARN (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=16 numCloses=16
[junit4] 2> 741039 INFO (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields
[junit4] 2> 741040 INFO (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
[junit4] 2> 741040 INFO (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /_he/qw
[junit4] 1> Formatting using clusterid: testClusterID
[junit4] 2> 741910 WARN (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
[junit4] 2> 742127 INFO (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
[junit4] 2> 742144 WARN (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 742200 INFO (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.m.log jetty-6.1.26
[junit4] 2> 742243 INFO (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_34039_hdfs____.7qm30h/webapp
[junit4] 2> 748386 INFO (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34039
[junit4] 2> 749134 WARN (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 749138 INFO (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.m.log jetty-6.1.26
[junit4] 2> 749157 INFO (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_60297_datanode____.1g624j/webapp
[junit4] 2> 750681 INFO (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:60297
[junit4] 2> 750980 WARN (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 750981 INFO (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.m.log jetty-6.1.26
[junit4] 2> 750997 INFO (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_50359_datanode____oua3k3/webapp
[junit4] 2> 751506 INFO (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:50359
[junit4] 2> 751745 ERROR (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9B90977827D05434-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9B90977827D05434-001/tempDir-001/hdfsBaseDir/data/data2/]] heartbeating to localhost/127.0.0.1:44381) [ ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
[junit4] 2> 751805 ERROR (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9B90977827D05434-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9B90977827D05434-001/tempDir-001/hdfsBaseDir/data/data4/]] heartbeating to localhost/127.0.0.1:44381) [ ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
[junit4] 2> 751934 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x482208b6e4a526: from storage DS-eab6854f-eb25-4268-bea0-79217409eda2 node DatanodeRegistration(127.0.0.1:50213, datanodeUuid=6a562b5d-a072-43a4-97e3-4a80b6c44971, infoPort=33173, infoSecurePort=0, ipcPort=51085, storageInfo=lv=-56;cid=testClusterID;nsid=428486816;c=0), blocks: 0, hasStaleStorage: true, processing time: 4 msecs
[junit4] 2> 751934 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x482208b6fcfe37: from storage DS-930c0b2c-1f48-4966-b1e9-eb403480c793 node DatanodeRegistration(127.0.0.1:44050, datanodeUuid=bab998d6-c809-4db4-8e14-6e9b79b0a53e, infoPort=58299, infoSecurePort=0, ipcPort=58930, storageInfo=lv=-56;cid=testClusterID;nsid=428486816;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
[junit4] 2> 751934 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x482208b6e4a526:
[...truncated too long message...]
hdfs_home/solrj_collection/core_node2/data
[junit4] 2> 945873 WARN (zkCallback-644-thread-3-processing-n:127.0.0.1:35226__he%2Fqw) [n:127.0.0.1:35226__he%2Fqw ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 945874 INFO (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[9B90977827D05434]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@11f1ec0b{HTTP/1.1,[http/1.1]}{127.0.0.1:35226}
[junit4] 2> 945874 INFO (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[9B90977827D05434]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@255b5eb5{/_he/qw,null,UNAVAILABLE}
[junit4] 2> 945875 INFO (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[9B90977827D05434]) [ ] o.a.s.c.ChaosMonkey monkey: stop shard! 60627
[junit4] 2> 945875 ERROR (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[9B90977827D05434]) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 945876 INFO (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[9B90977827D05434]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:60154 60154
[junit4] 2> 946144 INFO (Thread-927) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:60154 60154
[junit4] 2> 946146 WARN (Thread-927) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 12 /solr/aliases.json
[junit4] 2> 10 /solr/security.json
[junit4] 2> 10 /solr/configs/conf1
[junit4] 2> 6 /solr/clusterprops.json
[junit4] 2> 4 /solr/collections/collection1/state.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 96 /solr/collections/solrj_collection2/state.json
[junit4] 2> 96 /solr/collections/solrj_collection3/state.json
[junit4] 2> 93 /solr/collections/solrj_collection/state.json
[junit4] 2> 80 /solr/collections/solrj_collection4/state.json
[junit4] 2> 12 /solr/clusterprops.json
[junit4] 2> 12 /solr/clusterstate.json
[junit4] 2> 3 /solr/overseer_elect/election/99618109152231432-127.0.0.1:35149__he%2Fqw-n_0000000001
[junit4] 2> 2 /solr/collections/solrj_collection/leader_elect/shard2/election/99618109152231432-core_node3-n_0000000000
[junit4] 2> 2 /solr/overseer_elect/election/99618109152231448-127.0.0.1:35226__he%2Fqw-n_0000000006
[junit4] 2> 2 /solr/collections/solrj_collection2/leader_elect/shard1/election/99618109152231445-core_node4-n_0000000002
[junit4] 2> 2 /solr/overseer_elect/election/99618109152231428-127.0.0.1:34228__he%2Fqw-n_0000000000
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 12 /solr/collections
[junit4] 2> 7 /solr/live_nodes
[junit4] 2> 3 /solr/overseer/queue
[junit4] 2> 3 /solr/overseer/queue-work
[junit4] 2> 3 /solr/overseer/collection-queue-work
[junit4] 2>
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SharedFSAutoReplicaFailoverTest -Dtests.method=test -Dtests.seed=9B90977827D05434 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/test-data/enwiki.random.lines.txt -Dtests.locale=th-TH-u-nu-thai-x-lvariant-TH -Dtests.timezone=Pacific/Wallis -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
[junit4] FAILURE 194s J1 | SharedFSAutoReplicaFailoverTest.test <<<
[junit4] > Throwable #1: java.lang.AssertionError
[junit4] > at __randomizedtesting.SeedInfo.seed([9B90977827D05434:13C4A8A2892C39CC]:0)
[junit4] > at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:307)
[junit4] > at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:136)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 946148 WARN (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
[junit4] 2> 946159 INFO (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
[junit4] 2> 946261 WARN (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9B90977827D05434-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9B90977827D05434-001/tempDir-001/hdfsBaseDir/data/data4/]] heartbeating to localhost/127.0.0.1:44381) [ ] o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager interrupted
[junit4] 2> 946261 WARN (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9B90977827D05434-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9B90977827D05434-001/tempDir-001/hdfsBaseDir/data/data4/]] heartbeating to localhost/127.0.0.1:44381) [ ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-1675768844-10.41.0.5-1520051703658 (Datanode Uuid 6a562b5d-a072-43a4-97e3-4a80b6c44971) service to localhost/127.0.0.1:44381
[junit4] 2> 946263 WARN (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
[junit4] 2> 946270 INFO (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
[junit4] 2> 946371 WARN (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9B90977827D05434-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9B90977827D05434-001/tempDir-001/hdfsBaseDir/data/data2/]] heartbeating to localhost/127.0.0.1:44381) [ ] o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager interrupted
[junit4] 2> 946371 WARN (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9B90977827D05434-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9B90977827D05434-001/tempDir-001/hdfsBaseDir/data/data2/]] heartbeating to localhost/127.0.0.1:44381) [ ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-1675768844-10.41.0.5-1520051703658 (Datanode Uuid bab998d6-c809-4db4-8e14-6e9b79b0a53e) service to localhost/127.0.0.1:44381
[junit4] 2> 946386 INFO (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
[junit4] 2> 946490 INFO (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9B90977827D05434]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9B90977827D05434-001
[junit4] 2> Mar 03, 2018 4:38:28 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 34 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene62): {id=PostingsFormat(name=LuceneFixedGap), text=BlockTreeOrds(blocksize=128)}, docValues:{_version_=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=258, maxMBSortInHeap=5.347407300784594, sim=RandomSimilarity(queryNorm=false,coord=no): {}, locale=th-TH-u-nu-thai-x-lvariant-TH, timezone=Pacific/Wallis
[junit4] 2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=6,free=202325336,total=532152320
[junit4] 2> NOTE: All tests run in this JVM: [TestSolrCloudSnapshots, MoreLikeThisHandlerTest, TestRandomFlRTGCloud, TestSolrCloudWithSecureImpersonation, TestFastOutputStream, BasicFunctionalityTest, TestReloadAndDeleteDocs, IndexBasedSpellCheckerTest, FileUtilsTest, TestSizeLimitedDistributedMap, TestCoreContainer, TestSearchPerf, ChaosMonkeySafeLeaderTest, TestBlendedInfixSuggestions, TestPointFields, CollectionsAPISolrJTest, TestMinMaxOnMultiValuedField, TestManagedSchema, TestMacros, DocValuesTest, BadCopyFieldTest, MissingSegmentRecoveryTest, SolrIndexSplitterTest, SharedFSAutoReplicaFailoverTest]
[junit4] Completed [147/714 (2!)] on J1 in 214.36s, 1 test, 1 failure <<< FAILURES!
[...truncated 48150 lines...]