You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2016/11/21 14:30:45 UTC

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 960 - Unstable!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/960/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

3 tests failed.
FAILED:  org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail

Error Message:
expected:<200> but was:<404>

Stack Trace:
java.lang.AssertionError: expected:<200> but was:<404>
	at __randomizedtesting.SeedInfo.seed([D2B24768EBA4448B:BA0D72423B3E5667]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.cancelDelegationToken(TestSolrCloudWithDelegationTokens.java:140)
	at org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail(TestSolrCloudWithDelegationTokens.java:294)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.StressHdfsTest.test

Error Message:
Error from server at http://127.0.0.1:58922: Could not fully remove collection: delete_data_dir

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:58922: Could not fully remove collection: delete_data_dir
	at __randomizedtesting.SeedInfo.seed([D2B24768EBA4448B:5AE678B245582973]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:610)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:262)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:439)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:391)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1288)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1058)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1000)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.hdfs.StressHdfsTest.createAndDeleteCollection(StressHdfsTest.java:220)
	at org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:103)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.update.AutoCommitTest.testMaxTime

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([D2B24768EBA4448B:48463A8A753ED8B7]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:812)
	at org.apache.solr.update.AutoCommitTest.testMaxTime(AutoCommitTest.java:270)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
	xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
</response>

	request was:q=id:529&qt=standard&start=0&rows=20&version=2.2
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:805)
	... 40 more




Build Log:
[...truncated 10881 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.StressHdfsTest
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001/init-core-data-001
   [junit4]   2> 309051 INFO  (SUITE-StressHdfsTest-seed#[D2B24768EBA4448B]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 309052 INFO  (SUITE-StressHdfsTest-seed#[D2B24768EBA4448B]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 309096 WARN  (SUITE-StressHdfsTest-seed#[D2B24768EBA4448B]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 309103 WARN  (SUITE-StressHdfsTest-seed#[D2B24768EBA4448B]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 309104 INFO  (SUITE-StressHdfsTest-seed#[D2B24768EBA4448B]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 309116 INFO  (SUITE-StressHdfsTest-seed#[D2B24768EBA4448B]-worker) [    ] o.m.log Extract jar:file:/export/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_solaris.vm_36538_hdfs____hc90h5/webapp
   [junit4]   2> 309654 INFO  (SUITE-StressHdfsTest-seed#[D2B24768EBA4448B]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@solaris-vm:36538
   [junit4]   2> 309803 WARN  (SUITE-StressHdfsTest-seed#[D2B24768EBA4448B]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 309806 INFO  (SUITE-StressHdfsTest-seed#[D2B24768EBA4448B]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 309834 INFO  (SUITE-StressHdfsTest-seed#[D2B24768EBA4448B]-worker) [    ] o.m.log Extract jar:file:/export/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_37060_datanode____nju1v1/webapp
   [junit4]   2> 310402 INFO  (SUITE-StressHdfsTest-seed#[D2B24768EBA4448B]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37060
   [junit4]   2> 310533 WARN  (SUITE-StressHdfsTest-seed#[D2B24768EBA4448B]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 310534 INFO  (SUITE-StressHdfsTest-seed#[D2B24768EBA4448B]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 310547 INFO  (SUITE-StressHdfsTest-seed#[D2B24768EBA4448B]-worker) [    ] o.m.log Extract jar:file:/export/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_53431_datanode____8ybgnv/webapp
   [junit4]   2> 310644 INFO  (IPC Server handler 7 on 53673) [    ] BlockStateChange BLOCK* processReport: from storage DS-a57cbd08-be16-4d21-beef-e70654a2beca node DatanodeRegistration(127.0.0.1:64527, datanodeUuid=d513892c-1281-4c6f-ba14-d1978bf07f44, infoPort=38584, infoSecurePort=0, ipcPort=34735, storageInfo=lv=-56;cid=testClusterID;nsid=418953656;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 310644 INFO  (IPC Server handler 7 on 53673) [    ] BlockStateChange BLOCK* processReport: from storage DS-1eb28edb-c233-45b4-aed1-56402e4dd49c node DatanodeRegistration(127.0.0.1:64527, datanodeUuid=d513892c-1281-4c6f-ba14-d1978bf07f44, infoPort=38584, infoSecurePort=0, ipcPort=34735, storageInfo=lv=-56;cid=testClusterID;nsid=418953656;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 311151 INFO  (SUITE-StressHdfsTest-seed#[D2B24768EBA4448B]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:53431
   [junit4]   2> 311314 INFO  (IPC Server handler 8 on 53673) [    ] BlockStateChange BLOCK* processReport: from storage DS-fbe16b57-9c72-4e8d-b88a-5f4dd17c8c90 node DatanodeRegistration(127.0.0.1:50389, datanodeUuid=25623a32-71cb-4984-b6d1-1056f122fe60, infoPort=63333, infoSecurePort=0, ipcPort=41165, storageInfo=lv=-56;cid=testClusterID;nsid=418953656;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 311314 INFO  (IPC Server handler 8 on 53673) [    ] BlockStateChange BLOCK* processReport: from storage DS-48319414-c43d-4d18-806d-e246910e8177 node DatanodeRegistration(127.0.0.1:50389, datanodeUuid=25623a32-71cb-4984-b6d1-1056f122fe60, infoPort=63333, infoSecurePort=0, ipcPort=41165, storageInfo=lv=-56;cid=testClusterID;nsid=418953656;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 311438 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 311439 INFO  (Thread-616) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 311439 INFO  (Thread-616) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 311538 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.ZkTestServer start zk server on port:55352
   [junit4]   2> 311557 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 311565 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 311567 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 311571 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 311573 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 311576 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 311578 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 311581 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 311584 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 311586 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 311589 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 311754 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001/control-001/cores/collection1
   [junit4]   2> 311755 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 311757 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@28cf0d9{/,null,AVAILABLE}
   [junit4]   2> 311757 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.e.j.s.ServerConnector Started ServerConnector@a5e54d3{HTTP/1.1,[http/1.1]}{127.0.0.1:34057}
   [junit4]   2> 311757 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.e.j.s.Server Started @315475ms
   [junit4]   2> 311757 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://solaris-vm:53673/hdfs__solaris-vm_53673__export_home_jenkins_workspace_Lucene-Solr-master-Solaris_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001_tempDir-002_control_data, hostContext=/, hostPort=34057, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001/control-001/cores}
   [junit4]   2> 311758 ERROR (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 311759 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 311759 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 311759 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 311759 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2016-11-21T13:34:39.950Z
   [junit4]   2> 311764 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 311765 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001/control-001/solr.xml
   [junit4]   2> 311771 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:55352/solr
   [junit4]   2> 311817 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [n:127.0.0.1:34057_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34057_
   [junit4]   2> 311819 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [n:127.0.0.1:34057_    ] o.a.s.c.Overseer Overseer (id=96975931285897220-127.0.0.1:34057_-n_0000000000) starting
   [junit4]   2> 311831 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [n:127.0.0.1:34057_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34057_
   [junit4]   2> 311861 INFO  (zkCallback-356-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 311998 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [n:127.0.0.1:34057_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001/control-001/cores
   [junit4]   2> 311998 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [n:127.0.0.1:34057_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 312005 INFO  (OverseerStateUpdate-96975931285897220-127.0.0.1:34057_-n_0000000000) [n:127.0.0.1:34057_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 313018 WARN  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 313019 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 313030 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 313109 WARN  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 313111 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 313132 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 313144 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://solaris-vm:53673/solr_hdfs_home
   [junit4]   2> 313144 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 313145 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001/control-001/cores/collection1], dataDir=[hdfs://solaris-vm:53673/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 313145 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@71e3acdb
   [junit4]   2> 313145 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://solaris-vm:53673/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 313167 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 313167 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 313167 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new global HDFS BlockCache
   [junit4]   2> 315189 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 315197 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://solaris-vm:53673/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 315225 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://solaris-vm:53673/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 315233 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 315233 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 315239 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 315240 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=97.2197265625, floorSegmentMB=0.69140625, forceMergeDeletesPctAllowed=3.9669638217323113, segmentsPerTier=49.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 315279 INFO  (IPC Server handler 2 on 53673) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:64527 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-48319414-c43d-4d18-806d-e246910e8177:NORMAL:127.0.0.1:50389|RBW], ReplicaUC[[DISK]DS-a57cbd08-be16-4d21-beef-e70654a2beca:NORMAL:127.0.0.1:64527|FINALIZED]]} size 0
   [junit4]   2> 315280 INFO  (IPC Server handler 1 on 53673) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50389 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-48319414-c43d-4d18-806d-e246910e8177:NORMAL:127.0.0.1:50389|RBW], ReplicaUC[[DISK]DS-a57cbd08-be16-4d21-beef-e70654a2beca:NORMAL:127.0.0.1:64527|FINALIZED]]} size 0
   [junit4]   2> 315309 WARN  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 315331 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 315331 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 315331 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 315364 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 315364 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 315366 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=38, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 315457 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@126fa84e[collection1] main]
   [junit4]   2> 315458 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 315459 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 315462 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 315464 INFO  (searcherExecutor-1086-thread-1-processing-n:127.0.0.1:34057_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@126fa84e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 315464 INFO  (coreLoadExecutor-1085-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1551614904793825280
   [junit4]   2> 315477 INFO  (coreZkRegister-1078-thread-1-processing-n:127.0.0.1:34057_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 315477 INFO  (coreZkRegister-1078-thread-1-processing-n:127.0.0.1:34057_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 315477 INFO  (coreZkRegister-1078-thread-1-processing-n:127.0.0.1:34057_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:34057/collection1/
   [junit4]   2> 315477 INFO  (coreZkRegister-1078-thread-1-processing-n:127.0.0.1:34057_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 315478 INFO  (coreZkRegister-1078-thread-1-processing-n:127.0.0.1:34057_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:34057/collection1/ has no replicas
   [junit4]   2> 315487 INFO  (coreZkRegister-1078-thread-1-processing-n:127.0.0.1:34057_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:34057/collection1/ shard1
   [junit4]   2> 315512 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 315514 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 315514 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 315538 INFO  (coreZkRegister-1078-thread-1-processing-n:127.0.0.1:34057_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:34057_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 315688 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001/shard-1-001/cores/collection1
   [junit4]   2> 315689 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001/shard-1-001
   [junit4]   2> 315690 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 315691 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@51c98454{/,null,AVAILABLE}
   [junit4]   2> 315692 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.e.j.s.ServerConnector Started ServerConnector@4ed0b631{HTTP/1.1,[http/1.1]}{127.0.0.1:54513}
   [junit4]   2> 315693 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.e.j.s.Server Started @319411ms
   [junit4]   2> 315693 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://solaris-vm:53673/hdfs__solaris-vm_53673__export_home_jenkins_workspace_Lucene-Solr-master-Solaris_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=54513, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001/shard-1-001/cores}
   [junit4]   2> 315693 ERROR (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 315694 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 315694 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 315694 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 315694 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2016-11-21T13:34:43.885Z
   [junit4]   2> 315699 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 315699 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001/shard-1-001/solr.xml
   [junit4]   2> 315705 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:55352/solr
   [junit4]   2> 315723 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [n:127.0.0.1:54513_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 315730 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [n:127.0.0.1:54513_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54513_
   [junit4]   2> 315732 INFO  (zkCallback-356-thread-2-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 315733 INFO  (zkCallback-360-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 315736 INFO  (zkCallback-366-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 315811 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [n:127.0.0.1:54513_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001/shard-1-001/cores
   [junit4]   2> 315811 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [n:127.0.0.1:54513_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 315815 INFO  (OverseerStateUpdate-96975931285897220-127.0.0.1:34057_-n_0000000000) [n:127.0.0.1:34057_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 315921 INFO  (zkCallback-366-thread-2-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 316836 WARN  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 316838 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 316850 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 316941 WARN  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 316943 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 316962 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 316962 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://solaris-vm:53673/solr_hdfs_home
   [junit4]   2> 316962 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 316962 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001/shard-1-001/cores/collection1], dataDir=[hdfs://solaris-vm:53673/solr_hdfs_home/collection1/core_node1/data/]
   [junit4]   2> 316962 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@71e3acdb
   [junit4]   2> 316963 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://solaris-vm:53673/solr_hdfs_home/collection1/core_node1/data/snapshot_metadata
   [junit4]   2> 316972 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 316972 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 316978 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 316979 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://solaris-vm:53673/solr_hdfs_home/collection1/core_node1/data
   [junit4]   2> 317005 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://solaris-vm:53673/solr_hdfs_home/collection1/core_node1/data/index
   [junit4]   2> 317013 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 317013 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 317019 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 317019 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=97.2197265625, floorSegmentMB=0.69140625, forceMergeDeletesPctAllowed=3.9669638217323113, segmentsPerTier=49.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 317050 INFO  (IPC Server handler 5 on 53673) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50389 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a57cbd08-be16-4d21-beef-e70654a2beca:NORMAL:127.0.0.1:64527|RBW], ReplicaUC[[DISK]DS-fbe16b57-9c72-4e8d-b88a-5f4dd17c8c90:NORMAL:127.0.0.1:50389|RBW]]} size 0
   [junit4]   2> 317052 INFO  (IPC Server handler 7 on 53673) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:64527 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-fbe16b57-9c72-4e8d-b88a-5f4dd17c8c90:NORMAL:127.0.0.1:50389|RBW], ReplicaUC[[DISK]DS-1eb28edb-c233-45b4-aed1-56402e4dd49c:NORMAL:127.0.0.1:64527|FINALIZED]]} size 0
   [junit4]   2> 317086 WARN  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 317154 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 317154 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 317154 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 317167 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 317167 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 317169 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=38, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 317181 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7a8b9196[collection1] main]
   [junit4]   2> 317182 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 317183 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 317183 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 317184 INFO  (searcherExecutor-1097-thread-1-processing-n:127.0.0.1:54513_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7a8b9196[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 317184 INFO  (coreLoadExecutor-1096-thread-1-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1551614906597376000
   [junit4]   2> 317197 INFO  (coreZkRegister-1091-thread-1-processing-n:127.0.0.1:54513_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 317197 INFO  (coreZkRegister-1091-thread-1-processing-n:127.0.0.1:54513_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 317198 INFO  (coreZkRegister-1091-thread-1-processing-n:127.0.0.1:54513_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:54513/collection1/
   [junit4]   2> 317198 INFO  (coreZkRegister-1091-thread-1-processing-n:127.0.0.1:54513_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 317199 INFO  (coreZkRegister-1091-thread-1-processing-n:127.0.0.1:54513_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:54513/collection1/ has no replicas
   [junit4]   2> 317206 INFO  (coreZkRegister-1091-thread-1-processing-n:127.0.0.1:54513_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:54513/collection1/ shard1
   [junit4]   2> 317311 INFO  (zkCallback-366-thread-2-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 317360 INFO  (coreZkRegister-1091-thread-1-processing-n:127.0.0.1:54513_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54513_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 317468 INFO  (zkCallback-366-thread-2-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 317485 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001/shard-2-001/cores/collection1
   [junit4]   2> 317486 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001/shard-2-001
   [junit4]   2> 317487 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 317489 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@77921d2{/,null,AVAILABLE}
   [junit4]   2> 317490 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.e.j.s.ServerConnector Started ServerConnector@4643c81b{HTTP/1.1,[http/1.1]}{127.0.0.1:58922}
   [junit4]   2> 317490 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.e.j.s.Server Started @321210ms
   [junit4]   2> 317491 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://solaris-vm:53673/hdfs__solaris-vm_53673__export_home_jenkins_workspace_Lucene-Solr-master-Solaris_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001_tempDir-002_jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=58922, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001/shard-2-001/cores}
   [junit4]   2> 317491 ERROR (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 317491 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 317492 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 317492 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 317492 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2016-11-21T13:34:45.683Z
   [junit4]   2> 317496 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 317497 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001/shard-2-001/solr.xml
   [junit4]   2> 317504 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:55352/solr
   [junit4]   2> 317521 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [n:127.0.0.1:58922_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 317529 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [n:127.0.0.1:58922_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58922_
   [junit4]   2> 317531 INFO  (zkCallback-360-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 317532 INFO  (zkCallback-356-thread-2-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 317531 INFO  (zkCallback-366-thread-2-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 317536 INFO  (zkCallback-372-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 317568 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [n:127.0.0.1:58922_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001/shard-2-001/cores
   [junit4]   2> 317568 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [n:127.0.0.1:58922_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 317573 INFO  (OverseerStateUpdate-96975931285897220-127.0.0.1:34057_-n_0000000000) [n:127.0.0.1:34057_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 317679 INFO  (zkCallback-366-thread-2-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 317679 INFO  (zkCallback-372-thread-2-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 318587 WARN  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 318588 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 318600 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 318698 WARN  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 318699 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 318716 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 318716 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://solaris-vm:53673/solr_hdfs_home
   [junit4]   2> 318716 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 318716 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001/shard-2-001/cores/collection1], dataDir=[hdfs://solaris-vm:53673/solr_hdfs_home/collection1/core_node2/data/]
   [junit4]   2> 318717 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@71e3acdb
   [junit4]   2> 318717 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://solaris-vm:53673/solr_hdfs_home/collection1/core_node2/data/snapshot_metadata
   [junit4]   2> 318726 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 318726 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 318730 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 318732 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://solaris-vm:53673/solr_hdfs_home/collection1/core_node2/data
   [junit4]   2> 318754 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://solaris-vm:53673/solr_hdfs_home/collection1/core_node2/data/index
   [junit4]   2> 318761 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 318761 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 318766 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 318766 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=97.2197265625, floorSegmentMB=0.69140625, forceMergeDeletesPctAllowed=3.9669638217323113, segmentsPerTier=49.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 318807 INFO  (IPC Server handler 8 on 53673) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:64527 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-48319414-c43d-4d18-806d-e246910e8177:NORMAL:127.0.0.1:50389|RBW], ReplicaUC[[DISK]DS-a57cbd08-be16-4d21-beef-e70654a2beca:NORMAL:127.0.0.1:64527|RBW]]} size 0
   [junit4]   2> 318817 INFO  (IPC Server handler 1 on 53673) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50389 is added to blk_1073741827_1003 size 71
   [junit4]   2> 318825 WARN  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 318859 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 318859 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 318859 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 318872 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 318872 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 318875 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=38, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 318885 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@d735472[collection1] main]
   [junit4]   2> 318887 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 318888 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 318888 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 318890 INFO  (searcherExecutor-1108-thread-1-processing-n:127.0.0.1:58922_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@d735472[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 318890 INFO  (coreLoadExecutor-1107-thread-1-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1551614908386246656
   [junit4]   2> 318897 INFO  (coreZkRegister-1102-thread-1-processing-n:127.0.0.1:58922_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 318897 INFO  (updateExecutor-369-thread-1-processing-n:127.0.0.1:58922_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 318897 INFO  (recoveryExecutor-370-thread-1-processing-n:127.0.0.1:58922_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 318898 INFO  (recoveryExecutor-370-thread-1-processing-n:127.0.0.1:58922_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 318898 INFO  (recoveryExecutor-370-thread-1-processing-n:127.0.0.1:58922_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
   [junit4]   2> 318898 INFO  (recoveryExecutor-370-thread-1-processing-n:127.0.0.1:58922_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. HDFSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 318898 INFO  (recoveryExecutor-370-thread-1-processing-n:127.0.0.1:58922_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:54513/collection1/] and I am [http://127.0.0.1:58922/collection1/]
   [junit4]   2> 318902 INFO  (recoveryExecutor-370-thread-1-processing-n:127.0.0.1:58922_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:58922_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:54513]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:58922_&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 318907 INFO  (qtp1462804224-2532) [n:127.0.0.1:54513_    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 318907 INFO  (qtp1462804224-2532) [n:127.0.0.1:54513_    ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 318907 INFO  (qtp1462804224-2532) [n:127.0.0.1:54513_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:58922_, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"http://127.0.0.1:58922","node_name":"127.0.0.1:58922_","state":"down"}
   [junit4]   2> 319007 INFO  (zkCallback-366-thread-2-processing-n:127.0.0.1:54513_) [n:127.0.0.1:54513_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 319007 INFO  (zkCallback-372-thread-2-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 319080 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 319080 INFO  (TEST-StressHdfsTest.test-seed#[D2B24768EBA4448B]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Turning on auto soft commit: 1000
   [junit4]   2> 319085 INFO  (qtp1462804224-2534) [n:127.0.0.1:54513_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&maxShardsPerNode=1&name=delete_data_dir&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 319089 INFO  (OverseerThreadFactory-1083-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_    ] o.a.s.c.CreateCollectionCmd Create collection delete_data_dir
   [junit4]   2> 319090 INFO  (OverseerThreadFactory-1083-thread-1-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_    ] o.a.s.c.CreateCollectionCmd Only one config set found in zk - using it:conf1
   [junit4]   2> 319302 INFO  (qtp601681391-2566) [n:127.0.0.1:58922_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=delete_data_dir_shard1_replica2&action=CREATE&numShards=1&collection=delete_data_dir&shard=shard1&wt=javabin&version=2
   [junit4]   2> 319303 INFO  (qtp67929762-2482) [n:127.0.0.1:34057_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=delete_data_dir_shard1_replica1&action=CREATE&numShards=1&collection=delete_data_dir&shard=shard1&wt=javabin&version=2
   [junit4]   2> 319418 INFO  (zkCallback-356-thread-3-processing-n:127.0.0.1:34057_) [n:127.0.0.1:34057_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delete_data_dir/state.json] for collection [delete_data_dir] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 319418 INFO  (zkCallback-372-thread-2-processing-n:127.0.0.1:58922_) [n:127.0.0.1:58922_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delete_data_dir/state.json] for collection [delete_data_dir] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 319910 INFO  (qtp1462804224-2532) [n:127.0.0.1:54513_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:58922_, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","dataDir":"hdfs://solaris-vm:53673/solr_hdfs_home/collection1/core_node2/data/","base_url":"http://127.0.0.1:58922","node_name":"127.0.0.1:58922_","state":"recovering","ulogDir":"hdfs://solaris-vm:53673/solr_hdfs_home/collection1/core_node2/data/tlog"}
   [junit4]   2> 319910 INFO  (qtp1462804224-2532) [n:127.0.0.1:54513_    ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 319910 INFO  (qtp1462804224-2532) [n:127.0.0.1:54513_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:58922_&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1004
   [junit4]   2> 320325 WARN  (qtp67929762-2482) [n:127.0.0.1:34057_ c:delete_data_dir s:shard1  x:delete_data_dir_shard1_replica1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 320326 INFO  (qtp67929762-2482) [n:127.0.0.1:34057_ c:delete_data_dir s:shard1  x:delete_data_dir_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 320327 WARN  (qtp601681391-2566) [n:127.0.0.1:58922_ c:delete_data_dir s:shard1  x:delete_data_dir_shard1_replica2] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 320327 INFO  (qtp601681391-2566) [n:127.0.0.1:58922_ c:delete_data_dir s:shard1  x:delete_data_dir_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 320353 INFO  (qtp67929762-2482) [n:127.0.0.1:34057_ c:delete_data_dir s:shard1  x:delete_data_dir_shard1_replica1] o.a.s.s.IndexSchema [delete_data_dir_shard1_replica1] Schema name=test
   [junit4]   2> 320353 INFO  (qtp601681391-2566) [n:127.0.0.1:58922_ c:delete_data_dir s:shard1  x:delete_data_dir_shard1_replica2] o.a.s.s.IndexSchema [delete_data_dir_shard1_replica2] Schema name=test
   [junit4]   2> 320459 WARN  (qtp67929762-2482) [n:127.0.0.1:34057_ c:delete_data_dir s:shard1  x:delete_data_dir_shard1_replica1] o.a.s.s.IndexSchema [delete_data_dir_shard1_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 320474 INFO  (qtp67929762-2482) [n:127.0.0.1:34057_ c:delete_data_dir s:shard1  x:delete_data_dir_shard1_replica1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 320489 INFO  (qtp67929762-2482) [n:127.0.0.1:34057_ c:delete_data_dir s:shard1  x:delete_data_dir_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'delete_data_dir_shard1_replica1' using configuration from collection delete_data_dir
   [junit4]   2> 320489 INFO  (qtp67929762-2482) [n:127.0.0.1:34057_ c:delete_data_dir s:shard1 r:core_node1 x:delete_data_dir_shard1_replica1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://solaris-vm:53673/solr_hdfs_home
   [junit4]   2> 320489 INFO  (qtp67929762-2482) [n:127.0.0.1:34057_ c:delete_data_dir s:shard1 r:core_node1 x:delete_data_dir_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 320490 INFO  (qtp67929762-2482) [n:127.0.0.1:34057_ c:delete_data_dir s:shard1 r:core_node1 x:delete_data_dir_shard1_replica1] o.a.s.c.SolrCore [[delete_data_dir_shard1_replica1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001/control-001/cores/delete_data_dir_shard1_replica1], dataDir=[hdfs://solaris-vm:53673/solr_hdfs_home/delete_data_dir/core_node1/data/]
   [junit4]   2> 320490 INFO  (qtp67929762-2482) [n:127.0.0.1:34057_ c:delete_data_dir s:shard1 r:core_node1 x:delete_data_dir_shard1_replica1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@71e3acdb
   [junit4]   2> 320490 INFO  (qtp67929762-2482) [n:127.0.0.1:34057_ c:delete_data_dir s:shard1 r:core_node1 x:delete_data_dir_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://solaris-vm:53673/solr_hdfs_home/delete_data_dir/core_node1/data/snapshot_metadata
   [junit4]   2> 320499 INFO  (qtp67929762-2482) [n:127.0.0.1:34057_ c:delete_data_dir s:shard1 r:core_node1 x:delete_data_dir_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 320499 INFO  (qtp67929762-2482) [n:127.0.0.1:34057_ c:delete_data_dir s:shard1 r:core_node1 x:delete_data_dir_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 320526 INFO  (qtp67929762-2482) [n:127.0.0.1:34057_ c:delete_data_dir s:shard1 r:core_node1 x:delete_data_dir_shard1_replica1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 320528 WARN  (qtp601681391-2566) [n:127.0.0.1:58922_ c:delete_data_dir s:shard1  x:delete_data_dir_shard1_replica2] o.a.s.s.IndexSchema [delete_data_dir_shard1_replica2] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 320529 INFO  (qtp67929762-2482) [n:127.0.0.1:34057_ c:delete_data_dir s:shard1 r:core_node1 x:delete_data_dir_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://solaris-vm:53673/solr_hdfs_home/delete_data_dir/core_node1/data
   [junit4]   2> 320540 INFO  (qtp601681391-2566) [n:127.0.0.1:58922_ c:delete_data_dir s:shard1  x:delete_data_dir_shard1_replica2] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 320552 INFO  (qtp67929762-2482) [n:127.0.0.1:34057_ c:delete_data_dir s:shard1 r:core_node1 x:delete_data_dir_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://solaris-vm:53673/solr_hdfs_home/delete_data_dir/core_node1/data/index
   [junit4]   2> 320560 INFO  (qtp67929762-2482) [n:127.0.0.1:34057_ c:delete_data_dir s:shard1 r:core_node1 x:delete_data_dir_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 320560 INFO  (qtp67929762-2482) [n:127.0.0.1:34057_ c:delete_data_dir s:shard1 r:core_node1 x:delete_data_dir_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 320580 INFO  (qtp601681391-2566) [n:127.0.0.1:58922_ c:delete_data_dir s:shard1  x:delete_data_dir_shard1_replica2] o.a.s.c.CoreContainer Creating SolrCore 'delete_data_dir_shard1_replica2' using configuration from collection delete_data_dir
   [junit4]   2> 320582 INFO  (qtp601681391-2566) [n:127.0.0.1:58922_ c:delete_data_dir s:shard1 r:core_node2 x:delete_data_dir_shard1_replica2] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://solaris-vm:53673/solr_hdfs_home
   [junit4]   2> 320582 INFO  (qtp601681391-2566) [n:127.0.0.1:58922_ c:delete_data_dir s:shard1 r:core_node2 x:delete_data_dir_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 320582 INFO  (qtp601681391-2566) [n:127.0.0.1:58922_ c:delete_data_dir s:shard1 r:core_node2 x:delete_data_dir_shard1_replica2] o.a.s.c.SolrCore [[delete_data_dir_shard1_replica2] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_D2B24768EBA4448B-001/shard-2-001/cores/delete_data_dir_shard1_replica2], dataDir=[hdfs://solaris-vm:53673/solr_hdfs_home/delete_data_dir/core_node2/data/]
   [junit4]   2> 320582 INFO  

[...truncated too long message...]

ooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 2175798 ERROR (Thread[Thread-6990,5,TGRP-TestSolrCloudWithDelegationTokens]) [n:127.0.0.1:56964_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 2175800 INFO  (jetty-launcher-4242-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@48fd6bae{/solr,null,UNAVAILABLE}
   [junit4]   2> 2175803 INFO  (jetty-launcher-4242-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@f7d0c4{/solr,null,UNAVAILABLE}
   [junit4]   2> 2175805 INFO  (SUITE-TestSolrCloudWithDelegationTokens-seed#[D2B24768EBA4448B]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:50228 50228
   [junit4]   2> 2175828 INFO  (Thread-6970) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:50228 50228
   [junit4]   2> 2175829 WARN  (Thread-6970) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	14	/solr/aliases.json
   [junit4]   2> 	14	/solr/clusterprops.json
   [junit4]   2> 	2	/solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	14	/solr/clusterstate.json
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_7
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_6
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_9
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_1
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_1
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_3
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_3
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_2
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_2
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_5
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_4
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_4
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_12
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_11
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_10
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	14	/solr/live_nodes
   [junit4]   2> 	14	/solr/collections
   [junit4]   2> 	2	/solr/overseer/queue
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot
   [junit4]   2> 	2	/solr/overseer/queue-work
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot
   [junit4]   2> 
   [junit4]   2> 2175829 INFO  (SUITE-TestSolrCloudWithDelegationTokens-seed#[D2B24768EBA4448B]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithDelegationTokens_D2B24768EBA4448B-001
   [junit4]   2> Nov 21, 2016 2:05:46 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 3 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=798, maxMBSortInHeap=7.238852504779036, sim=RandomSimilarity(queryNorm=false): {}, locale=be, timezone=Europe/Vatican
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_102 (64-bit)/cpus=3,threads=1,free=124415536,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestXmlQParser, ConcurrentDeleteAndCreateCollectionTest, SharedFSAutoReplicaFailoverTest, TestMergePolicyConfig, TestConfigSetsAPI, SpatialFilterTest, TestRandomDVFaceting, XmlUpdateRequestHandlerTest, TestJsonRequest, HdfsSyncSliceTest, QueryElevationComponentTest, DirectoryFactoryTest, PreAnalyzedFieldManagedSchemaCloudTest, TestSubQueryTransformerDistrib, TestIBSimilarityFactory, SolrCoreTest, TestSort, AddSchemaFieldsUpdateProcessorFactoryTest, TestFieldCollectionResource, TestBinaryField, StatsReloadRaceTest, LeaderElectionTest, XsltUpdateRequestHandlerTest, HttpSolrCallGetCoreTest, TestReqParamsAPI, TestSolrConfigHandlerCloud, TestFunctionQuery, SuggesterFSTTest, TestSolrJ, TestDistributedSearch, TestStressLucene, TestJmxIntegration, SoftAutoCommitTest, BadCopyFieldTest, DistributedFacetPivotLargeTest, ChangedSchemaMergeTest, TestPartialUpdateDeduplication, PingRequestHandlerTest, TestDynamicFieldCollectionResource, DistributedQueryComponentOptimizationTest, TestRequestForwarding, SimplePostToolTest, TestConfigSetImmutable, SecurityConfHandlerTest, TestSolrCloudWithKerberosAlt, TestReplicationHandler, TestConfigSets, DirectUpdateHandlerTest, TestRebalanceLeaders, CollectionsAPIAsyncDistributedZkTest, ZkCLITest, ConnectionManagerTest, TestUpdate, TestFieldCacheSanityChecker, HdfsRecoveryZkTest, TestLeaderElectionZkExpiry, DistanceUnitsTest, SolrIndexConfigTest, TestSchemaSimilarityResource, TestSolrDeletionPolicy1, TestFastLRUCache, TestHdfsUpdateLog, BinaryUpdateRequestHandlerTest, TestDownShardTolerantSearch, SOLR749Test, TestQueryWrapperFilter, DistributedQueueTest, AnalyticsQueryTest, OverseerStatusTest, SearchHandlerTest, DOMUtilTest, MBeansHandlerTest, TestRemoteStreaming, DataDrivenBlockJoinTest, TestCursorMarkWithoutUniqueKey, AssignTest, DisMaxRequestHandlerTest, DistributedExpandComponentTest, DistributedFacetPivotSmallAdvancedTest, CollectionStateFormat2Test, MergeStrategyTest, TestSchemaNameResource, VMParamsZkACLAndCredentialsProvidersTest, TestDistributedStatsComponentCardinality, TestLFUCache, TestDefaultStatsCache, TestSolrQueryResponse, SynonymTokenizerTest, TestPerFieldSimilarityWithDefaultOverride, TestAuthenticationFramework, TestNumericTerms32, BlobRepositoryCloudTest, CloudExitableDirectoryReaderTest, DocExpirationUpdateProcessorFactoryTest, DateFieldTest, RollingRestartTest, ForceLeaderTest, TestGraphMLResponseWriter, TestTolerantSearch, RecoveryZkTest, PolyFieldTest, DistributedQueryElevationComponentTest, OutputWriterTest, TestJsonFacetRefinement, TestHighlightDedupGrouping, TestPostingsSolrHighlighter, TestScoreJoinQPNoScore, TestSolrXml, FullHLLTest, BasicAuthStandaloneTest, SimpleFacetsTest, TestBlendedInfixSuggestions, CoreAdminCreateDiscoverTest, TestSortByMinMaxFunction, CachingDirectoryFactoryTest, SuggestComponentContextFilterQueryTest, TestSubQueryTransformer, TestCloudInspectUtil, TestLRUStatsCache, HighlighterTest, UniqFieldsUpdateProcessorFactoryTest, TestUninvertingReader, AnalysisAfterCoreReloadTest, DeleteReplicaTest, TestMacroExpander, CurrencyFieldXmlFileTest, HLLUtilTest, DocValuesMissingTest, TestFieldCache, TestSurroundQueryParser, SortSpecParsingTest, DeleteLastCustomShardedReplicaTest, FastVectorHighlighterTest, TestMiniSolrCloudClusterSSL, TestSQLHandler, TestSearcherReuse, TestCSVLoader, TestHdfsCloudBackupRestore, TestPseudoReturnFields, OverriddenZkACLAndCredentialsProvidersTest, LeaderElectionContextKeyTest, TestBinaryResponseWriter, TestHashPartitioner, TestBulkSchemaAPI, HdfsBasicDistributedZkTest, TestClusterProperties, DistributedSpellCheckComponentTest, StandardRequestHandlerTest, TestSchemaManager, HdfsTlogReplayBufferedWhileIndexingTest, StatelessScriptUpdateProcessorFactoryTest, HdfsChaosMonkeySafeLeaderTest, TestGraphTermsQParserPlugin, TestSolrFieldCacheMBean, TestCloudPseudoReturnFields, PeerSyncReplicationTest, PrimUtilsTest, TestSweetSpotSimilarityFactory, TestImplicitCoreProperties, TestDocBasedVersionConstraints, BasicDistributedZkTest, UnloadDistributedZkTest, SyncSliceTest, OpenCloseCoreStressTest, OverseerTest, ShardRoutingTest, BasicZkTest, FullSolrCloudDistribCmdsTest, ZkSolrClientTest, ZkControllerTest, TestStressReorder, TestStressVersions, DistributedTermsComponentTest, PeerSyncTest, ConvertedLegacyTest, BasicFunctionalityTest, TestLazyCores, CurrencyFieldOpenExchangeTest, TestExtendedDismaxParser, SolrRequestParserTest, SuggesterTest, SuggesterWFSTTest, WordBreakSolrSpellCheckerTest, SchemaVersionSpecificBehaviorTest, TestAtomicUpdateErrorCases, TestWordDelimiterFilterFactory, QueryEqualityTest, DirectUpdateHandlerOptimizeTest, DocValuesMultiTest, DebugComponentTest, CacheHeaderTest, IndexBasedSpellCheckerTest, LukeRequestHandlerTest, TestQueryUtils, TestWriterPerf, DirectSolrSpellCheckerTest, DocumentAnalysisRequestHandlerTest, FileBasedSpellCheckerTest, RequestHandlersTest, TestValueSourceCache, MoreLikeThisHandlerTest, TestSolrQueryParser, LoggingHandlerTest, SolrPluginUtilsTest, ReturnFieldsTest, TestAnalyzedSuggestions, TestPHPSerializedResponseWriter, HighlighterConfigTest, TestQuerySenderNoQuery, TestStressRecovery, SolrTestCaseJ4Test, TestJettySolrRunner, BaseCdcrDistributedZkTest, CdcrBootstrapTest, DeleteInactiveReplicaTest, DeleteShardTest, DistributedVersionInfoTest, LeaderFailoverAfterPartitionTest, MultiThreadedOCPTest, RemoteQueryErrorTest, SharedFSAutoReplicaFailoverUtilsTest, SimpleCollectionCreateDeleteTest, SolrCloudExampleTest, TestCloudPivotFacet, TestConfigSetsAPIExclusivity, TestConfigSetsAPIZkFailure, TestCryptoKeys, TestDeleteCollectionOnDownNodes, TestDistribDocBasedVersion, TestLeaderElectionWithEmptyReplica, TestLocalFSCloudBackupRestore, TestLockTree, TestMiniSolrCloudCluster, TestOnReconnectListenerSupport, TestRandomFlRTGCloud, TestReplicaProperties, TestSolrCloudWithDelegationTokens]
   [junit4] Completed [498/652 (3!)] on J0 in 2.73s, 8 tests, 1 failure <<< FAILURES!

[...truncated 62850 lines...]


[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 962 - Still unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/962/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.embedded.LargeVolumeJettyTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [TransactionLog] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:43)  at org.apache.solr.update.TransactionLog.<init>(TransactionLog.java:188)  at org.apache.solr.update.UpdateLog.newTransactionLog(UpdateLog.java:344)  at org.apache.solr.update.UpdateLog.ensureLog(UpdateLog.java:859)  at org.apache.solr.update.UpdateLog.add(UpdateLog.java:428)  at org.apache.solr.update.UpdateLog.add(UpdateLog.java:415)  at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:299)  at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:211)  at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:166)  at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)  at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)  at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:957)  at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1112)  at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:738)  at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)  at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:97)  at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:179)  at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:135)  at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:298)  at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121)  at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:263)  at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:181)  at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:186)  at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:107)  at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:54)  at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)  at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:153)  at org.apache.solr.core.SolrCore.execute(SolrCore.java:2228)  at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:303)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:113)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:518)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)  at java.lang.Thread.run(Thread.java:745)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [TransactionLog]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:43)
	at org.apache.solr.update.TransactionLog.<init>(TransactionLog.java:188)
	at org.apache.solr.update.UpdateLog.newTransactionLog(UpdateLog.java:344)
	at org.apache.solr.update.UpdateLog.ensureLog(UpdateLog.java:859)
	at org.apache.solr.update.UpdateLog.add(UpdateLog.java:428)
	at org.apache.solr.update.UpdateLog.add(UpdateLog.java:415)
	at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:299)
	at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:211)
	at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:166)
	at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
	at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
	at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:957)
	at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1112)
	at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:738)
	at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
	at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:97)
	at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:179)
	at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:135)
	at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:298)
	at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121)
	at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:263)
	at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:181)
	at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:186)
	at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:107)
	at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:54)
	at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
	at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:153)
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2228)
	at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:303)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:113)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)


	at __randomizedtesting.SeedInfo.seed([76A03F3D30E407EE]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:260)
	at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 13213 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.embedded.LargeVolumeJettyTest
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/init-core-data-001
   [junit4]   2> 75981 INFO  (SUITE-LargeVolumeJettyTest-seed#[76A03F3D30E407EE]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 76027 INFO  (SUITE-LargeVolumeJettyTest-seed#[76A03F3D30E407EE]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 76027 INFO  (SUITE-LargeVolumeJettyTest-seed#[76A03F3D30E407EE]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 76027 INFO  (SUITE-LargeVolumeJettyTest-seed#[76A03F3D30E407EE]-worker) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-002/cores/core
   [junit4]   2> 76028 INFO  (SUITE-LargeVolumeJettyTest-seed#[76A03F3D30E407EE]-worker) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 76029 INFO  (SUITE-LargeVolumeJettyTest-seed#[76A03F3D30E407EE]-worker) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1e9bb327{/solr,null,AVAILABLE}
   [junit4]   2> 76030 INFO  (SUITE-LargeVolumeJettyTest-seed#[76A03F3D30E407EE]-worker) [    ] o.e.j.s.ServerConnector Started ServerConnector@547d5b7b{HTTP/1.1,[http/1.1]}{127.0.0.1:51548}
   [junit4]   2> 76031 INFO  (SUITE-LargeVolumeJettyTest-seed#[76A03F3D30E407EE]-worker) [    ] o.e.j.s.Server Started @80636ms
   [junit4]   2> 76031 INFO  (SUITE-LargeVolumeJettyTest-seed#[76A03F3D30E407EE]-worker) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=51548, configSetBaseDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-002/cores}
   [junit4]   2> 76031 ERROR (SUITE-LargeVolumeJettyTest-seed#[76A03F3D30E407EE]-worker) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 76031 INFO  (SUITE-LargeVolumeJettyTest-seed#[76A03F3D30E407EE]-worker) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.0.0
   [junit4]   2> 76031 INFO  (SUITE-LargeVolumeJettyTest-seed#[76A03F3D30E407EE]-worker) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 76031 INFO  (SUITE-LargeVolumeJettyTest-seed#[76A03F3D30E407EE]-worker) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 76031 INFO  (SUITE-LargeVolumeJettyTest-seed#[76A03F3D30E407EE]-worker) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2016-11-22T09:40:59.845Z
   [junit4]   2> 76031 INFO  (SUITE-LargeVolumeJettyTest-seed#[76A03F3D30E407EE]-worker) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/solr.xml
   [junit4]   2> 76046 INFO  (SUITE-LargeVolumeJettyTest-seed#[76A03F3D30E407EE]-worker) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001
   [junit4]   2> 76046 INFO  (SUITE-LargeVolumeJettyTest-seed#[76A03F3D30E407EE]-worker) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 76180 WARN  (coreLoadExecutor-198-thread-1) [    ] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/../../../../contrib/extraction/lib filtered by .*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/../../../../contrib/extraction/lib
   [junit4]   2> 76203 WARN  (coreLoadExecutor-198-thread-1) [    ] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/../../../../dist filtered by solr-cell-\d.*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/../../../../dist
   [junit4]   2> 76203 WARN  (coreLoadExecutor-198-thread-1) [    ] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/../../../../contrib/clustering/lib filtered by .*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/../../../../contrib/clustering/lib
   [junit4]   2> 76203 WARN  (coreLoadExecutor-198-thread-1) [    ] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/../../../../dist filtered by solr-clustering-\d.*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/../../../../dist
   [junit4]   2> 76204 WARN  (coreLoadExecutor-198-thread-1) [    ] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/../../../../contrib/langid/lib filtered by .*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/../../../../contrib/langid/lib
   [junit4]   2> 76204 WARN  (coreLoadExecutor-198-thread-1) [    ] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/../../../../dist filtered by solr-langid-\d.*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/../../../../dist
   [junit4]   2> 76205 WARN  (coreLoadExecutor-198-thread-1) [    ] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/../../../../contrib/velocity/lib filtered by .*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/../../../../contrib/velocity/lib
   [junit4]   2> 76205 WARN  (coreLoadExecutor-198-thread-1) [    ] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/../../../../dist filtered by solr-velocity-\d.*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/../../../../dist
   [junit4]   2> 76217 INFO  (coreLoadExecutor-198-thread-1) [    ] o.a.s.u.SolrIndexConfig IndexWriter infoStream solr logging is enabled
   [junit4]   2> 76218 INFO  (coreLoadExecutor-198-thread-1) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 76276 INFO  (coreLoadExecutor-198-thread-1) [    ] o.a.s.s.IndexSchema [collection1] Schema name=example
   [junit4]   2> 76792 INFO  (coreLoadExecutor-198-thread-1) [    ] o.a.s.r.RestManager Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedWordSetResource for path /schema/analysis/stopwords/english
   [junit4]   2> 76792 INFO  (coreLoadExecutor-198-thread-1) [    ] o.a.s.r.RestManager Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory$SynonymManager for path /schema/analysis/synonyms/english
   [junit4]   2> 76793 INFO  (coreLoadExecutor-198-thread-1) [    ] o.a.s.s.IndexSchema Loaded schema example/1.6 with uniqueid field id
   [junit4]   2> 76793 INFO  (coreLoadExecutor-198-thread-1) [    ] o.a.s.r.RestManager Added observer of type org.apache.solr.rest.schema.analysis.ManagedStopFilterFactory to existing ManagedResource /schema/analysis/stopwords/english
   [junit4]   2> 76793 INFO  (coreLoadExecutor-198-thread-1) [    ] o.a.s.r.RestManager Added observer of type org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory to existing ManagedResource /schema/analysis/synonyms/english
   [junit4]   2> 76820 INFO  (coreLoadExecutor-198-thread-1) [    ] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1
   [junit4]   2> 76820 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/data/]
   [junit4]   2> 76820 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@38b1fe7b
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-198-thread-1]: init: current segments file is "segments"; deletionPolicy=org.apache.solr.core.IndexDeletionPolicyWrapper@67386612
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-198-thread-1]: delete []
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-198-thread-1]: now checkpoint "" [0 segments ; isCommit = false]
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-198-thread-1]: delete []
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-198-thread-1]: 0 msec to checkpoint
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: init: create=true
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: 
   [junit4]   2> dir=MockDirectoryWrapper(RAMDirectory@1f251e8e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@71e60058)
   [junit4]   2> index=
   [junit4]   2> version=7.0.0
   [junit4]   2> analyzer=org.apache.solr.update.SolrIndexConfig$DelayedSchemaAnalyzer
   [junit4]   2> ramBufferSizeMB=100.0
   [junit4]   2> maxBufferedDocs=-1
   [junit4]   2> maxBufferedDeleteTerms=-1
   [junit4]   2> mergedSegmentWarmer=null
   [junit4]   2> delPolicy=org.apache.solr.core.IndexDeletionPolicyWrapper
   [junit4]   2> commit=null
   [junit4]   2> openMode=CREATE
   [junit4]   2> similarity=org.apache.solr.search.similarities.SchemaSimilarityFactory$SchemaSimilarity
   [junit4]   2> mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=-1, maxMergeCount=-1, ioThrottle=true
   [junit4]   2> codec=Lucene70
   [junit4]   2> infoStream=org.apache.solr.update.LoggingInfoStream
   [junit4]   2> mergePolicy=[TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=5120.0, floorSegmentMB=2.0, forceMergeDeletesPctAllowed=10.0, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1
   [junit4]   2> indexerThreadPool=org.apache.lucene.index.DocumentsWriterPerThreadPool@64a638cc
   [junit4]   2> readerPooling=false
   [junit4]   2> perThreadHardLimitMB=1945
   [junit4]   2> useCompoundFile=false
   [junit4]   2> commitOnClose=true
   [junit4]   2> indexSort=null
   [junit4]   2> writer=org.apache.solr.update.SolrIndexWriter@53c08bb8
   [junit4]   2> 
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: MMapDirectory.UNMAP_SUPPORTED=true
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: now flush at close
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]:   start flush: applyAllDeletes=true
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]:   index before flush 
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][coreLoadExecutor-198-thread-1]: startFullFlush
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][coreLoadExecutor-198-thread-1]: coreLoadExecutor-198-thread-1 finishFullFlush success=true
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: apply all deletes during flush
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: now apply all deletes for all segments maxDoc=0
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][coreLoadExecutor-198-thread-1]: prune sis=segments:  minGen=9223372036854775807 packetCount=0
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [MS][coreLoadExecutor-198-thread-1]: initDynamicDefaults spins=false maxThreadCount=1 maxMergeCount=6
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [MS][coreLoadExecutor-198-thread-1]: updateMergeThreads ioThrottle=true targetMBPerSec=10240.0 MB/sec
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [MS][coreLoadExecutor-198-thread-1]: now merge
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [MS][coreLoadExecutor-198-thread-1]:   index: 
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [MS][coreLoadExecutor-198-thread-1]:   no more merges pending; now return
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: waitForMerges
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: waitForMerges done
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: commit: start
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: commit: enter lock
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: commit: now prepare
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: prepareCommit: flush
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]:   index before flush 
   [junit4]   2> 76831 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][coreLoadExecutor-198-thread-1]: startFullFlush
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: apply all deletes during flush
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: now apply all deletes for all segments maxDoc=0
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][coreLoadExecutor-198-thread-1]: prune sis=segments:  minGen=9223372036854775807 packetCount=0
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][coreLoadExecutor-198-thread-1]: coreLoadExecutor-198-thread-1 finishFullFlush success=true
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: startCommit(): start
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: startCommit index= changeCount=2
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: startCommit: wrote pending segments file "pending_segments_1"
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: done all syncs: []
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: commit: pendingCommit != null
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: commit: done writing segments file "segments_1"
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-198-thread-1]: now checkpoint "" [0 segments ; isCommit = true]
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-198-thread-1]: 0 msec to checkpoint
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-198-thread-1]: delete []
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: commit: took 0.4 msec
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: commit: done
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: rollback
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: all running merges have aborted
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: rollback: done finish merges
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][coreLoadExecutor-198-thread-1]: abort
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][coreLoadExecutor-198-thread-1]: done abort success=true
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: rollback: infos=
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-198-thread-1]: now checkpoint "" [0 segments ; isCommit = false]
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-198-thread-1]: delete []
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-198-thread-1]: 0 msec to checkpoint
   [junit4]   2> 76832 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-198-thread-1]: delete []
   [junit4]   2> 76833 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
   [junit4]   2> 76845 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 76845 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 76852 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms; 
   [junit4]   2> 76852 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 76853 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-198-thread-1]: init: current segments file is "segments_1"; deletionPolicy=org.apache.solr.core.IndexDeletionPolicyWrapper@67386612
   [junit4]   2> 76853 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-198-thread-1]: init: load commit "segments_1"
   [junit4]   2> 76853 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-198-thread-1]: delete []
   [junit4]   2> 76853 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-198-thread-1]: now checkpoint "" [0 segments ; isCommit = false]
   [junit4]   2> 76853 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-198-thread-1]: delete []
   [junit4]   2> 76853 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-198-thread-1]: 0 msec to checkpoint
   [junit4]   2> 76853 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: init: create=false
   [junit4]   2> 76853 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: 
   [junit4]   2> dir=MockDirectoryWrapper(RAMDirectory@1f251e8e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@71e60058)
   [junit4]   2> index=
   [junit4]   2> version=7.0.0
   [junit4]   2> analyzer=org.apache.solr.update.SolrIndexConfig$DelayedSchemaAnalyzer
   [junit4]   2> ramBufferSizeMB=100.0
   [junit4]   2> maxBufferedDocs=-1
   [junit4]   2> maxBufferedDeleteTerms=-1
   [junit4]   2> mergedSegmentWarmer=null
   [junit4]   2> delPolicy=org.apache.solr.core.IndexDeletionPolicyWrapper
   [junit4]   2> commit=null
   [junit4]   2> openMode=APPEND
   [junit4]   2> similarity=org.apache.solr.search.similarities.SchemaSimilarityFactory$SchemaSimilarity
   [junit4]   2> mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=-1, maxMergeCount=-1, ioThrottle=true
   [junit4]   2> codec=Lucene70
   [junit4]   2> infoStream=org.apache.solr.update.LoggingInfoStream
   [junit4]   2> mergePolicy=[TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=5120.0, floorSegmentMB=2.0, forceMergeDeletesPctAllowed=10.0, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1
   [junit4]   2> indexerThreadPool=org.apache.lucene.index.DocumentsWriterPerThreadPool@1a211a94
   [junit4]   2> readerPooling=false
   [junit4]   2> perThreadHardLimitMB=1945
   [junit4]   2> useCompoundFile=false
   [junit4]   2> commitOnClose=true
   [junit4]   2> indexSort=null
   [junit4]   2> writer=org.apache.solr.update.SolrIndexWriter@12274c76
   [junit4]   2> 
   [junit4]   2> 76853 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: MMapDirectory.UNMAP_SUPPORTED=true
   [junit4]   2> 76853 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: flush at getReader
   [junit4]   2> 76853 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][coreLoadExecutor-198-thread-1]: startFullFlush
   [junit4]   2> 76853 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: apply all deletes during flush
   [junit4]   2> 76853 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: now apply all deletes for all segments maxDoc=0
   [junit4]   2> 76853 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][coreLoadExecutor-198-thread-1]: prune sis=segments_1:  minGen=9223372036854775807 packetCount=0
   [junit4]   2> 76853 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: incRefDeleter for NRT reader version=2 segments=
   [junit4]   2> 76853 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: return reader version=2 reader=StandardDirectoryReader(segments_1:2:nrt)
   [junit4]   2> 76853 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][coreLoadExecutor-198-thread-1]: coreLoadExecutor-198-thread-1 finishFullFlush success=true
   [junit4]   2> 76853 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-198-thread-1]: getReader took 0 msec
   [junit4]   2> 76853 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7b7c0539[collection1] main]
   [junit4]   2> 76854 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/conf
   [junit4]   2> 76854 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Loaded LinkedHashMap at path _rest_managed.json using file:dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/conf
   [junit4]   2> 76854 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.r.ManagedResource Loaded initArgs {} for /rest/managed
   [junit4]   2> 76854 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Loaded LinkedHashMap at path _schema_analysis_stopwords_english.json using file:dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/conf
   [junit4]   2> 76854 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.r.ManagedResource Loaded initArgs {ignoreCase=true} for /schema/analysis/stopwords/english
   [junit4]   2> 76854 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.r.s.a.ManagedWordSetResource Loaded 35 words for /schema/analysis/stopwords/english
   [junit4]   2> 76854 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.r.ManagedResource Notified 2 observers of /schema/analysis/stopwords/english
   [junit4]   2> 76854 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Loaded LinkedHashMap at path _schema_analysis_synonyms_english.json using file:dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/conf
   [junit4]   2> 76854 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.r.ManagedResource Loaded initArgs {ignoreCase=true,format=solr} for /schema/analysis/synonyms/english
   [junit4]   2> 76855 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.r.s.a.ManagedSynonymFilterFactory Loaded 3 synonym mappings for /schema/analysis/synonyms/english
   [junit4]   2> 76855 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.r.ManagedResource Notified 2 observers of /schema/analysis/synonyms/english
   [junit4]   2> 76855 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 76855 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=text,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
   [junit4]   2> 76855 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.h.c.SuggestComponent Initializing SuggestComponent
   [junit4]   2> 76855 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.s.s.SolrSuggester init: {name=mySuggester,lookupImpl=FuzzyLookupFactory,dictionaryImpl=DocumentDictionaryFactory,field=cat,weightField=price,suggestAnalyzerFieldType=string,buildOnStartup=false}
   [junit4]   2> 76855 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.s.s.SolrSuggester Dictionary loaded with params: {name=mySuggester,lookupImpl=FuzzyLookupFactory,dictionaryImpl=DocumentDictionaryFactory,field=cat,weightField=price,suggestAnalyzerFieldType=string,buildOnStartup=false}
   [junit4]   2> 76855 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.h.c.QueryElevationComponent Loading QueryElevation from: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.LargeVolumeJettyTest_76A03F3D30E407EE-001/tempDir-001/collection1/conf/elevate.xml
   [junit4]   2> 76860 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 76860 INFO  (searcherExecutor-199-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@7b7c0539[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 76861 INFO  (searcherExecutor-199-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=static+firstSearcher+warming+in+solrconfig.xml&distrib=false&event=firstSearcher} hits=0 status=0 QTime=0
   [junit4]   2> 76861 INFO  (searcherExecutor-199-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.QuerySenderListener QuerySenderListener done.
   [junit4]   2> 76861 INFO  (searcherExecutor-199-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 76861 INFO  (searcherExecutor-199-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 76861 INFO  (searcherExecutor-199-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7b7c0539[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 76863 INFO  (coreLoadExecutor-198-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1551690797602045952
   [junit4]   2> 77049 INFO  (SUITE-LargeVolumeJettyTest-seed#[76A03F3D30E407EE]-worker) [    ] o.a.s.SolrJettyTestBase Jetty Assigned Port#51548
   [junit4]   2> 77053 INFO  (TEST-LargeVolumeJettyTest.testMultiThreaded-seed#[76A03F3D30E407EE]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testMultiThreaded
   [junit4]   2> 77059 INFO  (qtp464770005-887) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-887]: anyChanges? numDocsInRam=0 deletes=true hasTickets:false pendingChangesInFullFlush: false
   [junit4]   2> 77059 INFO  (qtp464770005-887) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-887]: anyChanges? numDocsInRam=0 deletes=true hasTickets:false pendingChangesInFullFlush: false
   [junit4]   2> 77059 INFO  (qtp464770005-887) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-887]: nrtIsCurrent: infoVersion matches: true; DW changes: true; BD changes: false
   [junit4]   2> 77059 INFO  (qtp464770005-887) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-887]: flush at getReader
   [junit4]   2> 77059 INFO  (qtp464770005-887) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-887]: startFullFlush
   [junit4]   2> 77059 INFO  (qtp464770005-887) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-887]: anyChanges? numDocsInRam=0 deletes=true hasTickets:false pendingChangesInFullFlush: false
   [junit4]   2> 77059 INFO  (qtp464770005-887) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-887]: qtp464770005-887: flush naked frozen global deletes
   [junit4]   2> 77059 INFO  (qtp464770005-887) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-887]: push deletes  1 deleted queries bytesUsed=80 segmentPrivate?=false delGen=2 packetCount=1 totBytesUsed=80
   [junit4]   2> 77059 INFO  (qtp464770005-887) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-887]: flush: push buffered updates:  1 deleted queries bytesUsed=80
   [junit4]   2> 77059 INFO  (qtp464770005-887) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-887]: apply all deletes during flush
   [junit4]   2> 77059 INFO  (qtp464770005-887) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-887]: now apply all deletes for all segments maxDoc=0
   [junit4]   2> 77059 INFO  (qtp464770005-887) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-887]: prune sis=segments_1:  minGen=9223372036854775807 packetCount=1
   [junit4]   2> 77059 INFO  (qtp464770005-887) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-887]: pruneDeletes: prune 1 packets; 0 packets remain
   [junit4]   2> 77059 INFO  (qtp464770005-887) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-887]: incRefDeleter for NRT reader version=2 segments=
   [junit4]   2> 77059 INFO  (qtp464770005-887) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-887]: return reader version=2 reader=StandardDirectoryReader(segments_1:2:nrt)
   [junit4]   2> 77059 INFO  (qtp464770005-887) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-887]: qtp464770005-887 finishFullFlush success=true
   [junit4]   2> 77059 INFO  (qtp464770005-887) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-887]: getReader took 0 msec
   [junit4]   2> 77059 INFO  (qtp464770005-887) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][qtp464770005-887]: delete []
   [junit4]   2> 77059 INFO  (qtp464770005-887) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-887]: decRefDeleter for NRT reader version=2 segments=
   [junit4]   2> 77060 INFO  (qtp464770005-887) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:* (-1551690797805469696)} 0 3
   [junit4]   2> 77063 INFO  (TEST-LargeVolumeJettyTest.testMultiThreaded-seed#[76A03F3D30E407EE]) [    ] o.a.s.c.s.LargeVolumeTestBase Started thread: 0
   [junit4]   2> 77064 INFO  (TEST-LargeVolumeJettyTest.testMultiThreaded-seed#[76A03F3D30E407EE]) [    ] o.a.s.c.s.LargeVolumeTestBase Started thread: 1
   [junit4]   2> 77065 INFO  (TEST-LargeVolumeJettyTest.testMultiThreaded-seed#[76A03F3D30E407EE]) [    ] o.a.s.c.s.LargeVolumeTestBase Started thread: 2
   [junit4]   2> 77066 INFO  (TEST-LargeVolumeJettyTest.testMultiThreaded-seed#[76A03F3D30E407EE]) [    ] o.a.s.c.s.LargeVolumeTestBase Started thread: 3
   [junit4]   2> 77067 INFO  (TEST-LargeVolumeJettyTest.testMultiThreaded-seed#[76A03F3D30E407EE]) [    ] o.a.s.c.s.LargeVolumeTestBase Started thread: 4
   [junit4]   2> 77070 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[T0:0 (1551690797815955456), T0:1 (1551690797818052608), T0:2 (1551690797818052609), T0:3 (1551690797818052610), T0:4 (1551690797818052611), T0:5 (1551690797818052612), T0:6 (1551690797818052613), T0:7 (1551690797818052614), T0:8 (1551690797818052615), T0:9 (1551690797818052616), ... (100 adds)]} 0 3
   [junit4]   2> 77073 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[T2:0 (1551690797820149760), T2:1 (1551690797820149761), T2:2 (1551690797820149762), T2:3 (1551690797820149763), T2:4 (1551690797820149764), T2:5 (1551690797820149765), T2:6 (1551690797820149766), T2:7 (1551690797820149767), T2:8 (1551690797820149768), T2:9 (1551690797820149769), ... (100 adds)]} 0 2
   [junit4]   2> 77073 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 77074 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@12274c76
   [junit4]   2> 77074 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: commit: start
   [junit4]   2> 77074 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: commit: enter lock
   [junit4]   2> 77074 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: commit: now prepare
   [junit4]   2> 77074 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: prepareCommit: flush
   [junit4]   2> 77074 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]:   index before flush 
   [junit4]   2> 77074 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-888]: startFullFlush
   [junit4]   2> 77074 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-888]: anyChanges? numDocsInRam=200 deletes=true hasTickets:false pendingChangesInFullFlush: false
   [junit4]   2> 77074 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [DWFC][qtp464770005-888]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0 199 deleted terms (unique count=199) bytesUsed=33612, segment=_0, aborted=false, numDocsInRAM=200, deleteQueue=DWDQ: [ generation: 2 ]]
   [junit4]   2> 77074 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-888]: flush postings as segment _0 numDocs=200
   [junit4]   2> 77074 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: 0 msec to write norms
   [junit4]   2> 77075 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: 0 msec to write docValues
   [junit4]   2> 77075 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: 0 msec to write points
   [junit4]   2> 77075 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: 0 msec to finish stored fields
   [junit4]   2> 77076 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: 0 msec to write postings and finish vectors
   [junit4]   2> 77076 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: 0 msec to write fieldInfos
   [junit4]   2> 77076 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-888]: new segment has 0 deleted docs
   [junit4]   2> 77076 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-888]: new segment has no vectors; norms; docValues; prox; freqs
   [junit4]   2> 77076 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-888]: flushedFiles=[_0_Lucene50_0.doc, _0_Lucene50_0.tim, _0_Lucene70_0.dvd, _0_Lucene50_0.pos, _0.nvd, _0.fdx, _0_Lucene50_0.tip, _0.fdt, _0.nvm, _0_Lucene70_0.dvm, _0.fnm]
   [junit4]   2> 77076 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-888]: flushed codec=Lucene70
   [junit4]   2> 77076 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-888]: flushed: segment=_0 ramUsed=0.133 MB newFlushedSize=0.006 MB docs/MB=35,359.164
   [junit4]   2> 77076 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-888]: flush time 1.855031 msec
   [junit4]   2> 77076 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-888]: publishFlushedSegment seg-private updates=null
   [junit4]   2> 77076 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: publishFlushedSegment
   [junit4]   2> 77076 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-888]: push deletes  200 deleted terms (unique count=200) bytesUsed=1072 segmentPrivate?=false delGen=4 packetCount=1 totBytesUsed=1072
   [junit4]   2> 77076 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: publish sets newSegment delGen=5 seg=_0(7.0.0):C200
   [junit4]   2> 77076 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][qtp464770005-888]: now checkpoint "_0(7.0.0):C200" [1 segments ; isCommit = false]
   [junit4]   2> 77076 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][qtp464770005-888]: delete []
   [junit4]   2> 77076 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][qtp464770005-888]: 0 msec to checkpoint
   [junit4]   2> 77076 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: apply all deletes during flush
   [junit4]   2> 77076 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: now apply all deletes for all segments maxDoc=200
   [junit4]   2> 77076 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-888]: applyDeletes: open segment readers took 0 msec
   [junit4]   2> 77076 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-888]: applyDeletes: infos=[_0(7.0.0):C200] packetCount=1
   [junit4]   2> 77077 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-888]: applyDeletes took 0 msec for 1 segments, 0 newly deleted docs (query deletes), 0 visited terms, allDeleted=null
   [junit4]   2> 77077 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-888]: prune sis=segments_1: _0(7.0.0):C200 minGen=5 packetCount=1
   [junit4]   2> 77077 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-888]: pruneDeletes: prune 1 packets; 0 packets remain
   [junit4]   2> 77077 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-888]: qtp464770005-888 finishFullFlush success=true
   [junit4]   2> 77077 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [TMP][qtp464770005-888]: findMerges: 1 segments
   [junit4]   2> 77077 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [TMP][qtp464770005-888]:   seg=_0(7.0.0):C200 size=0.006 MB [floored]
   [junit4]   2> 77077 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [TMP][qtp464770005-888]:   allowedSegmentCount=1 vs count=1 (eligible count=1) tooBigCount=0
   [junit4]   2> 77077 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [MS][qtp464770005-888]: initDynamicDefaults spins=false maxThreadCount=1 maxMergeCount=6
   [junit4]   2> 77077 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [MS][qtp464770005-888]: now merge
   [junit4]   2> 77077 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [MS][qtp464770005-888]:   index: _0(7.0.0):C200
   [junit4]   2> 77077 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [MS][qtp464770005-888]:   no more merges pending; now return
   [junit4]   2> 77077 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: startCommit(): start
   [junit4]   2> 77077 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: startCommit index=_0(7.0.0):C200 changeCount=4
   [junit4]   2> 77077 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: startCommit: wrote pending segments file "pending_segments_2"
   [junit4]   2> 77077 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: done all syncs: [_0.si, _0_Lucene50_0.doc, _0_Lucene50_0.tim, _0_Lucene70_0.dvd, _0_Lucene50_0.pos, _0.nvd, _0.fdx, _0_Lucene50_0.tip, _0.fdt, _0_Lucene70_0.dvm, _0.nvm, _0.fnm]
   [junit4]   2> 77077 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: commit: pendingCommit != null
   [junit4]   2> 77077 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: commit: done writing segments file "segments_2"
   [junit4]   2> 77077 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][qtp464770005-888]: now checkpoint "_0(7.0.0):C200" [1 segments ; isCommit = true]
   [junit4]   2> 77078 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][qtp464770005-888]: deleteCommits: now decRef commit "segments_1"
   [junit4]   2> 77078 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][qtp464770005-888]: delete [segments_1]
   [junit4]   2> 77078 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][qtp464770005-888]: 0 msec to checkpoint
   [junit4]   2> 77078 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][qtp464770005-888]: delete []
   [junit4]   2> 77078 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: commit: took 4.1 msec
   [junit4]   2> 77078 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: commit: done
   [junit4]   2> 77078 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: nrtIsCurrent: infoVersion matches: false; DW changes: false; BD changes: false
   [junit4]   2> 77078 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: flush at getReader
   [junit4]   2> 77078 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-888]: startFullFlush
   [junit4]   2> 77078 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: apply all deletes during flush
   [junit4]   2> 77078 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: now apply all deletes for all segments maxDoc=200
   [junit4]   2> 77078 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-888]: applyDeletes: open segment readers took 0 msec
   [junit4]   2> 77078 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-888]: applyDeletes: no segments; skipping
   [junit4]   2> 77078 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-888]: prune sis=segments_2: _0(7.0.0):C200 minGen=5 packetCount=0
   [junit4]   2> 77079 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: incRefDeleter for NRT reader version=6 segments=_0(7.0.0):C200
   [junit4]   2> 77079 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: return reader version=6 reader=StandardDirectoryReader(segments_2:6:nrt _0(7.0.0):C200)
   [junit4]   2> 77079 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-888]: qtp464770005-888 finishFullFlush success=true
   [junit4]   2> 77079 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-888]: getReader took 1 msec
   [junit4]   2> 77079 INFO  (qtp464770005-888) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1bb1b719[collection1] main]
   [junit4]   2> 77079 INFO  (qtp464770005-888) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 77080 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 77081 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@12274c76
   [junit4]   2> 77081 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: commit: start
   [junit4]   2> 77081 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: commit: enter lock
   [junit4]   2> 77081 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: commit: now prepare
   [junit4]   2> 77081 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: prepareCommit: flush
   [junit4]   2> 77081 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]:   index before flush _0(7.0.0):C200
   [junit4]   2> 77081 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-896]: startFullFlush
   [junit4]   2> 77081 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DWFC][qtp464770005-896]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0, segment=_1, aborted=false, numDocsInRAM=1, deleteQueue=DWDQ: [ generation: 4 ]]
   [junit4]   2> 77081 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-896]: flush postings as segment _1 numDocs=1
   [junit4]   2> 77082 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: 0 msec to write norms
   [junit4]   2> 77082 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: 0 msec to write docValues
   [junit4]   2> 77082 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: 0 msec to write points
   [junit4]   2> 77082 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: 0 msec to finish stored fields
   [junit4]   2> 77082 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: 0 msec to write postings and finish vectors
   [junit4]   2> 77082 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: 0 msec to write fieldInfos
   [junit4]   2> 77082 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-896]: new segment has 0 deleted docs
   [junit4]   2> 77082 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-896]: new segment has no vectors; norms; docValues; prox; freqs
   [junit4]   2> 77082 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-896]: flushedFiles=[_1_Lucene50_0.doc, _1_Lucene50_0.tim, _1_Lucene70_0.dvd, _1.nvd, _1_Lucene50_0.pos, _1.fdx, _1.nvm, _1.fnm, _1.fdt, _1_Lucene70_0.dvm, _1_Lucene50_0.tip]
   [junit4]   2> 77082 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-896]: flushed codec=Lucene70
   [junit4]   2> 77082 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-896]: flushed: segment=_1 ramUsed=0.095 MB newFlushedSize=0.001 MB docs/MB=670.445
   [junit4]   2> 77082 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-896]: flush time 0.878651 msec
   [junit4]   2> 77082 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-896]: publishFlushedSegment seg-private updates=null
   [junit4]   2> 77082 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: publishFlushedSegment
   [junit4]   2> 77082 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-896]: push deletes  1 deleted terms (unique count=1) bytesUsed=1072 segmentPrivate?=false delGen=8 packetCount=1 totBytesUsed=1072
   [junit4]   2> 77082 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: publish sets newSegment delGen=9 seg=_1(7.0.0):C1
   [junit4]   2> 77082 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][qtp464770005-896]: now checkpoint "_0(7.0.0):C200 _1(7.0.0):C1" [2 segments ; isCommit = false]
   [junit4]   2> 77083 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][qtp464770005-896]: delete []
   [junit4]   2> 77083 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][qtp464770005-896]: 0 msec to checkpoint
   [junit4]   2> 77083 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: apply all deletes during flush
   [junit4]   2> 77083 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: now apply all deletes for all segments maxDoc=229
   [junit4]   2> 77083 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-896]: applyDeletes: open segment readers took 0 msec
   [junit4]   2> 77083 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-896]: applyDeletes: infos=[_0(7.0.0):C200, _1(7.0.0):C1] packetCount=1
   [junit4]   2> 77083 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-896]: applyTermDeletes took 0.1 msec for 2 segments and 1 packets; 1 del terms visited; 2 seg terms visited
   [junit4]   2> 77083 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-896]: applyDeletes: 0 new deleted documents
   [junit4]   2> 77083 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-896]: applyDeletes took 0 msec for 2 segments, 0 newly deleted docs (query deletes), 1 visited terms, allDeleted=null
   [junit4]   2> 77083 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-896]: prune sis=segments_2: _0(7.0.0):C200 _1(7.0.0):C1 minGen=10 packetCount=1
   [junit4]   2> 77083 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-896]: pruneDeletes: prune 1 packets; 0 packets remain
   [junit4]   2> 77083 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-896]: qtp464770005-896 finishFullFlush success=true
   [junit4]   2> 77083 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [TMP][qtp464770005-896]: findMerges: 2 segments
   [junit4]   2> 77083 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [TMP][qtp464770005-896]:   seg=_0(7.0.0):C200 size=0.006 MB [floored]
   [junit4]   2> 77083 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [TMP][qtp464770005-896]:   seg=_1(7.0.0):C1 size=0.002 MB [floored]
   [junit4]   2> 77083 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [TMP][qtp464770005-896]:   allowedSegmentCount=1 vs count=2 (eligible count=2) tooBigCount=0
   [junit4]   2> 77084 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [MS][qtp464770005-896]: now merge
   [junit4]   2> 77084 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [MS][qtp464770005-896]:   index: _0(7.0.0):C200 _1(7.0.0):C1
   [junit4]   2> 77084 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [MS][qtp464770005-896]:   no more merges pending; now return
   [junit4]   2> 77084 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: startCommit(): start
   [junit4]   2> 77084 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: startCommit index=_0(7.0.0):C200 _1(7.0.0):C1 changeCount=9
   [junit4]   2> 77084 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: startCommit: wrote pending segments file "pending_segments_3"
   [junit4]   2> 77084 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: done all syncs: [_0.si, _0_Lucene50_0.doc, _0_Lucene50_0.tim, _1.si, _1_Lucene70_0.dvd, _1.nvd, _0_Lucene50_0.pos, _0.nvd, _0.fdx, _0_Lucene50_0.tip, _1.nvm, _1.fnm, _0.fdt, _0_Lucene70_0.dvm, _1_Lucene50_0.doc, _1_Lucene50_0.tim, _0_Lucene70_0.dvd, _1_Lucene50_0.pos, _1.fdx, _0.nvm, _0.fnm, _1.fdt, _1_Lucene70_0.dvm, _1_Lucene50_0.tip]
   [junit4]   2> 77084 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: commit: pendingCommit != null
   [junit4]   2> 77084 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: commit: done writing segments file "segments_3"
   [junit4]   2> 77084 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][qtp464770005-896]: now checkpoint "_0(7.0.0):C200 _1(7.0.0):C1" [2 segments ; isCommit = true]
   [junit4]   2> 77084 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][qtp464770005-896]: deleteCommits: now decRef commit "segments_2"
   [junit4]   2> 77084 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][qtp464770005-896]: delete [segments_2]
   [junit4]   2> 77084 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][qtp464770005-896]: 0 msec to checkpoint
   [junit4]   2> 77084 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][qtp464770005-896]: delete []
   [junit4]   2> 77084 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: commit: took 3.6 msec
   [junit4]   2> 77084 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: commit: done
   [junit4]   2> 77085 WARN  (qtp464770005-896) [    x:collection1] o.a.s.c.SolrCore [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
   [junit4]   2> 77085 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-896]: anyChanges? numDocsInRam=80 deletes=true hasTickets:false pendingChangesInFullFlush: false
   [junit4]   2> 77085 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: nrtIsCurrent: infoVersion matches: false; DW changes: true; BD changes: false
   [junit4]   2> 77085 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: flush at getReader
   [junit4]   2> 77085 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-896]: startFullFlush
   [junit4]   2> 77085 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-896]: anyChanges? numDocsInRam=80 deletes=true hasTickets:false pendingChangesInFullFlush: false
   [junit4]   2> 77092 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DWFC][qtp464770005-896]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0 79 deleted terms (unique count=79) bytesUsed=13343, segment=_2, aborted=false, numDocsInRAM=80, deleteQueue=DWDQ: [ generation: 5 ]]
   [junit4]   2> 77093 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-896]: flush postings as segment _2 numDocs=80
   [junit4]   2> 77093 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: 0 msec to write norms
   [junit4]   2> 77093 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: 0 msec to write docValues
   [junit4]   2> 77093 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: 0 msec to write points
   [junit4]   2> 77093 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: 0 msec to finish stored fields
   [junit4]   2> 77094 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: 0 msec to write postings and finish vectors
   [junit4]   2> 77094 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: 0 msec to write fieldInfos
   [junit4]   2> 77094 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-896]: new segment has 0 deleted docs
   [junit4]   2> 77094 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-896]: new segment has no vectors; norms; docValues; prox; freqs
   [junit4]   2> 77094 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-896]: flushedFiles=[_2.nvd, _2_Lucene70_0.dvd, _2.fdt, _2.nvm, _2_Lucene50_0.pos, _2.fnm, _2.fdx, _2_Lucene50_0.doc, _2_Lucene50_0.tim, _2_Lucene70_0.dvm, _2_Lucene50_0.tip]
   [junit4]   2> 77094 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-896]: flushed codec=Lucene70
   [junit4]   2> 77094 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-896]: flushed: segment=_2 ramUsed=0.11 MB newFlushedSize=0.003 MB docs/MB=24,628.914
   [junit4]   2> 77094 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-896]: flush time 1.419006 msec
   [junit4]   2> 77094 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-896]: publishFlushedSegment seg-private updates=null
   [junit4]   2> 77094 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: publishFlushedSegment
   [junit4]   2> 77094 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-896]: push deletes  80 deleted terms (unique count=80) bytesUsed=1072 segmentPrivate?=false delGen=11 packetCount=1 totBytesUsed=1072
   [junit4]   2> 77094 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: publish sets newSegment delGen=12 seg=_2(7.0.0):C80
   [junit4]   2> 77094 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][qtp464770005-896]: now checkpoint "_0(7.0.0):C200 _1(7.0.0):C1 _2(7.0.0):C80" [3 segments ; isCommit = false]
   [junit4]   2> 77094 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][qtp464770005-896]: delete []
   [junit4]   2> 77094 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][qtp464770005-896]: 0 msec to checkpoint
   [junit4]   2> 77094 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: apply all deletes during flush
   [junit4]   2> 77094 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: now apply all deletes for all segments maxDoc=339
   [junit4]   2> 77094 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-896]: applyDeletes: open segment readers took 0 msec
   [junit4]   2> 77094 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-896]: applyDeletes: infos=[_0(7.0.0):C200, _1(7.0.0):C1, _2(7.0.0):C80] packetCount=1
   [junit4]   2> 77095 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-896]: applyTermDeletes took 0.1 msec for 3 segments and 1 packets; 80 del terms visited; 161 seg terms visited
   [junit4]   2> 77095 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-896]: applyDeletes: 0 new deleted documents
   [junit4]   2> 77095 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-896]: applyDeletes took 1 msec for 3 segments, 0 newly deleted docs (query deletes), 80 visited terms, allDeleted=null
   [junit4]   2> 77095 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-896]: prune sis=segments_3: _0(7.0.0):C200 _1(7.0.0):C1 _2(7.0.0):C80 minGen=13 packetCount=1
   [junit4]   2> 77095 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][qtp464770005-896]: pruneDeletes: prune 1 packets; 0 packets remain
   [junit4]   2> 77095 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[T3:0 (1551690797843218432), T3:1 (1551690797843218433), T3:2 (1551690797843218434), T3:3 (1551690797843218435), T3:4 (1551690797843218436), T3:5 (1551690797843218437), T3:6 (1551690797843218438), T3:7 (1551690797843218439), T3:8 (1551690797843218440), T3:9 (1551690797843218441), ... (100 adds)]} 0 2
   [junit4]   2> 77095 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: incRefDeleter for NRT reader version=13 segments=_0(7.0.0):C200 _1(7.0.0):C1 _2(7.0.0):C80
   [junit4]   2> 77096 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: return reader version=13 reader=StandardDirectoryReader(segments_3:13:nrt _0(7.0.0):C200 _1(7.0.0):C1 _2(7.0.0):C80)
   [junit4]   2> 77096 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-896]: qtp464770005-896 finishFullFlush success=true
   [junit4]   2> 77096 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [TMP][qtp464770005-896]: findMerges: 3 segments
   [junit4]   2> 77096 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [TMP][qtp464770005-896]:   seg=_0(7.0.0):C200 size=0.006 MB [floored]
   [junit4]   2> 77096 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [TMP][qtp464770005-896]:   seg=_2(7.0.0):C80 size=0.003 MB [floored]
   [junit4]   2> 77096 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [TMP][qtp464770005-896]:   seg=_1(7.0.0):C1 size=0.002 MB [floored]
   [junit4]   2> 77096 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [TMP][qtp464770005-896]:   allowedSegmentCount=1 vs count=3 (eligible count=3) tooBigCount=0
   [junit4]   2> 77096 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [MS][qtp464770005-896]: now merge
   [junit4]   2> 77096 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [MS][qtp464770005-896]:   index: _0(7.0.0):C200 _1(7.0.0):C1 _2(7.0.0):C80
   [junit4]   2> 77096 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [MS][qtp464770005-896]:   no more merges pending; now return
   [junit4]   2> 77096 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-896]: getReader took 11 msec
   [junit4]   2> 77097 INFO  (qtp464770005-896) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@76368420[collection1] main]
   [junit4]   2> 77097 INFO  (qtp464770005-896) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 77098 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[T1:0 (1551690797846364160), T1:1 (1551690797846364161), T1:2 (1551690797846364162), T1:3 (1551690797846364163), T1:4 (1551690797846364164), T1:5 (1551690797846364165), T1:6 (1551690797847412736), T1:7 (1551690797847412737), T1:8 (1551690797847412738), T1:9 (1551690797847412739), ... (100 adds)]} 0 2
   [junit4]   2> 77100 INFO  (qtp464770005-885) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[T4:0 (1551690797829586944), T4:1 (1551690797831684096), T4:2 (1551690797831684097), T4:3 (1551690797831684098), T4:4 (1551690797831684099), T4:5 (1551690797831684100), T4:6 (1551690797831684101), T4:7 (1551690797831684102), T4:8 (1551690797831684103), T4:9 (1551690797831684104), ... (100 adds)]} 0 19
   [junit4]   2> 77100 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 77100 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@12274c76
   [junit4]   2> 77100 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-895]: commit: start
   [junit4]   2> 77100 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-895]: commit: enter lock
   [junit4]   2> 77100 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-895]: commit: now prepare
   [junit4]   2> 77100 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-895]: prepareCommit: flush
   [junit4]   2> 77100 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-895]:   index before flush _0(7.0.0):C200 _1(7.0.0):C1 _2(7.0.0):C80
   [junit4]   2> 77100 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-895]: startFullFlush
   [junit4]   2> 77100 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][qtp464770005-895]: anyChanges? numDocsInRam=219 deletes=true hasTickets:false pendingChangesInFullFlush: false
   [junit4]   2> 77100 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [DWFC][qtp464770005-895]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0 18 deleted terms (unique count=18) bytesUsed=3042, segment=_4, aborted=false, numDocsInRAM=19, deleteQueue=DWDQ: [ generation: 6 ]]
   [junit4]   2> 77100 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [DWFC][qtp464770005-895]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0 199 deleted terms (unique count=199) bytesUsed=33612, segment=_3, aborted=false, numDocsInRAM=200, deleteQueue=DWDQ: [ generation: 6 ]]
   [junit4]   2> 77101 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-895]: flush postings as segment _4 numDocs=19
   [junit4]   2> 77101 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-895]: 0 msec to write norms
   [junit4]   2> 77101 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-895]: 0 msec to write docValues
   [junit4]   2> 77101 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-895]: 0 msec to write points
   [junit4]   2> 77101 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-895]: 0 msec to finish stored fields
   [junit4]   2> 77101 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-895]: 0 msec to write postings and finish vectors
   [junit4]   2> 77102 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-895]: 0 msec to write fieldInfos
   [junit4]   2> 77102 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-895]: new segment has 0 deleted docs
   [junit4]   2> 77102 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-895]: new segment has no vectors; norms; docValues; prox; freqs
   [junit4]   2> 77102 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-895]: flushedFiles=[_4.fdt, _4.nvm, _4_Lucene50_0.pos, _4.fnm, _4_Lucene70_0.dvd, _4_Lucene50_0.tip, _4.nvd, _4_Lucene70_0.dvm, _4.fdx, _4_Lucene50_0.doc, _4_Lucene50_0.tim]
   [junit4]   2> 77102 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-895]: flushed codec=Lucene70
   [junit4]   2> 77102 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-895]: flushed: segment=_4 ramUsed=0.099 MB newFlushedSize=0.002 MB docs/MB=9,747.037
   [junit4]   2> 77102 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-895]: flush time 1.217473 msec
   [junit4]   2> 77102 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-895]: flush postings as segment _3 numDocs=200
   [junit4]   2> 77102 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-895]: 0 msec to write norms
   [junit4]   2> 77102 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-895]: 0 msec to write docValues
   [junit4]   2> 77102 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-895]: 0 msec to write points
   [junit4]   2> 77102 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-895]: 0 msec to finish stored fields
   [junit4]   2> 77103 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-895]: 0 msec to write postings and finish vectors
   [junit4]   2> 77104 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][qtp464770005-895]: 0 msec to write fieldInfos
   [junit4]   2> 77104 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-895]: new segment has 0 deleted docs
   [junit4]   2> 77104 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-895]: new segment has no vectors; norms; docValues; prox; freqs
   [junit4]   2> 77104 INFO  (qtp464770005-895) [    x:collection1] o.a.s.u.LoggingInfoStream [DWPT][qtp464770005-895]: flushedFiles=[_3.fdx, _3.nvd, _3_Lucene50_0.pos, _3_Lucene70_0.dvd, _3_Lucene50_0.doc, _3_Lucene50_0.tim, _3_Lucene70_0.dvm, _3.nvm, _3.fnm, _3.fdt, _3_Lucene50_0.tip]
   [junit4]   2> 7710

[...truncated too long message...]

) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-204-thread-1]: commit: done
   [junit4]   2> 77275 INFO  (coreCloseExecutor-204-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-204-thread-1]: rollback
   [junit4]   2> 77275 INFO  (coreCloseExecutor-204-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-204-thread-1]: all running merges have aborted
   [junit4]   2> 77275 INFO  (coreCloseExecutor-204-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-204-thread-1]: rollback: done finish merges
   [junit4]   2> 77275 INFO  (coreCloseExecutor-204-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][coreCloseExecutor-204-thread-1]: abort
   [junit4]   2> 77275 INFO  (coreCloseExecutor-204-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][coreCloseExecutor-204-thread-1]: done abort success=true
   [junit4]   2> 77275 INFO  (coreCloseExecutor-204-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-204-thread-1]: rollback: infos=_5(7.0.0):C500
   [junit4]   2> 77275 INFO  (coreCloseExecutor-204-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreCloseExecutor-204-thread-1]: now checkpoint "_5(7.0.0):C500" [1 segments ; isCommit = false]
   [junit4]   2> 77276 INFO  (coreCloseExecutor-204-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreCloseExecutor-204-thread-1]: delete []
   [junit4]   2> 77276 INFO  (coreCloseExecutor-204-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreCloseExecutor-204-thread-1]: 0 msec to checkpoint
   [junit4]   2> 77276 INFO  (coreCloseExecutor-204-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreCloseExecutor-204-thread-1]: delete []
   [junit4]   2> 77276 INFO  (coreCloseExecutor-204-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreCloseExecutor-204-thread-1]: delete []
   [junit4]   2> 77369 INFO  (SUITE-LargeVolumeJettyTest-seed#[76A03F3D30E407EE]-worker) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1e9bb327{/solr,null,UNAVAILABLE}
   [junit4]   2> 77371 INFO  (SUITE-LargeVolumeJettyTest-seed#[76A03F3D30E407EE]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=24, maxMBSortInHeap=5.844482272295528, sim=RandomSimilarity(queryNorm=false): {}, locale=de-DE, timezone=America/St_Vincent
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_102 (64-bit)/cpus=3,threads=1,free=37292952,total=101449728
   [junit4]   2> NOTE: All tests run in this JVM: [SolrExceptionTest, TestFastInputStream, CommonParamsTest, SolrExampleJettyTest, JdbcTest, TestNamedListCodec, GraphTest, StreamExpressionToExplanationTest, SolrExampleBinaryTest, SolrQueryTest, TestClusteringResponse, AnlysisResponseBaseTest, TestCloudSolrClientConnections, SolrParamTest, TestJsonRecordReader, HttpSolrClientBuilderTest, TestConfigSetAdminRequest, SolrPingTest, FacetFieldTest, SolrZkClientTest, ClientUtilsTest, LargeVolumeBinaryJettyTest, SolrExampleStreamingBinaryTest, SolrExampleXMLTest, SolrExampleEmbeddedTest, TestBatchUpdate, LargeVolumeEmbeddedTest, LargeVolumeJettyTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=LargeVolumeJettyTest -Dtests.seed=76A03F3D30E407EE -Dtests.slow=true -Dtests.locale=de-DE -Dtests.timezone=America/St_Vincent -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | LargeVolumeJettyTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [TransactionLog]
   [junit4]    > org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
   [junit4]    > 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:43)
   [junit4]    > 	at org.apache.solr.update.TransactionLog.<init>(TransactionLog.java:188)
   [junit4]    > 	at org.apache.solr.update.UpdateLog.newTransactionLog(UpdateLog.java:344)
   [junit4]    > 	at org.apache.solr.update.UpdateLog.ensureLog(UpdateLog.java:859)
   [junit4]    > 	at org.apache.solr.update.UpdateLog.add(UpdateLog.java:428)
   [junit4]    > 	at org.apache.solr.update.UpdateLog.add(UpdateLog.java:415)
   [junit4]    > 	at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:299)
   [junit4]    > 	at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:211)
   [junit4]    > 	at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:166)
   [junit4]    > 	at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
   [junit4]    > 	at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
   [junit4]    > 	at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:957)
   [junit4]    > 	at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1112)
   [junit4]    > 	at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:738)
   [junit4]    > 	at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
   [junit4]    > 	at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:97)
   [junit4]    > 	at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:179)
   [junit4]    > 	at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:135)
   [junit4]    > 	at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:298)
   [junit4]    > 	at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121)
   [junit4]    > 	at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:263)
   [junit4]    > 	at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:181)
   [junit4]    > 	at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:186)
   [junit4]    > 	at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:107)
   [junit4]    > 	at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:54)
   [junit4]    > 	at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
   [junit4]    > 	at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
   [junit4]    > 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:153)
   [junit4]    > 	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2228)
   [junit4]    > 	at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)
   [junit4]    > 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
   [junit4]    > 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:303)
   [junit4]    > 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)
   [junit4]    > 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
   [junit4]    > 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:113)
   [junit4]    > 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
   [junit4]    > 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
   [junit4]    > 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]    > 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
   [junit4]    > 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
   [junit4]    > 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]    > 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
   [junit4]    > 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]    > 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
   [junit4]    > 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]    > 	at org.eclipse.jetty.server.Server.handle(Server.java:518)
   [junit4]    > 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
   [junit4]    > 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
   [junit4]    > 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]    > 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]    > 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]    > 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
   [junit4]    > 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([76A03F3D30E407EE]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:260)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4] Completed [44/87 (1!)] on J1 in 32.54s, 1 test, 1 failure <<< FAILURES!

[...truncated 61810 lines...]


[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 961 - Failure!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/961/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

All tests passed

Build Log:
[...truncated 56916 lines...]
changes-to-html:
    [mkdir] Created dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/docs/changes
     [exec] Can't locate XML/Simple.pm in @INC (@INC contains: /usr/perl5/site_perl/5.12/i86pc-solaris-64int /usr/perl5/site_perl/5.12 /usr/perl5/vendor_perl/5.12/i86pc-solaris-64int /usr/perl5/vendor_perl/5.12 /usr/perl5/5.12/lib/i86pc-solaris-64int /usr/perl5/5.12/lib .) at /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/site/changes/changes2html.pl line 26.
     [exec] BEGIN failed--compilation aborted at /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/site/changes/changes2html.pl line 26.

BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:765: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:101: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build.xml:138: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build.xml:482: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:2513: exec returned: 2

Total time: 95 minutes 16 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any